packages.sury.org returns a high level of 500

[lyrixx]

Frequently asked questions

• [x] I have read Frequently Asked Questions
• [x] I have looked at the list of the existing issues (including closed issues) and searched if my issue has been already reported
• [x] I have tried to resolve the issue myself and will describe what I did in clear and concise manner

Describe the bug

Hello, I think there is an issue with the https://packages.sury.org/ server. We good some 500, or hash mismatch. I read the FAQ, and the notice about hash mismatch, but I think there is another problem, because we got 500 too

Some samples from our CI:

#63 15.50 E: Failed to fetch https://packages.sury.org/php/pool/main/p/php7.4/php7.4-common_7.4.33-8+0~20230904.88+debian10~1.gbp87c414_amd64.deb  500  Internal Server Error [IP: 169.150.221.147 443]

or

2024-04-04T08:29:57.2425239Z #112 14.59 E: Failed to fetch https://packages.sury.org/php/pool/main/libg/libgd2/libgd3_2.3.3-9+0~20230207.13+debian10~1.gbp649015_amd64.deb  Hash Sum mismatch
2024-04-04T08:29:57.2426855Z #112 14.59    Hashes of expected file:
2024-04-04T08:29:57.2427933Z #112 14.59     - SHA256:a7289d6505a91c7562b934125d67c55ea3f6cbad0aa55c44d437b40b6f5b1522
2024-04-04T08:29:57.2429567Z #112 14.59     - SHA1:100b6d3076357b243a9115d5ec8f80441e925d5d [weak]
2024-04-04T08:29:57.2430574Z #112 14.59     - MD5Sum:d53c6c10bf8b320f2092554dc3f29a66 [weak]
2024-04-04T08:29:57.2431364Z #112 14.59     - Filesize:136844 [weak]
2024-04-04T08:29:57.2432564Z #112 14.59    Hashes of received file:
2024-04-04T08:29:57.2433621Z #112 14.59     - SHA256:460b4b813d55bfc11debfc9b20bda88b9999759f0d70bdfc4122346feb2bd573
2024-04-04T08:29:57.2434808Z #112 14.59     - SHA1:ed015112f13e6faf54c9fc26c9631d8de4c06e31 [weak]
2024-04-04T08:29:57.2435850Z #112 14.59     - MD5Sum:4145bed4da1a2ad082846182766fe155 [weak]
2024-04-04T08:29:57.2436678Z #112 14.60     - Filesize:136844 [weak]

or

#63 136.9 E: Failed to fetch https://packages.sury.org/php/pool/main/libz/libzip/libzip4_1.7.3-1+0~20210114.10+debian10~1.gbp4c125d_amd64.deb  Hash Sum mismatch
#63 136.9    Hashes of expected file:
#63 136.9     - SHA256:7a3c53422002633bd9797b1cae58241ae0b440c0f96cbead1939532ffe6d42e4
#63 136.9     - SHA1:35596d80f87a9b02952d3a651f9a1e26606f02dd [weak]
#63 136.9     - MD5Sum:e7091e8a5761b32653c345961e70727c [weak]
#63 136.9     - Filesize:55096 [weak]
#63 136.9    Hashes of received file:
#63 136.9     - SHA256:2d8646feadc4f6c7bbf4b080ab9e16b4792cef2b94feb05ba8a1092102838917
#63 136.9     - SHA1:77f0ca6f6af91afd0fb0b8c430333af51c017c11 [weak]
#63 136.9     - MD5Sum:58e8d723c23da28f95de35dcee680bae [weak]
#63 136.9     - Filesize:55096 [weak]

To Reproduce

This issue is transient. and hard to reproduce. But we have a high number a failing PR due to this "bug".

So it's hard to reproduce. But I managed to get an error with the following script:

#!/usr/bin/env bash

expected=d53c6c10bf8b320f2092554dc3f29a66

date

while true ; do
    hash=`curl -sq https://packages.sury.org/php/pool/main/libg/libgd2/libgd3_2.3.3-9+0~20230207.13+debian10~1.gbp649015_amd64.deb  | md5sum | awk '{print $1}'`
    if [ "$hash" == "$expected" ]; then
        echo -n "."
    else
        echo
        echo "Hash is incorrect"
        echo $hash
        date
        exit 1
    fi
    sleep .1
done

I got the following hash

d41d8cd98f00b204e9800998ecf8427e

For the record:

>/home/gregoire echo  -n  | md5sum
d41d8cd98f00b204e9800998ecf8427e  -

So I got an empty content

Your understanding of what is happening

I think there is an issue with the origin.

What steps did you take to resolve issue yourself before reporting it here

• I curl the URL on my computer but it works
• I retried many time, same on the CI

Expected behavior

A 200

Distribution (please complete the following information):

• OS: Debian
• Architecture: amd64
• Repository: packages.sury.org

Package(s) (please complete the following information): Output of apt-cache policy <package> works best.

[oerdnj]

But I managed to get an error with the following script

I wonder in what universe you could think it’s ok to just bash the server with HTTP requests in a loop? Do you realize that the bandwidth costs money?

I am so angry now that I don’t even want to pay attention to this.

[lyrixx]

Oh, I'm deeply sorry for that! I did not realize it'll cost you money, and I wanted to give you a reproducer as simple as possible to mimic what is done in our CI.

I'll transfer you some money right away!

Do you realize that the bandwidth costs money?

For the record, I use many hosting platform that do not charge bandwidth. For example GitHub, https://www.ovhcloud.com, https://www.scaleway.com/, etc

[oerdnj]

I'll transfer you some money right away!

You don’t have to do that. I have a generous credit from the CDN nowadays. But nothing is free and it’s important to realize that.

Just look at the current graph. I don’t know if the 2M difference daily is symptom or cause, but I’ll open the ticket with the CDN.

[oerdnj]

Can you give me full http headers?

[Greg4040]

French sysAdmin here.

It is entirely true that French (and most European) hosts and ISPs do not count the data consumed but this is rather an exception in the world. Furthermore, we have no idea of the download load required to maintain this project. What bothers me in your script is especially the clearly disrespectful “sleep .1”.

@oerdnj To err is human and we all appreciate your great work. Please don't be too angry. I give from time to time with Paypal, but do not hesitate to contact me, I can set up a dedicated server in France at OVH for free and without bandwidth costs if that helps.

[oerdnj]

Ideally for both the 2xx and the 5xx error code. You can spin this in the loop now.

[oerdnj]

I'm ok now, but the sleep .1 was the thing that through me off balance.

[oerdnj]

FTR It was never about the money that this could have potentially cost. But I wish people would think more about the impact their actions could cause.

I don't want to spend more time on this. Apologies accepted, let's move on and try to solve the real issue now.

[lyrixx]

I'm not able to reproduce anymore.

But I think I found something:

1. I first try to reproduce it on my computer with a simple curl repeated 5-10 times => OKAY
2. Then I did the same in GitHubAction (to check with another IP, network stack) => OKAY
3. Then I extracted a minimal reproducer from our Dockerfile, and executed it locally, without cache, 5-10 times => OKAY
4. Same on github => OKAY
5. So I though it was solved, rollback all our CI to the initial state => KO
6. WHAT? I repeated this 2/3 times, and it was always the same results

Then I noticed in our docker-compose.yaml file we forced the DNS to 8.8.8.8.

So I removed theses an it seems OK.

I tried to reproduced it by forcing the resolution via 8.8.8.8, on GHA and locally, but I was not able to reproduce it anymore 😢.

So let's close it for now.

[lyrixx]

For the record, The issue is still here. But I think it's an issue with github. I managed to create a reproducer and I contacted github support