search

armbian / mirror

Armbian Linux Mirrors Infrastructure
45 stars 8 forks source link

Inconsisten sync state on some mirrors (at least netix, stpete, xogium) #7

Closed matthijskooijman closed 2 years ago

matthijskooijman commented 2 years ago

I'm reporting this here in the assumption that repo is for tracking issues with the sync state of various apt.armbian.com mirrors. If this should be elsewhere, let me know.

While investigating some build failures related to failed apt updates, I noticed that some mirrors seem to be inconsistent. I originally thought this was caused by apt.armbian.com doing inconsistent redirects (which I still think is an issue, reported at https://github.com/armbian/dl-router/issues/29, but not the biggest cause for these problems right now), but it seems that some mirrors are also inconsistent.

In short, it seems that at least the netix, stpete and xogium mirrors have an outdated InRelease file (dated Feb 1), while other files do seem up-to-date. To reproduce, use either of these sources.list lines:

deb http://stpete-mirror.armbian.com/apt focal main focal-utils focal-desktop
deb http://mirrors.netix.net/armbian/apt focal main focal-utils focal-desktop
deb http://xogium.performanceservers.nl/apt focal main focal-utils focal-desktop

Which produces either of these errors (depending, I think, on whether the mirror has by-hash enabled):

Err:6 https://stpete-mirror.armbian.com/apt focal/main armhf Packages
  404  Not Found [IP: 188.227.12.37 443]                   
Fetched 50.2 kB in 26s (1,903 B/s)                                                                                                                             
Reading package lists... Done                                                                                                                                  
E: Failed to fetch https://stpete-mirror.armbian.com/apt/dists/focal/main/binary-armhf/by-hash/SHA512/7df5a6040bc2fc97457a149e7901b816af1299a7441353104c3f07fe7
345dffbe154ecca08f7c12f5bf7206e06c39ad070df355c475ea6a0d65f110d6733ea8d  404  Not Found [IP: 188.227.12.37 443]
E: Failed to fetch https://stpete-mirror.armbian.com/apt/dists/focal/focal-utils/binary-armhf/by-hash/SHA512/f419d6ec3f1cd578e203dcf2dac89a8db42a9694649bc0edce
7119db12b1fe05b574b5ceeac06630b84a1f1758018d7cc219d75ced2677271dc86ec6ac2036b3  404  Not Found [IP: 188.227.12.37 443]

or

Err:6 http://mirrors.netix.net/armbian/apt focal/main armhf Packages
  File has unexpected size (199533 != 123546). Mirror sync in progress? [IP: 87.121.121.2 80]
  Hashes of expected file:
   - Filesize:123546 [weak]
   - SHA512:cfa454cb1bbcb61b47ca00d2b6970feaf9ab7085740b864d7e021b5a6d2f830097703c7fe86a4e591508bab6bc3fce7a8edc1484df62767cdce169b076712226
   - SHA256:65c20abb7cdf3a958056f6e6c8420b6c64d2972db8ff06a3c4ae1f839665a831
   - SHA1:12fb81cbdd36bc04236551dcee503ed42ff25d09 [weak]
   - MD5Sum:06a04d0d482b380776dc4fb5f292d5a1 [weak]
  Release file created at: Tue, 01 Feb 2022 17:25:15 +0000
E: Failed to fetch http://mirrors.netix.net/armbian/apt/dists/focal/main/binary-armhf/Packages.bz2  File has unexpected size (199533 != 123546). Mirror sync in progress? [IP: 87.121.121.2 80]

To confirm that these mirrors are indeed inconsistent, I manually downloaded some files:

$ for i in {stpete-mirror,imola}.armbian.com xogium.performanceservers.nl mirrors.netix.net/armbian; do mkdir -p $i; wget http://$i/apt/dists/focal/InRelease -O $i/InRelease; wget http://$i/apt/dists/focal/main/binary-armhf/Packages.bz2 -O $i/Packages.bz2; done
$ mv mirrors.netix.net/armbian/* mirrors.netix.net/ # Remove one extra directory level for easier scripting

Then you can see that only imola is up-to-date:

$ grep Date */InRelease
imola.armbian.com/InRelease:Date: Mon, 7 Mar 2022 09:45:17 UTC
mirrors.netix.net/InRelease:Date: Tue, 1 Feb 2022 17:25:15 UTC
stpete-mirror.armbian.com/InRelease:Date: Tue, 1 Feb 2022 17:25:15 UTC
xogium.performanceservers.nl/InRelease:Date: Tue, 1 Feb 2022 17:25:15 UTC

Looking at the MD5Sums of the Packages.bz2 files in the InRelease file also shows imola differs:

$ for f in */InRelease; do grep --max-count=1 --with-filename main/binary-armhf/Packages.bz2 "$f";done
imola.armbian.com/InRelease: f7971fdc3eeb0cf0c81b77870e447057   199533 main/binary-armhf/Packages.bz2
mirrors.netix.net/InRelease: 06a04d0d482b380776dc4fb5f292d5a1   123546 main/binary-armhf/Packages.bz2
stpete-mirror.armbian.com/InRelease: 06a04d0d482b380776dc4fb5f292d5a1   123546 main/binary-armhf/Packages.bz2
xogium.performanceservers.nl/InRelease: 06a04d0d482b380776dc4fb5f292d5a1   123546 main/binary-armhf/Packages.bz2

However, the actual files seem to all match the imola MD5sum:

md5sum */Packages.bz2
f7971fdc3eeb0cf0c81b77870e447057  imola.armbian.com/Packages.bz2
f7971fdc3eeb0cf0c81b77870e447057  mirrors.netix.net/Packages.bz2
f7971fdc3eeb0cf0c81b77870e447057  stpete-mirror.armbian.com/Packages.bz2
f7971fdc3eeb0cf0c81b77870e447057  xogium.performanceservers.nl/Packages.bz2

IOW, it seems that the Packages files have been synced correctly, but not the InRelease files (for 5 weeks now).

This causes issues for anyone using these mirrors, including when redirected to them through apt.armbian.com (as I was).

To confirm my setup is otherwise ok, I tried:

deb http://imola.armbian.com/apt focal main focal-utils focal-desktop

Which succeeds every time.

igorpecovnik commented 2 years ago

We had some strange drive crash (moved now to safer but slower storage) on a server where repository is held which could explain some malfunction.

When did you check this? Cause now it seems alright.

rsync -av rsync://rsync.armbian.com/apt/dists/jammy/
drwxrwsr-x          4,096 2022/03/07 11:16:55 by-hash            -rw-rw-r--     22,151,392 2022/03/07 11:17:12 Contents-arm64.gz
drwxrwsr-x          4,096 2022/03/07 11:17:17 jammy-desktop      -rw-rw-r--         25,950 2022/03/07 11:17:44 Release          
drwxrwsr-x          4,096 2022/03/07 11:17:44 .                  -rw-rw-r--         26,799 2022/03/07 11:17:44 InRelease        
drwxrwsr-x          4,096 2022/03/07 11:17:44 jammy-utils        -rw-rw-r--      3,521,412 2022/03/07 11:17:15 Contents-amd64.gz
drwxrwsr-x          4,096 2022/03/07 11:17:44 main               -rw-rw-r--            800 2022/03/07 11:17:44 Release.gpg      
-rw-rw-r--     15,940,852 2022/03/07 11:16:55 Contents-armhf.gz

imola is origin and stpete is currently serving rsync. Others need 12-24h.

matthijskooijman commented 2 years ago

When did you check this? Cause now it seems alright.

Just an hour ago. Checked again, still wrong.

With rsync, I do get recent timestamps like you, but the file contents is still wrong:

$ rsync -av rsync://rsync.armbian.com/apt/dists/focal/InRelease .
receiving incremental file list
InRelease

sent 43 bytes  received 26,895 bytes  17,958.67 bytes/sec
total size is 26,799  speedup is 0.99
$ grep Date InRelease 
Date: Tue, 1 Feb 2022 17:25:15 UTC

I tested with focal, since that is what I tested before, but rsyncing Jammy shows the same result.

igorpecovnik commented 2 years ago

Thank you!!

tl;dr; Failed stuck rsync, copy / paste bug in updating script.

Main repository wasn't changing much if anything until last week so it was not possible to notice. Perhaps we need to implement unit testing to cover this ;)

Sync has to catch up ... still working.

matthijskooijman commented 2 years ago

Redoing my earlier tests, it seems that all mirrors I tested are now up-to-date (March 10 in the release file), so I'm going to close this. Thanks!