search

RPi-Distro / repo

Issue tracking for the archive.raspberrypi.com repo
37 stars 1 forks source link

Mirrors not in sync? #136

Closed der-eismann closed 5 years ago

der-eismann commented 5 years ago

Hey everyone,

we are building our own Raspberry images for internal use, but the builds are failing for a while now while installing various packages. This happened on 8th March, but it was the only time we got a 500.

E: Failed to fetch http://archive.raspberrypi.org/debian/pool/main/r/raspberrypi-firmware/raspberrypi-bootloader_1.20190401-1_armhf.deb  500  Internal Server Error [IP: 93.93.128.133 80]
E: Failed to fetch http://archive.raspberrypi.org/debian/pool/main/r/raspberrypi-firmware/raspberrypi-kernel_1.20190401-1_armhf.deb  500  Internal Server Error [IP: 93.93.128.133 80]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

This happened on 1st and on 15th March while still on the 2018-11-13 branch from pi-gen

E: Failed to fetch http://archive.raspberrypi.org/debian/pool/ui/l/lxplug-ptbatt/lxplug-ptbatt_0.5_armhf.deb  404  Not Found [IP: 93.93.130.214 80]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

Today we wanted to fix it by switching to the 2019-04-08 branch, but now other packages are failing (~5 times in a row now):

E: Failed to fetch http://archive.raspberrypi.org/debian/pool/main/r/raspberrypi-firmware/libraspberrypi-dev_1.20190517-1_armhf.deb  404  Not Found [IP: 93.93.128.230 80]
E: Failed to fetch http://archive.raspberrypi.org/debian/pool/main/r/raspberrypi-firmware/libraspberrypi-doc_1.20190517-1_armhf.deb  404  Not Found [IP: 93.93.128.230 80]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

The package seems to be indeed missing on that specific mirror so it seems like they are not in sync. Can you fix this please?

XECDesign commented 5 years ago

Thanks for the report. I've forwarded it on.

XECDesign commented 5 years ago

I was told the underlying problem was fixed and the repo should've been syncing about half an hour ago, so I suspect it should be resolved now.

der-eismann commented 5 years ago

Unfortunately it is not.

Steps to reproduce:

  1. Verify the malicious server is still in the DNS record for archive.raspberrypi.org
  2. Edit /etc/hosts and add the following line: 
    93.93.128.230 archive.raspberrypi.org
  3. Open the package link in the browser => results in 404
der-eismann commented 5 years ago

Alright, I was able to do one successful build, so it seems at least the missing packages are back.

However it still fails regularly with a new error:

Get:1 http://raspbian.raspberrypi.org/raspbian stretch InRelease [15.0 kB]
Get:2 http://archive.raspberrypi.org/debian stretch InRelease [25.4 kB]
Get:3 http://raspbian.raspberrypi.org/raspbian stretch/main armhf Packages [11.7 MB]
Get:4 http://raspbian.raspberrypi.org/raspbian stretch/contrib armhf Packages [56.9 kB]
Get:5 http://raspbian.raspberrypi.org/raspbian stretch/non-free armhf Packages [95.5 kB]
Get:6 http://raspbian.raspberrypi.org/raspbian stretch/rpi armhf Packages [1360 B]
Get:7 http://archive.raspberrypi.org/debian stretch/main armhf Packages [223 kB]
Err:7 http://archive.raspberrypi.org/debian stretch/main armhf Packages
  Hash Sum mismatch
  Hashes of expected file:
   - Filesize:222777 [weak]
   - SHA256:7f544abdfa05d8c53081fa1685f2ca626d20d5a54254440a581ad305360413e0
   - SHA1:5209e2b0b68ef0f15afdfd6c5fb86a5d32ed0b09 [weak]
   - MD5Sum:0d49d6a1151c968d629a5c9e1975f260 [weak]
  Hashes of received file:
   - SHA256:ae723d811f8b00588f6bb0e765fd03a7bcf30c340f0229c6173d22e74fc2bbf0
   - SHA1:567dea177aa56f9ab883892f39c01b9230b11d94 [weak]
   - MD5Sum:d3df0eb8773b63449da374c60afe02d2 [weak]
   - Filesize:221206 [weak]
  Last modification reported: Mon, 20 May 2019 13:15:28 +0000
  Release file created at: Thu, 25 Apr 2019 09:55:18 +0000
Get:8 http://archive.raspberrypi.org/debian stretch/ui armhf Packages [44.9 kB]
Err:8 http://archive.raspberrypi.org/debian stretch/ui armhf Packages

Fetched 12.1 MB in 8s (1394 kB/s)
Reading package lists...
E: Failed to fetch http://archive.raspberrypi.org/debian/dists/stretch/main/binary-armhf/Packages.gz  Hash Sum mismatch
   Hashes of expected file:
    - Filesize:222777 [weak]
    - SHA256:7f544abdfa05d8c53081fa1685f2ca626d20d5a54254440a581ad305360413e0
    - SHA1:5209e2b0b68ef0f15afdfd6c5fb86a5d32ed0b09 [weak]
    - MD5Sum:0d49d6a1151c968d629a5c9e1975f260 [weak]
   Hashes of received file:
    - SHA256:ae723d811f8b00588f6bb0e765fd03a7bcf30c340f0229c6173d22e74fc2bbf0
    - SHA1:567dea177aa56f9ab883892f39c01b9230b11d94 [weak]
    - MD5Sum:d3df0eb8773b63449da374c60afe02d2 [weak]
    - Filesize:221206 [weak]
   Last modification reported: Mon, 20 May 2019 13:15:28 +0000
   Release file created at: Thu, 25 Apr 2019 09:55:18 +0000
E: Failed to fetch http://archive.raspberrypi.org/debian/dists/stretch/ui/binary-armhf/Packages.gz  
E: Some index files failed to download. They have been ignored, or old ones used instead.

Any idea why this happens?

joolswills commented 5 years ago

Still a problem. Mirror 93.93.130.214 is not up to date - 10 days ago I had the same problem with 93.93.128.130

$ sudo apt-get update; sudo apt-get dist-upgrade
Get:1 http://raspbian.raspberrypi.org/raspbian stretch InRelease [15.0 kB]
Get:2 http://archive.raspberrypi.org/debian stretch InRelease [25.4 kB]
Hit:3 http://pipplware.pplware.pt/pipplware/dists/stretch/main/binary ./ InRelease
Get:4 http://raspbian.raspberrypi.org/raspbian stretch/main armhf Packages [11.7 MB]
Ign:5 https://apt.tvheadend.org/stable raspbian-stretch InRelease
Get:6 https://apt.tvheadend.org/stable raspbian-stretch Release [470 B]
Get:7 https://apt.tvheadend.org/stable raspbian-stretch Release.gpg [455 B]    
Get:8 http://archive.raspberrypi.org/debian stretch/main armhf Packages [221 kB]
Get:9 http://archive.raspberrypi.org/debian stretch/main armhf Contents (deb) [717 kB]
Get:10 http://raspbian.raspberrypi.org/raspbian stretch/main armhf Contents (deb) [35.6 MB]
Fetched 48.2 MB in 37s (1,271 kB/s)                                                                                 
Reading package lists... Done
Reading package lists... Done
Building dependency tree        
Reading state information... Done
Calculating upgrade... Done
The following package was automatically installed and is no longer required:
  libyajl2
Use 'sudo apt autoremove' to remove it.
The following packages will be upgraded:
  libavcodec-dev libavcodec57 libavdevice-dev libavdevice57 libavfilter-dev libavfilter6 libavformat-dev
  libavformat57 libavresample-dev libavresample3 libavutil-dev libavutil55 libpostproc-dev libpostproc54
  libraspberrypi-bin libraspberrypi-dev libraspberrypi-doc libraspberrypi0 libswresample-dev libswresample2
  libswscale-dev libswscale4 raspberrypi-bootloader raspberrypi-kernel raspberrypi-kernel-headers
25 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 102 MB of archives.
After this operation, 9,744 kB of additional disk space will be used.
Do you want to continue? [Y/n] y
Err:1 http://archive.raspberrypi.org/debian stretch/main armhf libavdevice-dev armhf 7:3.2.14-1~deb9u1+rpt1
  404  Not Found [IP: 93.93.130.214 80]
Err:2 http://archive.raspberrypi.org/debian stretch/main armhf libavfilter-dev armhf 7:3.2.14-1~deb9u1+rpt1
  404  Not Found [IP: 93.93.130.214 80]
Err:3 http://archive.raspberrypi.org/debian stretch/main armhf libavformat-dev armhf 7:3.2.14-1~deb9u1+rpt1
  404  Not Found [IP: 93.93.130.214 80]
Err:4 http://archive.raspberrypi.org/debian stretch/main armhf libavformat57 armhf 7:3.2.14-1~deb9u1+rpt1
  404  Not Found [IP: 93.93.130.214 80]
Err:5 http://archive.raspberrypi.org/debian stretch/main armhf libavcodec-dev armhf 7:3.2.14-1~deb9u1+rpt1
  404  Not Found [IP: 93.93.130.214 80]
Err:6 http://archive.raspberrypi.org/debian stretch/main armhf libswresample-dev armhf 7:3.2.14-1~deb9u1+rpt1
  404  Not Found [IP: 93.93.130.214 80]
Err:7 http://archive.raspberrypi.org/debian stretch/main armhf libswresample2 armhf 7:3.2.14-1~deb9u1+rpt1
  404  Not Found [IP: 93.93.130.214 80]
Err:8 http://archive.raspberrypi.org/debian stretch/main armhf libswscale-dev armhf 7:3.2.14-1~deb9u1+rpt1
  404  Not Found [IP: 93.93.130.214 80]
Err:9 http://archive.raspberrypi.org/debian stretch/main armhf libswscale4 armhf 7:3.2.14-1~deb9u1+rpt1
  404  Not Found [IP: 93.93.130.214 80]
Err:10 http://archive.raspberrypi.org/debian stretch/main armhf libpostproc-dev armhf 7:3.2.14-1~deb9u1+rpt1
  404  Not Found [IP: 93.93.130.214 80]
Err:11 http://archive.raspberrypi.org/debian stretch/main armhf libpostproc54 armhf 7:3.2.14-1~deb9u1+rpt1
  404  Not Found [IP: 93.93.130.214 80]
Err:12 http://archive.raspberrypi.org/debian stretch/main armhf libavresample-dev armhf 7:3.2.14-1~deb9u1+rpt1
  404  Not Found [IP: 93.93.130.214 80]
Err:13 http://archive.raspberrypi.org/debian stretch/main armhf libavresample3 armhf 7:3.2.14-1~deb9u1+rpt1
  404  Not Found [IP: 93.93.130.214 80]
Err:14 http://archive.raspberrypi.org/debian stretch/main armhf libavutil-dev armhf 7:3.2.14-1~deb9u1+rpt1
  404  Not Found [IP: 93.93.130.214 80]
Err:15 http://archive.raspberrypi.org/debian stretch/main armhf libavutil55 armhf 7:3.2.14-1~deb9u1+rpt1
  404  Not Found [IP: 93.93.130.214 80]
Get:16 http://archive.raspberrypi.org/debian stretch/main armhf libraspberrypi-doc armhf 1.20190517-1 [31.4 MB]
Get:17 http://archive.raspberrypi.org/debian stretch/main armhf libraspberrypi-bin armhf 1.20190517-1 [332 kB]
Get:18 http://archive.raspberrypi.org/debian stretch/main armhf libraspberrypi-dev armhf 1.20190517-1 [404 kB]
Get:19 http://archive.raspberrypi.org/debian stretch/main armhf raspberrypi-kernel armhf 1.20190517-1 [35.1 MB]
Get:20 http://archive.raspberrypi.org/debian stretch/main armhf libraspberrypi0 armhf 1.20190517-1 [839 kB]
Get:21 http://archive.raspberrypi.org/debian stretch/main armhf raspberrypi-bootloader armhf 1.20190517-1 [3,567 kB]
Err:22 http://archive.raspberrypi.org/debian stretch/main armhf libavcodec57 armhf 7:3.2.14-1~deb9u1+rpt1
  404  Not Found [IP: 93.93.130.214 80]
Err:23 http://archive.raspberrypi.org/debian stretch/main armhf libavfilter6 armhf 7:3.2.14-1~deb9u1+rpt1
  404  Not Found [IP: 93.93.130.214 80]
Err:24 http://archive.raspberrypi.org/debian stretch/main armhf libavdevice57 armhf 7:3.2.14-1~deb9u1+rpt1
  404  Not Found [IP: 93.93.130.214 80]
Get:25 http://archive.raspberrypi.org/debian stretch/main armhf raspberrypi-kernel-headers armhf 1.20190517-1 [16.7 MB]
Fetched 88.3 MB in 4s (18.2 MB/s)                       
E: Failed to fetch http://archive.raspberrypi.org/debian/pool/main/f/ffmpeg/libavdevice-dev_3.2.14-1~deb9u1+rpt1_armhf.deb  404  Not Found [IP: 93.93.130.214 80]
E: Failed to fetch http://archive.raspberrypi.org/debian/pool/main/f/ffmpeg/libavfilter-dev_3.2.14-1~deb9u1+rpt1_armhf.deb  404  Not Found [IP: 93.93.130.214 80]
E: Failed to fetch http://archive.raspberrypi.org/debian/pool/main/f/ffmpeg/libavformat-dev_3.2.14-1~deb9u1+rpt1_armhf.deb  404  Not Found [IP: 93.93.130.214 80]
E: Failed to fetch http://archive.raspberrypi.org/debian/pool/main/f/ffmpeg/libavformat57_3.2.14-1~deb9u1+rpt1_armhf.deb  404  Not Found [IP: 93.93.130.214 80]
E: Failed to fetch http://archive.raspberrypi.org/debian/pool/main/f/ffmpeg/libavcodec-dev_3.2.14-1~deb9u1+rpt1_armhf.deb  404  Not Found [IP: 93.93.130.214 80]
E: Failed to fetch http://archive.raspberrypi.org/debian/pool/main/f/ffmpeg/libswresample-dev_3.2.14-1~deb9u1+rpt1_armhf.deb  404  Not Found [IP: 93.93.130.214 80]
E: Failed to fetch http://archive.raspberrypi.org/debian/pool/main/f/ffmpeg/libswresample2_3.2.14-1~deb9u1+rpt1_armhf.deb  404  Not Found [IP: 93.93.130.214 80]
E: Failed to fetch http://archive.raspberrypi.org/debian/pool/main/f/ffmpeg/libswscale-dev_3.2.14-1~deb9u1+rpt1_armhf.deb  404  Not Found [IP: 93.93.130.214 80]
E: Failed to fetch http://archive.raspberrypi.org/debian/pool/main/f/ffmpeg/libswscale4_3.2.14-1~deb9u1+rpt1_armhf.deb  404  Not Found [IP: 93.93.130.214 80]
E: Failed to fetch http://archive.raspberrypi.org/debian/pool/main/f/ffmpeg/libpostproc-dev_3.2.14-1~deb9u1+rpt1_armhf.deb  404  Not Found [IP: 93.93.130.214 80]
E: Failed to fetch http://archive.raspberrypi.org/debian/pool/main/f/ffmpeg/libpostproc54_3.2.14-1~deb9u1+rpt1_armhf.deb  404  Not Found [IP: 93.93.130.214 80]
E: Failed to fetch http://archive.raspberrypi.org/debian/pool/main/f/ffmpeg/libavresample-dev_3.2.14-1~deb9u1+rpt1_armhf.deb  404  Not Found [IP: 93.93.130.214 80]
E: Failed to fetch http://archive.raspberrypi.org/debian/pool/main/f/ffmpeg/libavresample3_3.2.14-1~deb9u1+rpt1_armhf.deb  404  Not Found [IP: 93.93.130.214 80]
E: Failed to fetch http://archive.raspberrypi.org/debian/pool/main/f/ffmpeg/libavutil-dev_3.2.14-1~deb9u1+rpt1_armhf.deb  404  Not Found [IP: 93.93.130.214 80]
E: Failed to fetch http://archive.raspberrypi.org/debian/pool/main/f/ffmpeg/libavutil55_3.2.14-1~deb9u1+rpt1_armhf.deb  404  Not Found [IP: 93.93.130.214 80]
E: Failed to fetch http://archive.raspberrypi.org/debian/pool/main/f/ffmpeg/libavcodec57_3.2.14-1~deb9u1+rpt1_armhf.deb  404  Not Found [IP: 93.93.130.214 80]
E: Failed to fetch http://archive.raspberrypi.org/debian/pool/main/f/ffmpeg/libavfilter6_3.2.14-1~deb9u1+rpt1_armhf.deb  404  Not Found [IP: 93.93.130.214 80]
E: Failed to fetch http://archive.raspberrypi.org/debian/pool/main/f/ffmpeg/libavdevice57_3.2.14-1~deb9u1+rpt1_armhf.deb  404  Not Found [IP: 93.93.130.214 80]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
XECDesign commented 5 years ago

Thanks, Jools. It should be okay now.

They've now found the root cause of the problem and are looking at also adding monitoring to detect if things get out of sync.