search

Nature40 / pimod

Reconfigure Raspberry Pi images with an easy, Docker-like configuration file
GNU General Public License v3.0
122 stars 19 forks source link

FROM remote URL not working #30

Closed aniongithub closed 3 years ago

aniongithub commented 3 years ago

When using FROM with the official URL for a lite image (https://downloads.raspberrypi.org/raspios_lite_armhf/images/raspios_lite_armhf-2021-03-25/2021-03-04-raspios-buster-armhf-lite.zip) from this page, I get

### FROM https://downloads.raspberrypi.org/raspios_lite_armhf/images/raspios_lite_armhf-2021-03-25/2021-03-04-raspios-buster-armhf-lite.zip
Fetching remote: https://downloads.raspberrypi.org/raspios_lite_armhf/images/raspios_lite_armhf-2021-03-25/2021-03-04-raspios-buster-armhf-lite.zip
--2021-04-26 17:01:43--  https://downloads.raspberrypi.org/raspios_lite_armhf/images/raspios_lite_armhf-2021-03-25/2021-03-04-raspios-buster-armhf-lite.zip%0D
Resolving downloads.raspberrypi.org (downloads.raspberrypi.org)... 93.93.130.212, 93.93.135.141, 46.235.230.122, ...
Connecting to downloads.raspberrypi.org (downloads.raspberrypi.org)|93.93.130.212|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2021-04-26 17:01:44 ERROR 404: Not Found.

but a wget with the same url gives me

wget https://downloads.raspberrypi.org/raspios_lite_armhf/images/raspios_lite_armhf-2021-03-25/2021-03-04-raspios-buster-armhf-lite.zip
--2021-04-26 10:03:01--  https://downloads.raspberrypi.org/raspios_lite_armhf/images/raspios_lite_armhf-2021-03-25/2021-03-04-raspios-buster-armhf-lite.zip
Resolving downloads.raspberrypi.org (downloads.raspberrypi.org)... 176.126.240.86, 176.126.240.84, 93.93.135.118, ...
Connecting to downloads.raspberrypi.org (downloads.raspberrypi.org)|176.126.240.86|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 463405209 (442M) [application/zip]
Saving to: ‘2021-03-04-raspios-buster-armhf-lite.zip.1’

buster-armhf-lite.zip.1           11%[=====>                                                    ]  51.60M  7.53MB/s    eta 58s
oxzi commented 3 years ago

Thanks for trying pimod and report your experience.

First, could you please provide your Pifile or at least its head regarding the FROM command?

Internally pimod also uses wget to fetch remote images, have a look at the from_remote_fetch function: https://github.com/Nature40/pimod/blob/dd10904fbf39bd0f20c22c22d46fc2fe0a5e5632/modules/from_remote.sh#L42-L44

However, the IP addresses are indicating that you have used two different mirror servers from the CDN. I successfully tried to access the file from your faulty mirror with:

curl \
  --header 'Host:downloads.raspberrypi.org' \
  --insecure \
  -o foo \
  https://93.93.130.212/raspios_lite_armhf/images/raspios_lite_armhf-2021-03-25/2021-03-04-raspios-buster-armhf-lite.zip

Thus, I would suggest that either the CDN server was faulty at that specific time or there was some something wrong with your Pifile.

aniongithub commented 3 years ago 
FROM https://downloads.raspberrypi.org/raspios_lite_armhf/images/raspios_lite_armhf-2021-03-25/2021-03-04-raspios-buster-armhf-lite.zip

RUN curl -sSL get.docker.com | sh
RUN usermod -aG docker pi

Here is my pifile. I can't imagine that the CDN being down makes sense because I tried subsequent invocations of wget and docker run in the same console, right after one another (no time-gap) - multiple times. The wget always succeeds, but the

docker run --rm --privileged -v $PWD:/pimod pimod pimod.sh examples/Rpi-Docker.Pifile

always fails with the log shown above. Also note that both invocations used the exact same URL. How it resolves to 2 different CDNs is a mystery to me (not something I did) - maybe a side-effect of using Docker?

aniongithub commented 3 years ago

I also confirm that

running

root@dc9cb31b2187:/# wget https://downloads.raspberrypi.org/raspios_lite_armhf/images/raspios_lite_armhf-2021-03-25/2021-03-04-raspios-buster-armhf-lite.zip
--2021-04-26 20:17:21--  https://downloads.raspberrypi.org/raspios_lite_armhf/images/raspios_lite_armhf-2021-03-25/2021-03-04-raspios-buster-armhf-lite.zip
Resolving downloads.raspberrypi.org (downloads.raspberrypi.org)... 176.126.240.86, 176.126.240.84, 46.235.230.122, ...
Connecting to downloads.raspberrypi.org (downloads.raspberrypi.org)|176.126.240.86|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 463405209 (442M) [application/zip]
Saving to: '2021-03-04-raspios-buster-armhf-lite.zip'

                2021-03-04-ra  15%[======>                                           ]  68.55M  4.71MB/s    eta 81s

works fine, which is a docker container I ran using: docker run -it --rm ubuntu. I did this to eliminate docker being the cause.

aniongithub commented 3 years ago

This was my own fault - some quirk of the WSL (Windows Subsystem for Linux caused VS code to use CRLF line endings. Closing.

oxzi commented 3 years ago

I added a check to warn about potentially wrong encodings in 8b9db075c493a2f52215b588a3f8fd5c766ae328.

Thanks a lot for reporting this issue!