search

TuxML / tuxml

The main repository of the TuxML project contains the scripts for building Linux kernel configurations in the large.
Apache License 2.0
4 stars 2 forks source link

Failure of Docker image build due to apt updates? #6

Open FAMILIAR-project opened 5 years ago

FAMILIAR-project commented 5 years ago 
vmacher:ProjetIrma macher1$ python3 kernel_generator.py --dev 1
You aren't in the docker group, hence you will be ask superuser access.
You are using the development version, whose can be unstable.
Trying to update docker image...
An error occured when updating. Cleaning and force update...
dev: Pulling from tuxml/tartuxml
c5e155d5a1d1: Pull complete
0f46b6428436: Pull complete
a78ef8847ff8: Pull complete
5e3a58262a73: Pull complete
2c6b891b2208: Pull complete
495aaf83b9dc: Pull complete
e9545ca8a354: Pull complete
c91d8bcbca94: Pull complete
82d7cfb0263a: Pull complete
d37fbdc2feea: Pull complete
cc7e9666ca75: Pull complete
79c7f4a0b1d3: Pull complete
aab94b7e23f3: Pull complete
9f4433c0b806: Pull complete
8616539818c7: Pull complete
b0fc83a59e1c: Pull complete
Digest: sha256:0e64786721969d196b3acfed1e2f42822499cebe5c84712549d8568170113e78
Status: Downloaded newer image for tuxml/tartuxml:dev
Password:
Sending build context to Docker daemon  360.9MB
Step 1/4 : FROM tuxml/tartuxml:dev
 ---> fbb0a783e08b
Step 2/4 : RUN tar xf /TuxML/linux-4.13.3.tar.xz -C /TuxML && rm /TuxML/linux-4.13.3.tar.xz
 ---> Running in 127cb0fa3508
Removing intermediate container 127cb0fa3508
 ---> 5a3808f1bda9
Step 3/4 : RUN tar xf /TuxML/TuxML.tar.xz -C /TuxML && rm /TuxML/TuxML.tar.xz
 ---> Running in 463deaaf5ec8
Removing intermediate container 463deaaf5ec8
 ---> dc2a4e496afa
Step 4/4 : RUN apt-get install -qq -y --no-install-recommends $(cat /dependencies.txt)
 ---> Running in fc30779d6ebc
E: Failed to fetch http://security.debian.org/debian-security/pool/updates/main/q/qemu/qemu-system_2.8+dfsg-6+deb9u6_amd64.deb  404  Not Found
E: Failed to fetch http://security.debian.org/debian-security/pool/updates/main/q/qemu/qemu-utils_2.8+dfsg-6+deb9u6_amd64.deb  404  Not Found
E: Failed to fetch http://security.debian.org/debian-security/pool/updates/main/q/qemu/qemu_2.8+dfsg-6+deb9u6_amd64.deb  404  Not Found
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
The command '/bin/sh -c apt-get install -qq -y --no-install-recommends $(cat /dependencies.txt)' returned a non-zero code: 100
An error as occured while building the image. Retrying...
Sending build context to Docker daemon  360.9MB
Step 1/4 : FROM tuxml/tartuxml:dev
 ---> fbb0a783e08b
Step 2/4 : RUN tar xf /TuxML/linux-4.13.3.tar.xz -C /TuxML && rm /TuxML/linux-4.13.3.tar.xz
 ---> Using cache
 ---> 5a3808f1bda9
Step 3/4 : RUN tar xf /TuxML/TuxML.tar.xz -C /TuxML && rm /TuxML/TuxML.tar.xz
 ---> Using cache
 ---> dc2a4e496afa
Step 4/4 : RUN apt-get install -qq -y --no-install-recommends $(cat /dependencies.txt)
 ---> Running in bb7b95250696
E: Failed to fetch http://security.debian.org/debian-security/pool/updates/main/q/qemu/qemu-system_2.8+dfsg-6+deb9u6_amd64.deb  404  Not Found
E: Failed to fetch http://security.debian.org/debian-security/pool/updates/main/q/qemu/qemu-utils_2.8+dfsg-6+deb9u6_amd64.deb  404  Not Found
E: Failed to fetch http://security.debian.org/debian-security/pool/updates/main/q/qemu/qemu_2.8+dfsg-6+deb9u6_amd64.deb  404  Not Found
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
The command '/bin/sh -c apt-get install -qq -y --no-install-recommends $(cat /dependencies.txt)' returned a non-zero code: 100
Traceback (most recent call last):
  File "kernel_generator.py", line 122, in get_id_docker_image
    universal_newlines=True
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 395, in check_output
    **kwargs).stdout
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 487, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command 'sudo docker image ls --format "{{.Repository}}:{{.Tag}} {{.ID}}" | grep "tuxml/tartuxml:dev"' returned non-zero exit status 1.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "kernel_generator.py", line 395, in docker_image_update
    id_image_base = get_id_docker_image(image=__COMPRESSED_IMAGE, tag=tag)
  File "kernel_generator.py", line 125, in get_id_docker_image
    raise NotImplementedError("No ID found") from ex
NotImplementedError: No ID found
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "kernel_generator.py", line 175, in docker_build
    subprocess.check_call(str_build, shell=True)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 347, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'sudo docker build -t tuxml/tuxml:dev .' returned non-zero exit status 100.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "kernel_generator.py", line 707, in <module>
    have_been_updated = docker_image_update(tag)
  File "kernel_generator.py", line 421, in docker_image_update
    docker_uncompress_image(tag)
  File "kernel_generator.py", line 380, in docker_uncompress_image
    path="."
  File "kernel_generator.py", line 180, in docker_build
    subprocess.check_call(str_build, shell=True)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 347, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'sudo docker build -t tuxml/tuxml:dev .' returned non-zero exit status 100.

indeed

E: Failed to fetch http://security.debian.org/debian-security/pool/updates/main/q/qemu/qemu-system_2.8+dfsg-6+deb9u6_amd64.deb  404  Not Found
E: Failed to fetch http://security.debian.org/debian-security/pool/updates/main/q/qemu/qemu-utils_2.8+dfsg-6+deb9u6_amd64.deb  404  Not Found
E: Failed to fetch http://security.debian.org/debian-security/pool/updates/main/q/qemu/qemu_2.8+dfsg-6+deb9u6_amd64.deb  404  Not Found

http://security-cdn.debian.org/debian-security/pool/updates/main/q/qemu/ packages about deb9u7 are available now, not u6

mipicard commented 5 years ago

Related to https://github.com/TuxML/ProjetIrma/issues/128

mipicard commented 5 years ago

Temporary measure would be to clean your local docker repository and fully rebuild the image with docker_image_tuxml.py and the full_rebuild option. After, update the image on docker hub.

FAMILIAR-project commented 5 years ago

This is what we've done, thanks! apt-get update seems necessary to update the "sources" see the two commits here: 5873a5216a05b427db8f2dd1e1f77ae091b9b758 aed9c261d2b585fa77c152e9c7591c525a69e7bc

mipicard commented 5 years ago

But here remains a little problem : when you use the apt-get update, you will retrieve newer version of the software available on the docker image, hence you won't be sure that 2 image build at different time but with the same tartuxml will be the same. Also, you will make an internet request, which is something that we wanted to avoid.

Further research with the issue TuxML/ProjetIrma#128 are needed, to be sure that we can't be entirely independant from the debian repository when we build the final tuxml image from an already build tartuml image. If we can't, then yes, running apt-get update will solve the problem, but with all the inconsistency it come with.

FAMILIAR-project commented 5 years ago

I think youre right its a temporary fix. The update in our case seems very lightweight and only needed for qemu packages... But un the long run it can be more problematic.

We are too much dependent on package servers. Some versions suddenly disappear :(