[Bug]: cloud build spends 15 minutes on Unpacking rootfs as cmd COPY pyproject.toml . requires it.

racinmat commented 3 months ago

Contact Details

No response

What happened?

cloud run spends 15 minutes on Unpacking rootfs as cmd COPY pyproject.toml . requires it. even when it's single file and it's already cached

wanna.yaml file

see the wanna-heur-eml

wanna-ml CLI command

anything that triggers the cloud build

Relevant log output

No response

jsilva commented 3 months ago

Addressed in https://github.com/avast/wanna-ml/pull/107 and this issue is mostly for big fat images.

Please try the 0.3.0 release and get back to me. Thx

jsilva commented 3 months ago

Pease also see the docs https://avast.github.io/wanna-ml/tutorial/docker/#parameters-for-docker-section where settings

cloud_build_kaniko_version - [str] (optional) which https://github.com/GoogleContainerTools/kaniko/ version to use
cloud_build_kaniko_flags - [str] (optional) which https://github.com/GoogleContainerTools/kaniko/ flags to use

can be used to tweak cloud build builder https://github.com/GoogleContainerTools/kaniko/

jsilva commented 3 months ago

After some more investigation there isn't much we can about this. I verified that the report is not fully accurate if there are no changes in the COPY ... files

INFO[0001] Using caching version of cmd: COPY pyproject.toml . 
....
INFO[0001] Using caching version of cmd: COPY poetry.lock .
...
INFO[0001] Using caching version of cmd: RUN poetry install --only main --no-root

When indeed there are changes let's say COPY . . indeed the INFO[0001] Unpacking rootfs as cmd COPY . . requires it. kicks in and it pulls all base image.

I have tried several kaniko options and did not find possible speed ups here.

For this cases users may prefer to build the container locally since the layers are locally cached, something just not possible with current kaniko and with the fact new worker boots fresh for each build.

avast / wanna-ml