search

opendatahub-io / notebooks

Notebook images for ODH
Apache License 2.0
17 stars 60 forks source link

RHOAIENG-10783: fix(Dockerfiles): combine `fix-permissions` RUN with the preceding RUN when possible #635

Closed jiridanek closed 2 months ago

jiridanek commented 2 months ago

Fixes #539

Makes changes to more images than

Description

The below step in jupyter-pytorch-ubi9-python-3_9 and probably other images creates a large layer. Fix the permissions in the previous step already.

STEP 5/5: RUN chmod -R g+w /opt/app-root/lib/python3.9/site-packages &&     fix-permissions /opt/app-root -P
COMMIT ghcr.io/jiridanek/notebooks/workbench-images:jupyter-pytorch-ubi9-python-3.9-2024a_20240523

Note that changing the metadata of files, for example, changing file permissions or ownership of a file, can also result in a copy_up operation, therefore duplicating the file to the writable layer.

from https://docs.docker.com/storage/storagedriver/

How Has This Been Tested?

Build images in CI, check sizes.

After:

jupyter-pytorch-ubi9-python-3.9-7d67798ec8625f47b5152882b27a2b79da8745dd           5bfda56883dc  2 minutes ago   20.6 GB

Before:

jupyter-pytorch-ubi9-python-3.9-main_4520c936792a92270f31044ac81e542a90ba6929           0c3707b5a463  5 days ago   25.9 GB

The pytorch images show the largest size difference, for the other images the impact is very small.

Merge criteria:

jiridanek commented 2 months ago

If possible, can we add some description, please.

I linked the Fixes https://github.com/opendatahub-io/notebooks/issues/539, isn't it sufficient? I'll copy the desc from there to here then.

also do we want to have these task to tracked under trivy, or separate JIRA ?

Not sure which is better. The previous RUN commands change (https://github.com/opendatahub-io/notebooks/pull/634) was not tracked under anything. This makes the image for trivy scan smaller, so it should be fine to track under trivy, i think.

jstourac commented 2 months ago

/lgtm

Thank you :guitar:

caponetto commented 2 months ago

@jiridanek could you please rebase this PR to pick up https://github.com/opendatahub-io/notebooks/pull/652?

caponetto commented 2 months ago

/lgtm

jiridanek commented 2 months ago

/override ci/prow/images /override ci/prow/habana-notebooks-e2e-tests /override ci/prow/notebooks-ubi9-e2e-tests /override ci/prow/rocm-notebooks-e2e-tests /override ci/prow/notebooks-ubi8-e2e-tests

openshift-ci[bot] commented 2 months ago

@jiridanek: Overrode contexts on behalf of jiridanek: ci/prow/habana-notebooks-e2e-tests, ci/prow/images, ci/prow/notebooks-ubi8-e2e-tests, ci/prow/notebooks-ubi9-e2e-tests, ci/prow/rocm-notebooks-e2e-tests

In response to [this](https://github.com/opendatahub-io/notebooks/pull/635#issuecomment-2273447254): >/override ci/prow/images >/override ci/prow/habana-notebooks-e2e-tests >/override ci/prow/notebooks-ubi9-e2e-tests >/override ci/prow/rocm-notebooks-e2e-tests >/override ci/prow/notebooks-ubi8-e2e-tests Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
jiridanek commented 2 months ago 
localhost:5000/workbench-images                                jupyter-pytorch-ubi9-python-3.9-3e35325d9e373ab7ab80f90407c64680be319b8c           339e75337381  2 minutes ago   20.6 GB
localhost:5000/workbench-images                                rocm-jupyter-pytorch-ubi9-python-3.9-3e35325d9e373ab7ab80f90407c64680be319b8c      26a5e8dfe20a  2 minutes ago   37.8 GB
localhost:5000/workbench-images                                cuda-jupyter-tensorflow-ubi9-python-3.9-3e35325d9e373ab7ab80f90407c64680be319b8c   47ddd578c751  59 seconds ago  18.7 GB
localhost:5000/workbench-images                                cuda-jupyter-datascience-ubi8-python-3.8-3e35325d9e373ab7ab80f90407c64680be319b8c  203356d021ed  13 seconds ago  14.7 GB

these are the first four newly built images after the changes

jiridanek commented 2 months ago 
ghcr.io/opendatahub-io/notebooks/workbench-images              jupyter-pytorch-ubi9-python-3.9-main_f84cf6981bcad1c9a81c9f5c600005fe4c4c5164           a7f31eb5e5ff  4 days ago  25.9 GB

looking at latest main, pytorch, where the savings are the most significant, shrank by ~5GB by doing this!

jiridanek commented 2 months ago

/approve

openshift-ci[bot] commented 2 months ago

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jiridanek

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files: - ~~[OWNERS](https://github.com/opendatahub-io/notebooks/blob/main/OWNERS)~~ [jiridanek] Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment
jiridanek commented 2 months ago

/override ci/prow/notebooks-ubi8-e2e-tests

failed due to failed image builds, known issue already present on main

openshift-ci[bot] commented 2 months ago

@jiridanek: Overrode contexts on behalf of jiridanek: ci/prow/notebooks-ubi8-e2e-tests

In response to [this](https://github.com/opendatahub-io/notebooks/pull/635#issuecomment-2273533176): >/override ci/prow/notebooks-ubi8-e2e-tests > >failed due to failed image builds Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
jiridanek commented 2 months ago

/override nosuchjob

openshift-ci[bot] commented 2 months ago

@jiridanek: /override requires failed status contexts, check run or a prowjob name to operate on. The following unknown contexts/checkruns were given:

Only the following failed contexts/checkruns were expected:

If you are trying to override a checkrun that has a space in it, you must put a double quote on the context.

In response to [this](https://github.com/opendatahub-io/notebooks/pull/635#issuecomment-2273579306): >/override nosuchjob Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
jiridanek commented 2 months ago

/override ci/prow/habana-notebooks-e2e-tests /override ci/prow/images /override ci/prow/notebook-cuda-jupyter-ds-ubi8-python-3-8-pr-image-mirror /override ci/prow/notebook-cuda-jupyter-ds-ubi9-python-3-9-pr-image-mirror /override ci/prow/notebook-cuda-jupyter-minimal-ubi8-python-3-8-pr-image-mirror /override ci/prow/notebook-cuda-jupyter-minimal-ubi9-python-3-9-pr-image-mirror /override ci/prow/notebook-cuda-jupyter-tf-ubi9-python-3-9-pr-image-mirror /override ci/prow/notebook-habana-1-10-0-ubi8-python-3-8-pr-image-mirror /override ci/prow/notebook-habana-1-13-0-ubi8-python-3-8-pr-image-mirror /override ci/prow/notebook-jupyter-datascience-ubi8-python-3-8-pr-image-mirror /override ci/prow/notebook-jupyter-datascience-ubi9-python-3-9-pr-image-mirror /override ci/prow/notebook-jupyter-minimal-ubi8-python-3-8-pr-image-mirror /override ci/prow/notebook-jupyter-minimal-ubi9-python-3-9-pr-image-mirror /override ci/prow/notebook-jupyter-pytorch-ubi9-python-3-9-pr-image-mirror /override ci/prow/notebook-jupyter-trustyai-ubi9-python-3-9-pr-image-mirror /override ci/prow/notebook-rocm-jupyter-pyt-ubi9-python-3-9-pr-image-mirror /override ci/prow/notebook-rocm-jupyter-tf-ubi9-python-3-9-pr-image-mirror /override ci/prow/notebooks-ubi8-e2e-tests /override ci/prow/notebooks-ubi9-e2e-tests /override ci/prow/rocm-notebooks-e2e-tests /override pull-ci-opendatahub-io-notebooks-2023a-images /override pull-ci-opendatahub-io-notebooks-main-habana-notebooks-e2e-tests /override pull-ci-opendatahub-io-notebooks-main-notebook-cuda-jupyter-ds-ubi8-python-3-8-pr-image-mirror /override pull-ci-opendatahub-io-notebooks-main-notebook-cuda-jupyter-ds-ubi9-python-3-9-pr-image-mirror /override pull-ci-opendatahub-io-notebooks-main-notebook-cuda-jupyter-minimal-ubi8-python-3-8-pr-image-mirror /override pull-ci-opendatahub-io-notebooks-main-notebook-cuda-jupyter-minimal-ubi9-python-3-9-pr-image-mirror /override pull-ci-opendatahub-io-notebooks-main-notebook-cuda-jupyter-tf-ubi9-python-3-9-pr-image-mirror /override pull-ci-opendatahub-io-notebooks-main-notebook-habana-1-10-0-ubi8-python-3-8-pr-image-mirror /override pull-ci-opendatahub-io-notebooks-main-notebook-habana-1-13-0-ubi8-python-3-8-pr-image-mirror /override pull-ci-opendatahub-io-notebooks-main-notebook-jupyter-datascience-ubi8-python-3-8-pr-image-mirror /override pull-ci-opendatahub-io-notebooks-main-notebook-jupyter-datascience-ubi9-python-3-9-pr-image-mirror /override pull-ci-opendatahub-io-notebooks-main-notebook-jupyter-minimal-ubi8-python-3-8-pr-image-mirror /override pull-ci-opendatahub-io-notebooks-main-notebook-jupyter-minimal-ubi9-python-3-9-pr-image-mirror /override pull-ci-opendatahub-io-notebooks-main-notebook-jupyter-pytorch-ubi9-python-3-9-pr-image-mirror /override pull-ci-opendatahub-io-notebooks-main-notebook-jupyter-trustyai-ubi9-python-3-9-pr-image-mirror /override pull-ci-opendatahub-io-notebooks-main-notebook-rocm-jupyter-pyt-ubi9-python-3-9-pr-image-mirror /override pull-ci-opendatahub-io-notebooks-main-notebook-rocm-jupyter-tf-ubi9-python-3-9-pr-image-mirror /override pull-ci-opendatahub-io-notebooks-main-notebooks-ubi8-e2e-tests /override pull-ci-opendatahub-io-notebooks-main-notebooks-ubi9-e2e-tests /override pull-ci-opendatahub-io-notebooks-main-rocm-notebooks-e2e-tests

openshift-ci[bot] commented 2 months ago

@jiridanek: Overrode contexts on behalf of jiridanek: ci/prow/habana-notebooks-e2e-tests, ci/prow/images, ci/prow/notebook-cuda-jupyter-ds-ubi8-python-3-8-pr-image-mirror, ci/prow/notebook-cuda-jupyter-ds-ubi9-python-3-9-pr-image-mirror, ci/prow/notebook-cuda-jupyter-minimal-ubi8-python-3-8-pr-image-mirror, ci/prow/notebook-cuda-jupyter-minimal-ubi9-python-3-9-pr-image-mirror, ci/prow/notebook-cuda-jupyter-tf-ubi9-python-3-9-pr-image-mirror, ci/prow/notebook-habana-1-10-0-ubi8-python-3-8-pr-image-mirror, ci/prow/notebook-habana-1-13-0-ubi8-python-3-8-pr-image-mirror, ci/prow/notebook-jupyter-datascience-ubi8-python-3-8-pr-image-mirror, ci/prow/notebook-jupyter-datascience-ubi9-python-3-9-pr-image-mirror, ci/prow/notebook-jupyter-minimal-ubi8-python-3-8-pr-image-mirror, ci/prow/notebook-jupyter-minimal-ubi9-python-3-9-pr-image-mirror, ci/prow/notebook-jupyter-pytorch-ubi9-python-3-9-pr-image-mirror, ci/prow/notebook-jupyter-trustyai-ubi9-python-3-9-pr-image-mirror, ci/prow/notebook-rocm-jupyter-pyt-ubi9-python-3-9-pr-image-mirror, ci/prow/notebook-rocm-jupyter-tf-ubi9-python-3-9-pr-image-mirror, ci/prow/notebooks-ubi8-e2e-tests, ci/prow/notebooks-ubi9-e2e-tests, ci/prow/rocm-notebooks-e2e-tests

In response to [this](https://github.com/opendatahub-io/notebooks/pull/635#issuecomment-2273581763): >/override ci/prow/habana-notebooks-e2e-tests >/override ci/prow/images >/override ci/prow/notebook-cuda-jupyter-ds-ubi8-python-3-8-pr-image-mirror >/override ci/prow/notebook-cuda-jupyter-ds-ubi9-python-3-9-pr-image-mirror >/override ci/prow/notebook-cuda-jupyter-minimal-ubi8-python-3-8-pr-image-mirror >/override ci/prow/notebook-cuda-jupyter-minimal-ubi9-python-3-9-pr-image-mirror >/override ci/prow/notebook-cuda-jupyter-tf-ubi9-python-3-9-pr-image-mirror >/override ci/prow/notebook-habana-1-10-0-ubi8-python-3-8-pr-image-mirror >/override ci/prow/notebook-habana-1-13-0-ubi8-python-3-8-pr-image-mirror >/override ci/prow/notebook-jupyter-datascience-ubi8-python-3-8-pr-image-mirror >/override ci/prow/notebook-jupyter-datascience-ubi9-python-3-9-pr-image-mirror >/override ci/prow/notebook-jupyter-minimal-ubi8-python-3-8-pr-image-mirror >/override ci/prow/notebook-jupyter-minimal-ubi9-python-3-9-pr-image-mirror >/override ci/prow/notebook-jupyter-pytorch-ubi9-python-3-9-pr-image-mirror >/override ci/prow/notebook-jupyter-trustyai-ubi9-python-3-9-pr-image-mirror >/override ci/prow/notebook-rocm-jupyter-pyt-ubi9-python-3-9-pr-image-mirror >/override ci/prow/notebook-rocm-jupyter-tf-ubi9-python-3-9-pr-image-mirror >/override ci/prow/notebooks-ubi8-e2e-tests >/override ci/prow/notebooks-ubi9-e2e-tests >/override ci/prow/rocm-notebooks-e2e-tests >/override pull-ci-opendatahub-io-notebooks-2023a-images >/override pull-ci-opendatahub-io-notebooks-main-habana-notebooks-e2e-tests >/override pull-ci-opendatahub-io-notebooks-main-notebook-cuda-jupyter-ds-ubi8-python-3-8-pr-image-mirror >/override pull-ci-opendatahub-io-notebooks-main-notebook-cuda-jupyter-ds-ubi9-python-3-9-pr-image-mirror >/override pull-ci-opendatahub-io-notebooks-main-notebook-cuda-jupyter-minimal-ubi8-python-3-8-pr-image-mirror >/override pull-ci-opendatahub-io-notebooks-main-notebook-cuda-jupyter-minimal-ubi9-python-3-9-pr-image-mirror >/override pull-ci-opendatahub-io-notebooks-main-notebook-cuda-jupyter-tf-ubi9-python-3-9-pr-image-mirror >/override pull-ci-opendatahub-io-notebooks-main-notebook-habana-1-10-0-ubi8-python-3-8-pr-image-mirror >/override pull-ci-opendatahub-io-notebooks-main-notebook-habana-1-13-0-ubi8-python-3-8-pr-image-mirror >/override pull-ci-opendatahub-io-notebooks-main-notebook-jupyter-datascience-ubi8-python-3-8-pr-image-mirror >/override pull-ci-opendatahub-io-notebooks-main-notebook-jupyter-datascience-ubi9-python-3-9-pr-image-mirror >/override pull-ci-opendatahub-io-notebooks-main-notebook-jupyter-minimal-ubi8-python-3-8-pr-image-mirror >/override pull-ci-opendatahub-io-notebooks-main-notebook-jupyter-minimal-ubi9-python-3-9-pr-image-mirror >/override pull-ci-opendatahub-io-notebooks-main-notebook-jupyter-pytorch-ubi9-python-3-9-pr-image-mirror >/override pull-ci-opendatahub-io-notebooks-main-notebook-jupyter-trustyai-ubi9-python-3-9-pr-image-mirror >/override pull-ci-opendatahub-io-notebooks-main-notebook-rocm-jupyter-pyt-ubi9-python-3-9-pr-image-mirror >/override pull-ci-opendatahub-io-notebooks-main-notebook-rocm-jupyter-tf-ubi9-python-3-9-pr-image-mirror >/override pull-ci-opendatahub-io-notebooks-main-notebooks-ubi8-e2e-tests >/override pull-ci-opendatahub-io-notebooks-main-notebooks-ubi9-e2e-tests >/override pull-ci-opendatahub-io-notebooks-main-rocm-notebooks-e2e-tests Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
openshift-ci[bot] commented 2 months ago

@jiridanek: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/runtime-rocm-tensorflow-ubi9-python-3-9-pr-image-mirror 1fdba81197db6cc58b9b03f3de3dfba0a46ccf2e link true /test runtime-rocm-tensorflow-ubi9-python-3-9-pr-image-mirror
ci/prow/runtime-rocm-pytorch-ubi9-python-3-9-pr-image-mirror 1fdba81197db6cc58b9b03f3de3dfba0a46ccf2e link true /test runtime-rocm-pytorch-ubi9-python-3-9-pr-image-mirror
ci/prow/rocm-runtimes-ubi9-e2e-tests 1fdba81197db6cc58b9b03f3de3dfba0a46ccf2e link true /test rocm-runtimes-ubi9-e2e-tests
ci/prow/runtimes-ubi8-e2e-tests 1fdba81197db6cc58b9b03f3de3dfba0a46ccf2e link true /test runtimes-ubi8-e2e-tests
ci/prow/runtimes-ubi9-e2e-tests 1fdba81197db6cc58b9b03f3de3dfba0a46ccf2e link true /test runtimes-ubi9-e2e-tests

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
jiridanek commented 2 months ago

/override ci/prow/notebook-cuda-jupyter-ds-ubi9-python-3-9-pr-image-mirror /override ci/prow/notebook-cuda-jupyter-minimal-ubi9-python-3-9-pr-image-mirror /override ci/prow/notebook-jupyter-pytorch-ubi9-python-3-9-pr-image-mirror

openshift-ci[bot] commented 2 months ago

@jiridanek: Overrode contexts on behalf of jiridanek: ci/prow/notebook-cuda-jupyter-ds-ubi9-python-3-9-pr-image-mirror, ci/prow/notebook-cuda-jupyter-minimal-ubi9-python-3-9-pr-image-mirror, ci/prow/notebook-jupyter-pytorch-ubi9-python-3-9-pr-image-mirror

In response to [this](https://github.com/opendatahub-io/notebooks/pull/635#issuecomment-2273586462): >/override ci/prow/notebook-cuda-jupyter-ds-ubi9-python-3-9-pr-image-mirror >/override ci/prow/notebook-cuda-jupyter-minimal-ubi9-python-3-9-pr-image-mirror >/override ci/prow/notebook-jupyter-pytorch-ubi9-python-3-9-pr-image-mirror Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.