[Bug Report] Docker container always rebuilds upon "start"

[vassil-atn]

Describe the bug

Whenever the docker container is started (with ./container.py start) it gets rebuilt despite no changes happening to the Dockerfile which takes about two minutes each time. Some parts do get loaded from cache but not all of them, which still takes a significant amount of time to build. Is this intended or a bug?

Steps to reproduce

$ ./docker/container.py start
[INFO] Using container profile: base
[INFO] X11 Forwarding is configured as '1' in '.container.cfg'.
    To disable X11 forwarding, set 'X11_FORWARDING_ENABLED=0' in '.container.cfg'.
[INFO] Building the docker image and starting the container 'isaac-lab-base' in the background...

[+] Building 94.9s (19/19) FINISHED                                                                                      docker:default
 => [isaac-lab-base internal] load build definition from Dockerfile.base                                                           0.0s
 => => transferring dockerfile: 4.53kB                                                                                             0.0s
 => WARN: InvalidDefaultArgInFrom: Default value for ARG nvcr.io/nvidia/isaac-sim:${ISAACSIM_VERSION_ARG} results in empty or inv  0.0s
 => [isaac-lab-base internal] load metadata for nvcr.io/nvidia/isaac-sim:4.2.0                                                     0.7s
 => [isaac-lab-base internal] load .dockerignore                                                                                   0.0s
 => => transferring context: 2B                                                                                                    0.0s
 => [isaac-lab-base base  1/13] FROM nvcr.io/nvidia/isaac-sim:4.2.0@sha256:X  0.0s
 => [isaac-lab-base internal] load build context                                                                                   0.3s
 => => transferring context: 1.08MB                                                                                                0.3s
 => CACHED [isaac-lab-base base  2/13] RUN --mount=type=cache,target=/var/cache/apt     apt-get update && apt-get install -y --no  0.0s
 => [isaac-lab-base base  3/13] COPY ../ /workspace/isaaclab                                                                      12.8s
 => [isaac-lab-base base  4/13] RUN ln -sf /isaac-sim /workspace/isaaclab/_isaac_sim                                               0.2s
 => [isaac-lab-base base  5/13] RUN --mount=type=cache,target=/var/cache/apt     /workspace/isaaclab/isaaclab.sh -p /workspace/is  0.4s
 => [isaac-lab-base base  6/13] RUN mkdir -p /isaac-sim/kit/cache &&     mkdir -p /root/.cache/ov &&     mkdir -p /root/.cache/pi  0.3s
 => [isaac-lab-base base  7/13] RUN touch /bin/nvidia-smi &&     touch /bin/nvidia-debugdump &&     touch /bin/nvidia-persistence  0.3s 
 => [isaac-lab-base base  8/13] RUN --mount=type=cache,target=/root/.cache/pip     /workspace/isaaclab/isaaclab.sh --install      53.7s 
 => [isaac-lab-base base  9/13] RUN cd /workspace/isaaclab/rsl_rl-algorithms && /workspace/isaaclab/_isaac_sim/python.sh -m pip i  5.0s 
 => [isaac-lab-base base 10/13] RUN cd /workspace/isaaclab/PMTG && /workspace/isaaclab/_isaac_sim/python.sh -m pip install -e .    3.5s 
 => [isaac-lab-base base 11/13] RUN echo "export ISAACLAB_PATH=/workspace/isaaclab" >> ${HOME}/.bashrc &&     echo "alias isaacla  0.2s 
 => [isaac-lab-base base 12/13] RUN echo "machine api.wandb.ai login user password X" >> $  0.2s 
 => [isaac-lab-base base 13/13] WORKDIR /workspace/isaaclab                                                                        0.1s 
 => [isaac-lab-base] exporting to image                                                                                           17.0s 
 => => exporting layers                                                                                                           16.9s 
 => => writing image sha256:X                                       0.0s 
 => => naming to docker.io/library/isaac-lab-base                                                                                  0.0s
 => [isaac-lab-base] resolving provenance for metadata file                                                                        0.0s
[+] Building 0.8s (19/19) FINISHED                                                                                       docker:default
 => [isaac-lab-base internal] load build definition from Dockerfile.base                                                           0.0s
 => => transferring dockerfile: 4.53kB                                                                                             0.0s
 => WARN: InvalidDefaultArgInFrom: Default value for ARG nvcr.io/nvidia/isaac-sim:${ISAACSIM_VERSION_ARG} results in empty or inv  0.0s
 => [isaac-lab-base internal] load metadata for nvcr.io/nvidia/isaac-sim:4.2.0                                                     0.3s
 => [isaac-lab-base internal] load .dockerignore                                                                                   0.0s
 => => transferring context: 2B                                                                                                    0.0s
 => [isaac-lab-base base  1/13] FROM nvcr.io/nvidia/isaac-sim:4.2.0@sha256:X  0.0s
 => [isaac-lab-base internal] load build context                                                                                   0.3s
 => => transferring context: 1.04MB                                                                                                0.3s
 => CACHED [isaac-lab-base base  2/13] RUN --mount=type=cache,target=/var/cache/apt     apt-get update && apt-get install -y --no  0.0s
 => CACHED [isaac-lab-base base  3/13] COPY ../ /workspace/isaaclab                                                                0.0s
 => CACHED [isaac-lab-base base  4/13] RUN ln -sf /isaac-sim /workspace/isaaclab/_isaac_sim                                        0.0s
 => CACHED [isaac-lab-base base  5/13] RUN --mount=type=cache,target=/var/cache/apt     /workspace/isaaclab/isaaclab.sh -p /works  0.0s
 => CACHED [isaac-lab-base base  6/13] RUN mkdir -p /isaac-sim/kit/cache &&     mkdir -p /root/.cache/ov &&     mkdir -p /root/.c  0.0s
 => CACHED [isaac-lab-base base  7/13] RUN touch /bin/nvidia-smi &&     touch /bin/nvidia-debugdump &&     touch /bin/nvidia-pers  0.0s
 => CACHED [isaac-lab-base base  8/13] RUN --mount=type=cache,target=/root/.cache/pip     /workspace/isaaclab/isaaclab.sh --insta  0.0s
 => CACHED [isaac-lab-base base  9/13] RUN cd /workspace/isaaclab/rsl_rl-algorithms && /workspace/isaaclab/_isaac_sim/python.sh -  0.0s
 => CACHED [isaac-lab-base base 10/13] RUN cd /workspace/isaaclab/PMTG && /workspace/isaaclab/_isaac_sim/python.sh -m pip install  0.0s
 => CACHED [isaac-lab-base base 11/13] RUN echo "export ISAACLAB_PATH=/workspace/isaaclab" >> ${HOME}/.bashrc &&     echo "alias   0.0s
 => CACHED [isaac-lab-base base 12/13] RUN echo "machine api.wandb.ai login user password X  0.0s
 => CACHED [isaac-lab-base base 13/13] WORKDIR /workspace/isaaclab                                                                 0.0s
 => [isaac-lab-base] exporting to image                                                                                            0.0s
 => => exporting layers                                                                                                            0.0s
 => => writing image sha256:X                                       0.0s
 => => naming to docker.io/library/isaac-lab-base                                                                                  0.0s
 => [isaac-lab-base] resolving provenance for metadata file                                                                        0.0s
[+] Running 1/1

System Info

Describe the characteristic of your environment:

• Commit:
• Isaac Sim Version: 4.2.0
• OS: Ubuntu 20.04
• GPU: RTX 4090
• CUDA: 12,4
• GPU Driver: 550.120

Additional context

Add any other context about the problem here.

Checklist

• [X] I have checked that there is no similar issue in the repo (required)
• [X] I have checked that the issue is not in running Isaac Sim itself and is related to the repo

Acceptance Criteria

Add the criteria for which this task is considered done. If not known at issue creation time, you can add this once the issue is assigned.

• [ ] Fixed if bug.

[RandomOakForest]

This is due to a relatively early COPY in the Dockerfile which invalidates the build cache and forces a new rebuild of the layers that follow. We will review. Thanks.