Open joshuataylor opened 1 year ago
Seems to be related to https://github.com/docker/build-push-action/issues/545 . Could we have this configurable?
For multistage python builds, you might have many thousands of files in the venv lib (I think it was ~30k for our smallish project, this could be many MANY more for larger projects).
edit:
Ended up fixing this via this action:
- name: Build, tag, and push image to Amazon ECR
env:
ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
ECR_REPOSITORY: my-repo-name
IMAGE_TAG: ${{ github.sha }}
run: |
docker build -t $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG .
docker push $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG
- name: Deploy to Dagster Cloud
uses: dagster-io/dagster-cloud-action/actions/utils/deploy@v0.1
id: deploy
with:
dagster_cloud_api_token: ${{ secrets.DAGSTER_CLOUD_API_TOKEN }}
location: ${{ toJson(matrix.location) }}
organization_id: ${{ secrets.ORGANIZATION_ID }}
deployment: prod
pr: "${{ github.event.number }}"
image_tag: ${{ github.sha }}
env:
DAGSTER_CLOUD_API_TOKEN: ${{ secrets.DAGSTER_CLOUD_API_TOKEN }}
We use multistage builds, and we're seeing slow COPY performance between stages. I think it's the configuration setup in
hybrid_prod_deploy
.Our Docker setup
It's pretty simple:
Our dockerfile looks like this:
Here we can see we've got /opt/dagster/app setup, and we then setup in
/app/.venv
. I changed this from the default dagster directory, as when trying to diagnose this issue this was my first thought. I will be changing this back to/opt/dagster/app
. Mydagster_cloud.yaml
hasexecutable_path: /app/.venv/bin/python3
setup to use this python3 venv.Problem
I think the problem lives here: https://github.com/dagster-io/dagster-cloud-action/blob/main/actions/hybrid_prod_deploy/action.yml#LL43-L52
Raw github actions with above
``` 2022-11-19T03:57:55.3698127Z #13 [production 1/3] COPY --from=builder-base /app /app 2022-11-19T03:57:55.5203146Z #13 sha256:4b961533c7c8c1f7cfe7d5d52e0df85cce8eb237f71529a68da807d07811fc73 0B / 255.74MB 0.2s 2022-11-19T03:57:55.5203594Z #13 sha256:c9855a2acee514e35c3d27eebd4b15600fea80656165910ad9ee61f0693c06c4 0B / 112.81MB 0.2s 2022-11-19T03:57:55.5204474Z #13 sha256:04a9993ebea101b8473401cf28834c76c4e7a97ef0f463f6707e1ac09ae68220 0B / 3.34MB 0.2s 2022-11-19T03:57:55.5204865Z #13 sha256:ddc9bdde4697fb636db1b96654f7566da0e2b2ae2c333d46339441d441adb21e 0B / 232B 0.2s 2022-11-19T03:57:56.4222799Z #13 sha256:ddc9bdde4697fb636db1b96654f7566da0e2b2ae2c333d46339441d441adb21e 232B / 232B 1.1s done 2022-11-19T03:57:56.5727087Z #13 sha256:20b28bf2dde1a1b9653fb71a8f832a47d4cb7450cec5f2d822c122e2f86e4dca 0B / 12.10MB 0.2s 2022-11-19T03:57:58.0757808Z #13 sha256:04a9993ebea101b8473401cf28834c76c4e7a97ef0f463f6707e1ac09ae68220 1.05MB / 3.34MB 2.7s 2022-11-19T03:57:58.0758371Z #13 sha256:20b28bf2dde1a1b9653fb71a8f832a47d4cb7450cec5f2d822c122e2f86e4dca 1.05MB / 12.10MB 1.7s 2022-11-19T03:57:58.5255015Z #13 sha256:20b28bf2dde1a1b9653fb71a8f832a47d4cb7450cec5f2d822c122e2f86e4dca 2.10MB / 12.10MB 2.1s 2022-11-19T03:57:58.6493233Z #13 sha256:04a9993ebea101b8473401cf28834c76c4e7a97ef0f463f6707e1ac09ae68220 3.34MB / 3.34MB 3.4s done 2022-11-19T03:57:58.7995681Z #13 sha256:20b28bf2dde1a1b9653fb71a8f832a47d4cb7450cec5f2d822c122e2f86e4dca 3.15MB / 12.10MB 2.4s 2022-11-19T03:57:58.7996157Z #13 sha256:a603fa5e3b4127f210503aaa6189abf6286ee5a73deeaab460f8f33ebc6b64e2 0B / 31.41MB 0.2s 2022-11-19T03:57:59.1000619Z #13 sha256:20b28bf2dde1a1b9653fb71a8f832a47d4cb7450cec5f2d822c122e2f86e4dca 4.19MB / 12.10MB 2.7s 2022-11-19T03:57:59.2503238Z #13 sha256:20b28bf2dde1a1b9653fb71a8f832a47d4cb7450cec5f2d822c122e2f86e4dca 5.24MB / 12.10MB 2.9s 2022-11-19T03:57:59.6997636Z #13 sha256:a603fa5e3b4127f210503aaa6189abf6286ee5a73deeaab460f8f33ebc6b64e2 2.10MB / 31.41MB 1.1s 2022-11-19T03:57:59.8502931Z #13 sha256:20b28bf2dde1a1b9653fb71a8f832a47d4cb7450cec5f2d822c122e2f86e4dca 6.29MB / 12.10MB 3.5s 2022-11-19T03:57:59.9996852Z #13 sha256:20b28bf2dde1a1b9653fb71a8f832a47d4cb7450cec5f2d822c122e2f86e4dca 7.34MB / 12.10MB 3.6s 2022-11-19T03:57:59.9997352Z #13 sha256:a603fa5e3b4127f210503aaa6189abf6286ee5a73deeaab460f8f33ebc6b64e2 4.19MB / 31.41MB 1.4s 2022-11-19T03:58:00.2994214Z #13 sha256:c9855a2acee514e35c3d27eebd4b15600fea80656165910ad9ee61f0693c06c4 6.29MB / 112.81MB 5.0s 2022-11-19T03:58:00.2994729Z #13 sha256:20b28bf2dde1a1b9653fb71a8f832a47d4cb7450cec5f2d822c122e2f86e4dca 8.39MB / 12.10MB 3.9s 2022-11-19T03:58:00.5996429Z #13 sha256:4b961533c7c8c1f7cfe7d5d52e0df85cce8eb237f71529a68da807d07811fc73 8.39MB / 255.74MB 5.3s 2022-11-19T03:58:00.5997208Z #13 sha256:20b28bf2dde1a1b9653fb71a8f832a47d4cb7450cec5f2d822c122e2f86e4dca 9.44MB / 12.10MB 4.2s 2022-11-19T03:58:00.9003177Z #13 sha256:a603fa5e3b4127f210503aaa6189abf6286ee5a73deeaab460f8f33ebc6b64e2 6.29MB / 31.41MB 2.1s 2022-11-19T03:58:01.0502544Z #13 sha256:20b28bf2dde1a1b9653fb71a8f832a47d4cb7450cec5f2d822c122e2f86e4dca 10.49MB / 12.10MB 4.7s 2022-11-19T03:58:01.1793717Z #13 sha256:20b28bf2dde1a1b9653fb71a8f832a47d4cb7450cec5f2d822c122e2f86e4dca 12.10MB / 12.10MB 4.9s done 2022-11-19T03:58:01.4803616Z #13 sha256:a603fa5e3b4127f210503aaa6189abf6286ee5a73deeaab460f8f33ebc6b64e2 8.39MB / 31.41MB 2.7s 2022-11-19T03:58:01.4804289Z #13 sha256:b00aaacf759c581712fa578a6b4e8e0b9fc780919a5d835a168457b754755644 0B / 1.08MB 0.3s 2022-11-19T03:58:01.7806661Z #13 sha256:b00aaacf759c581712fa578a6b4e8e0b9fc780919a5d835a168457b754755644 1.08MB / 1.08MB 0.5s done 2022-11-19T03:58:02.2323402Z #13 sha256:a603fa5e3b4127f210503aaa6189abf6286ee5a73deeaab460f8f33ebc6b64e2 10.49MB / 31.41MB 3.5s 2022-11-19T03:58:02.6838572Z #13 sha256:a603fa5e3b4127f210503aaa6189abf6286ee5a73deeaab460f8f33ebc6b64e2 12.58MB / 31.41MB 3.9s 2022-11-19T03:58:03.2850844Z #13 sha256:c9855a2acee514e35c3d27eebd4b15600fea80656165910ad9ee61f0693c06c4 12.58MB / 112.81MB 8.0s 2022-11-19T03:58:03.4357623Z #13 sha256:a603fa5e3b4127f210503aaa6189abf6286ee5a73deeaab460f8f33ebc6b64e2 14.68MB / 31.41MB 4.7s 2022-11-19T03:58:04.0371359Z #13 sha256:a603fa5e3b4127f210503aaa6189abf6286ee5a73deeaab460f8f33ebc6b64e2 16.78MB / 31.41MB 5.3s 2022-11-19T03:58:04.7908763Z #13 sha256:a603fa5e3b4127f210503aaa6189abf6286ee5a73deeaab460f8f33ebc6b64e2 18.87MB / 31.41MB 6.0s 2022-11-19T03:58:05.2414593Z #13 sha256:a603fa5e3b4127f210503aaa6189abf6286ee5a73deeaab460f8f33ebc6b64e2 20.97MB / 31.41MB 6.5s 2022-11-19T03:58:05.6921143Z #13 sha256:4b961533c7c8c1f7cfe7d5d52e0df85cce8eb237f71529a68da807d07811fc73 18.87MB / 255.74MB 10.4s 2022-11-19T03:58:05.9935279Z #13 sha256:a603fa5e3b4127f210503aaa6189abf6286ee5a73deeaab460f8f33ebc6b64e2 23.07MB / 31.41MB 7.2s 2022-11-19T03:58:06.1444050Z #13 sha256:c9855a2acee514e35c3d27eebd4b15600fea80656165910ad9ee61f0693c06c4 18.87MB / 112.81MB 10.8s 2022-11-19T03:58:06.5957798Z #13 sha256:a603fa5e3b4127f210503aaa6189abf6286ee5a73deeaab460f8f33ebc6b64e2 25.17MB / 31.41MB 7.8s 2022-11-19T03:58:07.3498501Z #13 sha256:a603fa5e3b4127f210503aaa6189abf6286ee5a73deeaab460f8f33ebc6b64e2 27.26MB / 31.41MB 8.6s 2022-11-19T03:58:07.7998944Z #13 sha256:a603fa5e3b4127f210503aaa6189abf6286ee5a73deeaab460f8f33ebc6b64e2 29.36MB / 31.41MB 9.0s 2022-11-19T03:58:08.2496297Z #13 sha256:a603fa5e3b4127f210503aaa6189abf6286ee5a73deeaab460f8f33ebc6b64e2 31.41MB / 31.41MB 9.5s done 2022-11-19T03:58:08.2496886Z #13 extracting sha256:a603fa5e3b4127f210503aaa6189abf6286ee5a73deeaab460f8f33ebc6b64e2 2022-11-19T03:58:08.5336169Z #13 extracting sha256:a603fa5e3b4127f210503aaa6189abf6286ee5a73deeaab460f8f33ebc6b64e2 0.4s done 2022-11-19T03:58:08.6846070Z #13 extracting sha256:b00aaacf759c581712fa578a6b4e8e0b9fc780919a5d835a168457b754755644 0.0s done 2022-11-19T03:58:08.6846515Z #13 extracting sha256:20b28bf2dde1a1b9653fb71a8f832a47d4cb7450cec5f2d822c122e2f86e4dca 2022-11-19T03:58:08.8021475Z #13 extracting sha256:20b28bf2dde1a1b9653fb71a8f832a47d4cb7450cec5f2d822c122e2f86e4dca 0.1s done 2022-11-19T03:58:08.8021951Z #13 extracting sha256:ddc9bdde4697fb636db1b96654f7566da0e2b2ae2c333d46339441d441adb21e 0.0s done 2022-11-19T03:58:08.8022344Z #13 extracting sha256:04a9993ebea101b8473401cf28834c76c4e7a97ef0f463f6707e1ac09ae68220 0.1s done 2022-11-19T03:58:09.1033624Z #13 sha256:c9855a2acee514e35c3d27eebd4b15600fea80656165910ad9ee61f0693c06c4 25.17MB / 112.81MB 13.8s 2022-11-19T03:58:10.7603179Z #13 sha256:4b961533c7c8c1f7cfe7d5d52e0df85cce8eb237f71529a68da807d07811fc73 29.36MB / 255.74MB 15.5s 2022-11-19T03:58:12.1153310Z #13 sha256:c9855a2acee514e35c3d27eebd4b15600fea80656165910ad9ee61f0693c06c4 31.46MB / 112.81MB 16.8s 2022-11-19T03:58:15.1265057Z #13 sha256:c9855a2acee514e35c3d27eebd4b15600fea80656165910ad9ee61f0693c06c4 37.75MB / 112.81MB 19.8s 2022-11-19T03:58:15.8793344Z #13 sha256:4b961533c7c8c1f7cfe7d5d52e0df85cce8eb237f71529a68da807d07811fc73 40.89MB / 255.74MB 20.6s 2022-11-19T03:58:18.1375749Z #13 sha256:c9855a2acee514e35c3d27eebd4b15600fea80656165910ad9ee61f0693c06c4 44.04MB / 112.81MB 22.8s 2022-11-19T03:58:20.9955744Z #13 sha256:4b961533c7c8c1f7cfe7d5d52e0df85cce8eb237f71529a68da807d07811fc73 51.38MB / 255.74MB 25.7s 2022-11-19T03:58:21.1464276Z #13 sha256:c9855a2acee514e35c3d27eebd4b15600fea80656165910ad9ee61f0693c06c4 50.33MB / 112.81MB 25.8s 2022-11-19T03:58:24.1576056Z #13 sha256:c9855a2acee514e35c3d27eebd4b15600fea80656165910ad9ee61f0693c06c4 56.62MB / 112.81MB 28.8s 2022-11-19T03:58:26.1140718Z #13 sha256:4b961533c7c8c1f7cfe7d5d52e0df85cce8eb237f71529a68da807d07811fc73 61.87MB / 255.74MB 30.8s 2022-11-19T03:58:27.1696632Z #13 sha256:c9855a2acee514e35c3d27eebd4b15600fea80656165910ad9ee61f0693c06c4 62.91MB / 112.81MB 31.8s 2022-11-19T03:58:30.1759811Z #13 sha256:c9855a2acee514e35c3d27eebd4b15600fea80656165910ad9ee61f0693c06c4 69.21MB / 112.81MB 34.8s 2022-11-19T03:58:31.2262626Z #13 sha256:4b961533c7c8c1f7cfe7d5d52e0df85cce8eb237f71529a68da807d07811fc73 72.35MB / 255.74MB 35.9s 2022-11-19T03:58:33.1757476Z #13 sha256:c9855a2acee514e35c3d27eebd4b15600fea80656165910ad9ee61f0693c06c4 75.50MB / 112.81MB 37.8s 2022-11-19T03:58:36.1763088Z #13 sha256:c9855a2acee514e35c3d27eebd4b15600fea80656165910ad9ee61f0693c06c4 81.79MB / 112.81MB 40.8s 2022-11-19T03:58:36.3261165Z #13 sha256:4b961533c7c8c1f7cfe7d5d52e0df85cce8eb237f71529a68da807d07811fc73 83.89MB / 255.74MB 41.1s 2022-11-19T03:58:39.3255317Z #13 sha256:c9855a2acee514e35c3d27eebd4b15600fea80656165910ad9ee61f0693c06c4 88.08MB / 112.81MB 44.0s 2022-11-19T03:58:41.4261433Z #13 sha256:4b961533c7c8c1f7cfe7d5d52e0df85cce8eb237f71529a68da807d07811fc73 94.37MB / 255.74MB 46.2s 2022-11-19T03:58:42.3257093Z #13 sha256:c9855a2acee514e35c3d27eebd4b15600fea80656165910ad9ee61f0693c06c4 94.37MB / 112.81MB 47.0s 2022-11-19T03:58:45.3255051Z #13 sha256:c9855a2acee514e35c3d27eebd4b15600fea80656165910ad9ee61f0693c06c4 100.66MB / 112.81MB 50.0s 2022-11-19T03:58:46.5266749Z #13 sha256:4b961533c7c8c1f7cfe7d5d52e0df85cce8eb237f71529a68da807d07811fc73 104.86MB / 255.74MB 51.3s 2022-11-19T03:58:48.3253348Z #13 sha256:c9855a2acee514e35c3d27eebd4b15600fea80656165910ad9ee61f0693c06c4 106.95MB / 112.81MB 53.0s 2022-11-19T03:58:50.8295858Z #13 sha256:c9855a2acee514e35c3d27eebd4b15600fea80656165910ad9ee61f0693c06c4 112.81MB / 112.81MB 55.6s done 2022-11-19T03:58:50.9803331Z #13 extracting sha256:c9855a2acee514e35c3d27eebd4b15600fea80656165910ad9ee61f0693c06c4 2022-11-19T03:58:51.7048999Z #13 sha256:4b961533c7c8c1f7cfe7d5d52e0df85cce8eb237f71529a68da807d07811fc73 115.34MB / 255.74MB 56.4s 2022-11-19T03:58:51.7049530Z #13 extracting sha256:c9855a2acee514e35c3d27eebd4b15600fea80656165910ad9ee61f0693c06c4 0.9s done 2022-11-19T03:58:56.8278343Z #13 sha256:4b961533c7c8c1f7cfe7d5d52e0df85cce8eb237f71529a68da807d07811fc73 126.88MB / 255.74MB 61.5s 2022-11-19T03:59:01.9486008Z #13 sha256:4b961533c7c8c1f7cfe7d5d52e0df85cce8eb237f71529a68da807d07811fc73 137.36MB / 255.74MB 66.6s 2022-11-19T03:59:07.0669824Z #13 sha256:4b961533c7c8c1f7cfe7d5d52e0df85cce8eb237f71529a68da807d07811fc73 147.85MB / 255.74MB 71.7s 2022-11-19T03:59:12.1761499Z #13 sha256:4b961533c7c8c1f7cfe7d5d52e0df85cce8eb237f71529a68da807d07811fc73 158.33MB / 255.74MB 76.8s 2022-11-19T03:59:17.1260185Z #13 sha256:4b961533c7c8c1f7cfe7d5d52e0df85cce8eb237f71529a68da807d07811fc73 169.87MB / 255.74MB 81.9s 2022-11-19T03:59:22.2259871Z #13 sha256:4b961533c7c8c1f7cfe7d5d52e0df85cce8eb237f71529a68da807d07811fc73 180.36MB / 255.74MB 87.0s 2022-11-19T03:59:27.3259496Z #13 sha256:4b961533c7c8c1f7cfe7d5d52e0df85cce8eb237f71529a68da807d07811fc73 190.84MB / 255.74MB 92.1s 2022-11-19T03:59:32.4255573Z #13 sha256:4b961533c7c8c1f7cfe7d5d52e0df85cce8eb237f71529a68da807d07811fc73 201.33MB / 255.74MB 97.2s 2022-11-19T03:59:37.6763769Z #13 sha256:4b961533c7c8c1f7cfe7d5d52e0df85cce8eb237f71529a68da807d07811fc73 211.81MB / 255.74MB 102.3s 2022-11-19T03:59:42.7761971Z #13 sha256:4b961533c7c8c1f7cfe7d5d52e0df85cce8eb237f71529a68da807d07811fc73 223.35MB / 255.74MB 107.6s 2022-11-19T03:59:47.8755511Z #13 sha256:4b961533c7c8c1f7cfe7d5d52e0df85cce8eb237f71529a68da807d07811fc73 233.83MB / 255.74MB 112.7s 2022-11-19T03:59:53.1262318Z #13 sha256:4b961533c7c8c1f7cfe7d5d52e0df85cce8eb237f71529a68da807d07811fc73 244.32MB / 255.74MB 117.8s 2022-11-19T03:59:57.9261716Z #13 sha256:4b961533c7c8c1f7cfe7d5d52e0df85cce8eb237f71529a68da807d07811fc73 255.74MB / 255.74MB 122.6s done 2022-11-19T03:59:57.9262418Z #13 extracting sha256:4b961533c7c8c1f7cfe7d5d52e0df85cce8eb237f71529a68da807d07811fc73 2022-11-19T04:00:01.3821154Z #13 extracting sha256:4b961533c7c8c1f7cfe7d5d52e0df85cce8eb237f71529a68da807d07811fc73 3.5s done 2022-11-19T04:00:01.3821606Z #13 DONE 127.2s ```Now if I do just a "normal" docker build/push:
Fast.
Workaround
I'm building the docker image in a previous step, then using this action:
Annoying, but works :shrug:.
Thought I would share, as I was really confused why this was happening as I thought that it was GHA, so I setup a hosted runner as a test, and that was having problems too...