satackey / action-docker-layer-caching

[CAUTION] This repository is not actively maintained. / Enable Docker layer caching in your GitHub Actions workflow.
https://github.com/marketplace/actions/docker-layer-caching
MIT License
421 stars 56 forks source link

Add a filter parameter - to select for the images to cache #121

Open jamie-wearsafe opened 3 years ago

jamie-wearsafe commented 3 years ago

Is your feature request related to a problem? Please describe. Not all images need caching, and can take up valuable room. Some images are loaded before the cache is loaded such as the images in services. So excluding those images would help reduce the size and speed up the save/load process.

Describe the solution you'd like When doing docker save $(docker images -q) you can provide a list of the image ID's to save. If we could provide a filter at that point, it would mean we were only saving the images that we needed and so instead of this list:

REPOSITORY                                    TAG                       IMAGE ID       CREATED         SIZE
codacy/codacy-prospector                      2.3.9                     2ed875f0c09d   2 days ago      267MB
codacy/codacy-analysis-cli                    latest                    9d412c689197   2 days ago      421MB
snyk/snyk                                     maven-3-jdk-11            882d05531c51   3 days ago      734MB
codacy/codacy-rubocop                         5.1.6                     88f504735018   7 days ago      166MB
codacy/codacy-tsqllint                        1.2.4                     92b7ddf6589f   7 days ago      91.8MB
codacy/codacy-pylint-python3                  2.2.16                    504237065f5b   7 days ago      161MB
api                                           11                        413eb2357edb   9 days ago      512MB
codacy/codacy-bandit                          2.3.1                     5c95af379274   12 days ago     264MB
codacy/codacy-pmd                             3.11.0                    bb50d6e2a005   2 weeks ago     162MB
tomcat                                        8.5-jdk11-corretto        ae7b0918b39c   4 weeks ago     473MB
codacy/codacy-remark-lint                     2.5.59                    ef1454679208   6 weeks ago     155MB
codacy/codacy-bundler-audit                   4.2.12                    8d9828916ae8   2 months ago    86MB
redis                                         5-alpine                  caa6655434a5   2 months ago    29.7MB
codacy/codacy-spotbugs                        1.10.5                    999b0c7e7945   2 months ago    128MB
amazonlinux                                   xray                      b88042e00023   3 months ago    395MB
codacy/codacy-checkstyle                      2.3.0                     5b7b8322d896   4 months ago    118MB
codacy/codacy-hadolint                        1.5.2                     04e2b64ed099   4 months ago    9.31MB
codacy/codacy-metrics-pmd                     0.2.2                     8b5299fc6999   4 months ago    111MB
codacy/codacy-shellcheck                      2.4.1                     509fffd67432   4 months ago    33.5MB
codacy/codacy-sqlint                          2.3.2                     f21a8ce8c730   4 months ago    148MB
codacy/codacy-jackson-linter                  5.2.2                     d06c7b65bd22   4 months ago    104MB
codacy/codacy-brakeman                        1.3.2                     f95377ddb330   4 months ago    146MB
codacy/codacy-pylint                          3.2.1                     582388b48c67   4 months ago    242MB
codacy/codacy-codenarc                        0.4.2                     88739f96d424   4 months ago    111MB
mysql                                         5.6                       c580203d8753   4 months ago    302MB
codacy/codacy-metrics-cloc                    0.2.4                     0f47e48065ab   7 months ago    197MB
amazon/amazon-ecs-local-container-endpoints   latest                    b005329f50b1   7 months ago    180MB
codacy/codacy-duplication-pmdcpd              2.0.145                   e121e0516f79   11 months ago   132MB
codacy/codacy-metrics-rubocop                 0.2.2                     ee82a4e0ba6a   11 months ago   269MB
codacy/codacy-metrics-radon                   0.1.66                    19f7a80d4001   11 months ago   154MB
codacy/codacy-duplication-flay                2.0.154                   ba95f1e54cfa   11 months ago   166MB
codacy/codacy-pmdjava                         2.0.0-pmdlegacy.57fdbf2   dee6d8e1c0e5   20 months ago   231MB
boxfuse/flyway                                5.2.4-alpine              df49c5fb49bb   2 years ago     115MB
memcached                                     1.5.10-alpine             fb0df03da449   2 years ago     8.97MB

A filter would be added at this point:

docker save $(docker images -q --filter=reference='codacy/*')

and we would get this list instead:

REPOSITORY                         TAG                       IMAGE ID       CREATED         SIZE
codacy/codacy-prospector           2.3.9                     2ed875f0c09d   2 days ago      267MB
codacy/codacy-analysis-cli         latest                    9d412c689197   2 days ago      421MB
codacy/codacy-rubocop              5.1.6                     88f504735018   7 days ago      166MB
codacy/codacy-tsqllint             1.2.4                     92b7ddf6589f   7 days ago      91.8MB
codacy/codacy-pylint-python3       2.2.16                    504237065f5b   7 days ago      161MB
codacy/codacy-bandit               2.3.1                     5c95af379274   12 days ago     264MB
codacy/codacy-pmd                  3.11.0                    bb50d6e2a005   2 weeks ago     162MB
codacy/codacy-remark-lint          2.5.59                    ef1454679208   6 weeks ago     155MB
codacy/codacy-bundler-audit        4.2.12                    8d9828916ae8   2 months ago    86MB
codacy/codacy-spotbugs             1.10.5                    999b0c7e7945   2 months ago    128MB
codacy/codacy-checkstyle           2.3.0                     5b7b8322d896   4 months ago    118MB
codacy/codacy-hadolint             1.5.2                     04e2b64ed099   4 months ago    9.31MB
codacy/codacy-metrics-pmd          0.2.2                     8b5299fc6999   4 months ago    111MB
codacy/codacy-shellcheck           2.4.1                     509fffd67432   4 months ago    33.5MB
codacy/codacy-sqlint               2.3.2                     f21a8ce8c730   4 months ago    148MB
codacy/codacy-jackson-linter       5.2.2                     d06c7b65bd22   4 months ago    104MB
codacy/codacy-brakeman             1.3.2                     f95377ddb330   4 months ago    146MB
codacy/codacy-pylint               3.2.1                     582388b48c67   4 months ago    242MB
codacy/codacy-codenarc             0.4.2                     88739f96d424   4 months ago    111MB
codacy/codacy-metrics-cloc         0.2.4                     0f47e48065ab   7 months ago    197MB
codacy/codacy-duplication-pmdcpd   2.0.145                   e121e0516f79   11 months ago   132MB
codacy/codacy-metrics-rubocop      0.2.2                     ee82a4e0ba6a   11 months ago   269MB
codacy/codacy-metrics-radon        0.1.66                    19f7a80d4001   11 months ago   154MB
codacy/codacy-duplication-flay     2.0.154                   ba95f1e54cfa   11 months ago   166MB
codacy/codacy-pmdjava              2.0.0-pmdlegacy.57fdbf2   dee6d8e1c0e5   20 months ago   231MB
jamie-wearsafe commented 3 years ago

as in at this part of the code change it to this:


import * as exec from 'actions-exec-listener'
import * as core from '@actions/core'

export class ImageDetector {
  async getExistingImages(): Promise<string[]> {
    const _filter = core.getInput(`filter`)
    const filter = _filter ? `--filter=${_filter}` : ''
    const existingSet = new Set<string>([])
    const ids = (await exec.exec(`docker image ls -q ${filter}`, [], { silent: true, listeners: { stderr: console.warn }})).stdoutStr.split(`\n`).filter(id => id !== ``)
    const repotags = (await exec.exec(`docker`, `image ls --format {{.Repository}}:{{.Tag}} ${filter} --filter=dangling=false`.split(' '), { silent: true, listeners: { stderr: console.warn }})).stdoutStr.split(`\n`).filter(id => id !== ``);
    core.debug(JSON.stringify({ log: "getExistingImages", ids, repotags }));
    ([...ids, ...repotags]).forEach(image => existingSet.add(image))
    core.debug(JSON.stringify({ existingSet }))
    return Array.from(existingSet)
  }

  async getImagesShouldSave(alreadRegisteredImages: string[]): Promise<string[]> {
    const resultSet = new Set(await this.getExistingImages())
    alreadRegisteredImages.forEach(image => resultSet.delete(image))
    return Array.from(resultSet)
  }
}
leigh-hobbii commented 3 years ago

this would be cool, at the moment i'm caching many images that really dont need to be (like mysql etc), but OTOH I really want to cache the images that are built locally.. (I'm running this as docker-compose)

MostefaKamalLala commented 3 years ago

Imagine in windows containers, this would be amazing! Have you tested it @jamie-wearsafe ? would be nice to have a pull request if you've done it 🙏

jamie-wearsafe commented 3 years ago

Imagine in windows containers, this would be amazing! Have you tested it @jamie-wearsafe ? would be nice to have a pull request if you've done it 🙏

I am using a fork where it's working in production.

https://github.com/Broadshield/action-docker-layer-caching

cb109 commented 3 years ago

@jamie-wearsafe Thanks! Using your fork for now, hopefully the filter input can be merged into this project.

For anyone curious on how to use this here is an example:

- name: Cache docker images
  uses: Broadshield/action-docker-layer-caching@main
  continue-on-error: true
  with:
    filter: reference=my_custom_images_prefix*
jamie-wearsafe commented 3 years ago

@jamie-wearsafe Thanks! Using your fork for now, hopefully the filter input can be merged into this project.

For anyone curious on how to use this here is an example:


- name: Cache docker images

  uses: Broadshield/action-docker-layer-caching@main

  continue-on-error: true

  with:

    filter: reference=my_custom_images_prefix*

Ahh sorry I didn't provide an example!

madhavajay commented 3 years ago

@jamie-wearsafe Great work, however im seeing this:

Post job cleanup.
/usr/bin/docker image ls --format={{.ID}} {{.Repository}}:{{.Tag}} --filter=dangling=false --filter=reference='openmined/*'
There is no image to save.

However when using tunshell.com to debug the CI process I can run that exact command and it outputs several hashes... https://github.com/OpenMined/PySyft/runs/3511338358?check_suite_focus=true

jsirianni commented 2 years ago

@jamie-wearsafe your fork worked great for me. I needed to cache a single image to be restored by a job later in my workflow. Without the filter, the cache would load all images from previous runs (because they are uniquely tagged).

      - name: Set Image Tag
        run: echo "GITHUB_SHA_SHORT=$(git rev-parse --short HEAD)" >> $GITHUB_ENV
      - name: Cache Container Image
        uses: Broadshield/action-docker-layer-caching@main
        with:
          filter: reference=account/imagename:${{ env.GITHUB_SHA_SHORT }}
jamie-wearsafe commented 2 years ago

@jsirianni Nice to see :)

jamie-wearsafe commented 2 years ago

@madhavajay i know its been a long time but do you need that issue addressed?

madhavajay commented 2 years ago

Hi @jamie-wearsafe yeah that would be great, I am currently just using cache all and my guess is the cache is getting regularly evicted:

      - name: docker cache
        uses: actions/cache@v3
        if: steps.changes.outputs.stack == 'true'
        continue-on-error: true
        with:
          path: .docker-cache
          key: ${{ runner.os }}-docker
          restore-keys: |
            ${{ runner.os }}-docker
jamie-wearsafe commented 2 years ago

@madhavajay so, can you try adjusting your filter for me?

      - name: Cache OpenmineD Container Image
        uses: Broadshield/action-docker-layer-caching@main
        with:
          filter: reference="*/openmined*:*"
          key: docker-layer-caching-${{ github.workflow }}-${{ github.head_ref || github.ref }}-${{ github.event_name }}-{hash}
          restore-keys: |
            docker-layer-caching-${{ github.workflow }}-${{ github.head_ref || github.ref }}-${{ github.event_name }}-{hash}
            docker-layer-caching-${{ github.workflow }}-${{ github.head_ref || github.ref }}-${{ github.event_name }}
            docker-layer-caching-${{ github.workflow }}-${{ github.head_ref || github.ref }}
            docker-layer-caching-${{ github.workflow }}
            docker-layer-caching-
madhavajay commented 2 years ago

@jamie-wearsafe im still seeing no cache. Could this be because of dockerx or something?

https://github.com/OpenMined/PySyft/runs/7245735124?check_suite_focus=true

Screen Shot 2022-07-11 at 9 41 42 am
jamie-wearsafe commented 2 years ago

@madhavajay there will be no cache until the run completes successfully the first time. Is this a consecutive run?

madhavajay commented 2 years ago

@jamie-wearsafe shouldnt that happen in the final post step:

Post job cleanup.
/usr/bin/docker image ls --format={{.ID}} {{.Repository}}:{{.Tag}} --filter=dangling=false --filter=reference="*/openmined*:*"
There is no image to save.
Screen Shot 2022-07-12 at 8 42 22 am