Cache "Install AWS CLI" step

[raine]

Is there a way to cache the installation step?

What's the point of using the orb instead of a docker image that has the command built in, if you have to spend 15 seconds installing the tool in every build?

Cheers.

[jspri]

Alternatively could the default executor come with the cli installed?

[lokst]

@raine It might be possible to cache the installation step by caching the Python dependencies installed with pip. @Crazometer Thanks for the suggestion. The default executor is set to one of CircleCI's Python images for users' convenience. Although pre-installing the CLI into the image would make this orb perform faster, we have to consider the tradeoffs for other users of the image who do not use the aws CLI.

[jspri]

@lokst Not sure how the build process works there/how easy it would be - I was thinking more along the lines of you could add a layer to the circleci python image.

I know it might seem like a small deal but a huge amount of our build time is spent installing the cli 😆 . Proabably like 10% overall.

[lokst]

@Crazometer I understand the importance of reducing your build time 🙂 At the same time we do need to keep the circleci python image as slim as possible in the interests of other users, so I think it's more feasible to enhance the aws-cli orb instead, for example, to cache the python packages. I am adding this as a feature request to our backlog.

[jspri]

Glad it’s on the backlog.

I think you are misunderstanding me though, the base image for this orb could be something built on top of the standard python image. Hence non aws users would not be affected.

On Fri, 6 Dec 2019 at 9:12 pm, Stella Lok notifications@github.com wrote:

@Crazometer https://github.com/Crazometer I understand the importance of reducing your build time 🙂 At the same time we do need to keep the circleci python image as slim as possible in the interests of other users, so I think it's more feasible to enhance the aws-cli orb instead, for example, to cache the python packages. I am adding this as a feature request to our backlog.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/CircleCI-Public/aws-cli-orb/issues/25?email_source=notifications&email_token=ABJRG4KKHY5TPPSIOO6M4X3QXIQP5A5CNFSM4IYUNZ62YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGDURPY#issuecomment-562514111, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABJRG4ONXMGAVSXZHZB6ROTQXIQP5ANCNFSM4IYUNZ6Q .

[lokst]

@Crazometer I understand what you mean. It's a great approach but considering the maintenance upkeep needed for every new convenience image we support, I would recommend filing a suggestion on our Ideas portal on https://ideas.circleci.com/ so that our product team can gauge interest in this 🙂

[aleyan]

"Install AWS CLI" across two jobs takes up to 22% of the time in our most common workflow.

Per first comment suggestion, I tried caching via pip and circleci tooling. Unfortunately, aws-cli doesn't get installed in a virtualenv, making caching difficult.

orbs:
  aws-cli: circleci/aws-cli@0.1.19

references:
  save_awscli_cache: &save_awscli_cache
    save_cache:
      key: orb-aws-cli-v0_1_19
      paths:
        - /usr/local/lib/python3.7/site-packages
  restore_awscli_cache: &restore_awscli_cache
    restore_cache:
      keys:
        - orb-aws-cli-v0_1_19

  some-job:
    working_directory: ~/repo
    docker:
      - image: circleci/python:3.7
    steps:
      - *restore_awscli_cache
      - aws-cli/setup
      - *save_awscli_cache

This does not work due to the following error:

Found a cache from build 12499 at orb-aws-cli-v0_1_19
Size: 28 MiB
Cached paths:
  * /usr/local/lib/python3.7/site-packages
Downloading cache archive...
Validating cache...
Unarchiving cache...
Failed to unarchive cache
Error untarring cache: Error extracting tarball /tmp/cache069355973 : tar: usr/local/lib/python3.7/site-packages/.libs_cffi_backend/libffi-806b1a9d.so.6.0.4: Cannot open: File exists tar:  [elided]

Any ideas?

[raine]

Can you make a home brew docker image that contains aws cli and other things you need in CI environment?

[aleyan]

That was my first thought, however I am concerned that then the "Spin Up Environment" will take ~30s because my custom image will be rarely cached on the machine. Also I am concerned about the maintenance.

[lokst]

Closing this issue for now as we are exploring adding a convenience image with aws pre-installed, which will help with this.

[aleyan]

Is there a new issue tracking the status convenience image with aws pre-installed? Closing this issue otherwise is premature as we are all still having the problem of "Install AWS CLI" taking a non-trivial amount of time in our CI pipelines.

[ledniy]

@lokst any updates on convenience image with aws?

[invisiblefunnel]

As a workaround I've created a custom command that caches the AWS CLI installation. The linked gist also includes replacements for aws-s3/copy and aws-s3/sync. Assumes a linux install and env var config, but should be easy to adapt. Hope this is a helpful pattern for others to follow until an official alternative is available.

https://gist.github.com/invisiblefunnel/b719b199317916fe2816959a2cd7dfa0