In some environments, a `/tmp/jets/[project]/stage/vendor/bundle` directory remains and is included in zip, causing deploys to fail

sam0x17 commented 3 years ago

Checklist

[X] Upgrade Jets: Are you using the latest version of Jets? This allows Jets to fix issues fast. There's a jets upgrade command that makes this a simple task. There's also an Upgrading Guide: http://rubyonjets.com/docs/upgrading/
[X] Reproducibility: Are you reporting a bug others will be able to reproduce and not asking a question. If you're unsure or want to ask a question, do so on https://community.rubyonjets.com
[X] Code sample: Have you put together a code sample to reproduce the issue and make it available? Code samples help speed up fixes dramatically. If it's an easily reproducible issue, then code samples are not needed. If you're unsure, please include a code sample.

My Environment

Software	Version
Operating System	ubuntu-latest in Github Actions
Jets	3.0.8
Ruby	2.7.0

Expected Behaviour

During jets deploy, the compute_size method in lib/jets/builders/code_size.rb is used to calculate the total size of the code directory and the total size of the opt directory in bytes. These two numbers are added together and checked against the maximum Lambda code size (currently 250 MB).

The compute_size method currently shells out to du to calculate size. Crucially, there is a symlinked directory containing all the compiled gems for the app located in opt (gem layer) that is symlinked within code/vendor/gems/ruby/[ruby-version-here] (code layer). Because of the symlink, the size of the gems within the code/vendor/gems/ruby/[ruby version here] symlink should not count towards the total size of the code directory, since these files are already contained in the opt directory (the gem layer).

In most environments I've tried, this works fine and compute_size (really just du -ks) returns the size of the code directory minus the size of the symlinked files within code/vendor/gems/[ruby-version-here].

Current Behavior

In some environments, including MacOS (haven't directly confirmed if this is always the case, but is the case for some of my colleagues) and GitHub Actions using ubuntu-latest (this could apply more broadly to a variety of docker environments -- I don't know the scope of it), du -ks and thus compute_size will include the symlinked gems when calculating the size of the code directory, and, if compute_size('code') + compute_size('opt') is greater than 250 MB, will cause jets deploy to abort complaining that the deployment exceeds the 250 MB limit (even though it does not).

Step-by-step reproduction instructions

set up a new jets project with enough gems such that if the gems directory size was doubled, we would exceed 250 MB
add a manual GitHub Actions action using ubuntu-latest that deploys the project (sample shown below)
push up your code and try running the deploy action
the deploy will fail complaining that you are exceeding the 250 MB limit, even though you are well under it

Code Sample

name: Deploy Staging
on: workflow_dispatch
jobs:
  deploy:
    name: Deploy Staging
    runs-on: ubuntu-latest
    env:
      JETS_ENV: staging
      JETS_AGREE: YES
      AWS_REGION: ${{ secrets.STAGING_REGION }}
      AWS_DEFAULT_REGION: ${{ secrets.STAGING_REGION }}
      AWS_ACCESS_KEY_ID: ${{ secrets.STAGING_AWS_ACCESS_KEY_ID }}
      AWS_SECRET_ACCESS_KEY: ${{ secrets.STAGING_AWS_SECRET_ACCESS_KEY }}
    steps:
      - uses: actions/checkout@v1
      - uses: ruby/setup-ruby@v1
        with:
          bundler-cache: true
      - name: Configure AWS Credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          aws-access-key-id: ${{ secrets.STAGING_AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.STAGING_AWS_SECRET_ACCESS_KEY }}
          aws-region: ${{ secrets.STAGING_AWS_SECRET_ACCESS_KEY }}
      - name: Install JS Packages
        run: sudo npm install -g yarn
      - name: Install Gems
        run: |
          gem install bundler
          bundle install --jobs 4 --retry 3
      - name: Set Serverless Gems Token
        run: bundle exec jets configure ${{ secrets.STAGING_JETS_TOKEN }}
      - name: Deploy to AWS
        run: bundle exec jets deploy
      - name: Debug Discrepancy
        if: always()
        run: |
          echo "Code:"
          du -sh /tmp/jets/arist/stage/code
          ls -la /tmp/jets/arist/stage/code/vendor/gems/ruby
          echo "Gem Layer:"
          du -sh /tmp/jets/arist/stage/opt

Solution Suggestion

This seems to be cross-platform inconsistency with du, which is super out of scope for this project. The best solution would be a more portable implementation for compute_size written in pure Ruby that is specially crafted to skip symlinked directories (but still count the size of the symlink towards total sie).

sam0x17 commented 3 years ago

update: I think I have found the discrepancy:

When running du -sh /tmp/jets/[project]/stage/code/vendor/*

On my local system after running jets deploy I get the following output:

8.0K    /tmp/jets/arist/stage/code/vendor/gems

This makes sense because the only thing that should be in there should be the symlinked reference to the gem layer

In my Docker Action after running jets deploy (after it fails due to size check as usual) I get the following output for the same command:

163M    /tmp/jets/arist/stage/code/vendor/bundle
8.0K    /tmp/jets/arist/stage/code/vendor/gems

This extra bundle directory is causing all my problems and seems to only appear on specific systems. I am going to try to patch my fork of jets so this directory is deleted before this stage and see if that fixes things.

sam0x17 commented 3 years ago

Update: yup, that fixes it! Successful deploy! I'll open a pull request shortly

tongueroo commented 3 years ago

Closed by #577

rubyonjets / jets