Caching won't work with node_modules.

[dashmug]

CodeBuild keeps its cache locally and then symlinks it to our target location as evidenced by this line:

[Container] 2019/02/27 02:24:38 Symlinking: /codebuild/output/src718408252/src/github.com/dashmug/api-tests/node_modules => /codebuild/local-cache/custom/dba27c31aad935787bb275c3e5e4e957708f15386de599eff1db476022cd7e4c

This behaviour, however, breaks Node's module resolution causing our Node apps to fail.

internal/modules/cjs/loader.js:596
throw err;
^
Error: Cannot find module 'source-map-support'
at Function.Module._resolveFilename (internal/modules/cjs/loader.js:594:15)
at Function.Module._load (internal/modules/cjs/loader.js:520:25)
at Module.require (internal/modules/cjs/loader.js:650:17)
at require (internal/modules/cjs/helpers.js:20:18)

This is a widely-known issue. https://github.com/npm/npm/issues/9479 https://github.com/nodejs/node-v0.x-archive/issues/25090

TLDR: node's require() doesn't work with a symlinked node_modules directory if the source directory is not named node_modules.

Because of this bug in node, having the following in buildspec.yml will fail:

cache:
  paths:
    - 'node_modules/**/*'

[josephvusich]

We're investigating this issue, thanks for bringing it to our attention.

[dashmug]

@josephvusich Thanks for looking into it.

[josephvusich]

This issue should now be resolved. Let us know if you continue to experience problems.

[dashmug]

Thank you very much.

[MrStarktastic]

Seems like it doesn't cache all modules for me, and npm install requires "a peer of @babel/core@^7.0.0-0 but none is installed". Consequently, webpack build fails.

[mparent61]

Having same issue using latest CodeBuild v2.0 image and "node 10" runtime installed. Build works fine with caching disabled.

[jeanlks]

The same with me, even tried with recursive folders.

[mparent61]

The solution for me was to NOT use CodeBuild local caching, but instead use S3 caching mode. Not sure if this applies to your own situations, but hoping this helps someone else.

[AditModi]

@josephvusich , still facing issues while trying to cache node_modules inside the buildspec.yml file. I am trying to use codebuild local caching and it doesn't seem to work. any suggestions??

[christianaranda]

@josephvusich , still facing issues while trying to cache node_modules inside the buildspec.yml file. I am trying to use codebuild local caching and it doesn't seem to work. any suggestions??

Hi @AditModi - just wanted to say I came here for exactly this. If you find a solution, please post here. I'll do the same.

[soplan]

same issue for us. Using S3 as cache. yet in the logs I can see that it fetches packages every time...

[joshhoegen]

@mparent61, I've been trying the same, but have the same result as @soplan... any tips?

[mparent61]

@joshhoegen - Not sure what you're missing, but I can just give you my basic approach in case this helps -

My aws_codebuild_project Terraform config sets up S3 cache -

cache {
   type = "S3"
   location = "my-bucket-name/my-key"
}

I make sure the CodeBuild role can read/write to that bucket, and my buildspec is roughly -

version: 0.2
phases:
  install:
    runtime-versions:
      nodejs: 14
    commands:
      - npm install
  build:
    commands:
      - CI=true npm test
cache:
  paths:
    - node_modules/**/*

[cybercussion]

Anyone have a issue where it states "Uploading S3 cache..." and times out? I've gone over and over it and I can't figure out why it can't status an error. I've reviewed all the policy and the Source writes fine but anytime I add a cache or artifact it just hangs.

[joshhoegen]

I've been trying Codebuild with both s3 and local caching since before November... Eventually, I reached out to support from a federated work account. This was the last response.

Thank for the reaching out. As per your response, I understand that you were able to successfully use caching with Github Actions which it takes less than 6 min to build while your existing CodeBuild still takes ~14min to build.

I regret the inconvenience that this may have been causing you. I have added this points to the Service Team who are currently working on your case on priority and I will soon update you with the workaround, if any.

So, now we use Github actions for testing the integrity of our builds, and allow a very slow AWS build time for env promotion.

[cybercussion]

Thanks @joshhoegen I ended up removing the ‘node_modules/*/’ and the hang time ended. I used pnpm on this project so I tweaked how that whole thing would work but still tuning. Least it finally let the lint, test jobs run built in < 6 minutes.

phases:
  install:
    commands:
      - n 14.19.0
      - echo Installing source NPM dependencies...
      # This project uses angular/cli and pnpm.  "aws/codebuild/standard:5.0" comes with Node 14
      - npm install -g @angular/cli@13.3.6 pnpm@7.1.5
      - pnpm config set store-dir .pnpm-store
      - npm config -g set prefer-offline true
      - npm config -g set cache /root/.npm
      - npm config get cache