search

netlify / zip-it-and-ship-it

Intelligently prepare Node.js Lambda functions for deployment
https://www.npmjs.com/package/@netlify/zip-it-and-ship-it
MIT License
316 stars 35 forks source link

Strange bundling error when including sub directory #16

Closed DavidWells closed 4 years ago

DavidWells commented 5 years ago

Including a file from a sub folder is crashing my netlify deploy with this error

. Hashing files...Error: Could not find "encoding" module in file: /unsubscribe-center/functions/redirect.js.
Please ensure "encoding" is installed in the project.
    at handle (~/.nvm/versions/node/v10.4.1/lib/node_modules/netlify-cli/node_modules/@netlify/zip-it-and-ship-it/src/finders.js:76:17)
    at precinct.paperwork.forEach.dependency (~/.nvm/versions/node/v10.4.1/lib/node_modules/netlify-cli/node_modules/@netlify/zip-it-and-ship-it/src/finders.js:102:11)
    at Array.forEach (<anonymous>)
    at getDependencies (~/.nvm/versions/node/v10.4.1/lib/node_modules/netlify-cli/node_modules/@netlify/zip-it-and-ship-it/src/finders.js:95:8)
    at filesForFunctionZip (~/.nvm/versions/node/v10.4.1/lib/node_modules/netlify-cli/node_modules/@netlify/zip-it-and-ship-it/src/zip.js:65:7)
    at zipJs (~/.nvm/versions/node/v10.4.1/lib/node_modules/netlify-cli/node_modules/@netlify/zip-it-and-ship-it/src/zip.js:110:3)
    at zipFunction (~/.nvm/versions/node/v10.4.1/lib/node_modules/netlify-cli/node_modules/@netlify/zip-it-and-ship-it/src/zip.js:143:12)
    at ~/.nvm/versions/node/v10.4.1/lib/node_modules/netlify-cli/node_modules/@netlify/zip-it-and-ship-it/src/zip.js:181:9
    at element (~/.nvm/versions/node/v10.4.1/lib/node_modules/netlify-cli/node_modules/p-all/index.js:4:67)
    at Promise.resolve.then.element (~/.nvm/versions/node/v10.4.1/lib/node_modules/netlify-cli/node_modules/p-map/index.js:47:21)
npm ERR! code ELIFECYCLE
npm ERR! errno 2
npm ERR! unsubscribe-center@0.1.0 deploy: `npm run build && netlify deploy -p`
npm ERR! Exit status 2
npm ERR!
npm ERR! Failed at the unsubscribe-center@0.1.0 deploy script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/davidwells/.npm/_logs/2019-03-08T00_15_27_349Z-debug.log

The file is in a sub folder and has all the dependancies installed locally.

image

Here is the dep tree

image

I'm using node-fetch as a dependancy in the /functions/customerio/api/index.js file

biilmann commented 5 years ago

hmm, where's the encoding module imported?

DavidWells commented 5 years ago

No idea. I'm just importing node-fetch. Somewhere deep down the tree perhaps.

├── node-fetch@2.3.0
└─┬ simple-oauth2@2.2.1
  ├── date-fns@1.30.1
  ├─┬ debug@3.2.6
  │ └── ms@2.1.1
  ├─┬ joi@13.7.0
  │ ├── hoek@5.0.4
  │ ├─┬ isemail@3.2.0
  │ │ └── punycode@2.1.1
  │ └─┬ topo@3.0.3
  │   └── hoek@6.1.2
  └─┬ wreck@14.2.0
    ├─┬ boom@7.3.0
    │ └── hoek@6.1.2
    ├── bourne@1.1.2
    └── hoek@6.1.2

The modules aren't corrupted, the functions deploy fine if they don't require files from sibling folders.

I ended up hoisting everything to the parent folder for now to get around this

bcomnes commented 5 years ago

Did this get fixed?

DavidWells commented 5 years ago

Not yet. I fixed this by reorganizing my file structure.

Seemed like module resolution was not looking in sibling folder for dep. Which I think is the correct behavior.

Haven't see this issue since I reorganized my files

zanona commented 5 years ago

Also having this issue now, but it started from nothing, no change to netlify functions codebase whatsoever. In fact deploy previews were passing without issues and when it got merged to production, this poped up.

bcomnes commented 5 years ago

The solution is to install encoding. Its an undeclared peer dependency of node-fetch. Our tree shaker may be a little to aggressive right now, but thats the solution for now.

Auspicus commented 5 years ago

Bump. Also ran into this issue with cross-fetch which uses node-fetch.

ehmicky commented 4 years ago

This should be fixed by d3e4b2aae9578f631fb6dd78b55cf2404817e11d, please let us know if this is still happening.

polarbirke commented 4 years ago

Just happened to me on a preview deployment with node-fetch being a dependency of my lambda function.

jon301 commented 4 years ago

Hi

I'm encountering a similar error (not encoding but @nestjs/microservices module) when trying to deploy a Function exposing a NestJS app bundled with Nx : 

11:43:52 PM: Function Dir: /opt/build/repo/dist/apps/api-lambda
11:43:52 PM: TempDir: /tmp/zisi-5db619af1f4f9f00071d3713
11:43:53 PM: Prepping functions with zip-it-and-ship-it 0.4.0-5
11:43:56 PM: Error: In file "/opt/build/repo/dist/apps/api-lambda/main.js": Cannot find module '@nestjs/microservices' from '/opt/build/repo/node_modules/@nestjs/core'
11:43:56 PM: Error prepping functions

More info in this comment from @kamilmysliwiec (creator of NestJS) in this thread : https://github.com/nestjs/nest/issues/1706#issuecomment-474514484

TLDR :

webpack traverse modules graph and tries to resolve cache-manager dependency even though(!) CacheModule might not be used (tree-shaking is being performed later on, once bundle is created). Consequently, it breaks Nest idea of lazy require() calls, because webpack tries to load them all during the bundling process

From what I saw later in the thread, there are potentially other lazy imports : 

      const lazyImports = [
        '@nestjs/microservices',
        '@nestjs/microservices/microservices-module',
        '@nestjs/websockets',
        '@nestjs/websockets/socket-module',
        '@nestjs/platform-express',
        'cache-manager',
        'class-validator',
        'class-transformer',
      ];

Is there a way to tell zip-it-and-ship-it to ignore those dependencies ?

Installing all those peerDependencies as suggested by @bcomnes https://github.com/netlify/zip-it-and-ship-it/issues/16#issuecomment-484615282 could work, but there are plenty + not sure this is an exhaustive list (could break anytime if NestJS code evolves)

ehmicky commented 4 years ago

Hi @jon301 and @polarbirke. I've just made a new release. Could you please check if it now works?

Also you should see the following line in your log:

Prepping functions with zip-it-and-ship-it 0.4.0-6

If not, please let me know. Thanks!

jon301 commented 4 years ago

Hi @ehmicky, thanks for checking this out I've just tried to trigger new deployment but it's still running in the 0.4.0-5 version :

6:36:59 PM: Prepping functions with zip-it-and-ship-it 0.4.0-5
6:37:02 PM: Error: In file "/opt/build/repo/dist/apps/api-lambda/main.js": Cannot find module '@nestjs/microservices' from '/opt/build/repo/node_modules/@nestjs/core'
6:37:02 PM: Error prepping functions
ehmicky commented 4 years ago

Can you try again? You should now see:

Prepping functions with zip-it-and-ship-it 0.3.1

And your initial issue should now be solved.

akando42 commented 4 years ago

I have the same build issues from yesterday with the zip-it-and-ship-it 0.3.1

1:49:58 PM: TempDir: /tmp/zisi-5dbc7d8c336cc100087702aa
1:49:59 PM: Prepping functions with zip-it-and-ship-it 0.3.1
1:50:14 PM: [ { path: '/tmp/zisi-5dbc7d8c336cc100087702aa/account.zip',
1:50:14 PM:     runtime: 'js' },
1:50:14 PM:   { path: '/tmp/zisi-5dbc7d8c336cc100087702aa/addons.zip',
1:50:14 PM:     runtime: 'js' },
1:50:14 PM:   { path: '/tmp/zisi-5dbc7d8c336cc100087702aa/availability.zip',
1:50:14 PM:     runtime: 'js' },
1:50:14 PM:   { path: '/tmp/zisi-5dbc7d8c336cc100087702aa/booked.zip',
1:50:14 PM:     runtime: 'js' },
1:50:14 PM:   { path: '/tmp/zisi-5dbc7d8c336cc100087702aa/booking.zip',
1:50:14 PM:     runtime: 'js' },
1:50:14 PM:   { path: '/tmp/zisi-5dbc7d8c336cc100087702aa/cards-read.zip',
1:50:14 PM:     runtime: 'js' },
1:50:14 PM:   { path: '/tmp/zisi-5dbc7d8c336cc100087702aa/cards.zip',
1:50:14 PM:     runtime: 'js' },
1:50:14 PM:   { path: '/tmp/zisi-5dbc7d8c336cc100087702aa/health.zip',
1:50:14 PM:     runtime: 'js' },
1:50:14 PM:   { path: '/tmp/zisi-5dbc7d8c336cc100087702aa/instagram.zip',
1:50:14 PM:     runtime: 'js' },
1:50:14 PM:   { path: '/tmp/zisi-5dbc7d8c336cc100087702aa/purchase.zip',
1:50:14 PM:     runtime: 'js' },
1:50:14 PM: failed during stage 'building site': Build script returned non-zero exit code: 127
1:50:14 PM:   { path: '/tmp/zisi-5dbc7d8c336cc100087702aa/server.zip',
1:50:14 PM:     runtime: 'js' },
1:50:14 PM:   { path: '/tmp/zisi-5dbc7d8c336cc100087702aa/services.zip',
1:50:14 PM:     runtime: 'js' },
1:50:14 PM: Shutting down logging, 31 messages pending
ehmicky commented 4 years ago

@troydo42 thanks for reporting this. Could you please open a separate issue? This seems to be a different problem. Thanks.

jon301 commented 4 years ago

Tried just now and still with old version :(

11:54:28 PM: Prepping functions with zip-it-and-ship-it 0.4.0-5
jon301 commented 4 years ago

Hi @ehmicky , thanks for your help It seems like the previous error is gone with 0.4.0-6, but a new one appeared :

8:02:59 AM: Prepping functions with zip-it-and-ship-it 0.4.0-6
8:03:02 AM: Error: In file "/opt/build/repo/dist/apps/api/main.js": Cannot find module 'graphql-tag-pluck/package.json' from '/opt/build/repo/node_modules/graphql-toolkit'

Edit : I think it's because of this line : https://github.com/ardatan/graphql-toolkit/blob/master/packages/loaders/code-file/src/extract-document-string-from-code-file.ts#L52

ehmicky commented 4 years ago

Since that's a separate issue, I have opened a new issue at #66.

It looks like the original issue is solved. Please let me know if this appears again.

ehmicky commented 4 years ago

This issue is symptomatic of a more general problem described at #68. Please refer to this issue for further discussion.

EdwardGoomba commented 4 years ago

The solution is to install encoding. Its an undeclared peer dependency of node-fetch. Our tree shaker may be a little to aggressive right now, but thats the solution for now.

Thanks - ran into this issue. Resolved.