Open stesie opened 6 years ago
Could you elaborate a bit more on the implications of replacing symlinkSync
with copySync
? If there are no downsides, we'd be happy to merge a PR for this.
So my current playground setup looks like this ...
excerpt from package.json
:
{
...
"devDependencies": {
"@types/aws-lambda": "0.0.18",
"@types/node": "^8.0.47",
"aws-sdk": "^2.141.0",
"serverless": "^1.23.0",
"serverless-plugin-typescript": "^1.1.3"
},
"dependencies": {}
}
(so only dev dependencies, serverless installed locally)
With the replacement mentioned above in place, .build
folder has a node_modules
which is a real directory + package.json a copy of the file from the main folder.
Within .build
folder npm ls
just works as it should:
budgetier@1.0.0 /home/stesie/Projekte/budgetier/.build
├── @types/aws-lambda@0.0.18
├── @types/node@8.0.47
├─┬ aws-sdk@2.141.0
│ ├─┬ buffer@4.9.1
│ │ ├── base64-js@1.2.1
│ │ ├── ieee754@1.1.8
│ │ └── isarray@1.0.0
│ ├── crypto-browserify@1.0.9
│ ├── events@1.1.1
│ ├── jmespath@0.15.0
│ ├── querystring@0.2.0
│ ├── sax@1.2.1
│ ├─┬ url@0.10.3
│ │ ├── punycode@1.3.2
│ │ └── querystring@0.2.0 deduped
│ ├── uuid@3.1.0
│ ├─┬ xml2js@0.4.17
│ │ ├── sax@1.2.1 deduped
│ │ └── xmlbuilder@4.2.1 deduped
│ └─┬ xmlbuilder@4.2.1
│ └── lodash@4.17.4 deduped
├─┬ serverless@1.23.0
│ ├─┬ @serverless/fdk@0.5.1
│ │ ├─┬ isomorphic-fetch@2.2.1
│ │ │ ├── node-fetch@1.7.3 deduped
│ │ │ └── whatwg-fetch@2.0.3 deduped
│ │ ├── ramda@0.24.1
│ │ └─┬ url-parse@1.1.9
│ │ ├── querystringify@1.0.0
│ │ └── requires-port@1.0.0
│ ├─┬ apollo-client@1.9.3
│ │ ├── @types/graphql@0.10.2
│ │ ├─┬ apollo-link-core@0.5.4
│ │ │ ├── graphql@0.10.5 deduped
... and so on ...
yet without the change/fix in place both node_modules
and package.json
are symlinks to the parent's folders files. In this case npm ls
fails like this:
budgetier@1.0.0 /home/stesie/Projekte/budgetier/.build
├── @serverless/fdk@0.5.1 extraneous
├── @types/aws-lambda@0.0.18
├── @types/graphql@0.10.2 extraneous
├── @types/node@8.0.47
├── agent-base@2.1.1 extraneous
├── ansi@0.3.1 extraneous
├── ansi-escapes@1.4.0 extraneous
├── ansi-regex@2.1.1 extraneous
├── ansi-styles@3.2.0 extraneous
├── apollo-client@1.9.3 extraneous
├── apollo-link-core@0.5.4 extraneous
├── archiver@1.3.0 extraneous
├── archiver-utils@1.3.0 extraneous
├── are-we-there-yet@1.1.4 extraneous
├── argparse@1.0.9 extraneous
├── array-union@1.0.2 extraneous
├── array-uniq@1.0.3 extraneous
├── async@1.5.2 extraneous
├── asynckit@0.4.0 extraneous
├─┬ aws-sdk@2.141.0
│ ├── UNMET DEPENDENCY buffer@4.9.1
│ ├── UNMET DEPENDENCY crypto-browserify@1.0.9
│ ├── UNMET DEPENDENCY events@^1.1.1
│ ├── UNMET DEPENDENCY jmespath@0.15.0
│ ├── UNMET DEPENDENCY querystring@0.2.0
│ ├── UNMET DEPENDENCY sax@1.2.1
│ ├── UNMET DEPENDENCY url@0.10.3
│ ├── uuid@3.1.0
│ ├── UNMET DEPENDENCY xml2js@0.4.17
│ └── UNMET DEPENDENCY xmlbuilder@4.2.1
├── balanced-match@1.0.0 extraneous
... again many more ...
Notice how it reports unmet dependencies with the actual dependencies + extraneous directories in node_modules folder. This seems to be due to some special handling in npm for symlinks...
Regarding the downsides of this change I think it's just about copying around lots of data instead of creating a two symlinks. Even in my simple case it's copying around 100 MBs on each build, be it invoke local
or deploy
. Yet with SSDs and the OS's file system cache it shouldn't hurt much. At least on my laptop it's negligible
In a way this would also "solve" issue #23 :-)
@pmuens could you share your view on this?
Thanks for pinging @schickling 👍
Sounds like a feasible solution! Wondering if this is an edge case or smth. common 🤔.
Maybe a good solution is to use copy when deploying and linking while building for development?
Also having issues with this.
Is there any update on this, or related issues (like https://github.com/prismagraphql/serverless-plugin-typescript/issues/23, https://github.com/prismagraphql/serverless-plugin-typescript/issues/89, https://github.com/prismagraphql/serverless-plugin-typescript/issues/90) around using serverless-plugin-typescript on windows?
For whatever reason, the workaround that previously worked (being an administrator) isn't working for me on Windows 10 build 17134.
I'm experiencing the exact same issue described by @stesie.
I came across a workaround in a different issue that fixes this for me (https://github.com/prisma/serverless-plugin-typescript/issues/89#issuecomment-398754969) – set NODE_PRESERVE_SYMLINKS=1
in your env.
I'd still love to see this get fixed though!
I have the same issue: the "excludeDevDependencies
" feature in Serverless is not working, because npm ls
cannot properly read the node_modules
from the symlinked dir in the .build
dir.
This is not a Windows issue; npm ls
will give incorrect results on a unix box when run inside the .build
dir as currently created by serverless-plugin-typescript
.
This largely breaks excludeDevDependencies
. I am getting most of my dev dependencies included in my final zip output. It seems that NODE_PRESERVE_SYMLINKS=1
does fix this though.
I'm not sure what the best fix is from serverless-plugin-typescript
's point of view is. It seems to me that the way excludeDevDependencies
is coded up in zipService.js in base serverless is a bit fragile. And fixing npm
itself seems a bit out of scope for serverless-plugin-typescript
. Some ideas:
serverless-plugin-typescript
should just do a cp -r
rather than a symlink for now? That would be a more expensive build than the current symlink approachserverless-plugin-typescript
should explicitly copy only prod dependencies? That would complicate things here and might lead to different problems later.serverless-plugin-typescript
can convince the dependency examination code in zipService.js to run in the "real" node_modules dir, then use the outputted exclusion paths against the .build node_modules dir later?I think the title of this issue should be changed to be more like "excludeDevDependencies
broken by .build/node_modules symlink vs. npm". This issue is hard to find at the moment.
Hey,
I've just recently started trying out serverless framework and this plugin just today. So sorry in case things are a bit inaccurate :)
My problem is that, together with the typescript-plugin, the core's package plugin fails to properly exclude development dependencies from the build/package. This way even a simple "hello world" function has a footprint of roughly 15MB, where it should be just about 2 kB.
Problem seems to be, that package plugin just calls
npm ls --dev=true ...
andnpm ls --prod=true ...
and diffs the output of those two. With this plugin these shell calls are done from the.build
folder, yetnpm ls --dev=true ...
fails to properly list all the dependencies as it stumbles over the symlinking. Seems like it doesn't expect deduped/flattened dir structure then.Replacing both
symlinkSync
calls bycopySync
immediately fixes the problem so :) ... yet I'm unsure if there's a better way to do that ...regards