search

Butterwire / serverless-plugin-monorepo

A Serverless plugin that allows use of serverless in a JS mono repo.
Mozilla Public License 2.0
167 stars 31 forks source link

operation not permitted, symlink -- even as administrator, even with group policy updated, running mklink in the same terminal works #11

Closed sebastiangug closed 5 years ago

sebastiangug commented 5 years ago

I've been trying to get this plugin to work for creating symlinks, any ideas on why this might happen?

martin-css commented 5 years ago

From the description you're getting this error on Windows. What versions of Windows. Node and thjis plugin are you using? Does mklink /d work as well?

I know older versions needed administrative privileges but believe this isn't the case using Developer mode in the most recent Windows 10 builds.

sebastiangug commented 5 years ago

https://i.imgur.com/UlW6EEQ.png

Works as well, I am indeed using developer mode and I am on build: Version 10.0.18362 Build 18362 been stuck with this for 5 hours straight :(

martin-css commented 5 years ago

What version of Node are you using?

sebastiangug commented 5 years ago 
v10.16.3
hakimio commented 5 years ago

Facing the same issue. Is Windows actually supported since "symlink" is *nix only?

hakimio commented 5 years ago

Ok, I can see that you are using fs.symlink which should work on Windows. Did you check the limitations for Windows and have you tested on Windows?

sebastiangug commented 5 years ago

keep in mind it's using 'fs-extra' not the default 'fs'

hakimio commented 5 years ago

A suggestion from another github thread:

FWIW, I actually needed to pass type=junction to fs.symlink, that made running as admin unnecessary and made the symlinks actually work.

Also, I can see that when running in admin mode some of the symlinks are created but it chokes on @nestjs\common:

Error: EPERM: operation not permitted, stat 'C:\Users\Foo\PhpstormProjects\green-click-crm\backend\node_modules\@nestjs\common'
martin-css commented 5 years ago

A quick bit of testing in a Windows 10 virtual machine suggests the native symlink method isn't working all the time. This could do with having type set to dir as the auto-detection in V12 of Node is not in V10.

We could create hard (junction) links but these can have unintended consequences if not careful.

Try playing with running node in the terminal and creating simple links, e.g. require('fs').symlinkSync('test','C:\\some_path','dir'). I get a EPERM error for certain paths but not others - not ascertained the root reason yet.

sebastiangug commented 5 years ago

@hakimio you're choking with nestjs as well? man I've been having issues left and right trying to deploy this for ~12 hours straight now. I think I'm somewhere at 10 github issues, 6 stackoverflow threads, 4 friends harassed and 20 discord channels spammed. No luck.

@martin-css Node v12 is not yet supported in aws, won't running it via node 12 cause issues? I've got no problem upgrading to v12 on my local machine as long as the actual deployment works.

Also I'm not getting any EPERM with your example. I've created a 'test.js' file with that code and replaced the path to some actual file, the first time it ran, after that i'm getting 'file already exists' which is normal.

martin-css commented 5 years ago

Try pointing the serverless-plugin-monorepo dependency directly to the master branch on Github, e.g. Butterwire/serverless-plugin-monorepo to see if adding the dir type fixes your issue.

You're correct, you can't run V12 of Node on AWS. You could update to V12 locally and still use V10 in your Lambda function. It doesn't matter what version of Node you run locally, it is only used to create the zip package and interact with the AWS apis. The only caveat to this approach is to make sure you don't introduce any unexpected bugs by using V12 features which might pass local unit tests but would fail in production (due to running in V10 runtime).

sebastiangug commented 5 years ago

@martin-css never used that syntax before, how would I write that in my package.json?

martin-css commented 5 years ago

See the docs for more info. Just as I wrote. It's handy for testing, can also target specific branches, commits, tags etc.

I don't want to push out another release until confirmed this issue is resolved.

sebastiangug commented 5 years ago

I've done it, and it works! Althought my function is now 40mb compared to the ~2mb it was when using the optimize plugin, although the optimize one has its own issues around the mongoose driver.

martin-css commented 5 years ago

Good to hear it's working. Issue #8 might help reduce the size a little by hoisting common dependencies like the underlying package manager if/when implemented.

In the meantime, you could look into using layers to move all your node_modules dependencies. This is supported by Serverless but can sometimes require a little monkeying around with links or creating a separate service to manage these.

However, it's normally worth doing as it means you only need to upload dependencies when they change and makes your normal function deployments a lot smaller & hence quicker - which makes a happier developer :smile:

hakimio commented 5 years ago

While the deployment succeeds with the latest version, it still needs admin rights.

martin-css commented 5 years ago

Happy to make hard links (junction type) if everyone would find that preferable. There are some subtle differences that can sometimes trip unsuspecting users up but it may be better for Windows to avoid the need for admin rights (with exception of newer builds of Windows 10).

I'll modify but make the type configurable so it can be set as a symbolic link as well.

martin-css commented 5 years ago

@hakimio please see if the latest version works for you without admin privileges.

@michaelbats please feel free to test the latest version as well to make sure this still works for you as this is the version I intend on releasing to NPM.

hakimio commented 5 years ago

@martin-css it still chokes for me on @nestjs\common when running fs.symlink with junction type and without admin rights. @michaelbats Does it work for you?

sebastiangug commented 5 years ago

@hakimio sadly I've done so many things with my system around symlink permissions that my setup wouldn't be a good pointer. Running as 'admin' usually works because by default only administrators have access to create symlink, but I've added my own user next to the administrators in the local policy in windows. Then I've also switched to developer mode which is supposed to do that for you anyway.

Are you in developer mode in windows?

sebastiangug commented 5 years ago

@martin-css a bit unrelated to this issue, but do you happen to know if there's a way to mimify/optimize the node_modules themselves when used as layers?

I've moved all my main dependencies on a layer (~67mb, .zip compressed to 16mb) I can imagine mimified with tree-shaking this could be easily under 5mb though.