Zip deploy fails randomly on node_modules files for Linux web app

[AltarBeastiful]

Repro steps.

I'm building a Nodejs 10.14 webservice using Azure Devops and deploying on Azure web app. It works in local or in Azure container, now i'm trying to deploy it in a classic Web App for the sake of simplicity.

Right now my Zip deploys are flaky, giving me errors 3 out of 4 times, always on node_modules files.

My webservice is a classic express backend. The bcrypt module is the only fancyness in it. I'm deploying a zip containing all node_modules installed and files built.

Project structures.

The closest boilerplate I could find : https://github.com/icebob/vue-express-sql-boilerplate

The log/error given by the failure.

~75% of my zip deployments including a node_module ends with an error on a random file:

2019-03-15T10:00:28.6009617Z Command: "/home/site/deployments/tools/deploy.sh"
2019-03-15T10:00:28.6010078Z Handling Basic Web Site deployment.
2019-03-15T10:00:28.6010454Z Kudu sync from: '/tmp/zipdeploy/extracted' to: '/home/site/wwwroot'
2019-03-15T10:00:28.6010711Z Copying file: '.dockerignore'
....
2019-03-15T10:00:28.6026397Z Copying file: 'dist/models/index.js'
2019-03-15T10:00:28.6026635Z Omitting next output lines...

2019-03-15T10:00:28.6026871Z Error: ENOENT: no such file or directory, utime '/home/site/wwwroot/node_modules/.bin/escodegen'

2019-03-15T10:00:28.6027124Z An error has occurred during web site deployment.
2019-03-15T10:00:28.6027354Z Kudu Sync failed
2019-03-15T10:00:28.6027580Z \n/opt/Kudu/Scripts/starter.sh "/home/site/deployments/tools/deploy.sh"
2019-03-15T10:00:28.6027824Z App container will begin restart within 10 seconds.
2019-03-15T10:00:28.6113580Z ##[error]Failed to deploy web package to App Service.
2019-03-15T10:00:28.6123485Z ##[error]Error: Package deployment using ZIP Deploy failed. 

In the 5 failed deploy i had errors on: node_modules/.bin/uglifyjs, node_modules/.bin/sshpk-conv, node_modules/.bin/escodegen

Debug your Azure website remotely.

My sub is cb53**9346a and the site name qa*****pi

Other informations

I also tried to exclude the node_modules and add npm install in my post deployment steps. This doesn't work as bcrypt fails to compile. I didn't try much to make it work as I found documentation indicating the prefered way is to publish the node_modules folder.

Right now I find it hard to publish a node app on Azure, is anybody else experiencing this ?

Thanks all !

[nubianMONK]

facing same issue via Local Git deploy - EAST US + EAST US 2 regions. @sanchitmehta Is there an update for my region??

[rabellino12]

👍 same issue here with nodejs express app deployed by github repository

[dougfoo]

I also was receiving the similar error suddenly and wasted hours redeploying to try to figure out what is going on: Error: EACCES: permission denied, mkdir '/home/site/wwwroot/node_modules/.bin'

Looking at my filesystem it indeed had site/wwwroot/node_modules soft linked to /node_modules with permission issues

I removed the soft link (rm node_modules), redeployed and presto works. But only once since the soft link is recreated. I suppose we can wait for this fix or put in the delete soft link script into the pipeline. VERY annoying...

I was releasing almost daily, this started affecting my node.js webapp between Thurs 10/17 1130-1140pm EST if that helps w/ any debugging.

[nshumoogum]

Im also getting the same error as @varunnandu - python function apps on linux app service plan, in region UK South.

I have several function apps and I've tried redeploying (via Azure Devops) the same code on applications that have not been touched in a couple of weeks and they return the same error as those that Im trying to deploy new code to.

The deploy scripts (template) have not changed.

Below are the logs from the deploy:

2019-10-21T15:26:56.4139719Z Copying file: '.python_packages/lib/python3.6/site-packages/chardet/__pycache__/utf8prober.cpython-36.pyc'
2019-10-21T15:26:56.4140113Z Copying file: '.python_packages/lib/python3.6/site-packages/chardet/__pycache__/version.cpython-36.pyc'
2019-10-21T15:26:56.4141047Z Error: ENOENT: no such file or directory, stat '/tmp/zipdeploy/extracted/.python_packages/lib/python3.6/site-packages/chardet-3.0.4.dist-info/chardetect,sha256=SBUuV0vL03okWYc8Dq9UIcSzS-cxKdXBdegOLJoxIbQ,255
2019-10-21T15:26:56.4141384Z '
2019-10-21T15:26:56.4141463Z An error has occurred during web site deployment.
2019-10-21T15:26:56.4141561Z Kudu Sync failed
2019-10-21T15:26:56.4141633Z \n/opt/Kudu/Scripts/starter.sh "/home/site/deployments/tools/deploy.sh"
2019-10-21T15:26:56.4141708Z App container will begin restart within 10 seconds.
2019-10-21T15:26:56.4208986Z ##[error]Failed to deploy web package to App Service.
2019-10-21T15:26:56.4221227Z ##[error]Error: Package deployment using ZIP Deploy failed. Refer logs for more details.

I noticed this started happening yesterday (Monday 21st October) at midday at the latest.

[sanchitmehta]

We are investigating the python Zip Deploy issue, the bug with node_modules has been fixed and patched across all the regions

[craig-miller]

I'm also seeing this as of 10/22/2019. Building on linux, deploying to linux.

Error: EACCES: permission denied, mkdir '/home/site/wwwroot/node_modules/.bin' 2019-10-22T15:44:47.1523830Z \n/opt/Kudu/Scripts/starter.sh kudusync -v 50 -f /tmp/zipdeploy/extracted -t /home/site/wwwroot -n /home/site/deployments/7c4c70484ebb4d788b50257230b3340d/manifest -p /home/site/deployments/1ca21aa686af4aac9cfa661a145b4e77/manifest -i ".git;.hg;.deployment;.deploy.sh"

[ustun]

It seems I have exactly the ZipDeploy Symlinks problem. Currently I can't publish my node app on Linux. RUN_FROM_PACKAGE is not available on Linux. FTP upload would take over a day. When will this be fixed?

@ltwlf Contrary to what docs say (maybe it has since been fixed, haven't checked), we have been using RUN_FROM_PACKAGE for about a month.

We too have the symlink issue though, the symlinks are not symlinks but normal files and this causes relative imports to fail for some packages like sequelize-cli.

[craig-miller]

We are investigating the python Zip Deploy issue, the bug with node_modules has been fixed and patched across all the regions

@sanchitmehta Unfortunately, this hasn't been fixed for node_modules. Just tried it again. Note the timestamp from the logs.

2019-10-22T16:46:42.9510471Z Error: EACCES: permission denied, mkdir '/home/site/wwwroot/node_modules/.bin' 2019-10-22T16:46:42.9510611Z \n/opt/Kudu/Scripts/starter.sh kudusync -v 50 -f /tmp/zipdeploy/extracted -t /home/site/wwwroot -n /home/site/deployments/ca89f0184d7a41d5bd84e8218c8bfe4b/manifest -p /home/site/deployments/1ca21aa686af4aac9cfa661a145b4e77/manifest -i ".git;.hg;.deployment;.deploy.sh"

[jordync3]

@craig-miller Same thing is happening with me. Where are you finding the logs?

@sanchitmehta It is still happening for me.

[jordync3]

Temporary fix for azure app services. Create a new deployment slot, push your app to the new slot, and then swap the slots --> then delete the temporary slot.

[craig-miller]

@jordync3

@craig-miller Same thing is happening with me. Where are you finding the logs?

Azure DevOps->Your project->Pipelines->Releases->Your Release. Mouse over your release task and a couple of buttons will appear including "Logs". Click on Logs. Mouse over the failed deploy task and on the right side of it, you'll see a small blue icon that looks like a zip archive. Click on it, and it will download the complete logs (not just the short section shown in the window).

[roycornelissen]

Same thing here... after creating a new WebApp on Linux, I can do a few successful deployments, but at some random point they start failing and never recover from that until I delete the WebApp and create a new one.

Logs:

2019-10-23T14:17:46.7753068Z Kudu sync from: '/tmp/zipdeploy/extracted' to: '/home/site/wwwroot'
2019-10-23T14:17:46.7753600Z Error: ENOENT: no such file or directory, stat '/home/site/wwwroot/node_modules/.bin/seek-bunzip'
2019-10-23T14:17:46.7753904Z An error has occurred during web site deployment.
2019-10-23T14:17:46.7754154Z Kudu Sync failed
2019-10-23T14:17:46.7754382Z \n/opt/Kudu/Scripts/starter.sh "/home/site/deployments/tools/deploy.sh"
2019-10-23T14:17:46.7754477Z App container will begin restart within 10 seconds.
2019-10-23T14:17:46.7815628Z ##[error]Failed to deploy web package to App Service.
2019-10-23T14:17:46.7838244Z ##[error]Error: Package deployment using ZIP Deploy failed. Refer logs for more details.

[gecode-es]

Same issue here for past week.

Error: EACCES: permission denied, mkdir '/home/site/wwwroot/node_modules/.bin'

Been using Azure for both .NET Core and Node projects for a few years now and I've never had any confidence in the services. I always deploy and worry that tomorrow something will break and I'll find myself in a world of hell again. This latest issue is the final straw I think. :-( my life is complicated enough without this nonsense.

[roycornelissen]

No real issues with .NET Core till now but this past week makes me wonder how much Microsoft ❤️ Linux at all. Very frustrating.

[ustun]

Have you tried WEBSITE_RUN_FROM_PACKAGE=1 env var?

It worked wonders for us (didn't fix the symlink issue, but we circumvented that by specifying the full path.)

https://docs.microsoft.com/en-us/azure/azure-functions/run-functions-from-deployment-package#benefits-of-running-from-a-package-file

it says functions, but works with node on linux app service.

[roycornelissen]

@ustun, thanks, that’s interesting. Need to figure out how to build and deploy that package but I’ll try that. Quite new to NodeJS so it’s an adventure.

[gecode-es]

@ustun much appreciated this appears to have worked perfectly for me. Has saved me a world of hurt so thank you.

[ustun]

@gecode-es Glad to hear it helped you!

Here is my understanding of azure web apps and kudu (I'm an azure customer, not an MSFT employee, so please correct me if any of these assumptions are wrong):

App Service runs kind of a docker container, however kudu sync treats the container as a mutable file system. That's why at every deployment, you're basically copying some of the files to the target file system. Also, you will see that if you create a file in the container, let's say via touch foobar, that file will be preserved across deploys. So it does not start with a fresh container as in docker each time, it is more similar to how a normal machine would work, even though it is docker based.

With this env var though, nothing gets copied except the zip file itself, which is mounted as a filesystem, therefore it is atomic (all or none)

With kudusync, when you deploy a node app, it looks at the timestamps of each file in your app and node_modules, determines which files are newer, and copies those. This is problematic in node ecosystem as some packages have been built such that their timestamp is a fixed point back in time (1985 specifically.) node guys do this for reproducible builds, but this confuses kudu sync as it thinks that those files are either too old or the same.

Therefore, in general, mutating the file system is very problematic and error proen and this flag makes it as if a layer (somewhat like a docker layer) consisting of the contents of the zip have been applied.

So, it is a very clean approach and works almost without issues.

One of the issues we ran into was the following though, when the very same container is accessed via kudu's rest api, kudu sees the system as if that zip file has not been mounted. We were using kudu rest api to run some one of commands, but when kudu access the app service, the folder is blank since it is not mounted.

Another issue was that the symlinks inside node_modules/.bin were not proper, so to run a migration via sequelize for example, we had to do the following:

node node_modules/sequelize-cli/lib/sequelize db:migrate

normally, npm run dbmigrate which is specified as sequelize db:migrate would work, but doesn't and fails with the following issue: Error: Cannot find module './core/yargs'

In general, I recommend WEBSITE_RUN_FROM_PACKAGE=1 overall. One other benefit was that our deploys, which were taking 15-20 minutes due to node_modules now take 30-40 seconds.

[gecode-es]

@ustun some great insight thanks for taking the time to explain your understanding and experience. Really appreciated.

[studioab]

Same issue. Deploying node app to Web App service on Linux stopped working recently. Zip deploy fails
Error: EACCES: permission denied, mkdir '/home/site/wwwroot/node_modules/.bin'

is this Azure issue or kudu? Any ETA on fix?

[sanchitmehta]

@ustun Thanks for explaining the problem. We have been working to improve Kudu on Linux and the path forward we see is mounting the deployment artifacts as a read only file system and versioning them (enabling rollbacks and rollforwards) - this would automatically plug in RunFromPackage with any deployment type (built on Kudu or elsewhere).

For more insights, how kudu works on App Service on Linux.

1. We redeveloped/ported Kudu in ASP NETCore which is used on Linux platform -> KuduLite.
2. Kudu Builds on Linux are now done using Oryx Build System
3. KuduLite runs as a sidecar docker container and mounts a shared file system at /home

For the same reason, we have already changed Kudu NodeJS(and Python Environment) build process with zip deploy/git deploy to compress the node_modules folder (or antenv for python environment) and mount this zip as a read only file system(you can try this by setting an App Setting BUILD_FLAGS=UseExpressBuild) at /node_modules or runtime containers simply fallback to extracting them at /node_modules during startup (in the container's ephemeral file system). This problem is with artifacts that are built from dev ops pipelines is preserving symlinks depend on the zip client(and OS) type resulting in file file errors. We recently fixed the symlinks issue with the zip artifacts(Kudu tries to detect a symlink and creates it).

The builds traditionally on Kudu have been carried out on the shared file system in the destination directory (or /home/site/wwwroot) , resulting in much slower deployments. We have changed this in KuduLite to carry out builds in an intermediate local directory and copy them over finally to /home/site/wwwroot.

We recommend usingWEBSITE_RUN_FROM_PACKAGE=1.Any folder under /home is still writable and preserved between the deployments but /home/site/wwwroot is not. If this app setting is switched on, we don't extract the zip file but mount the zip artifact as a read only file system at /home/site/wwwroot (we may change this path in future).

When the above app setting is switched on Kudu saves the zip artifact at /home/data/SitePackages (this path would change with the versioning changes described above, and I would update this post then). This zip is then mounted at /home/site/wwwroot to the runtime container only. This can be seen by navigating to /home/site/wwwroot on WebSSH. We are tracking a backlog Work Item to mount this to Kudu Container as well.

[clewiston]

I started hitting this issue 2 days ago with my node/express deployment on Azure Linux. I'm using git deploy. Is the consensus now that git deploy is broken, and I should only use FTP? this is rather inconvenient if it's the case.

[sanchitmehta]

@clewiston There are known issues with Zip Deploy (where we recommend the use of WEBSITES_RUN_FROM_PACKAGE=1 app setting) but not Git Deployment. Could you please share the deployment error trace ?

[clewiston]

Thanks @sanchitmehta
It was failing on the step "Removing existing modules directory from root..." I don't have the exact error log, but it was related to moving the node_modules folder to del_node_modules folder.

I just ripped down and set up the web app again, twice, and now i'm deploying just fine, so naturally I can't recreate this error right now.

One question about node_modules (maybe this isn't the right place to ask) -- I don't commit my node_modules when I deploy, and then my startup script performs an "npm install" before starting up the node process. Is this the correct approach?

[ustun]

One question about node_modules (maybe this isn't the right place to ask) -- I don't commit my node_modules when I deploy, and then my startup script performs an "npm install" before starting up the node process. Is this the correct approach?

Ideally you should have a build step when you zip everything and use that during release (deploy) stage, otherwise it will be flaky. Also you should not use npm install but npm ci since npm install does not honor package-lock.json.

[nshumoogum]

I am still hitting this issue with my python function app running on an app service plan.

I built a new function app from scratch from within the azure portal as well as a new app service plan P1V2. I added the WEBSITE_RUN_FROM_PACKAGE=1 env var to function app.

Then I created a new CI/CD pipeline in Azure Devops but the deploy fails for the same reason as before:

2019-10-25T18:23:41.4700631Z Command: "/home/site/deployments/tools/deploy.sh"
2019-10-25T18:23:41.4700786Z Handling Basic Web Site deployment.
2019-10-25T18:23:41.4701185Z Kudu sync from: '/tmp/zipdeploy/extracted' to: '/home/site/wwwroot'
2019-10-25T18:23:41.4701259Z Ignoring: .git
2019-10-25T18:23:41.4701711Z Error: ENOENT: no such file or directory, stat '/tmp/zipdeploy/extracted/.python_packages/lib/python3.6/site-packages/chardet-3.0.4.dist-info/chardetect,sha256=SBUuV0vL03okWYc8Dq9UIcSzS-cxKdXBdegOLJoxIbQ,255
2019-10-25T18:23:41.4701991Z '
2019-10-25T18:23:41.4702052Z An error has occurred during web site deployment.
2019-10-25T18:23:41.4702132Z Kudu Sync failed
2019-10-25T18:23:41.4703788Z \n/opt/Kudu/Scripts/starter.sh "/home/site/deployments/tools/deploy.sh"
2019-10-25T18:23:41.4704265Z App container will begin restart within 10 seconds.
2019-10-25T18:23:41.4776354Z ##[error]Failed to deploy web package to App Service.
2019-10-25T18:23:41.4786174Z ##[error]Error: Package deployment using ZIP Deploy failed. Refer logs for more details.
2019-10-25T18:23:43.5134595Z Successfully added release annotation to the Application Insight : dev-dataset-api-app

Im trying to understand whether I need to do something else to allow my CI/CD pipelines to work or am I waiting on Microsoft to fix zip deploy?

[willmtemple]

I was having this issue on deployments (EACCESS on node_modules/.bin) from Azure DevOps for a NodeJS web app on linux after moving to that from using git deploy. Deleting and re-creating the app service in the Azure Portal resolved the issue.

[sanchitmehta]

@nshumoogum If you're using Python 3 and creating the environment yourself you could try adding --copies : python -m venv antenv --copies.

From your logs it doesn't look like Run From package is enabled and Kudu is trying to do a normal zip deployment and extracting the node_modules. Could you please check the Kudu Environment page, if the app setting WEBSITE_RUN_FROM_PACKAGE=1 is present and then redeploy.

@willmtemple this could also be that a different user (kudu runs as a dedicated user/group) that is creating installing npm modules in the pipeline and kudu user does not have access to override them. You could add npm config set unsafe-perm=true to allow all users to override these files.

[ustun]

@nshumoogum I think you can also set the env var in azure devops release pipeline too. Give that a try as well. That's what I had tried initially, being unsure where I should set it. But I think setting it in portal should suffice.

[nshumoogum]

@ustun - thanks I tried adding the app setting in App settings likes so -WEBSITE_RUN_FROM_PACKAGE 1 and re-deployed but got the same error again.

@sanchitmehta - I've gone to the Kudu environment page and I can see the variable set in https://<app-name>.scm.azurewebsites.net/api/settings

See below for app settings:

{
"deployment_branch": "master",
"SCM_TRACE_LEVEL": "Verbose",
"SCM_COMMAND_IDLE_TIMEOUT": "60",
"SCM_LOGSTREAM_TIMEOUT": "7200",
"SCM_BUILD_ARGS": "",
"WEBSITES_RUN_FROM_PACKAGE": "1",
"ScmType": "None",
"WEBSITE_AUTH_LOGOUT_PATH": "/.auth/logout",
"FUNCTIONS_RUNTIME_SCALE_MONITORING_ENABLED": "0",
"WEBSITE_SITE_NAME": "dev-dataset-api-app",
"REMOTEDEBUGGINGVERSION": "15.0.28307.222",
"WEBSITE_AUTH_ENABLED": "False",
"SCM_USE_LIBGIT2SHARP_REPOSITORY": "0",
"WEBSITES_ENABLE_APP_SERVICE_STORAGE": "true",
"WEBSITE_ENABLE_SYNC_UPDATE_SITE": "true",
"WEBSITE_SLOT_NAME": "Production",
"FUNCTIONS_EXTENSION_VERSION": "~2",
"WEBSITE_AUTH_AUTO_AAD": "False",
"APPINSIGHTS_INSTRUMENTATIONKEY": <instrumentation key>,
"AzureWebJobsStorage": <connection string>,
"FUNCTIONS_WORKER_RUNTIME": "python"
}

[cbb330]

@sanchitmehta Had the same as error @nshumoogum with a python app failing because of symlinks. Adding the '--copies' option to venv creation fixed it! Hopefully adding WEBSITE_RUN_FROM_PACKAGE will add some consistency as well with other issues for us. Thank you and thanks for the transparency on these issues.

[superfliege]

Same here: Kudu sync from: '/home/site/repository' to: '/home/site/wwwroot' Error: ENOENT: no such file or directory, stat '/home/site/wwwroot/_del_node_modules'

[sanchitmehta]

@nshumoogum We are looking at your app. @superfliege We have fixed this for the next Kudu Version. This fix would be a part of our next milestone which would start deploying in a week.

If you're using Zip Deploy, a work around is to use Run From Package : Enable it by setting the app setting WEBSITE_RUN_FROM_PACKAGE=1

[nshumoogum]

@sanchitmehta - thank you - I did add the option --copies in the build script but also no luck. My github repositories are all public with several azure-pipelines.yml files to represent different builds (mostly for the different environments).

Also I ended up deploying via visual studio code (as a work around) as i needed to get some new code out onto staging environment to performance test. Let me know what you find @sanchitmehta

[studioab]

node_modules is locked and can't be deleted via SSH or Bash on Linux machine :( swapping slot with an empty one deletes everything, here is my workaround for Node app zip deploy

az webapp deployment slot create --name YOURAPP --resource-group YOURGROUP --slot temp

az webapp deployment slot swap -g YOURGROUP -n YOURAPP --slot temp --target-slot Development

az webapp deployment slot delete --name YOURAPP --resource-group YOURGROUP --slot temp

works like a charm

[ustun]

@nshumoogum WEBSITES_RUN_FROM_PACKAGE should be WEBSITE_RUN_FROM_PACKAGE

[nshumoogum]

@ustun - i am embarrassed, yes this worked for me; i should have double checked the variable name

Thanks everyone who tried to help me :)

[jwr456]

this fixed my Python Flask issue

• script: | python3.6 -m --copies venv .env source .env/bin/activate pip3.6 install setuptools pip3.6 install -r requirements.txt

[RamunasAdamonis]

I was having the same issue (no such file or directory, open '/home/site/wwwroot/node_modules/.bin/sshpk-conv), and then switched to WEBSITE_RUN_FROM_PACKAGE=1. But after this app was not working anymore. Reason being that app startup failed with error: Cannot find module './oryx-appinsightsloader.js'. My Web App had application insights enabled and this was injecting appinsightsloader script to confugure app insights. But this does not work if you run from package. Workaround was to disable Application Insights on Web App and configure it manually in code.

[nitinmb]

@sanchitmehta - We are noticing similar error related to a missing module during the app startup - after using zip deploy option. The same version of code works fine in one environment, but gives an error related to a missing module in other.

Per your recommendation , setting "WEBSITE_RUN_FROM_PACKAGE=1" helped fix the issue. But this makes the FS read only.

Following up with you to see if the team has identified the root cause for the issue with zip deploy and possible timeline for fix rollout.

Thanks for your help.

[hicksdavies]

@sanchitmehta - We are too now experiencing the (no such file or directory, open '/home/site/wwwroot/node_modules/.bin/xxx ) error. I have tried everything, recreated the Linux App, added WEBSITE_RUN_FROM_PACKAGE=1 into app settings, but nothing fixes the issue. Any ideas ? This is also not intermittant it seems to be every time ? Thanks John

[hicksdavies]

@nitinmb have you got any further in solving ? After i added the setting "WEBSITE_RUN_FROM_PACKAGE=1" I too get a missing module error ?

[HoLengZai]

Facing the same issue. With Azure Web App on Linux

2020-02-13T13:20:28.940273214Z > svg-sprite --config sprite-config.json src/svgs/*.svg 2020-02-13T13:20:28.940277614Z 2020-02-13T13:20:29.588164612Z internal/modules/cjs/loader.js:797 2020-02-13T13:20:29.588189713Z throw err; 2020-02-13T13:20:29.588201314Z ^ 2020-02-13T13:20:29.588206014Z 2020-02-13T13:20:29.588210314Z Error: Cannot find module '../lib/svg-sprite' 2020-02-13T13:20:29.588214615Z Require stack: 2020-02-13T13:20:29.588218715Z - /home/site/wwwroot/node_modules/.bin/svg-sprite 2020-02-13T13:20:29.588223015Z at Function.Module._resolveFilename (internal/modules/cjs/loader.js:794:15) 2020-02-13T13:20:29.588227315Z at Function.Module._load (internal/modules/cjs/loader.js:687:27) 2020-02-13T13:20:29.588231516Z at Module.require (internal/modules/cjs/loader.js:849:19) 2020-02-13T13:20:29.588235616Z at require (internal/modules/cjs/helpers.js:74:18) 2020-02-13T13:20:29.588239916Z at Object. (/home/site/wwwroot/node_modules/.bin/svg-sprite:25:15) 2020-02-13T13:20:29.588244616Z at Module._compile (internal/modules/cjs/loader.js:956:30) 2020-02-13T13:20:29.588248817Z at Object.Module._extensions..js (internal/modules/cjs/loader.js:973:10) 2020-02-13T13:20:29.588253017Z at Module.load (internal/modules/cjs/loader.js:812:32) 2020-02-13T13:20:29.588268718Z at Function.Module._load (internal/modules/cjs/loader.js:724:14) 2020-02-13T13:20:29.588272718Z at Function.Module.runMain (internal/modules/cjs/loader.js:1025:10) { 2020-02-13T13:20:29.588276618Z code: 'MODULE_NOT_FOUND', 2020-02-13T13:20:29.588280918Z requireStack: [ '/home/site/wwwroot/node_modules/.bin/svg-sprite' ] 2020-02-13T13:20:29.588284919Z } 2020-02-13T13:20:29.601757090Z npm ERR! code ELIFECYCLE 2020-02-13T13:20:29.602693244Z npm ERR! errno 1

Any progress on this issue? I went to the Bash console, and the file exist

Thanks for your help.

[pat-flew]

I am having a similar issue when using WEBSITE_RUN_FROM_PACKAGE=1. Some of the symbolic links are not being set correctly during deployment

e.g.

/node_modules/html-webpack-plugin/node_modules/loader-utils/node_modules/.bin/json5

is being linked to

../../../json5/lib/cli.js

where it should be

../../../../../json5/lib/cli.js

consequently, the deploy fails with Error: ENOENT: no such file or directory, stat '/home/site/wwwroot/node_modules/html-webpack-plugin/node_modules/loader-utils/node_modules/.bin/json5'

[pat-flew]

I cleaned up the build artifact to only include packages necessary for production which had the side-affect of fixing this, as we don't need webpack there. But is the above expected behaviour? This is a build that was working previously (at least until it suddenly stopped working which is what led me to this issue)

[jbeckton]

Azure Sucks... thats the problem. On AWS people can get work done plain and simple. Microsoft's warped architectures does nothing but hold you back. Maybe we just need a few more layers of crap on top of the other layers of crap to fix it?

[capraynor]

@jbeckton Agreed. Totally agreed.

[sanchitmehta]

@pat-flew we don't extract the zip package and recreate the symlinks. We use fuse-zip file system to mount this zip, when the app is run using RunFromPackage

[dsherret]

I've had so much trouble with Azure App Service on Linux. It's not a great experience setting up a node app there compared to deploying a .net application on a Windows service. I'm kind of upset about how much time I've wasted on this (I usually wouldn't express this on a GitHub issue, but it's a paid product). I'm concerned about what issues I may encounter in the future if setup was this bad.

For this specific problem I encountered, I managed to workaround this after a lot of trial and error.

Problem 1: After doing a deploy with symlinks, any deploy I tried after even without symlinks would fail, and web ssh didn't work. Was getting a EHOSTUNREACH error. Solution: Open up the App Service in the portal, Development Tools -> Advanced Tools, then click on the "Bash" tab. For me, that worked for some reason and I was displayed a prompt. From there I was able to rm -rf the contents of the /home/site/wwwroot folder.

Problem 2: Since a deploy with symlinks wrecked the server, what could be done to not deploy a zip with symlinks? Solution: I tried a lot of different things (including different deployment strategies) and ran into so many different problems. The only thing that ended up working for me was to delete all the symlinks and copy the target of the symlink to where the symlink used to be, then zip the folder for deployment to the server. This wasn't a big issue for me to do since the size of the files was small.

I used this powershell script in my azure pipeline to do this:

# Azure App Service Zip Deploy doesn't handle symlinks so inline
# all of them (https://github.com/projectkudu/kudu/issues/2946)
$links = dir . -recurse -force | ?{$_.LinkType}
foreach ($link in $links) {
  # store these before removing the link, otherwise the properties will go to null
  $source = $link.Target
  $dest = $link.FullName

  echo "Removing symbolic link at ${dest}..."
  (Get-Item $dest).Delete()

  echo "Copying ${source} to ${dest}..."
  Copy-Item -Path $source -Destination $dest -Recurse
}

[jake-subvrsive]

For people still running into this while trying to deploy a Node app from Github, I thought I'd add the easiest work around I've found.

I just created a startup.sh script in the root of my projects which runs npm rebuild to recreate all the node_modules symlinks followed by my actual project's run command. I then go into the app service configuration in the Azure portal and change the custom startup command to startup.sh.

This way the container will regenerate all of the symlinks before actually running your project. An example startup.sh script follows:

npm rebuild

npm run production