Zip deploy fails randomly on node_modules files for Linux web app

AltarBeastiful commented 5 years ago

Repro steps.

I'm building a Nodejs 10.14 webservice using Azure Devops and deploying on Azure web app. It works in local or in Azure container, now i'm trying to deploy it in a classic Web App for the sake of simplicity.

Right now my Zip deploys are flaky, giving me errors 3 out of 4 times, always on node_modules files.

My webservice is a classic express backend. The bcrypt module is the only fancyness in it. I'm deploying a zip containing all node_modules installed and files built.

Project structures.

The closest boilerplate I could find : https://github.com/icebob/vue-express-sql-boilerplate

The log/error given by the failure.

~75% of my zip deployments including a node_module ends with an error on a random file:

2019-03-15T10:00:28.6009617Z Command: "/home/site/deployments/tools/deploy.sh"
2019-03-15T10:00:28.6010078Z Handling Basic Web Site deployment.
2019-03-15T10:00:28.6010454Z Kudu sync from: '/tmp/zipdeploy/extracted' to: '/home/site/wwwroot'
2019-03-15T10:00:28.6010711Z Copying file: '.dockerignore'
....
2019-03-15T10:00:28.6026397Z Copying file: 'dist/models/index.js'
2019-03-15T10:00:28.6026635Z Omitting next output lines...

2019-03-15T10:00:28.6026871Z Error: ENOENT: no such file or directory, utime '/home/site/wwwroot/node_modules/.bin/escodegen'

2019-03-15T10:00:28.6027124Z An error has occurred during web site deployment.
2019-03-15T10:00:28.6027354Z Kudu Sync failed
2019-03-15T10:00:28.6027580Z \n/opt/Kudu/Scripts/starter.sh "/home/site/deployments/tools/deploy.sh"
2019-03-15T10:00:28.6027824Z App container will begin restart within 10 seconds.
2019-03-15T10:00:28.6113580Z ##[error]Failed to deploy web package to App Service.
2019-03-15T10:00:28.6123485Z ##[error]Error: Package deployment using ZIP Deploy failed.

In the 5 failed deploy i had errors on: node_modules/.bin/uglifyjs, node_modules/.bin/sshpk-conv, node_modules/.bin/escodegen

Debug your Azure website remotely.

My sub is cb53**9346a and the site name qa*****pi

Other informations

I also tried to exclude the node_modules and add npm install in my post deployment steps. This doesn't work as bcrypt fails to compile. I didn't try much to make it work as I found documentation indicating the prefered way is to publish the node_modules folder.

Right now I find it hard to publish a node app on Azure, is anybody else experiencing this ?

Thanks all !

suwatch commented 5 years ago

@michimune this is npm on Linux - could you help investigate this?

satheeshpayoda commented 5 years ago

I'm also facing this issue. For me, KuduSync not able to copy following files under /home/site/wwwroot/node_modules/.bin acorn atob errno

Even i modified the deployment script to delete whole folder using this command " rm -rf /home/site/wwwroot/node_modules " Gettings this error: rm: cannot remove '/home/site/wwwroot/node_modules/.bin': Directory not empty

Then i tried to ls from .bin folder: total 0 -????????? ? ? ? ? ? acorn -????????? ? ? ? ? ? atob -????????? ? ? ? ? ? errno

Anyway to solve this?

this is npm on Linux - could you help investigate this? Yes for me.

blueelvis commented 5 years ago

Try mounting a storage file share from a Storage account and see how it goes.

satheeshpayoda commented 5 years ago

Try mounting a storage file share from a Storage account and see how it goes.

How can i do that? No Storage account is associated with App service. I tried to check the /etc/fstab. But i can't check that from Kudu's bash.

blueelvis commented 5 years ago

Open Azure portal, find the web app and go to its application settings. Inside application settings, scroll to the bottom and you will see an option to mount storage.

On Tue 19 Mar, 2019, 9:21 AM Satheeshkumar Ravichandran, < notifications@github.com> wrote:

Try mounting a storage file share from a Storage account and see how it goes.

How can i do that? No Storage account is associated with App service. I tried to check the /etc/fstab. But i can't check that from Kudu's bash.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/projectkudu/kudu/issues/2946#issuecomment-474188275, or mute the thread https://github.com/notifications/unsubscribe-auth/AGqYI0VeLNq2MaJVBKAP2lScCPcJM-iMks5vYF61gaJpZM4b2sK_ .

satheeshpayoda commented 5 years ago

Open Azure portal, find the web app and go to its application settings. Inside application settings, scroll to the bottom and you will see an option to mount storage. … On Tue 19 Mar, 2019, 9:21 AM Satheeshkumar Ravichandran, < @.***> wrote: Try mounting a storage file share from a Storage account and see how it goes. How can i do that? No Storage account is associated with App service. I tried to check the /etc/fstab. But i can't check that from Kudu's bash. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2946 (comment)>, or mute the thread https://github.com/notifications/unsubscribe-auth/AGqYI0VeLNq2MaJVBKAP2lScCPcJM-iMks5vYF61gaJpZM4b2sK_ .

i don't see any mounted partition there. Sorry but Why do you want to mount Storage Account?

satheeshpayoda commented 5 years ago

I fixed this issue by increasing the Instance Count to 2 and after few mins i reverted back to 1. So old instance got removed and new one stayed. Now the same deployment script is working fine.

AltarBeastiful commented 5 years ago

Didn't try the instance count hack. We ended up going for Windows Web app. Still it would be nice to fix this issue on Zip deploy for Linux.

simkessy commented 5 years ago

I'm getting this error all of a sudden:

ENOENT: no such file or directory, stat '/home/site/wwwroot/node_modules/.bin/npm'

ethubert commented 5 years ago

I second this thread, any idea on how to solve this issue?

suwatch commented 5 years ago

@rramachand21, could you help triage this?

brother-donkey commented 5 years ago

Also experiencing this. Different node module but same problem.

2019-06-30T00:43:28.7646515Z Error: ENOENT: no such file or directory, stat '/home/site/wwwroot/node_modules/.bin/is-ci'

rramachand21-zz commented 5 years ago

@suwatch, we will investigate this.

ivan133 commented 5 years ago

I also have this problem:

Error: ENOENT: no such file or directory, stat '/home/site/wwwroot/bundle/programs/server/node_modules/.bin/js-yaml'

sanchitmehta commented 5 years ago

@AltarBeastiful @ivan133 @BrotherDonkey @ethubert @blueelvis : did it work before and is a recent problem, if yes since when did you start getting this problem? Could you please share your App Names so that we can investigate further ?

Alternatively, you could skip the local/pipeline build and add this App Setting SCM_DO_BUILD_DURING_DEPLOYMENT= true and Kudu will build the artifacts

blueelvis commented 5 years ago

@sanchitmehta - This has been happening since quite some time. The seems to arise because of so many files in an npm/php based project. When projects with less number of files is deployed using Zip-deployed, it doesn't happen. Seems like some sort of locking is done. We used to build locally and then push but it still used to fail.

I don't have access to the website now as it was with an earlier client.

ivan133 commented 5 years ago

I'm pretty sure it's connected with symlinks that are used in npm modules

satheeshpayoda commented 5 years ago

@ivan133 i'm also suspecting the same. Because when i tried to delete whole folder, except symlink everything get deleted. Also when i changed the App Service Plan from Standard to Premium i can able to delete those symlink files without any issue.

Laureian commented 5 years ago

I'm also facing similar issue. I'm trying to deploy standard generated strapi.io project from Azure DevOps to Azure Web App. It's node.js app. My steps:

Build package
- get project from github
- npm install
- create zip package
- publish zip package
Release project
- release zip package
- run npm start as a start command. Every task finishes successfully but when I try to access my app I always get errors like "Cannot find module '../package.json'".

What I found already:

trying to release project without node_modules and then run npm install on Kudu but npm install on Kudu is terribly slow even for small projects
changing Agent Poll from Hosted VS2017 to Hosted Ubuntu 1604 doesn't fix anything
when I download zip, unzip it locally and try to run it's works on win 10 but not on macOS - when I delete node_modules/.bin it starts working on macOS
it's for sure issue connected with differences between agent poll and kudu environment.

brother-donkey commented 5 years ago

I don't think this is the underlying issue, but my particular version of this problem was solved through the following steps:

Looking through the build, noticed that the build machine was a Windows image.
Changed it to hosted Ubuntu (16*), the same as the machine the app service is using.
Redeployed code via pushing to Github.

It did not work if yarn install was used to install things on the build machine. Again, can't say exactly why. Even though the project works well locally with yarn, the same commands on the build machine yield a different effect.

t3custom commented 5 years ago

I am having this exact same issue, and none of the suggestions are working for me. I suspect it has to do with Symlinks as well, but I can't confirm that.

sanchitmehta commented 5 years ago

We are investigating this issue and we would reply soon.

@Laureian are you using App service on Linux? we recently optimized the npm install times(via Kudu) on Linux platform. Have you tried Kudu Builds recently?

Laureian commented 5 years ago

Yup, I was using App Service on Linux. For now, I switched to Web App for Containers and it's working fine.

danjrwalsh commented 5 years ago

@sanchitmehta Any update on the investigation of this? I encountered this issue last week and can't seem to find a resolution. Switching to the Windows App Service isn't an option for my current project, unfortunately.

Laureian commented 5 years ago

I'm also waiting for an update as this is a blocker for Kudu Web apps. :(

ethubert commented 5 years ago

Context : I had to create a CI/CD pipeline for a project. The deployment failed on a linux machine on Azure. Fix : Create a new service plan on a Windows Machine. Kudu has a problem with linux, from what I remember, the messages, while not being precise enough, seemed to indicate 1 of/or 2 things :

No write rights for certain node modules
Path length was too big, which causes the deployment to fail

Hope this helps anyone having problems with it

fabiano commented 5 years ago

I was having a similar problem with a PHP project that is using composer:

Error: ENOENT: no such file or directory, stat '/home/site/wwwroot/vendor/bin/yii'

I had to remove the folders with symlinks from the zip package:

- task: DeleteFiles@1
  displayName: Delete vendor/bin folder from the staging directory
  inputs:
    sourceFolder: $(Build.StagingDirectory)
    contents: vendor/bin

sanchitmehta commented 5 years ago

We found a few issues with ZipDeploy :

Symlinks problem: The files in node_modules/.bin are symlinks. Depending on how users produce the .zip file for their deployment, it is likely that those symlinks are not included in the zipfile. npm automatically adds the files in node_modules/.bin to the PATH so they can be used, so if they are not in the .zip file the scripts might stop working. We have fixed this issue on Kudu for Linux, this should be available in our next platform release. Kudu would now recognize symlinks and defer their creation until the complete zip file is extracted. KuduSync ran into issues when Kudu failed to create these symlink files. If you have sample apps where you run into this issue frequently, let us know we can add this to our tests.
EACCESS issue : some of the users experienced this as well, files did not have correct permissions. Zip deploy now ensures that all the extracted files have correct file permissions.
One other possible cause is that the app is running and locking the binaries : We are considering versioned deployments to fix this - We recommend using slots for active sites currently.

We recently changed the fflow of Kudu based Node deployments and we compress the node_modules directory and extract it locally during runtime/mount it in the memory as a read only volume. (We use Microsoft Oryx to build artifacts on Kudu for Linux) . You could enable Kudu to build artifacts by enabling the app setting SCM_DO_BUILD_DURING_DEPLOYMENT= true. Or you could use RUN_FROM_PACKAGE feature, zip gets mounted on wwwroot as a read-only file system.

levenleven commented 5 years ago

@sanchitmehta Thanks! When next platform release is planned?

ltwlf commented 5 years ago

It seems I have exactly the ZipDeploy Symlinks problem. Currently I can't publish my node app on Linux. RUN_FROM_PACKAGE is not available on Linux. FTP upload would take over a day. When will this be fixed?

levenleven commented 5 years ago

@ltwlf the work around we currently use is running npm i or npm ci as post deploy script. Of course this is weird. Another option is deploying docker image

ltwlf commented 5 years ago

@levenleven thanks. I used the same workaround as @fabiano and and added an exclude for symlinks to my dev ops pipeline: !**/.bin/**/* !**/.env/**/*

usman-siddiqui-devops commented 5 years ago

I am getting the same issue, node application on linux web app on P2v2:1

yasso1am commented 4 years ago

@fabiano I'm very new to devops so I apologize for the ignorance in this question. What part of the process do you delete the bin folder? Is this after building but before zipping? Or is it at some other point?

Originally my post had a question regarding how to use your YAML because I was unfamiliar with using YAML and have only used the GUI provided in Azure Devops. I found there was a "Delete Files" task and it produced the same YAML you generated. Here is the order in which I am doing what you suggested, any guidance is appreciated:

@ltwlf Where/how did you add that in to your pipeline?

Thank you in advance

fabiano commented 4 years ago

@fabiano I'm very new to devops so I apologize for the ignorance in this question. What part of the process do you delete the bin folder? Is this after building but before zipping? Or is it at some other point?

Yes, I'm deleting the files before zipping it. I don't have access to the project anymore, but the order was something like:

Run composer install (it was a PHP project)
Copy files to the staging directory
Remove vendor/bin folder from the staging directory
Zip files
Deploy it to the Azure WebApp

Durisvk commented 4 years ago

Hey guys,

we're running Node JS application on the Linux machine (Azure Web App)

we're using Travis CI to deploy to Azure Web Apps: https://docs.travis-ci.com/user/deployment/azure-web-apps/

It started to happen all of a sudden (the deployments worked up until 2nd October without problems)

Here are the logs:

2019-10-04T12:17:31  Updating branch 'master'.
2019-10-04T12:17:32  Updating submodules.
2019-10-04T12:17:32  Preparing deployment for commit id '44a1913ace'.
2019-10-04T12:17:32  Oryx-Build: Running kudu sync...
2019-10-04T12:17:32    Command: kudusync -v 50 -f /home/site/repository -t /home/site/wwwroot -n /home/site/deployments/44a1913ace81612c66a02c93e583a9e1985d770a/manifest -p /home/site/deployments/02061404a9d2c7007a1b55db5a60210542a4e217/manifest -i ".git;.hg;.deployment;.deploy.sh"
2019-10-04T12:17:34    Kudu sync from: '/home/site/repository' to: '/home/site/wwwroot'
2019-10-04T12:17:34    Error: ENOENT: no such file or directory, stat '/home/site/wwwroot/node_modules'
2019-10-04T12:17:34    
/opt/Kudu/KuduConsole/Scripts/starter.sh kudusync -v 50 -f /home/site/repository -t /home/site/wwwroot -n /home/site/deployments/44a1913ace81612c66a02c93e583a9e1985d770a/manifest -p /home/site/deployments/02061404a9d2c7007a1b55db5a60210542a4e217/manifest -i ".git;.hg;.deployment;.deploy.sh"
2019-10-04T12:17:34  
2019-10-04T12:17:35  App container will begin restart within 10 seconds.

The unsatisfiable solution is to SSH into the machine, cd into the /home/site/wwwroot/ and remove whole node_modules folder (which is by the way present even though the Kudu sync command is complaining that the folder is missing) and rerun the Travis build (which is not how CI/CD should work)

Can someone explain me what could happen on 2nd October or why it stopped working without any change to deployment configuration? (it doesn't work till now - tried to create a new service - first deployment passed and the second immediately failed)

suwatch commented 4 years ago

@sanchitmehta could you help investigate this?

adamhwang commented 4 years ago

Same error as @Durisvk started happening to us today. No changes to package.json or any modules

sanchitmehta commented 4 years ago

@adamhwang @Durisvk We recently moved to compressing node_modules directory and carrying out builds in an local intermediate directory. This was done to optimize the build times. As a part of runtime, we mount/extract the compressed node modules directory to /node_modules. Oryx Release added a symbolic link to /node_modules from /home/site/wwwroot.

The fix for this is to create a /node_modules directory in the Kudu container. We are currently in the process of updating Kudu on Linux and have added a fix for this issue - You should get the fix soon.

Samerkassem commented 4 years ago

I am suddenly facing this issue today:

Command: kudusync -v 50 -f /home/site/repository -t /home/site/wwwroot -n /home/site/deployments/445e4e3aca6fd3719f9d2aa556f1a85f5620ce51/manifest -p /home/site/deployments/b7960ddaa5aac4d6e2a6bc37a315f1af7a199d2f/manifest -i ".git;.hg;.deployment;.deploy.sh"
Kudu sync from: '/home/site/repository' to: '/home/site/wwwroot'
Error: ENOENT: no such file or directory, stat '/home/site/wwwroot/node_modules'
\n/opt/Kudu/Scripts/starter.sh kudusync -v 50 -f /home/site/repository -t /home/site/wwwroot -n /home/site/deployments/445e4e3aca6fd3719f9d2aa556f1a85f5620ce51/manifest -p /home/site/deployments/b7960ddaa5aac4d6e2a6bc37a315f1af7a199d2f/manifest -i ".git;.hg;.deployment;.deploy.sh"

Yesterday it was working fine!!

Samerkassem commented 4 years ago

@adamhwang @Durisvk We recently moved to compressing node_modules directory and carrying out builds in an local intermediate directory. This was done to optimize the build times. As a part of runtime, we mount/extract the compressed node modules directory to /node_modules. Oryx Release added a symbolic link to /node_modules from /home/site/wwwroot.

The fix for this is to create a /node_modules directory in the Kudu container. We are currently in the process of updating Kudu on Linux and have added a fix for this issue - You should get the fix soon.

@sanchitmehta I have created node_modules folder in the very root, and another under "site" (site's root) , of course it won't allow me create node_modules manually inside wwwroot, and I am still having the same issue.

What's working for me is deleting the "node_modules" exensionless file (not folder) which appears inside site/wwwroot folder then the sync with bitbucket is working again but it fails again in the next push and have to redo this thing....it's slowing down deployment time by 10 folds at least because it's downloading and install all npm stuff every single time..

You mentioned a Kudu container, which I imagine is a docker container, where can I access it?

sanchitmehta commented 4 years ago

@Samerkassem The problem is Oryx Runtime Container creating a symlink on the shared file system (/home/site/wwwroot/node_modules) to the runtime container local directory(/node_modules) which is not present in Kudu Build Container. You can open Kudu Bash console <scm url>/newui/kududebug and execute mkdir -p /node_modules - only works on a single instance apps since this directory is local to a kudu instance. We are rolling the fix, kudu should have the fix in a few days.

App Service Kudu Docker Image : https://hub.docker.com/r/appsvc/kudulite Current Tag : appsvc/kudulite:1908260522

Samerkassem commented 4 years ago

@Samerkassem The problem is Oryx Runtime Container creating a symlink on the shared file system (/home/site/wwwroot/node_modules) to the runtime container local directory(/node_modules) which is not present in Kudu Build Container. You can open Kudu Bash console <scm url>/newui/kududebug and execute mkdir -p /node_modules - only works on a single instance apps since this directory is local to a kudu instance. We are rolling the fix, kudu should have the fix in a few days.

App Service Kudu Docker Image : https://hub.docker.com/r/appsvc/kudulite Current Tag : appsvc/kudulite:1908260522

So it wouldn't work to an a web app that has several deployment slots?

Durisvk commented 4 years ago

@Samerkassem Thanks for the explanation, we have currently resolved it by unlinking /home/site/wwwroot/node_modules from /node_modules in a file system.

Can you please keep us updated on when the fix from Kudu side will be up and running? (we're postponing a production release and I'd love to know if the same procedure needs to be applied to the production environment as well)

Samerkassem commented 4 years ago

@Durisvk I think you got me confused with sanchitmehta.

How did you unlink /home/site/wwwroot/node_modules from /node_modules?

adamhwang commented 4 years ago

@sanchitmehta Thanks for the update

For others, we've been able to work around this until the fix is in place by deleting out the /home/site/wwwroot/node_modules folder by running rm -rf node_modules from the wwwroot dir

capraynor commented 4 years ago

Yes same issue here.

Error: ENOENT: no such file or directory, stat '/home/site/wwwroot/node_modules local git deploy has the same issue.

sanchitmehta commented 4 years ago

@adamhwang @capraynor @Durisvk We have deployed the fix to all the affected regions. Some of the regions have already been fixed. We should have the fix deployed everywhere in a couple of days.

capraynor commented 4 years ago

@sanchitmehta

Thanks, this issue fixed on my side.

Japan East, F1 plan, Linux.

varunnandu commented 4 years ago

I am still having problems deploying the python function app on the Linux service app plan. The region is East US 2.

I get the following error:- `2019-10-18T11:42:45.9645416Z Kudu sync from: '/tmp/zipdeploy/extracted' to: '/home/site/wwwroot'

2019-10-18T11:42:45.9645651Z Ignoring: .git

2019-10-18T11:42:45.9645884Z Error: ENOENT: no such file or directory, stat '/tmp/zipdeploy/extracted/worker_venv/lib/python3.6/site-packages/PyJWT-1.7.1.dist-info/pyjwt,sha256=LSP-b8QOOKdE78y4YvciLjn0Mu_h63QoaA4XFXssfDw,244

2019-10-18T11:42:45.9646096Z '

2019-10-18T11:42:45.9646277Z An error has occurred during web site deployment.

2019-10-18T11:42:45.9646428Z Kudu Sync failed`

I am running a previously successful build and release from DevOps pipeline so I know it should work. However, during the last stage of Zip Deploy, it throws me this error. Also, I am on an Isolated plan in ASE.

projectkudu / kudu