Zip deploy fails randomly on node_modules files for Linux web app

[AltarBeastiful]

Repro steps.

I'm building a Nodejs 10.14 webservice using Azure Devops and deploying on Azure web app. It works in local or in Azure container, now i'm trying to deploy it in a classic Web App for the sake of simplicity.

Right now my Zip deploys are flaky, giving me errors 3 out of 4 times, always on node_modules files.

My webservice is a classic express backend. The bcrypt module is the only fancyness in it. I'm deploying a zip containing all node_modules installed and files built.

Project structures.

The closest boilerplate I could find : https://github.com/icebob/vue-express-sql-boilerplate

The log/error given by the failure.

~75% of my zip deployments including a node_module ends with an error on a random file:

2019-03-15T10:00:28.6009617Z Command: "/home/site/deployments/tools/deploy.sh"
2019-03-15T10:00:28.6010078Z Handling Basic Web Site deployment.
2019-03-15T10:00:28.6010454Z Kudu sync from: '/tmp/zipdeploy/extracted' to: '/home/site/wwwroot'
2019-03-15T10:00:28.6010711Z Copying file: '.dockerignore'
....
2019-03-15T10:00:28.6026397Z Copying file: 'dist/models/index.js'
2019-03-15T10:00:28.6026635Z Omitting next output lines...

2019-03-15T10:00:28.6026871Z Error: ENOENT: no such file or directory, utime '/home/site/wwwroot/node_modules/.bin/escodegen'

2019-03-15T10:00:28.6027124Z An error has occurred during web site deployment.
2019-03-15T10:00:28.6027354Z Kudu Sync failed
2019-03-15T10:00:28.6027580Z \n/opt/Kudu/Scripts/starter.sh "/home/site/deployments/tools/deploy.sh"
2019-03-15T10:00:28.6027824Z App container will begin restart within 10 seconds.
2019-03-15T10:00:28.6113580Z ##[error]Failed to deploy web package to App Service.
2019-03-15T10:00:28.6123485Z ##[error]Error: Package deployment using ZIP Deploy failed. 

In the 5 failed deploy i had errors on: node_modules/.bin/uglifyjs, node_modules/.bin/sshpk-conv, node_modules/.bin/escodegen

Debug your Azure website remotely.

My sub is cb53**9346a and the site name qa*****pi

Other informations

I also tried to exclude the node_modules and add npm install in my post deployment steps. This doesn't work as bcrypt fails to compile. I didn't try much to make it work as I found documentation indicating the prefered way is to publish the node_modules folder.

Right now I find it hard to publish a node app on Azure, is anybody else experiencing this ?

Thanks all !

[sanchitmehta]

We have been working on a new Feature called App Cache. This feature is designed specifically to create a per instance copy of the App - while ensuring the symlinks are created during the extraction. This would also solve the Kudu Sync and rsync issues that we see on this thread. Currently, the copy of the app code is only mounted in a R/O volume but we plan to make this R/W in coming months. Please try this and open an Azure Support case with us so we can help you immediately

[BenjaminGolba]

Thanks @jake-subvrsive, I ended up with your solution which was indeed the simplest of all that I tried.

[stambunan]

@sanchitmehta Are there any updates on this? Do we still need to rely on the App Cache to get symlinks to work?

[georgeOsdDev]

@sanchitmehta Is kudulite zipdeploy support extacting symLinks ?

Kudu version: 1.0.0.7 (e59ed50ca2)

When I deployed zip file contains symlinks, extracted symlink will result in a file with one line of content.

For example node_modules/.bin/nest is linked to ../@nestjs/cli/bin/nest.js will become a file contains ../@nestjs/cli/bin/nest.js

At local or GitHub Action Agent

$ ls -la node_modules/.bin | grep nest
lrwxrwxrwx   1 toshida toshida    26 Aug 15 10:26 nest -> ../@nestjs/cli/bin/nest.js

At Kudulite or Application Container

kudu_ssh_user@d22111dda621:~/site/wwwroot$  ls -la node_modules/.bin | grep nest
-rwxrwxrwx 1 nobody nogroup    26 Aug 18 07:11 nest
kudu_ssh_user@d22111dda621:~/site/wwwroot$ echo `cat node_modules/.bin/nest`
../@nestjs/cli/bin/nest.js
kudu_ssh_user@d22111dda621:~/site/wwwroot$ 

This file is not executable and cause error like below.

2022-08-18T07:24:09.794285335Z > my-new-nest-app@0.0.1 start /home/site/wwwroot
2022-08-18T07:24:09.794291835Z > nest start
2022-08-18T07:24:09.794295935Z 
2022-08-18T07:24:09.932729587Z /home/site/wwwroot/node_modules/.bin/nest: 1: /home/site/wwwroot/node_modules/.bin/nest: ../@nestjs/cli/bin/nest.js: not found
2022-08-18T07:24:09.943299691Z npm info lifecycle my-new-nest-app@0.0.1~start: Failed to exec start script

I found that there are WEBSITE_ZIP_PRESERVE_SYMLINKS option but it is not described official document. https://github.com/Azure-App-Service/KuduLite/blob/dev/Kudu.Services/Deployment/PushDeploymentController.cs#L903

BTW, I followed below document and have workaround by using dist/main.js instead of using node_modules/.bin/nest https://azureossd.github.io/2022/02/11/Nest-Deployment-on-App-Service-Linux/index.html

[jvano]

Hi

If the problem persists and is related to running it on Azure App Service, please open a support incident in Azure: https://learn.microsoft.com/en-us/azure/azure-portal/supportability/how-to-create-azure-support-request

This way we can better track and assist you on this case

Thanks,

Joaquin Vano Azure App Service