Deployment is *very* slow (nodeJS with publish profile)

0gust1 commented 4 years ago

I'm using a workflow file adapted from "build and deploy a Node.js Web app to Azure using publish profile".

(side remark) : I had issues with symbolic links too (see https://github.com/Azure/webapps-deploy/issues/54#issuecomment-694259266)

Upon analysis, it seems that the whole copy of the node_modules folders to the webroot of the webapp takes ages (10-12 minutes for the Azure Webapp deploy step !) It's very inefficient to process and send many little files over network.

=> Shouldn't the final npm install (and npm run build – I'm using typescript) commands be executed on the target machine instead of a github action container ?

IMHO, the workflow should be like this for the "nodeJS with publish profile" scenario :

the github action should only send source files to the targeted runtime machine
npm install, npm run build and npm start should be done on the targeted runtime machine
It's up to the developer to ensure that is app is building correctly before calling azure/webapps-deploy

Related questions :

Also how this Github action is related to Kudu ?
Does it use Kudu behind the scenes ? How ? (bare copy ? zip deploy ?)
If yes, are Kudu capabilities useable ? I.E. .deployment file, deployment hooks, etc.
references : https://github.com/projectkudu/kudu/wiki/Customizing-deployments and https://github.com/projectkudu/kudu/wiki/Deployment-hooks

sfabriece commented 4 years ago

My deployments are taking 10 minutes and it is slowing us down. How can we improve the situation?

rachael-ross commented 3 years ago

Our deployments are taking over 30 mins!

AmrutaKawade commented 3 years ago

@0gust1 Sorry for delayed response. action uses kudu zipdeploy api for deployment with zip file input https://github.com/projectkudu/kudu/wiki/REST-API#zip-deployment

sfabriece commented 3 years ago

Any way to speed it up @AmrutaKawade ?

AmrutaKawade commented 3 years ago

@sfabriece custom deloyment scripts should work. Do not include node_modules in zip. Instead add .deployment and deploy.cmd / deploy.sh files at root of your repo with npm install command. it will execute npm install on target machine

some examples https://docs.microsoft.com/en-us/azure/app-service/configure-language-nodejs?pivots=platform-windows#run-gruntbowergulp https://github.com/projectkudu/kudu/wiki/Deployment-hooks#easiest-way-to-get-started-with-a-custom-deployment-script https://github.com/projectkudu/kudu/wiki/Custom-Deployment-Script

0gust1 commented 3 years ago

Thanks @AmrutaKawade !

Ideally, hints could be given in documentations and workflow files (for example: https://github.com/Azure/actions-workflow-samples/blob/master/AppService/node.js-webapp-on-azure.yml), as every nodeJS app have the same behavior on this, and people using the actions-workflow-samples will encounter the same issue.

sfabriece commented 3 years ago

@AmrutaKawade as far as I can tell, the deploy scripts control how the app is deployment once on the target machine. How do I control what ends up in the zip file?

AmrutaKawade commented 3 years ago

@sfabriece you can provide directly zip as a input to action or folder location which will be zipped in action

- name: 'Run Azure webapp deploy action using publish profile credentials'
      uses: azure/webapps-deploy@v2
      with:
        app-name: <app-name>
        publish-profile: ${{ secrets.azureWebAppPublishProfile }}
        package: <provide path of your folder or zip file>

kaverma commented 3 years ago

@sfabriece you can provide directly zip as a input to action or folder location which will be zipped in action

- name: 'Run Azure webapp deploy action using publish profile credentials'
      uses: azure/webapps-deploy@v2
      with:
        app-name: <app-name>
        publish-profile: ${{ secrets.azureWebAppPublishProfile }}
        package: <provide path of your folder or zip file>

Closing the issue as the question has been addressed and for performance we already have npm install in the sample workflow which optimises the time

ywaserblat-excelforce commented 3 years ago

Had the same issue, tried with the suggested solution of creating a custom deploy (btw, custom deploy for a standard task?) but got the following error: /opt/Kudu/Scripts/starter.sh: line 2: exec: deploy.cmd: not found /opt/Kudu/Scripts/starter.sh: line 2: exec: deploy.cmd: not found\n/opt/Kudu/Scripts/starter.sh deploy.cmd so far I wasn't able to solve this issue, so I am falling back to the default inefficient procedure. I do believe that @0gust1's suggestion is the right approach: the github action should only send source files to the targeted runtime machine npm install, npm run build and npm start should be done on the targeted runtime machine It's up to the developer to ensure that is app is building correctly before calling azure/webapps-deploy this should be the default w/o any customization needed as it is the standard behavior for node.js apps

JonCognioDigital commented 3 years ago

Could somebody please explain how this can be solved?

I am using the latest version of the script automatically installed by the Azure portal when using the deployment center. It seems to be running npm install on Github then uploading every file individually. Did anyone find a way to (A) zip them up and unzip them on the target webapp or (B) just do the npm install on the target webapp (although that would stop you from doing any other steps using github actions like testing)?

ghost commented 3 years ago

Seconding this one I am having the same problem and the solutions provided above do not seem to be working. I am using the standard file automatically generated by the Azure portal as well.

kaverma commented 3 years ago

@BALAGA-GAYATRI , can you please help here with the above queries? If the solution isn't working, can you please reopen the issue?

sabeeshvk commented 3 years ago

Could somebody please explain how this can be solved?

I am using the latest version of the script automatically installed by the Azure portal when using the deployment center. It seems to be running npm install on Github then uploading every file individually. Did anyone find a way to (A) zip them up and unzip them on the target webapp or (B) just do the npm install on the target webapp (although that would stop you from doing any other steps using github actions like testing)?

This is still a problem. I am trying to deploy a sample angular app and the files are being pushed individually instead of getting packaged together. The build process is slow as well.

mcthick commented 3 years ago

I have a similar problem, but with just simple HTML. It seems that changing a single character in a single file means that the action goes ahead and grabs every file in the site structure to include in the zip rather than the files changed according to the last ref. So, a single character change takes 4-5 minutes to deploy. Setting the package directory as an option is not a functional solution since my devs could adjust any part of the site at any time

github-actions[bot] commented 3 years ago

This issue is idle because it has been open for 14 days with no activity.

mweel1 commented 3 years ago

+1. Very slow deployments.

zhughes3 commented 3 years ago

Just wanted to chime in here. Basically, I am seeing this be very slow on the initial deployment (~10min with "provisioningState": "InProgress") but am seeing deployment times of ~2 minutes on subsequent deploys. This deploy has the node app built with node_modules in the zip.

jsmircic commented 3 years ago

Can someone point to an example on how to setup npm install as post deployment step. Not much luck with google. This was possible with Azure DevOps release task but can't make it work with GitHub action. Tried the suggestion from @AmrutaKawade without success. looks like the Kudu is skipping .deployment and .deploy.sh files for some reason. This is written in logs

Kudu sync from: '/tmp/zipdeploy/extracted' to: '/home/site/wwwroot'
Ignoring: .deployment
Ignoring: deploy.sh

Having extremely slow deployment if entire node_modules is packaged (more than 40 min).

github-actions[bot] commented 3 years ago

This issue is idle because it has been open for 14 days with no activity.

Blakerunner commented 3 years ago

I have this same issue. Express server REST API and a React Client front end. The node_modules would be best built on the server rather than added in the zip deployment. Though I'm not really seeing much of a resource on how to do that. Any suggestions?

kizylirio commented 3 years ago

Same problem here

ywaserblat-excelforce commented 3 years ago

In the end I switched to Azure DevOps pipeline. cut deployment time from approx.. 1 hr. to about 8 min. Good job GitHub actions...

sabeeshvk commented 3 years ago

So GitHub actions won't fix and the recommendation is to use Azure dev ops ?

kaverma commented 3 years ago

Sorry for the delay on this issue. We will get back on an example here. We are validating a few examples and will put a recommended documentation for the same.

BALAGA-GAYATRI commented 3 years ago

So we are working on the examples to reduce the time using KUDU custom deployments and deployment hooks based upon the above suggestions by @AmrutaKawade and @0gust1 some examples https://docs.microsoft.com/en-us/azure/app-service/configure-language-nodejs?pivots=platform-windows#run-gruntbowergulp https://github.com/projectkudu/kudu/wiki/Deployment-hooks#easiest-way-to-get-started-with-a-custom-deployment-script https://github.com/projectkudu/kudu/wiki/Custom-Deployment-Script https://github.com/projectkudu/kudu/wiki/Deployment-hooks @0gust1 Can you please confirm if the suggested methods worked for you?

github-actions[bot] commented 3 years ago

This issue is idle because it has been open for 14 days with no activity.

0gust1 commented 3 years ago

Since we opened this issue (it's been some time!), we don't use the github action azure/webapps-deploy anymore:

we created a dedicated service principal to be used with az cli
we build in the github action and make a zip ourselve
we use the az cli action to deploy (see below)
(related?) since this issue opening we also changed our tech stack, we moved from ExpressJS+VueJS to Svelte-kit (with full OS-independent dependencies, on the contrary of our previous tech stack)

Deployment time are now acceptable for us, but we can't right now confirm that the improvements made on the deployment mechanisms described in this issue are working. We hope to have the time to test and confirm.

here the fragment of the github action we use: note to readers: this works for us, because we have OS independent deps in our code. Technically, in general, it's better to install and build on your runtime env**

     - name: Azure Login
        uses: azure/login@v1
        with:
          creds: ${{steps.vars.outputs.sp_credentials}}
      - uses: actions/checkout@v2
        with:
          fetch-depth: 0
      - uses: actions/setup-node@v2
        with:
          node-version: '14'
      - run: npm ci
      - run: VITE_BUILD_DATE="${{steps.vars.outputs.now}}" npm run build
      - run: zip -r -X Archive_to_deploy.zip node_modules/ node-dist/ package.json
      - name: Deploy
        uses: azure/CLI@v1
        with:
          azcliversion: 2.29.2
          inlineScript: |
            az webapp deployment source config-zip --resource-group "${{steps.vars.outputs.rg_name}}" --name "${{steps.vars.outputs.app_name}}" --src ./Archive_to_deploy.zip

github-actions[bot] commented 2 years ago

This issue is idle because it has been open for 14 days with no activity.

sven5 commented 2 years ago

You could even consider runFromPackage. See

This saved some seconds on my deployment using Azure Pipelines' task AzureWebApp@1

github-actions[bot] commented 2 years ago

This issue is idle because it has been open for 14 days with no activity.

bibekgaihre commented 2 years ago

It is taking one hour to deploy

sven5 commented 2 years ago

@bibekgaihre I think your content is too large. You should remove unnecessary files.

github-actions[bot] commented 2 years ago

This issue is idle because it has been open for 14 days with no activity.

danmana commented 2 years ago

I came across a solution for this accidentally. Basically split the workflow into 2 jobs: build & deploy (just like the default workflow template does), but copy between them only the needed files without node_modules. To do this, I create a zip at the end of the build and unzip it at the beginning of the deploy step.

I had to do this zipping anyway because Github was complaining that I was moving more than 10000 files between the jobs. https://stackoverflow.com/a/69402502/281252 https://github.com/actions/upload-artifact#too-many-uploads-resulting-in-429-responses

With this workflow, a Node+Angular app takes 1m for build and ~8m for deployment since Oryx/Kudu runs npm install and build again. Perhaps with a custom build, we might be able to pass it a zip with node_modules and everything built, and just run it without duplicating the build steps... Though there are some node_modules.tar.gz improvements that we might lose

Example workflow

# Docs for the Azure Web Apps Deploy action: https://github.com/Azure/webapps-deploy
# More GitHub Actions for Azure: https://github.com/Azure/actions

name: Build and deploy Node.js app to Azure Web App

on:
  push:
    branches:
      - main
  workflow_dispatch:

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v2

      - name: Set up Node.js version
        uses: actions/setup-node@v2
        with:
          node-version: '16'
          cache: 'npm' # github actions can cache node_modules automatically to speed the build by about ~45s

      - name: npm install, build, and test
        run: |
          npm install
          npm run build --if-present
          npm run test --if-present

      # Zip artifacts to speed things up
      - name: Zip artifact for deployment
        run: zip release.zip ./* -qr -x "node_modules/*" "dist/*"
      # We exclude node_modules and dist because the Azure Oryx/Kudu deploy runs npm install and rebuilds everything anyway...
      # If we zip them we just increase the size of the zip file and it's not needed

      - name: Upload artifact for deployment job
        uses: actions/upload-artifact@v2
        with:
          name: node-app
          path: release.zip

  deploy:
    runs-on: ubuntu-latest
    needs: build
    environment:
      name: 'Production'
      url: ${{ steps.deploy-to-webapp.outputs.webapp-url }}

    steps:
      - name: Download artifact from build job
        uses: actions/download-artifact@v2
        with:
          name: node-app

      - name: Unzip artifact for deployment
        run: |
          unzip -q release.zip
          rm release.zip

      - name: 'Deploy to Azure Web App'
        id: deploy-to-webapp
        uses: azure/webapps-deploy@v2
        with:
          app-name: 'my-app'
          slot-name: 'Production'
          publish-profile: ${{ secrets.AZUREAPPSERVICE_PUBLISHPROFILE_38C3A64C5A4A49F792EA62F5DCBC13ED }}
          package: .

danmana commented 2 years ago

The workflow described above runs in ~10 minutes, out of which the deploy part is the longest.

On my second attempt, I configured the workflow to deploy and run from ZIP, which is much faster - total time ~3 minutes (~1:45s build, ~30s deploy)

Step 1, in Azure configure the app to run from package, and disable the Oryx build (Github actions builds everything anyway)

WEBSITE_RUN_FROM_PACKAGE=1
SCM_DO_BUILD_DURING_DEPLOYMENT=false

Step 2, use this Github workflow that creates a zip after build (including node_modules)

# Docs for the Azure Web Apps Deploy action: https://github.com/Azure/webapps-deploy
# More GitHub Actions for Azure: https://github.com/Azure/actions

name: Build and deploy Node.js app to Azure Web App

on:
  push:
    branches:
      - main
  workflow_dispatch:

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v2

      - name: Set up Node.js version
        uses: actions/setup-node@v2
        with:
          node-version: '16'
          cache: 'npm'

      - name: npm install, build, and test
        run: |
          npm install
          npm run build --if-present
          npm run test:ci --if-present

      # Zip artifacts to speed things up
      - name: Zip artifact for deployment
        run: zip release.zip ./* -qr

      - name: Upload artifact for deployment job
        uses: actions/upload-artifact@v2
        with:
          name: node-app
          path: release.zip

  deploy:
    runs-on: ubuntu-latest
    needs: build
    environment:
      name: 'Production'
      url: ${{ steps.deploy-to-webapp.outputs.webapp-url }}

    steps:
      - name: Download artifact from build job
        uses: actions/download-artifact@v2
        with:
          name: node-app

      - name: 'Deploy to Azure Web App'
        id: deploy-to-webapp
        uses: azure/webapps-deploy@v2
        with:
          app-name: 'my-app'
          slot-name: 'Production'
          publish-profile: ${{ secrets.AZUREAPPSERVICE_PUBLISHPROFILE_38C3A64C5A4A49F792EA62F5DCBC13ED }}
          package: release.zip

An added benefit to this method is that you can download the Action artifacts later if you need to reproduce the environment exactly for debugging, as the Github job artifact named node-app here is exactly what is running on the server. There is no chance that the server would download a newer npm version or anything like that.

Plus it might reduce cold starts

chadmcgowan commented 2 years ago

@danmana thanks this was extremely helpful... I also had to update the azure startup command to npx serve -s /build to run the app... this is under Configuration/General Settings

trapgar commented 2 years ago

@danmana Were there any other changes you had to make to the app service, yml, etc.? Currently my website is failing to load. Logs reveal "Error: Cannot find module '../scripts/start'".

danmana commented 2 years ago

@trapgar no other relevant changes that I can think of.

Is scripts/start a file that you have defined?

Do you have a "start" script defined in package.json? If so, this is what Oryx will execute. If you have only the "main" property defined instead, make sure the path to the file is correct.

Check out the Oryx docs to see what commands it uses to run the app https://github.com/microsoft/Oryx/blob/main/doc/runtimes/nodejs.md#run

Maybe it will help if I tell you all the steps I did:

created a node express app
from VSCode I created an App Service and did a manual deployment to test it worked
I connected the App Service to Github (you can do this from VSCode or from the portal App Service > Deployment Center) - this step generated the default workflow
Made the changes I mentioned above to this workflow

0gust1 commented 2 years ago

A side note/improvement : in automated environments, before the build, it's better / quicker to do npm ci than npm install ref: https://docs.npmjs.com/cli/v8/commands/npm-ci

danmana commented 2 years ago

Yes, as @0gust1 mentioned, you can also use npm ci. When using npm ci you should remove the cache: 'npm' configuration, because it always performs a clean install so caching node_modules only wastes time.

Besides the functional difference between npm ci and npm install, in my experience caching and npm install works just as fast if not faster than npm ci. ymmv

trapgar commented 2 years ago

@danmana Thanks for the reply. scripts/start is a script file that is defined by a library. The application is a ReactJS app created using npx create-react-app my-app.

Since this is just for a personal app it's not particularly important that I get this faster than the ~10m per run I have it at now, so I've decided to shelve it until I finish the app itself.

github-actions[bot] commented 2 years ago

This issue is idle because it has been open for 14 days with no activity.

trapgar commented 2 years ago

Finally got it working. Build is now ~5m 51s down from ~13m 3s. Still not ideal as my app is pretty small, but I think it's as good as I'm going to get for a Node app given all the packages they usually pull in.

Issue ended being a mix of node_modules unnecessarily included in the zip, package.json & package-lock.json not being included in the zip (Onyx couldn't tell what Node version to use), and finally the zip folder containing a folder of files instead of just the files (release.zip/build/... vs. release.zip/...).

I think the main issue/confusion is that the sample yml provided attempts to copy over the directory contents after npm has already installed all the dependencies. I would suggest that it's updated to not do that.

Since I didn't have any unit tests to run, I no longer run npm install|build|test and skip straight to the zip & deploy. Those with tests could probably just run > rm node_modules or something so it's not zipped & copied.

For those using React my final deployment yml looked like below:

# Docs for the Azure Web Apps Deploy action: https://github.com/Azure/webapps-deploy
# More GitHub Actions for Azure: https://github.com/Azure/actions

name: build & deploy

on:
  push:
    branches:
      - master
  workflow_dispatch:

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v2

      - name: Set up Node.js version
        uses: actions/setup-node@v1
        with:
          node-version: '14.x'

      # Zip artifacts to speed things up
      - name: Zip artifact for deployment
        run: zip -r release.zip *

      - name: Upload artifact for deployment job
        uses: actions/upload-artifact@v2
        with:
          name: node-app
          path: release.zip

  deploy:
    runs-on: ubuntu-latest
    needs: build
    environment:
      name: 'Production'
      url: ${{ steps.deploy-to-webapp.outputs.webapp-url }}

    steps:
      - name: Download artifact from build job
        uses: actions/download-artifact@v2
        with:
          name: node-app

      - name: 'Deploy to Azure Web App'
        id: deploy-to-webapp
        uses: azure/webapps-deploy@v2
        with:
          app-name: 'my-appservice-name'
          slot-name: 'Production'
          publish-profile: ${{ secrets.AZUREAPPSERVICE_PUBLISHPROFILE_XXX }}
          package: release.zip

brugner commented 2 years ago

Any update on this issue? I stopped the process after more than 3hs. I'm using the default file created by Azure Portal.

rabollin commented 2 years ago

Hi - the defualt YML file created by azure portal builds file by file which will take longer time based on the no. of files. Can you try with the yml here : https://github.com/Azure/webapps-deploy#create-azure-web-app-and-deploy-using-github-actions

sven5 commented 2 years ago

This is really sort of annoying. We're deploying a Gatsby static App together with an express backend. The package is about 1 GB (500MB static files and 500MB node_modules). Deployment takes about 20 minutes on Linux App Service. 😢

The only way I can think of is using custom docker images. However, I'd like to avoid this as it hugely adds complexity.

rabollin commented 2 years ago

This is really sort of annoying. We're deploying a Gatsby static App together with an express backend. The package is about 1 GB (500MB static files and 500MB node_modules). Deployment takes about 20 minutes on Linux App Service. 😢

The only way I can think of is using custom docker images. However, I'd like to avoid this as it hugely adds complexity.

Can you provide your GITHUB workflow file used for deployment? If you have to many files then the recommendation is to zip and deploy that will expedite the process.

sven5 commented 2 years ago

@rabollin I'm using Azure Pipelines and I'm already using zip and ignoring unused files using .artifactignore file.

Here we have a part of my build pipeline script.

          - script: |
              yarn install --frozen-lockfile
              yarn build
            displayName: 'npm install and build for $(slotName)'
          - task: CopyFiles@2
            inputs:
              SourceFolder: $(System.DefaultWorkingDirectory)
              Contents: |
                *
                node_modules/**
                public/**
                routers/**
              TargetFolder: '$(Build.ArtifactStagingDirectory)'
          - publish: '$(Build.ArtifactStagingDirectory)'
            displayName: Publish artifacts
            artifact: WebApp

Keep in mind that the publish step already creates a Zip file.

The build runs in about 5 mins. But the AzureWebApp@1 task for deployment is really slow. See a snippet of my deploy pipeline:

          deploy:
            steps:
            - checkout: none
            - download: current
              artifact: WebApp
            - task: AzureWebApp@1
              inputs:
                azureSubscription: $(azureSubscription)
                appType: 'webAppLinux'
                appName: $(appName)
                package: '$(Pipeline.Workspace)/WebApp'
                runtimeStack: 'NODE|14-lts'
                startupCommand: 'node server.js'
                deployToSlotOrASE: false
                deploymentMethod: runFromPackage

github-actions[bot] commented 2 years ago

This issue is idle because it has been open for 14 days with no activity.

Azure / webapps-deploy