Yarn deployments for node apps

[mamaso]

Saw this today, basically facebook, google and others have worked on a package manager for node: https://code.facebook.com/posts/1840075619545360

Did some testing to install dependencies for the ‘Azure Mobile Apps Quickstart’ package on a web app with yarn vs npm3.

Yarn: 42.78s Npm3: 70.2s

Incorporating this into the kudu deployment script is something to consider as it could improve deployment speed for nodejs projects by close to 50%.

[davidebbo]

Yes, worth considering. First step would be to get it pre installed on the machine. That would allow users to write custom scripts based on it.

Then we can get fancier and make our default deploy scripts use it.

[koistya]

At Travis CI, for example, they have Yarn pre-installed and when you build a Node.js project containing yarn.lock file in the root of your project's source tree, it will run yarn install --production instead of npm install --production. It would be great to have the same setup in Kudu and Azure App Service.

Ref heroku/heroku-buildpack-nodejs/issues/337

[Yitzchok]

It would be really helpful to have yarnpkg installed like npm is.

[colltoaction]

Here's someone doing it manually: https://github.com/stefangordon/kudu-yarn

Would be great to have this built-in!

[davidebbo]

'built-in' could mean two things:

1. having yarn pre-installed on the VM
2. in addition, have the script generator automatically use it. I guess based on presence of yarn.lock?

Is the perf gain on Azure App Service that noticeable vs the latest npm?

[colltoaction]

For the first part, having it pre-installed works on my case. I have a csproj that calls npm install, so I could just change it to yarn if it was there.

Regarding performance, I haven't tested myself, but this (maybe old already?) blog post shows some good numbers favoring yarn: https://medium.freecodecamp.com/npm-vs-yarn-benchmark-9b456de4aa96

But aside from that, one of yarn's ability is to restore the exact package versions you need given a yarn.lock file. That build reproducibility would be great for people that deploy through Git.

[davidebbo]

I see. I wish npm could add support for the same so we don't have to deal with both.

My concern is that it wouldn't be good enough to just add one version of yarn. Users would then expect it to support many different versions as it evolves, with a selection mechanism. And that starts to add long term maintenance pain.

[colltoaction]

Totally understandable. Keep us updated if you decide on anything!

[cchamberlain]

@davidebbo I used yarn for a bit in one of our Azure app custom deployment scripts and it made the client build much faster (~4.5 minutes for yarn and webpack when dependencies don't change). When dependencies have not changed the yarn install time is within a second, the only problem is that the global yarn install gets wiped out about every third deploy (estimated) and reinstalling it tacks on time for the global install and voids the cache (and all benefits of yarn) - cc @tinchou.

Having yarn@latest preinstalled on the slot seems like a relatively small step on the road to auto-detection, and would cut down our build time by a large margin. It seems like the canonical approach to version detection would be to rely on the package.json engines section for the yarn version. See https://github.com/yarnpkg/yarn/issues/522

[cchamberlain]

@davidebbo now that npm 5 is out, we'd see some relief on this issue in regards to lockfile support if it were available on kudu - http://blog.npmjs.org/post/161081169345/v500

Looks like its got better caching to it as well.

[davidebbo]

npm 5 is now deployed to the West US 2 region (only there for now). @cchamberlain Would be great if you have the cycle to give it a spin there on a test site.

[lostintangent]

@davidebbo I just tried creating a new App Service Plan/Web App in the West US 2 region, and the default NPM installation is 3.10.8. Is there something else I need to do in order to test out the new NPM 5 deployment? I'd love to test out the potential improvements from that being available.

[davidebbo]

@lostintangent you can set a WEBSITE_NPM_DEFAULT_VERSION appsetting (see wiki).

[lostintangent]

@davidebbo Awesome, that worked!

[cchamberlain]

@davidebbo awesome thanks for getting this in so quick. Just got an email from npm stating 5 is the new standard so great timing 👏.

Any chance on getting it deployed to North Central US? Happy to do lots of testing over there.

[davidebbo]

@cchamberlain You're in luck as it just made it to North Central US as a second region. Also, we threw in Node 8 for good form since it's now the official 'Current' release: https://nodejs.org/

[davidebbo]

Note that this is now deployed everywhere. I'll close this issue, as hopefully npm 5 provides enough of the yarn benefits that we don't need yarn. I'd like to avoid having to maintain to Node package managers side by side.

[morsh]

@davidebbo - from personal experience, npm 5 doesn't supply a consistent version tracking as yarn

[xt0rted]

Yarn 1.0 was released today and with its feature set is becoming more compelling than NPM. Having this pre-installed in App Services would be nice.

[ahmelsayed]

you can always just get yarn through npm though.

[xt0rted]

My deployments already take around 10 minutes each because of downloading npm packages. I hate adding more because my builds just take longer and longer.

[ghost]

Btw, yarn seems to be installed by default in the Node.js images (at least for 8.x) in Azure Web Apps for Containers, which I think are coming out of preview these days. Perhaps that's an option?

[ahmelsayed]

@xt0rted it's a balancing act. These VMs have finite disk space and finite image size. we can't just throw every tool on there. we use yarn, I use yarn, but I firmly believe it's more appropriate to pull it down using npm than to expect it to be installed on the VMs. same thing with bower, gulp, angular-cli, react-cli, mocha, etc. if you want you can install it globally once in your deployment script

npm config set prefix "D:\home\tools"
npm i -g yarn
set path=%path%;D:\home\tools
yarn --version
:: or directly 
D:\home\tools\yarn --version

yarn will always be in that tools folder since your D:\home is persistent.

and as @sheepcount mentioned, containers provide more proper support for these scenarios where we don't have to litter every single VM with all these tools, but rather have container images for specific tools for people who needed. Unfortunately that doesn't apply to Windows apps though :/

[benmccallum]

I thought it interesting that the yarn website doesn't clearly indicate npm i -g yarn as an approach, rather points to installers. Sure enough, someone found out why: https://stackoverflow.com/a/43377805/725626

Yarn does seem a hell of a lot faster and offers lots of guarantees. Most devs now list their install process with yarn cmd first. It'd certainly be taking over as time progresses. But I can understand these VMs only have limited space... Tricky.

[pure180]

Unfortunately npm i -g yarn doesn't work on Azure Linux App Services. I made a quick and dirty work around to use yarn with Kudu and a custom deployment shell script.

For everyone who is interested, this is my approach deploying a react application :

#!/bin/bash

# ----------------------
# KUDU Deployment Script
# ----------------------
echo "Starting the KUDU deployment script"

# Helpers
# -------

exitWithMessageOnError () {
  if [ ! $? -eq 0 ]; then
    echo "An error has occurred during web site deployment."
    echo $1
    exit 1
  fi
}

# Prerequisites
# -------------

# Verify node.js installed
hash node 2>/dev/null
exitWithMessageOnError "Missing node.js executable, please install node.js, if already installed make sure it can be reached from current environment."

# Setup
# -----

SCRIPT_DIR="${BASH_SOURCE[0]%\\*}"
SCRIPT_DIR="${SCRIPT_DIR%/*}"
ARTIFACTS=$SCRIPT_DIR/../artifacts
KUDU_SYNC_CMD=${KUDU_SYNC_CMD//\"}

DEPLOYMENT_TOOLS=/home/site/deployments/tools
YARN_DIR=$DEPLOYMENT_TOOLS/yarn
TMP_DIR=/tmp

if [[ ! -n "$DEPLOYMENT_SOURCE" ]]; then
  DEPLOYMENT_SOURCE=$SCRIPT_DIR
fi

if [[ ! -n "$NEXT_MANIFEST_PATH" ]]; then
  NEXT_MANIFEST_PATH=$ARTIFACTS/manifest

  if [[ ! -n "$PREVIOUS_MANIFEST_PATH" ]]; then
    PREVIOUS_MANIFEST_PATH=$NEXT_MANIFEST_PATH
  fi
fi

if [[ ! -n "$DEPLOYMENT_TARGET" ]]; then
  DEPLOYMENT_TARGET=$ARTIFACTS/wwwroot
else
  KUDU_SERVICE=true
fi

if [[ ! -n "$KUDU_SYNC_CMD" ]]; then
  # Install kudu sync
  echo Installing Kudu Sync
  npm install kudusync -g --silent
  exitWithMessageOnError "Installing Kudu-Sync with npm failed"

  if [[ ! -n "$KUDU_SERVICE" ]]; then
    # In case we are running locally this is the correct location of kuduSync
    KUDU_SYNC_CMD=kuduSync
  else
    # In case we are running on kudu service this is the correct location of kuduSync
    KUDU_SYNC_CMD=$APPDATA/npm/node_modules/kuduSync/bin/kuduSync
  fi
fi

# Install Yarn
# ------------

echo "Installing yarn"
if [[ ! -e "$YARN_DIR" ]]; then
  cd "$TMP_DIR"
  eval wget https://github.com/yarnpkg/yarn/releases/download/v1.16.0/yarn-v1.16.0.tar.gz -O yarn.tar.gz
  exitWithMessageOnError "Downloading Yarn failed"

  eval tar -xzf yarn.tar.gz -C "$DEPLOYMENT_TOOLS"
  exitWithMessageOnError "Extracting archive failed"
  cd - > /dev/null

  cd "$DEPLOYMENT_TOOLS"
  eval ln -s yarn-v1.16.0/bin/yarn yarn
  exitWithMessageOnError "Symlink failed"
  cd - > /dev/null
fi

YARN_VERSION="$(yarn -v)"

if [[ -e "$DEPLOYMENT_TOOLS/yarn" ]]; then
  echo "Found Yarn v$YARN_VERSION"
else
  exitWithMessageOnError "Couldn't find Yarn"
fi

# Node Helpers
# ------------

selectNodeVersion () {
  if [[ -n "$KUDU_SELECT_NODE_VERSION_CMD" ]]; then
    SELECT_NODE_VERSION="$KUDU_SELECT_NODE_VERSION_CMD \"$DEPLOYMENT_SOURCE\" \"$DEPLOYMENT_TARGET\" \"$DEPLOYMENT_TEMP\""
    eval $SELECT_NODE_VERSION
    exitWithMessageOnError "select node version failed"

    if [[ -e "$DEPLOYMENT_TEMP/__nodeVersion.tmp" ]]; then
      NODE_EXE=`cat "$DEPLOYMENT_TEMP/__nodeVersion.tmp"`
      exitWithMessageOnError "getting node version failed"
    fi

    if [[ -e "$DEPLOYMENT_TEMP/.tmp" ]]; then
      NPM_JS_PATH=`cat "$DEPLOYMENT_TEMP/__npmVersion.tmp"`
      exitWithMessageOnError "getting npm version failed"
    fi

    if [[ ! -n "$NODE_EXE" ]]; then
      NODE_EXE=node
    fi

    NPM_CMD=yarn
  else
    NPM_CMD=yarn
    NODE_EXE=node
  fi
}

##################################################################################################################################
# Deployment
# ----------

echo "Handling node.js deployment, running on $DEPLOYMENT_SOURCE/deploy.sh"

# 1. Select NodeJs version 
echo "1. Select NodeJs Version."
selectNodeVersion
echo "Use $NPM_CMD to resolve packages"

# 2. Install Packages using Yarn
echo "2. Install $NPM_CMD packages"
if [ -e "$DEPLOYMENT_SOURCE/package.json" ]; then
  cd "$DEPLOYMENT_SOURCE"
  eval "$NPM_CMD" install --network-timeout 1000000 --ignore-engines
  exitWithMessageOnError "Installing npm packages failed"
  echo "Finished installing npm packages"
  cd - > /dev/null
fi

# 3. Build React App
echo "3. Build React App"
if [ -e "$DEPLOYMENT_SOURCE/node_modules" ]; then 
  cd "$DEPLOYMENT_SOURCE"
  eval "$NPM_CMD" build
  exitWithMessageOnError "Build failed!"
  echo "Build finished"
  cd - > /dev/null
fi

# 4. KuduSync
echo "4. KuduSync to '$DEPLOYMENT_TARGET'"
if [ -e "$DEPLOYMENT_SOURCE/build" ]; then
 cd "$DEPLOYMENT_SOURCE"
 eval "$KUDU_SYNC_CMD" -v 100 -f "$DEPLOYMENT_SOURCE/build" -t "$DEPLOYMENT_TARGET" -n "$NEXT_MANIFEST_PATH" -p "$PREVIOUS_MANIFEST_PATH"
 exitWithMessageOnError "Kudu Sync failed!"
 cd - > /dev/null
fi

# ##################################################################################################################################

# Post deployment stub
if [[ -n "$POST_DEPLOYMENT_ACTION" ]]; then
  POST_DEPLOYMENT_ACTION=${POST_DEPLOYMENT_ACTION//\"}
  cd "${POST_DEPLOYMENT_ACTION_DIR%\\*}"
  "$POST_DEPLOYMENT_ACTION"
   exitWithMessageOnError "post deployment action failed"
fi

echo "Finished successfully."