[Discussion] Run From Package: a new way to deploy your Azure Web Apps and Function Apps (in Preview)

[davidebbo]

Please use this issue to discuss https://github.com/Azure/app-service-announcements/issues/84

[Suchiman]

Sounds great!

• Whats the performance like?
  • Are reads always deflating or is it cached in locally?
  • How does this affect local cache?
• Is it compatible with all technologies (.NET Core, .NET Framework, etc.)?
• What happens when you point it to a new zip file?
  • The announcement says "its atomic", is it smooth like slot swap, or
  • Is the zip downloaded, app halted, zip remounted, app restarted?
• Is the transition to the new zip file synchronous or asynchronous?
  • If async, is there a way to detect when the transition to an updated version of the zip file is completed? (e.g., when slot swapping, the powershell cmdlets return, but requests may hit the old page for about another 30 seconds which is unfortunate for automation)

[davidebbo]

@Suchiman great questions:

• It does some local caching
• It shouldn't be used with the Local Cache feature (probably conflicts, untested)
• It should in theory work with all stacks
• The App Setting change will restart the app. So it is atomic in the sense that you can't be 'half-baked', but it is not smooth like slot. That's a potential future improvement, at least for apps that have multiple instances
• It's the same as modifying an Azure App Setting, which is async.

[shibayan]

It's cool feature.

• Is there a way to notify that deployment is complete?
  • Similarly, when the deployment fails?
• Is there a limit on the size of zip file?
• If I change App Setting while asynchronously deploying, will the deployment abort?
• Just an idea, How about the feature to deploy from a local machine use Azure CLI?
  • Transparently upload zip to storage and deploy it to App Service

[anthonychu]

Does it re-pull the zip file from the specified location on restart or scale out? Just wondering if we need to worry about the SAS URL expiring.

[devlead]

@davidebbo Will/is there be an option in the kudu API to do this kind of deploy this? Would be nice if similar to zip deploy something like post {"uri": "=https://davidebbostorage.blob.core.windows.net/content/.....D"} to /api/RunFromZip

to get a 202 or 200 if

1. Request received
2. Can access uri
3. Settings changed
4. Zip downloaded and mounted

[martinpeck]

This is a great feature! Thank you!

Is there some guidance for how this changes what should be deployed for your app? For example, I'm assuming that the zip file should include the node_modules folder, right?

[securityvoid]

Is there any way to set the appropriate environmental variable from a visualstudio.com build? If so, this should be able to be automated now since you can already zip up the source.

[securityvoid]

Alternatively, can this be used with git deploy? e.g. Zip file is returned to git, auto-deploys to environment; ZIP variable points to the file path where the Zip sits in the environment?

[davidebbo]

@shibayan Deploying completing is basically equivalent to the site restarting after changing an App Setting. If something goes wrong during zip download, you'll see a file called "FAILED TO DOWNLOAD ZIP FILE.txt" in wwwroot from Kudu with error details.

There is no strict size limit, but it does use the local storage, so in Free/Shared/Consumption really large zips could hit the limit.

The tooling part like Azure CLI is something that would come later and build on top of it.

[davidebbo]

@anthonychu Yes, it does repull the zip each time. So:

• Make the SAS valid as long as you want the site to work
• Use a storage account in the same region

[davidebbo]

@devlead Note that the zip is not uploaded a part of deployment, but instead that happens at runtime. There is no deployment per se, other than setting the App Setting.

Potentially, Kudu could build the zip from a source project, and do the upload. Though that does raise some questions about it needing write access to the blob storage.

[davidebbo]

@martinpeck Yes, it should absolutely include the node_module folder, and basically everything your app needs at runtime. Think of it this way: if you were to deploy via good ol' FTP, you'd copy a bunch of files. That exact same bunch of files is what you want in your zip.

[davidebbo]

@securityvoid similar to previous comments. Basically, right now we just have the low level intrinsic that knows how to handle the App Setting pointing to a zip. There can be all kind of deployment workflows built on top of that to create the zip, host it and set the App Setting. I don't know VSTS well, but presumably it's flexible enough that you could be this today with enough custom build logic.

[securityvoid]

@davidebbo Thank-you for the response. I'm going to play around with this some. As long as the variable supports pointing to the local file-path there are a lot of work flows that might work here. This should be a great feature!

FYI, in case someone is looking how to set an environment variable from VSTS: https://marketplace.visualstudio.com/items?itemName=pascalnaber.PascalNaber-Xpirit-WebAppConfiguration

[tony-gutierrez]

How would this zip contain node_modules that were built to be compatible with the environment? If I am on a mac and build a node module that uses gyp or is anything other than simple js, and deploy that zip, there is no way it is going to be compatible.

[devlead]

Note that he zip is not uploaded a part of deployment

@davidebbo I get that, but just setting the setting doesn't guarantee the web app can access the zip file. If an api could provide at least some minimal validation, that would give more confidence in the deployment.

And epic if an api could "like" the zip deploy api wait and give a 2xx when web is up or error if failed, that would give immediate feedback to fail or retry from CI.

[devlead]

How would this zip contain node_modules that were built to be compatible with the environment?

@tony-gutierrez Personally I would build and deploy from CI which would be using a Windows agent.

[davidebbo]

@tony-gutierrez Yes, that's true. This issue also exists today if you use FTP or msdeploy. The only case where it doesn't exist is when doing Kudu git deployment, since that has a build step that restores packages.

There is also the possibility of solving this via a CI server like VSTS. In such world

• your sources don't have the packages
• VSTS restores them at build time and has knowledge about the target environment
• It zips that up and all is well, without your client having to worry about it

So it may be that 'raw' Run-From-Zip' is a little too low level in some scenarios if used by itself. But the right tooling can make it work correctly (but really, this is no different from msdeploy/FTP today).

[davidebbo]

@devlead yes, I think this is something that can be built on top of the low level intrinsics. In fact, you could easily write a command line tool that takes the zip, uploads it to blob, sets the URL, and hits the Web App to make sure everything works.

[Suchiman]

@davidebbo but how does hitting the web app make sure everything works? Since its async, it sounds like it would need a Thread.Sleep(30000) to approximiate when the changes took effect.

[davidebbo]

@Suchiman is is no different from changing an App Setting in App Service today. You have FOO=1 and change it to FOO=2, and it can take 10 or 20 seconds before the change is live. The very same applies here.

[Suchiman]

@davidebbo Not really the answer i hoped for but i guess i can put a version.txt into the zip to workaround it and just poll it every second in the release process to block until the new release is live.

[securityvoid]

@Suchiman Why not simply return the environmental variable that points to the zip? If its the new value, you know the new release is in place. Then you don't need anything specifically in the release.

[Suchiman]

@securityvoid mh interesting trick, cool yeah that should work, thanks 👍

[davidebbo]

@securityvoid @Suchiman one small twist is that if you have multiple instances, they will not necessarily all big ready at the exact same time. So you might still want to wait a few seconds even if one instance tells you it's ready.

[dfinke]

The deploy worked. I can seen the function in the function apps blade. When I click on the function, get the error below. I deployed a PowerShell run.ps1

[davidebbo]

@dfinke Yep, that's a known issue in the Functions Runtime (not in this new feature) that just got fixed. Until the fix is deployed, just make sure to include a host.json at the root of your zip. It should just contain {}

[dfinke]

@davidebbo Thanks, yup, I just included it, was coming back to delete my comment and saw your response. This is pretty sweet and will simply function app DevOps.

[devlead]

@davidebbo Am I right to assume as this is an appsetting, you can't deploy via the Kudu api api/settings, but you must do it through the management APIs?

[joeizy]

If App_Data is read only, will the App Insights extension work properly? I thought the profiler webjob used that folder.

[davidebbo]

@devlead Indeed. There is another way to set it up which doesn't use App Settings, and where the zip file is instead dropped inside some special app folder. I have not included it in the announcement to keep things simpler, but if there is interest I can provide details. At this point, we're still exploring the best approach.

@joeizy Right, any scenario that involves writing to anywhere under wwwroot will not work. We will need to think through what that means to App Insights. Maybe we need a secondary WebJobs folder that doesn't live under wwwroot at all. That would also solve some longstanding deployment issues where the AppInsights WebJobs gets deleted or causes locking issues.

[devlead]

@davidebbo I'm interested😎 but kinda liked the idea of using a storage uri, as with the app setting, would zip file dropped in special folder you mean the full archive?

I've got a PoC zipping site, pushing to blob storage, creating sas uri and using management API to set app setting working on CI, but as many of our current deployments are kudu zip deploy this means a bit more configuration, i.e. publishing profile credentials isn't enough. If we could supply zip storage uri via kudu API or placing a "text" file with it somewhere on the app service, then it would be trivial to migrate from zip/MSDeploy/ftp/etc. to this.

[davidebbo]

@devlead see also what @anthonychu came up with for VSTS: https://anthonychu.ca/post/vsts-deploy-app-service-azure-functions-run-from-zip/.

It's awesome to see you guys help come up with deployment solutions!

I added the alternative approach below the Quick Start in the main issue.

[joeizy]

@davidebbo A separate folder is one option. I can see a world where people will also want to deploy the webjobs from zip, but you can’t have everything in v1 :)

On another topic, you mention that you’re considering an option to read from a local zip file and use that. Working with very security sensitive clients leads me to recommend that you do have some option available that doesn’t require sensitive info to be persisted in an App Setting.

The clients I work with would absolutely not allow the SAS (in the case of blog storage) to be kept in an App Setting. They would require it be in Key Vault. There are interesting options to consider such as integrate with MSI and then read the secret from Key Vault URI but that’s probably beyond the scope of v1 as well.

My point being, please do have an option for this use case. A zip file in the local drive would totally work and might even be preferable if it results in a faster boot time because it is local.

[joeizy]

I also noticed that in the benefits of this feature it mentions how it can help with validation of code for debugging and environment management but not how it could help meet compliance requirements. Some standards require verifying code in prod is what you intend it to be. I think PCI is one that requires it. I look at this as a big selling point to be advertised.

Docker makes this super easy with images because they’re read only (and you can sign them). They advertise the heck out it.

If you started to support hash verification that would be a plus too.

[davidebbo]

@joeizy I'm a bit confused by your reference to a 'local drive' here. See the Alternative Proposal in the announcement for a solution that involves dropping the zip file in the site files on App Service (or maybe that's what you mean by 'local'?).

For the App Setting, if you use Azure Functions, it already uses that to store storage connection strings, so this new feature changes nothing in this respect. Note that App Settings are secure and encrypted at rest. In any case, Key Vault support is a separate discussion, and is something that will be supported in the future (tracked here).

Yes, I agree that some optional code verification feature could be interesting on top of this feature (in the blob SAS URL case). Good feedback!

[devlead]

@davidebbo so updating is just a matter of uploading zip and changing siteversion.txt? And rollback should just be to set siteversion.txtto its previous value?

[markheath]

I like the idea of this, although I'd be a bit nervous about using a long-lived SAS token to the zip in production apps. Has potential to break the app randomly in the future if you cycled the storage account key. So having the SitePackages folder option is a helpful alternative.

I presume this rules out using the Kudu WebJob APIs to do things like upload new webjobs or even change the settings (e.g. cron expression) of a webjob.

Also, what happens if an app deployed this way does try to write to App_Data? Will it be denied write access?

[davidebbo]

so updating is just a matter of uploading zip and changing siteversion.txt? And rollback should just be to set siteversion.txtto its previous value?

@devlead Yes, exactly.

[davidebbo]

@markheath In the future, with KeyVault integration, the SAS secret rotation issue should be addressed (based on this). This would be a general new App Service integration and not specific to this feature.

Indeed, you generally cannot use any publishing API once in that mode, because the entire wwwroot becomes read-only. Everything you need should go in the zip file, including WebJobs.

Any attempts to write to App_Data will fail for the same reason. Generally, App_Data is an anti-pattern inherited from the old ASP.NET days, and there are better ways to solve this.

[joeizy]

@davidebbo thanks for all your interaction to everyone on this thread. This feature is a simple but AHA moment that just makes sense! "Why wasn't it always this way?" kinda thing...

I'm a bit confused by your reference to a 'local drive' here. See the Alternative Proposal in the announcement for a solution that involves dropping the zip file in the site files on App Service (or maybe that's what you mean by 'local'?).

Yes, that's what I mean 👍. Poor use of the word 'local' - my bad. Please make sure this (or something like it) gets into the prod release.

For the App Setting, if you use Azure Functions, it already uses that to store storage connection strings, so this new feature changes nothing in this respect.

Right, functions are approved via an 'exception process'. (aka. We acknowledge it doesn't meet the standards but we approve it anyway because there's no other way for now.) Looking forward to integration with Key Vault as you linked above.

Note that App Settings are secure and encrypted at rest. In any case, Key Vault support is a separate discussion, and is something that will be supported in the future (tracked here).

Right, makes sense. It's a question of 'What is the most secure way to store secret info?' The answer to that is what they want (and in many cases are required by law) to use.

[devlead]

@davidebbo so been playing around with this deploying using AppVeyor and Cake build scripts and it works really well, deployments are significantly faster, especially with static websites where I've found most of the time during deployment was spent kudu syncing files.

Did a quick blog for anyone interested using Cake scripts to deploy using "Run-From-Zip" and the just the Kudu api. https://medium.com/@devlead/run-from-zip-with-cake-kudu-client-5c063cd72b37

Will try this on a few projects just to see how it behaves and it'll be interesting to follow how the experience and tooling evolves around this.

[ScottHolden]

This is awesome!! 👍 Combing this with RA-GRS storage and Traffic Manager should lead to some really nice Active-Active HA designs 😃 I'm guessing this replaces the 'WEBSITE_CONTENTAZUREFILECONNECTIONSTRING' app setting for consumption plans? Or will we still need it for now?

[nphmuller]

In my current VSTS deployment, I deploy my Node app with the build-in Azure App Service Deploy task. Within this task I check the generate web.config option with the recommended parameters for node (-Handler iisnode -NodeStartFile main.js -appType node).

What is the recommended way to set these options with the Run From Zip feature? Can't I just set these options through one or more app settings?

[devlead]

@nphmuller Just zip the web.config a long with the other page artifacts and it'll be picked up by the app service. You can find an example here: https://github.com/projectkudu/kudu/wiki/Using-a-custom-web.config-for-Node-apps

[nphmuller]

@devlead We create the zip during the build and generate the web.config (by a build-in vsts task) during the deployment, so while it’s possible, it’s not ideal.

I’d also prefer the just keep my node app in the zip and keep server configuration like a web.config somewhere else.

Setting an appsetting would be ideal, but I guess one doesn’t exist for this purpose, right?

Edit: I also keep my app configuration/settings (connection strings, api keys, log levels, etc) out of the zip and set them through the app settings. A server configuration option seems in line with that.

[devlead]

@nphmuller how much does your web.config differ between environments? Mine are usually the same so just append the web.config to the zip file before pushing it to blob storage / app service. All environment unique settings we store in KeyVault or AppSettings.

[davidebbo]

@devlead Thanks for blogging, this is great!

@ScottHolden: great question. The answer is that right now, you still need Azure Files. Even though it won't be used for the wwwroot folder, there are other files that need to be in there (in particular, Function secrets). But I think we can get to a place where Azure Files is no longer necessary.

@nphmuller We're starting to chat with VSTS team, so hopefully we'll have some first class support for this. For now, the web.config (and every file you need to run) must be in the zip.

[nphmuller]

@devlead Yeah, it’s pretty much the same. Guess it’s just that it feels wrong to have deployment specific settings at that place. Especially since it didn’t need to be there before. But I’ll just add the file during the build for now. Hope it’s a temporary solution though.

Thanks for the advice @devlead & @davidebbo ! :)

EDIT Deployment for our node app went from 10 minutes to about 1 minute, per environment. Yay!

[PaitoAnderson]

I tried this out today with a Windows App Service and as soon as I entered the WEBSITE_USE_ZIP Application Setting it started throwing this error:

System.IO.IOException: Failed to bind to address http://localhost:11401.
 ---> System.AggregateException: One or more errors occurred.
 ---> System.BadImageFormatException: An attempt was made to load a program with an incorrect format. (Exception from HRESULT: 0x8007000B)
at Microsoft.AspNetCore.Server.Kestrel.Transport.Libuv.Internal.Networking.LibuvFunctions.NativeMethods.uv_loop_size()

I did test the SAS url and I could download the ZIP just fine that was uploaded from AppVeyor.