Introduce a "reset to empty" API endpoint

[atrauzzi]

As per my comment here: https://github.com/projectkudu/kudu/issues/2103#issuecomment-287624520

Basically, I'd really like an endpoint that I could call via the API that would reset the filesystem of the app to a new-state. The reasoning behind this is a bit layered as I've been encountering several snags in trying to make my deployments consistent. Much of it stemming from the (lack of) speed during filesystem access.

The premise is simple: This endpoint would take a POST and would quickly reset the wwwroot directory to a state like as if I had just created a fresh new App Service instance.

[davidebbo]

So you're only talking about resetting wwwroot, but not any of the files outside of it (e.g. logfiles, repository, ...)?

Note that this is easy to do without Kudu, e.g. by relying on webdeploy to deploy a simple index.html, and tell it to delete any unknown files. The wawsdeploy tool (built on WebDeploy), makes this even easier from the command line.

[atrauzzi]

Unfortunately I can't and really in the end don't want to use webdeploy. I'm running my deployment from a linux container, so this means I need to be able to get at Azure via standard means.
The idea for what I'm trying to do is to avoid the vendor lock in and workarounds that keep cropping up in trying to target Azure. So I'll put the usual boilerplate I seem to have to do here:

• I don't use git for deployment
• My development environment is not Windows-centric
• I'm deploying a nodejs application
• I need Azure Functions, so I can't use the Linux App Service

(I apologize for the directness, it's just very frustrating when every time I report an issue, I'm directed to either use VSTS, git deployment or Windows-only tooling. I'm told frequently that Azure is platform/tooling agnostic, but it's turning out to be very hard to put that into practice.)

While I don't mind abstracting certain operations that may be specific to Azure, the real crux here is that the interactions I have with the system either have to be through CURL, FTP or SSH.

(right now CURL wins because the FTP story is a slightly worse nightmare and I'm not aware of any SSH option for Windows-based App Service instances)

[atrauzzi]

For context, here's a link to the deployment script I'm using: https://github.com/atrauzzi/anecdote-azure/blob/master/bin/deploy-functions#L43-L49

I've highlighted the interesting bits, but I think at a glance this should give you an idea of what I'm trying to do.

[davidebbo]

The file system is slow, so if you have a huge tree of npm modules, deleting can take a while. Are you sure that the actual deletion is timeout, as opposed to just the REST API? i.e. if you run a single delete command, and wait even though the client timed out, does it eventually delete everything?

For FTP, you can get the hostname and creds simply by downloading the publishing xml, so not much of a nightmare. :) That being said, FTP deletion will be slow as well, per my previous paragraph (and probably slower since FTP is very inefficient). But authenticating is easily solved.

[atrauzzi]

It's likely the request timing out, but how can I know at that point when the delete is finished?

I can't upload files until that's done or else I end up deleting new files. I had the idea earlier to move the wwwroot directory and then create a new one, but that directory stays locked, even with my app stopped.

Seems like I'm cut off at every pass. Ideally the deletes would just be faster, but in lieu of that, I've got nothing else to fall back on.

[davidebbo]

If the delete is completing, then you're in good shape. You can just sleep/poll for directory presence using Kudu vfs API.

Closing here, since I don't think any new Kudu API will help. It would be subject to same 230 second timeout (which is not Kudu related).

[atrauzzi]

Are there any plans to make the filesystem faster in app service?

Moreover, how do all the other tools (like webdeploy) acheieve a faster outcome? It seems disingenuous to think that the same speeds can't be afforded to non-proprietary deployment techniques.

[atrauzzi]

I also don't think polling should be the solution here as that's quite heavy handed. Surely there is a better option than a second rate experience just because I want to use standard tools.

[davidebbo]

Make sure you set SCM_COMMAND_IDLE_TIMEOUT to a large value. On Azure, the client will still disconnect after 230 seconds, but the command will continue afterwards.

To your other points/question

• Unlikely that the file system will get faster, as it's storage based.
• I would not expect msdeploy to be faster. Why did you think it would be?

[atrauzzi]

Yet another esoteric Azure gotcha. Alright, I apologize, but I have to air some concerns. Before getting into them though, I want to really emphasize that I've I've been encouraged by the level of engagement from the Azure teams and the level of help I get - but this is just one boulder I need moved yesterday.

I'm feeling dismayed at the hours I've wasted and the difficulty I'm encountering trying to do something as simple as getting my files from a stable and repeatable build environment, into an App Service instance. I appreciate your suggestions, but none of this is unblocking me. I can set the timeout, and I'll hit it again, so I'm not sure what this is intended to get me.

The reason why I assumed msdeploy is faster is because it's the first thing you suggested and you mentioned that all I'd need to do is give it an index.html file.

But honestly, this isn't even talking about the same thing anymore. What I want to do here really shouldn't be this hard. Based on the searching around I've done, I'm also not the only person to suffer the slow speeds and very opinionated deployment flows either. Having succeeded with Heroku, AWS, Google Cloud and even my own Rancher farm in the past, the deployment story on Azure App Service really firmly brings up the rear.

As a customer consuming the service, the reason is pretty obvious: The path to getting a single file called functions.zip and expanding it in a clean directory on the Azure remote is something all the other platforms make a complete snap.

With Azure however, I'm told in order:

• Use git for deployment - Not an option as I don't use source control for deployment and getting that working from a clean and repeatable build environment requires a lot of unnecessary ceremony
• Use VSTS - Not an option because I'm using github and Travis CI
• Use a Windows-only tool - I'm not on Windows
• Download an XML file - I'm not sure why it's considered acceptable to force people designing automation flows to start hacking around in XML files
• Write a loop in my automation script to manually poll a path via CURL - At this point, clocks are starting to melt off the walls and I'm kind of just getting ready to let go

I've had the same signals from multiple teams on Azure and I gotta be honest - there is without a doubt a very urgent need for a review of the ease of use or "developer joy" when doing Kudu/App Service deployments. Where competing services have the same need, they've simply offered it through their CLI. No, seriously.

With Azure? The only thing I haven't tried yet at this point is a hunting dog and a ouija board.

This is supposed to be the era of Microsoft embracing all platforms but when I make my best efforts to take those assurances at face value, the only flows really getting attention are ones that promote vendor lock-in. When I try to communicate frustration, I'm funneled over to places like uservoice where the signal-to-noise is just right enough to disperse the most important suggestions.

I'm not going to get too much more abstract here, but this is how people get frustrated. There are just too many gotchas, secret flags, limitations to this entire orchestration.

Let's take this from the top: I know at least well enough to get to the point where I have:

• Prepared my infrastructure via resource templates
• Produced a file called functions.zip in a linux container
• Can authenticate to the Kudu API

I want to:

• Clean all existing files
• Extract functions.zip somewhere in my Azure App Service filesystem
• Not resort to goofy deployment kludges

Anyway, sorry for the wall of text, this is just getting too crazy.

🖋

[davidebbo]

I would suggest moving the question to StackOverflow (or MSDN). The problem here is that this is the issue tracker for Kudu, which is only a small piece of Azure App Service. So any discussion about alternate technology is out of scope.

I think the way you phrase it above is mostly what your SO question should look like. High level, your question is: On App Service, what's the best way of deploying new content from a zip file, such that it replaces any existing content? With additional twist that you can't use msdeploy on the client.

Please post link to question here for continuity in case someone finds this Kudu issue and has similar needs.

[atrauzzi]

I'm very skeptical about whether I'll get anything useful out of either.

Also, as a sidenote: My attempts to upload to kudu using the files API also seem to time out pretty regularly now, so this went from salvageable to "I can't trust this".

I'll start with SO, and in the meantime, I'm going to start comparing Google Cloud Functions and AWS Lambda. I'm really at my walking away point for the amount of time I've wasted chasing this.

Feel free to suggest improvements to the SO question.

[davidebbo]

Some more things I would add to the question:

• are you set up to call the Azure ARM API? I assume yes.
• are you able to host the zip package in the cloud before deployment (e.g. in a storage blob)?

[atrauzzi]

Added! Definitely angles to keep in mind. And yeah, I have the whole Azure CLI 2.0 at my disposal in this. That's how I get some of the credentials. That script I linked above is literally what gets run.

[davidebbo]

There is a possible solution that I think will work, but I need to research it. If it does work, I'll post to SO.

[atrauzzi]

That sounds great. I think when considering the flow I'm trying to establish, it's not overly complex.

The big gotcha is that I don't think a holistic analysis of what the App Service experience looks like from these inevitable angles has been done. Too often, it seems like Visual Studio and VSTS run off with the show and prematurely resolve the discussion. As a result, we have this hyper-optimized deployment pipeline that's been created to work around what I really think are things the unix world has been far more adept at streamlining in the past.

I don't think it's unreasonable to think that less-is-more when it comes to deployments and that there is a valid need for some internal championing of such. 😉

[davidebbo]

I agree that your question is perfectly reasonable.

[davidebbo]

Ok, I posted initial answer. Not quite there but close.

[atrauzzi]

@davidebbo - Brilliant, this appears to have worked! I commented at Stack Overflow, but just in case anyone lands here via search as well:

This is my build that successfully pushed code up to Azure.

Here are links to my project showing how I invoked everything:

• Upload my deployment image to storage
• Attempt a deployment Azure-side using the hosted msbuild tools

Other parts of that shell script are sure to be interesting to some as well. And of course, it's easily adapted to PowerShell if that's your preference.

[davidebbo]

Glad to hear. Thanks for sharing result!