Updating deployed app

[ramijarrar]

Has this been implemented/tested at all? At the moment it looks like it is just pushing the new files to wwwroot without consideration of what is already there.

[ramijarrar]

Had a look into the source for the 'deploy' command as well as the docs for the Kudu API and as I suspected the current method of deployment does not delete existing files before extracting the bundle into wwwroot.

[christopheranderson]

The current behavior just adds new content. We don't clear content or smartly upload new files. I think the clear content option could be added easily. Implementing a smart sync feature is pretty hefty and is likely to be bug prone. I'd need to see some hefty deployment times/issues before I'd pursue that with fervor.

[ramijarrar]

Yeah, that is all I am really proposing here - the clear content option. Right now I have to manually clear the directory before deployment so this addition would allow me to fully automate the delivery pipeline.

[christopheranderson]

So I'm thinking a new command clean which will remove .demeteorized locally and/or, if you provide your remote credentials, will clean out wwwroot. There's an open question of if I should keep app_data by default or not.

[ramijarrar]

Sounds good, but I don't think we would want to clear the .demeteorized directory as the idea was to run this before deployment (git clean handles this really well anyways)

[ramijarrar]

As for the app_data directory - since it's not relevant for Meteor apps I don't think we should worry about preserving it (at least not by default)

EDIT: Looks like `app_data`` is also used for WebJobs, so in retrospect I do think it's important not to wipe them if it exists.

[christopheranderson]

I'm the PM for WebJobs in Azure, so I'm always sensitive to just wiping away app_data. :)

Yeah, maybe that over complicates it. It should just be an option in deploy. --clean.

[ramijarrar]

Yeah, an option in deploy is definitely the way to go (unless you think of a valid use case for running this separately)

[ramijarrar]

Any ETA on this? Should I bother with my own fill-in solution?

[christopheranderson]

I'll probably get to this tomorrow/Friday night. Had some high priority stuff the last few days. I'll probably use the command api: https://github.com/projectkudu/kudu/wiki/REST-API#command

[ramijarrar]

:+1:

[christopheranderson]

So this has been a fun issue. Going to leave it open, for now, but it turns out fibers.node pretty much always gives off a Permission error because it is owned by w3wp. I've tried automatically killing the process via the processes api, but still getting problems.

The work around is to stop the site (i.e. via the portal).

Automating this requires migrating to using the azure APIs instead of the kudu APIs.

[ramijarrar]

It's easy to use the Azure CLI to stop the site, so the clean option would still be useful here without that feature.

[ramijarrar]

Another question is, I usually use the Kudu CMD to clear everything in wwwroot before uploading and I haven't needed to stop the process before. Does the Kudu API do something different here? It seems like when we push new code the app automatically restarts and just deleting the files on Kudu would do? You have more experience with this than I do, but I am just wondering if there would be a way to get this to work even if an manner that is less than ideal.

[christopheranderson]

It's a relatively unique issue I've seen with native node modules and node on Azure. There are hacks like putting a file watch inside the node app to look for sentinel and gracefully shutdown and stay down.

Real answer is to use the Azure APIs to turn off the site. Going to update the documentation with the work around for now. I'll need to spend some time migrating to the Azure APIs, but once I've done that, this and #8 should be do-able.

I've got my current work checked in, but I don't think it will be the right approach going forward. https://github.com/christopheranderson/azure-demeteorizer/tree/feature-clean-option

[ramijarrar]

This is still really helpful :)

[ramijarrar]

Btw - have you thought of interfacing with the Azure CLI directly (i.e just like we do with demeteorizer)?

[christopheranderson]

Yeah, haven't made up my mind yet. On one hand, might be less things to learn. On the other, it's another dependency I'm asking folks to install. I won't be able to get to this until after Wednesday, so I've got some time to think about it.

[ramijarrar]

Also, can we merge the --clean support and maybe get rid of the kill processes part which isn't working.

[ramijarrar]

And regarding the Azure CLI, I'm not sure you would have to ask users to install it is a dependency given its availability on npm. Authentication can be done through an external pem file, which the user can point to in the deploy command.

[christopheranderson]

The kill processes thing works, just not effectively enough. The cleanwwwroot piece is also only necessary if you need a totally fresh start - zip deploy will overwrite things for you. Cleanwwwroot takes FOREVER because of how many node_modules Meteor brings in.

I'm not going to merge this in yet because I think I'm going to end up redoing it with pure Azure APIs. You can use this command to install for now: npm install christopheranderson/azure-demeteorizer#b82d100 - it's not super stable; you've been warned. :)

I would definitely use the PEM file, regardless of using the Azure SDK or the Azure CLI directly.

[ramijarrar]

Sure, though the issue with just overwriting files is that it becomes really messy with the intersection of changes to a large app and not something you would feel comfortable doing for applications that people are depending on (it is an Alpha release at the moment, so I digress). Though, I guess if you are referencing the right files directly this won't be a big problem. In all cases, it certainly it isn't clean.

On a separate note, I think this addition along with #8 are currently the only major gaps in terms of a full feature set (just looking at other CLIs available today such as modulus) which you can start to publicize on Meteor channels.

I'll check in again later this week.

[christopheranderson]

Yeah, #9, #8, and #11 are all that I'm planning on doing, then I'll pop a version and move to beta. At that point, I'll write a doc for azure.com and publish a blog.

Before GA, I'm hoping to support CI & Linux/Mac - but it'll take some backend changes to Azure, which I'm working on.

[ramijarrar]

That's awesome! At the moment we are running through a dedicated build server running Windows (our only one :wink:) and we would love to be able to depreciate this in favor of built-in support. I also think you will get a really good response from other developers, since right now Azure provides by far the most powerful node PAAS compatible with Meteor.

[christopheranderson]

:+1: Glad you enjoy the Node experience. If you ever have any issues/feature requests for node in general, be sure to check out kudu, which is Azure Web App's backend management service.

[ramijarrar]

@christopheranderson Would be really great if you could post a quick comment regarding timelines for these changes. No hurry/urgency, just want to plan accordingly.

Thanks!

[christopheranderson]

Been slammed lately, so haven't gotten to this. Spent some time this evening and I think I have a plan mapped out to do this (plus some other goodness :)). Going to use the azure-cli to do a lot of the actions.

[ramijarrar]

I needed to go live with a more serious Meteor app that I was working on and ended up solving this by skipping the zip/deploy commands and writing my own script to deploy using Git. It's working well for me so far with no more issues during updates and significantly reduced build times (the zip command was very slow) as well as history in the portal.

Having something like this integrated in core could be a great compromise for the time-being (doesn't require interfacing with the azure-cli).

[avalanche1]

@ramijarrar could you share the script?

[ramijarrar]

@avalanche1

Looks something like this:

azure-demeteorizer build
azure-demeteorizer install
cd .demeteorized/bundle
git init
git add --all
git commit -m "%COMMIT_MESSAGE% (%CURRENT_BRANCH%)" --quiet
git push "https://%AZURE_GIT_USER%:%AZURE_GIT_PASS%@%AZURE_GIT_URL%" master --force --quiet

I initialize a repository in the bundle directory, then force push that to the azure git URL with a label that contains the current commit message and branch. If you are running this on a CI server, make sure to discard the output of the push so you don't unintentionally leak your credentials in the build logs. Also make sure you do a git clean on the original repo after you're done to get rid of the generated files.

If you have issues with long file-names, try setting this option as well:

git config --system core.longpaths true

[christopheranderson]

I went with FTP because it's not good practice to commit binaries, which you have to do in this case.

btw, sorry for being kinda absent here. I was working on Azure Functions so had my heads down over there.

[ramijarrar]

@christopheranderson That's fine, I just needed something to work in the immediate term. A few questions, now that you're back:

1) What do you think of moving to meteor build over demeteorizer which seems to be reaching its end-of-life?

2) Do you think it would be possible to build for the os.windows.x86_32 architecture and then compile on the Azure servers with a custom deployment script (since they already have Visual C++ and Python installed)

3) If we end up implementing #2 would moving to git make sense then?

[christopheranderson]

I wish we could do no.2, but npm install on 0.10.x doesn't seem to work with VC++ 15, which is the only version of VC++ that App Service can support (I can go into details on this, if necessary, but suffice it to say I've tried to get 12 and 10 working without success). That's pretty much the whole reason I built this tool to be a client side tool instead of server side.

meteor build is called by demeteorizer. I maybe out of the loop, what's wrong with demeteorizer? If I rewrote against just plain meteor, I'd basically be rewriting demeteorizer. :)

One thing I might look into is using VSTS to enable the git flow. VSTS doesn't have as restrictive of a sandbox as App Service (we should be glad App Service is restrictive - externally addressable servers should be picky). If you use VSTS, then install could run as part of the server flow.

I'm going to be sitting down with the team and thinking through our Node experience. I don't think I can "beta" this tool until git flow works. It's another reason this tool has kind of sat dormant - there are some other changes needed at the platform level to make meteor deployment to Azure as easy as I want it. This tool merely makes it possible, which isn't sufficient.

[ramijarrar]

@christopheranderson It may be worth mentioning that upgrading Node.js is the first priority for the next Meteor release (see here) so that might actually be the easiest way to move this forward. Any idea on timeline for changes to the underlying platform?

[christopheranderson]

Oh wow. Well if they are going to support 4, then we can probably just relax for now. The right way to solve this is letting the App Service platform do the install. If Meteor supports the 4.x LTS, then App Service install will just work. Then this tool will drastically change directions.

The dream is:

1. Git push meteor project to GitHub/VSTS/Web App
2. Via Custom deploy cmd
   1. Unpack site
   2. Install packages

The work required for this is:

1. Meteor supports 0.12.x or higher (4.2.x is awesome)
2. App Service pre-installs Meteor

As soon as Meteor supporting a high node version makes it to a pre-release, I'll get our dev team investigating adding it to App Service. Then I can replan this tool.

As it stands, I've been playing with adding more CLI stuff/etc. to this and for the most part, it's not really making anything less complicated. :( Our CLI and SDKs are hefty things. Maybe just expand the readme on how to use the xplat CLI for now.

[ramijarrar]

Sounds good. In the meantime, would it be possible to get clean updates working (assuming that the user turns off the app during deployments)?

[christopheranderson]

Sure. I've got that feature branch still open. Let me run some tests and I can merge with the warning that you need to turn off your site.

[ramijarrar]

Okay great. Upgrading to 1.3 broke the git-based method for me (I think it has something to do with long filenames: The data area passed to a system call is too small.\r\n) so I think it might be best to switch back to FTP if that works safely.

[christopheranderson]

That's why I chose FTP. It's cludgey, but it's reliable. Ultimately, I hope it is just a stop gap until we can get the App Service deploy install working.

[ramijarrar]

:+1:

I can manage with those changes for the time-being (we're looking to take a few apps to production over the coming weeks) and I don't think it will be a long wait for the next batch of beta releases to pop up.

BTW - would be awesome if you could test against 1.3

[ramijarrar]

Note: there is a 1.3.1 release out at the moment which addresses some issues with building on Windows. If that is a problem for you, just run meteor update --release 1.3.1

[christopheranderson]

I haven't been running into those problems.

[ramijarrar]

Cool, I think it's only an issue when Meteor tries to upgrade itself from a previous version and is unable to overwrite packages due to long filenames in node_modules (the recommended solution was just to delete the problematic packages manually).

[christopheranderson]

Could you give this a try before I merge it to master?

npm install -g christopheranderson/azure-demeteorizer#dev

[ramijarrar]

Yep, trying it atm

[ramijarrar]

Just to confirm, the only change should be: azure-demeteorizer deploy --clean

[christopheranderson]

Yep. Nothing else got touched besides the readme.

[ramijarrar]

@christopheranderson Ignore my earlier comments, at the moment I am getting this error while deploying with Meteor 1.3 :

[christopheranderson]

Yeah I was seeing something similar. I need to do some hacky powershell stuff to get rid of it.

[kgao]

@christopheranderson, How's this going? I notice this issue too. For a redeployment, it won't let you change the file, it's not because the kudu API PUT, neither your scripting. The 2 process running in azure web app - w3wp.exe actually lock the files, which blocks your for updating. (You can use kudu process explorer to watch this). So simply kill the processes - by stopping the server. You can deploy your new bundle files. After azure-demeteorizer deploy, just start your server from Azure portal again. Then you should see your updates.