search

Azure / app-service-announcements-discussions

Discussions for announcements from https://github.com/Azure/app-service-announcements/
MIT License
44 stars 5 forks source link

[Discussion] Run From Package: a new way to deploy your Azure Web Apps and Function Apps (in Preview) #32

Closed davidebbo closed 4 years ago

davidebbo commented 6 years ago

Please use this issue to discuss https://github.com/Azure/app-service-announcements/issues/84

davidebbo commented 6 years ago

@PaitoAnderson any chance you can share a zip that repros the issue? Thanks!

PaitoAnderson commented 6 years ago

@davidebbo here you go! I have a feeling it's the RuntimeIdentifier of win7-x86

Source: https://github.com/AppNouveau/repro-run-from-zip

Zip: https://github.com/AppNouveau/repro-run-from-zip/releases/tag/v1.0

App Service: (Edit: Deleted) https://run-from-zip.azurewebsites.net/

davidebbo commented 6 years ago

@PaitoAnderson Thanks! I'll talk to the experts and we'll try to figure out what's going on.

nphmuller commented 6 years ago

Today our ASP.NET Core 2.0 web app suddenly started only responding with 502 The specified CGI application encountered an error and the server terminated the process.. I suspect the source of the error was the Run-from-Zip feature for the following reasons:

What is the recommended way for me to provide more details?

nphmuller commented 6 years ago

From Log Stream:

2018-03-07T14:32:57 sandboxproc.exe complete successfully. Elapsed = 2913.00 ms
[2018/3/7 14:32:58]: Successfully downloaded zip file.
-- This is normal output of my application starting up
2018-03-07 14:33:05.301 +00:00 [Information] AWSSDK: Using AWS credentials specified with the AWSOptions.Credentials property
2018-03-07 14:33:05.356 +00:00 [Information] AWSSDK: Using AWS credentials specified with the AWSOptions.Credentials property
2018-03-07 14:33:08.030 +00:00 [Information] Microsoft.EntityFrameworkCore.Infrastructure: Entity Framework Core 2.0.1-rtm-125 initialized 'MyContext' using provider 'Microsoft.EntityFrameworkCore.SqlServer' with options: None
-- The error
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"> 
<html xmlns="http://www.w3.org/1999/xhtml"> 
<head> 
<title>IIS Detailed Error - 502.3 - Bad Gateway</title> 
....
</html>
-- And my application logs continue here, so it starts succesfully.
2018-03-07 14:33:08.241 +00:00 [Information] AWSSDK: Using AWS credentials specified with the AWSOptions.Credentials property
2018-03-07 14:33:08.596 +00:00 [Information] My.Library.DataServiceInitialization.Initializers.DatabaseInitializer: Initializing databases.

Update: Managed to get an exception. Not sure if it's related, but it happens right at the start, which is why Log Stream didn't capture it before (because it hadn't connected yet).

2018-03-07T14:38:29      Error occurred, type: error, text: Invalid remote feed url: https://www.nuget.org/api/v2/, stackTrace:    at Kudu.Core.SiteExtensions.FeedExtensions.<GetResourceAndValidateAsync>d__6`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Kudu.Core.SiteExtensions.FeedExtensions.<GetLatestPackageByIdFromSrcRepo>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Kudu.Core.SiteExtensions.SiteExtensionManager.<TryCheckLocalPackageLatestVersionFromRemote>d__44.MoveNext()

The only extension that is installed is ASP.NET Core Extensions version 2.0.1. When I click Site extensions in Kudu the exception above also appears. Again: not sure if its related:

Failed to retrieve site extensions from Gallery - An error has occurred. - Kudu.Contracts.SiteExtensions.InvalidEndpointException - Invalid remote feed url: https://www.nuget.org/api/v2/

Update 2: Might be related after all! The InvalidEndpointException disappears when I remove the WEBSITE_USE_ZIP appsetting. And reappears when I add it again.

davidebbo commented 6 years ago

@PaitoAnderson @nphmuller well, this is embarrassing, but the reason .NET Core apps don't work is because something got regressed. :( The team is looking into what happened here.

The good news is that the issue does not affect the alternate workflow given in the announcement (where the zip gets uploaded to the site files instead of using an App Setting pointing to it).

So for now, please try switching to that.

This feature is still in early preview, so there are rough spots, and clearly we need to improve our processes to prevent such regressions. Apologies for the inconvenience!

nphmuller commented 6 years ago

@davidebbo Well, that explains it. :)

Because the feature is in preview we just enabled it for our test environment. So it's not that big a deal, luckily. Don't really want to implement the alternative flow in our ci/cd environment though. So we'll just disable the appsetting flow for now until the regression has been fixed.

markheath commented 6 years ago

I'm seeing rather temperamental performance with a .NET Core 2.0 site and SitePackages. I used this technique with a Cake task to perform the deploy.

The cake task has a bug which means it reports an error even though it worked ok. I can see that the zip is uploaded successfully to SitePackages and the contents of siteversion.txt are updated correctly.

But even though KuduClientZipRunFromDirectoryVersion.txt shows the correct version in my wwwroot, I still see the old version of my ASP.NET Core site. Even 10 minutes later it still shows the old version. But after restarting the web app the updated version of my site is available.

This is repeatable - has happened on all the updated versions of the zip I've pushed so far.

davidebbo commented 6 years ago

@markheath to help isolate, do you see the same behavior if you manually update the zip and siteversion.txt (e.g. via Drag and Drop in Kudu console)? This would completely remove the Cake task from the equation.

Also, make sure you use a different zip name for each deployment. e.g. putting the date/time in the file name is a good practice.

markheath commented 6 years ago

Yes, they all have different names based on timestamp. I did try updating siteversion.txt, but didn't seem to help. Unfortunately I've literally just torn down my resource group (wasn't expecting such a quick response!), but will experiment a bit more and report back next time I work on this.

devlead commented 6 years ago

The Cake addin will invariant name zip $"{DateTime.UtcNow:yyyyMMdd_HHmmss}_{Guid.NewGuid():N}.zip"

Bug is currently an validation step after deployment that won't work with a .NET Core web, I'll address this in next release. Could be .NET Core related as I obviously hadn't tested that fully 😎

markheath commented 6 years ago

I've had a chance to try this again. It seems the first deploy against a brand new web app works fine, but problems begin when I try to update to a new version. Some fairly lengthy notes on what I tried...

D:\home\data\SitePackages>ls 
20180326_192907_3c4753c2605d47c1ac5ad5bbae9e192a.zip 
siteversion.txt 
D:\home\data\SitePackages>cat siteversion.txt 
20180326_192907_3c4753c2605d47c1ac5ad5bbae9e192a.zip

This is "v2" of my site. I then make a "v3" deployment:

D:\home\data\SitePackages>ls 
20180326_192907_3c4753c2605d47c1ac5ad5bbae9e192a.zip 
20180326_193825_119504df44834982bc734228f7b3f128.zip 
siteversion.txt 
D:\home\data\SitePackages>cat siteversion.txt 
20180326_193825_119504df44834982bc734228f7b3f128.zip

And it appears to have been picked up:

D:\home\site\wwwroot>cat KuduClientZipRunFromDirectoryVersion.txt 
20180326_193825_119504df44834982bc734228f7b3f128.zip

And yet v2 of my site was still visible. So I downloaded both zips from Kudu, renaming them to v2.zip and v3.zip and uploaded them via drag and drop in the kudu web console. And then I edited siteversion.txt to have the contents v2.zip

Strangely, KuduClientZipRunFromDirectoryVersion.txt gets updated, but not to v2.zip. Instead to the filename of the other zip that happens to have the same contents.

D:\home>cat D:\home\site\wwwroot\kuduclientziprunfromdirectoryversion.txt
20180326_192907_3c4753c2605d47c1ac5ad5bbae9e192a.zip

And then if I edit siteversion.txt in the Kudu web console to have the contents v3.zip and immediately the contents of kuduclientziprunfromdirectoryversion.txt change again but to the filename of the other zip:

D:\home>cat D:\home\site\wwwroot\kuduclientziprunfromdirectoryversion.txt
20180326_193825_119504df44834982bc734228f7b3f128.zip

However, v2 of my site is still showing. I changed back and forth a few times and only twice out of multiple modifications to siteversion.txt did I see the deployed version of the site actually change.

As a final test I set siteversion.txt to v4.zip which doesn't exist. Immediately this took effect:

D:\home>cat D:\home\site\wwwroot\kuduclientziprunfromdirectoryversion.txt
cat
: 'D:\home\site\wwwroot\kuduclientziprunfromdirectoryversion.txt': No medium found

and when I visit the site

The page cannot be displayed because an internal server error has occurred.

finally I updated site.txt back to v3.zip and now my site returned, correctly showing v3 of my website as expected. I was still getting the filename of the other zip (which has the same contents) in kuduclientziprunfromdirectoryversion.txt

D:\home>cat D:\home\site\wwwroot\kuduclientziprunfromdirectoryversion.txt
20180326_193825_119504df44834982bc734228f7b3f128.zip

So it appears to me that changing siteversion.txt directly from one version to another is not reliable, but if you change it via an intermediate invalid version it does work.

Final observation - it seems that changing from v3.zip to v2.zip does seem to work and take effect quickly, but going the other way is where I have the real problems.

Hope this is useful. I'm around for the next hour or so to try other experiments if that helps.

davidebbo commented 6 years ago

@markheath Hmmm, this does sound like quirky behavior. If you have cycles to test a little bit more, there is actually some news: we have the new version of this features deployed to West Central US (rest will follow later). The changes are:

You can get to be the first one to try it there! And the feedback will be very useful to us. Thanks!

markheath commented 6 years ago

OK cool, my site is in westeurope but will give it quick try in west central us

davidebbo commented 6 years ago

@markheath Thanks! Please also share your West Central US test App name if you run into any issues, so we can better investigate.

markheath commented 6 years ago

Can't reproduce my original scenario as West Central US doesn't support ACI yet, but I've created an app named acicake2 and can't get run from zip to work at all now just manually uploading zip files in kudu and editing siteversion.txt.

D:\home\data\SitePackages>ls
siteversion.txt
v2.zip
v3.zip

D:\home\data\SitePackages>cat siteversion.txt
v3.zip

but not getting picked up

D:\home\data\SitePackages>cat D:\home\site\wwwroot\kuduclientziprunfromdirectoryversion.txt
cat
: 'D:\home\site\wwwroot\kuduclientziprunfromdirectoryversion.txt': No such file or directory

I've also set up an app setting called WEBSITE_USE_ZIP with value 1

Can't see anything obvious I'm missing.

davidebbo commented 6 years ago

@markheath I think the problem is that it struggles to transition from not having a valid siteversion.txt to having one. i.e. if you first turn on WEBSITE_USE_ZIP=1 and don't yet have siteversion.txt, it gets into a state where it doesn't monitor the appearance of siteversion.txt. I restarted your site and it seems fine now. I tried going back and force v3->v2->v3 and all seems fine. Please give it shot now!

markheath commented 6 years ago

Great, yes it seems to be able to switch forwards and backwards just fine now. I was creating the WEBSITE_USE_ZIP=1 app setting as part of provisioning my app service before doing the first kudu deploy to site packages.

Let me know when this version is available in westeurope and I can re-test my end to end scenario with Cake and the kudu deploy api (the actual thing I'm trying to test is the gitRepo volume mounting of ACI, so it's a bit of a pain to replicate my test scenario locally)

davidebbo commented 6 years ago

@markheath should normally be everywhere by end of next week. We'll need to fix the initial experience in a future update so it doesn't get into that strange state.

devlead commented 6 years ago

@markheath Post deployment validation is now optional and configurable in just released Cake.Kudu.Client 0.5.0, read more in issue cake-contrib/Cake.Kudu.Client#1

tmasternak commented 6 years ago

I've tried this out 2 weeks ago and got it working without any problems (I'm using WEBSITE_USE_ZIP=SAS_URI). I've tried it one more time today and it failed. I'm hosting my function in North Europe and what I get at the client calling the function is HTTP 500.

Looking into kudu logs and host logs in particular I can see:

2018-03-28T20:53:26.489 [Error] The remote name could not be resolved: ''

2018-03-28T20:53:48.695 [Verbose] Host 'a90972140dbe2ddc70a86a29fa962c16' failed to acquire host lock lease: One or more errors occurred. ... 2018-03-28T20:54:14.049 [Verbose] Host 'a90972140dbe2ddc70a86a29fa962c16' failed to acquire host lock lease: One or more errors occurred.

In addition when I try to open functions logs in kudu (via WebJobs dashobard) I get error message:

The configuration is not properly set for the Microsoft Azure WebJobs Dashboard. ...

Removing WEBSITE_USE_ZIP=SAS_URI and using MsDeploy resolves this.

I'm pretty sure this kind of configuration worked when I tried it previously but I might be missing something.

davidebbo commented 6 years ago

@tmasternak Yep, there is an issue with the WEBSITE_USE_ZIP=SAS_URI workflow. It is being fixed, and the fix is already in the West Central US region. It should propagate to the rest of the world within a week or two. If you have cycles to help validate that it all works for you in West Central US, that would also be appreciated :)

tmasternak commented 6 years ago

@davidebbo I've tried in West Central US and it worked without any problem :).

davidebbo commented 6 years ago

Note that the issue with the WEBSITE_USE_ZIP=SAS_URI workflow is now resolved. If you ran into this earlier, please give it a try again.

xumx commented 6 years ago

I tried deploying a NodeJS app.

The server didn't start after changing WEBSITE_USE_ZIP.

Log stream shows Successfully downloaded zip file.

in a loop. 

[2018/4/30 12:38:57]: Successfully downloaded zip file.
[2018/4/30 12:39:0]: Successfully downloaded zip file.
[2018/4/30 12:39:4]: Successfully downloaded zip file.
[2018/4/30 12:39:8]: Successfully downloaded zip file.
[2018/4/30 12:39:11]: Successfully downloaded zip file.
[2018/4/30 12:39:15]: Successfully downloaded zip file.
[2018/4/30 12:39:19]: Successfully downloaded zip file.
[2018/4/30 12:39:22]: Successfully downloaded zip file.
[2018/4/30 12:39:26]: Successfully downloaded zip file.
[2018/4/30 12:39:30]: Successfully downloaded zip file.
[2018/4/30 12:39:33]: Successfully downloaded zip file.
[2018/4/30 12:39:37]: Successfully downloaded zip file.
[2018/4/30 12:39:41]: Successfully downloaded zip file.

Any idea what is causing this? @davidebbo

davidebbo commented 6 years ago

@xumx can you share the Web App name (either directly or indirectly)? We will investigate.

xumx commented 6 years ago

It was contactone.azurewebsites.net I had to remove the WEBSITE_USE_ZIP for now because it was downloading in loops. @davidebbo

davidebbo commented 6 years ago

@xumx any chance you can create a new test site that has the zip setting? We will need this to properly investigate. You can email me directly at david.ebbo (at) microsoft.com if you like.

davidebbo commented 6 years ago

@xumx: would be really great to get a repro from you, in case there is a bug at the root.

davidebbo commented 6 years ago

Everyone, please note the recent renames:

The old names will still work fine for a while to help transition, but please switch over asap.

robdmoore commented 6 years ago

Any news on what the thinking is with app insights siteextension and this feature?

davidebbo commented 6 years ago

@robdmoore Yep. We added support of an alternate WebJobs folder under D:\home\site\jobs (as opposed to the standard D:\home\site\wwwroot\App_Data\jobs). In fact, now if you add a WebJob from Portal on an app that uses Run-From-Zip, it will automatically go there.

And the App Insights team has already changed their extension to use the new folder (starting with v2.5.4).

Bottom line: this should be a solved problem now!

robdmoore commented 6 years ago

<3

jsheetzati commented 6 years ago

I have the new app setting and removed WEBSITE_USE_ZIP. Also renamed siteversion.txt -> packagename.txt. However, after doing a zip deploy, it recreated the siteversion.txt file.

davidebbo commented 6 years ago

@jsheetzati that's a bug, we'll fix it!

davidebbo commented 6 years ago

@jsheetzati It is now fixed in Kudu. Can you try again? Restart your app to be sure you get the fix.

jsheetzati commented 6 years ago

Yep, looking good now. Thanks @davidebbo!

davidebbo commented 6 years ago

Please see https://github.com/Azure/app-service-announcements/issues/110 for related announcement.

sopelt commented 6 years ago

Hey @davidebbo , do you happen to have an idea about a timeline when Run-From-Zip will be production ready/generally available? Thank you.

davidebbo commented 6 years ago

@sopelt I would expect July or August.

mattduffield commented 6 years ago

Hi @davidebbo, I am using VS Code on a mac and it appears to deploy fine. I have node_modules and package.json as .gitignore settings so they are picked up when I commit. I have noticed that now that I am trying to use the app setting WEBSITE_RUN_FROM_ZIP, my function complains that it cannot find the dependency that it tries to require. What configuration or setting do I need to have in place for VS Code to include the dependencies necessary?

davidebbo commented 6 years ago

@mattduffield when using Run-From-Zip, all the files that you need to have at runtime must be in the zip. Not a VS Code expert, so can't comment on how you'd build the zip from there. But in the end, it's just a plain old zip file with all your files.

mattduffield commented 6 years ago

@davidebbo I tturned out that I needed to update my package-lock.json file by running the command npm install. Once I did that and re-deployed, everything started working! This is such a great improvement over all the manual Webpack bundler setup! Thanks!

davidebbo commented 6 years ago

@mattduffield Thanks for the feedback. Glad to hear it is working well!

mmdriley commented 6 years ago

Is Linux support for this feature on the roadmap?

davidebbo commented 6 years ago

@mmdriley yes, Linux is on the roadmap, though we don't have an ETA to share quite yet,

skanwat commented 6 years ago

Great feature!

mirandazh commented 6 years ago

Hey @davidebbo do you know whether it is available in Azure for US Government?

davidebbo commented 6 years ago

Yes, it is available in US Gov cloud as well.

jsheetzati commented 6 years ago

@davidebbo Is there a way to enable stdout logging to troubleshoot an ASP.Net Core api that isn't starting? Previously, I would follow the directions in this guide. With run-from-zip, I can't really edit the web.config to set stdoutLogEnabled=true. Any recommendations?