ASP.NET Core 3 preview 7 - AspNetCoreModuleV2 not working on Windows Azure Pack

[tbprince]

Describe the bug

When publishing a self-contained application based on ASP.NET Core 3 preview 7 (I read that is supported in production since preview7) on a Windows Azure Pack WebsitesV2 account, application won't start when using module AspNetCoreModuleV2 which has been installed on server roles (we installed the latest preview7 bundle available).

The actual error is reported as: Handler "aspNetCore" has a bad module "AspNetCoreModuleV2" in its module list

However, switching to AspNetCoreModule (which had also been installed) by the WAP runtime, works fine. Do we need extra steps in WAP to enable module V2 ?

Not sure that using AspNetCoreModule instead of V2 is safe enough. We will surely miss some functionalities.

Thanks.

[jkotalik]

This usually means the AspNetCoreModuleV2 isn't installed on the target machine. I'm not familiar with the Windows Azure Pack WebsitesV2, @anurse ?

[analogrelay]

I'm not either but I'll make inquiries :).

@tbprince, as @jkotalik said, this usually means that the ASP.NET Core module isn't properly installed. This is a global installation that must be done on the Web Site base image. In Azure Web Apps (in the cloud) we work with the team to do this, but I don't know how/if that translates in Azure Pack. Do you have the ability to modify the base image used for your Web Site? If so, you could try installing the Windows Hosting Bundle (from https://dotnet.microsoft.com/download/dotnet-core/3.0) and that should install ANCM for you.

If you don't have the ability to affect the base image in Azure Pack, you may have to wait for Azure Pack's images to support ASP.NET Core 3.0.

[analogrelay]

Not sure that using AspNetCoreModule instead of V2 is safe enough. We will surely miss some functionalities.

You should be able to do this safely and run an ASP.NET Core 3.0 app. You're right that you'll miss out on some new functionality. Specifically you won't be able to run in the in-process mode added in ASP.NET Core 2.2. There are also some diagnostics and error reporting improvements in AspNetCoreModuleV2. However, the core scenarios should work find with AspNetCoreModule and ASP.NET Core 3.0.

We'll work with the Azure Pack team to see what the best way forward is here in the long-term.

[analogrelay]

Looks like Azure Pack V2 Web Sites are no longer supported: https://support.microsoft.com/en-us/lifecycle/search?alpha=Windows%20Azure%20Pack%20V2%20Web%20Sites

The docs also indicate this: https://docs.microsoft.com/en-us/previous-versions/azure/windows-server-azure-pack/dn457745(v=technet.10)

As a result, Azure Pack is not going to be updated to include this. If it's possible to install the Hosting Bundle on the VMs themselves, you'll be able to get it working but it won't be supported by Azure Pack natively.

Closing this issue.

[tbprince]

@anurse,

that's exactly what we wanted to do and indeed we installed the hosting bundle on our test servers. Please look at this image:

it seems to me that the module is installed since we manually did it:

Any hint ? Is there something else that we should install ?

Thanks.

[tbprince]

@anurse,

regarding ASP.NET Core Module v1, yes, it works fine. App runs when we do it through Module v1 but of course it would be better if we could just run it through v2.

Thanks again.

[analogrelay]

Any hint ? Is there something else that we should install ?

That does look like it would work. @jkotalik are there some places @tbprince could look on disk to confirm the installation is correct?

[jkotalik]

Check the global applicationhost.config file in C:\Windows\System32\inetsrv\config.

[tbprince]

What should I look for ?

Attaching the file

applicationHost.zip

This environment (WAP) should be similar to Azure App Service. At least a while ago it was.

[analogrelay]

I see the module installed there:

            <add name="AspNetCoreModuleV2" image="%ProgramFiles%\IIS\Asp.Net Core Module\V2\aspnetcorev2.dll" />

@tbprince can you check for this file and double-check the bitness? You can do that with dumpbin /headers [filepath] (which is available from the VS command prompt). We don't need the whole output, just the value for machine, which is right at the top. For example:

Dump of file ...\path\to\aspnetcorev2.dll

PE signature found

File Type: DLL

FILE HEADER VALUES
             14C machine (x86)

Where is the error being reported? The Windows Event Log? Is there any more detail in the error message you can provide?

[tbprince]

@anurse,

here you go:

`Microsoft (R) COFF/PE Dumper Version 14.00.23506.0 Copyright (C) Microsoft Corporation. All rights reserved.

Dump of file c:\dev\aspnetcorev2.dll

PE signature found

File Type: DLL

FILE HEADER VALUES 8664 machine (x64) 7 number of sections 5D2D53FB time date stamp Tue Jul 16 06:35:07 2019 0 file pointer to symbol table 0 number of symbols F0 size of optional header 2022 characteristics Executable Application can handle large (>2GB) addresses DLL `

We have failed trace logs active. The error is being reported there. Not sure which other details to report. Anything I can provide via Kudu ?

Running in Kudu seems to be working fine.

[tbprince]

I can tell you that our project is using Kestrel, not HTTP.sys since I know that is not compatible with Module v2. By the way, this is just a default ASP.NET Core app in Visual Studio 2019 Preview. Our init code seems to confirm that we are using Kestrel:

public static IHostBuilder CreateHostBuilder(string[] args) => Host.CreateDefaultBuilder(args) .ConfigureWebHostDefaults(webBuilder => { webBuilder.UseStartup<Startup>(); });

[tbprince]

Not sure this can help but this is the failed traced log that gets generated:

IIS Detailed Error - 500.21 - Internal Server Error.html.zip

[analogrelay]

The app configuration doesn't really affect this. The problem is that IIS can't load the aspnetcorev2.dll module on the machine running the website. Assuming the XML entry I referenced is present in the apphost.config on the app server and the DLL is physically present there and the bitness matches, I don't know why it wouldn't be loading.

[tbprince]

@anurse,

any log I could check for specific errors ? Since the v1 module seems to be working fine, generally speaking, there should be no problem with v2 either. Any runtime to install or any dependency ?

BTW, I'm sure you noticed but the host OS is Windows Server 2012 R2, which I believe should be ok.

Thanks.

[analogrelay]

The Windows Event Log is probably your best source for more information.

I'm going to ask a fairly obvious question just to make sure we haven't missed something: The Web Site is running directly on these machines, correct? I don't know if Azure Pack does anything with VMs or has multiple tiers (i.e. a front-end and web roles, etc.).

[tbprince]

@anurse,

yes, website is running on those machines. If you aren't familiar with WAP layout, it has frontends accepting connections and load balancing worker roles where websites will be executed inside a specific process. Websites are not directly configured in IIS though. There's a worker process, similar to Azure App Service. In facts, I believe that WAP and Azure App Service share the same heritage.

However, ASP.NET Core Module v1 is working fine so I don't believe there's anything for that.

Anyway, logs haven't been helpful. Nothing wrong there.

[analogrelay]

yes, website is running on those machines. If you aren't familiar with WAP layout, it has frontends accepting connections and load balancing worker roles where websites will be executed inside a specific process.

Ok, that's what I thought. I just wanted to make absolutely sure you were installing the module on the backend worker roles actually running the websites (which it sounds like you were, but it's always good to double check 😄)

Websites are not directly configured in IIS though. There's a worker process, similar to Azure App Service

That may be part of the problem. I just don't know what WAP is running to host the app. If they're in IIS, then the module needs to be installed into IIS (which is what our installers do). If they run in a special IIS process of some kind that uses a different set of module configurations then it might need to be configured there.

If you aren't seeing App Pools in IIS Manager then it would definitely strongly indicate that WAP uses some custom IIS hosting that may not be targeted by our Hosting Bundle.

In facts, I believe that WAP and Azure App Service share the same heritage.

Yeah, and we have a bunch of special stuff we do with the App Service team to deploy the module which why I think this might just be a limitation of WAP (which, since it's now out of support, isn't likely to change).

However, ASP.NET Core Module v1 is working fine so I don't believe there's anything for that.

I think sticking with ANCMv1 is probably your best bet for now. It should continue to work for ASP.NET Core 3.0 apps, you'll just be missing out on bug fixes and features (like In-Proc support).

Since WAP is out of support, I don't expect you'll see official support for ANCMv2 there. Migrating to Azure Stack (which is still supported I believe) or just plain Virtual/Physical Machines may be your best plan moving forward to get future releases running.

[tbprince]

@anurse,

first of all, thank you for your replies. I don't want to bother and I believe that sticking with Module v1 is our best option but do you mind to explain (even roughly) what you had to do with the App Service team ? Maybe we could use such info to perform changes for our deployment since those two environments are quite similar, at least in their first forms.

I believe that WAP and App Service skip configuring multiple app pools and websites and only run a single process that spawns other processes based on website configuration. So it might well be that the list of enabled modules is dynamically loaded somewhere using configuration and that's why we do have the module installed but that's not picked up by the "master" process. It might not be included into the configuration even when installed.

Thanks.

[analogrelay]

what you had to do with the App Service team

Mostly just installing the MSI on their base image. I'm not aware of specifics beyond that but I know we work closely with them to deploy. I also know that while WAP is based on some of that technology, it's not the same, so there definitely could be other things going on.

[tbprince]

There must be something else that we need to do because we installed the module on worker servers. Thanks anyway for looking into this.