Functions using custom extensions work in emulator but fail indexing when deployed (c#, v2.0.12050.0)

[stevedenman]

Following the recent update (2.0.12050.0) I made the change suggested here, using IWebJobsStartup to add extensions, however functions that use these extensions are not being indexed when deployed (works in emulator).

Investigative information

Please provide the following:

• Timestamp:
• Function App version (1.0 or 2.0-beta): 2.0-beta
• Function App name:
• Function name(s) (as appropriate):
• Invocation ID:
• Region: Australia East

Repro steps

Provide the steps required to reproduce the problem:

I deployed to a new function app with a custom extension registered in IWebJobsStartup, but no functions could be indexed;

... 2018-09-04T23:28:17.933 [Information] Host Status: { "id": "version-two-test", "state": "Running", "version": "2.0.12050.0", "versionDetails": "2.0.12050.0-beta1 Commit hash: 2db3b7e26f4c10b455afe385f66f67d1b4959b3a" } .... 2018-09-04T23:29:22.188 [Information] Starting JobHost 2018-09-04T23:29:22.188 [Information] Starting Host (HostId=version-two-test, InstanceId=8f722119-650d-430b-8e82-9cb9e447e180, Version=2.0.12050.0, ProcessId=5276, AppDomainId=1, Debug=True, FunctionsExtensionVersion=beta) 2018-09-04T23:29:22.230 [Information] Generating 1 job function(s) 2018-09-04T23:29:22.231 [Error] Error indexing method 'TestFunction.Run' 2018-09-04T23:29:22.277 [Warning] Function 'TestFunction.Run' failed indexing and will be disabled. 2018-09-04T23:29:22.277 [Warning] No job functions found. Try making your job classes and methods public. If you're using binding extensions (e.g. ServiceBus, Timers, etc.) make sure you've called the registration method for the extension(s) in your startup code (e.g. config.UseServiceBus(), config.UseTimers(), etc.). 2018-09-04T23:29:22.277 [Information] Host initialized (89ms) 2018-09-04T23:29:22.278 [Information] Host started (89ms) 2018-09-04T23:29:22.278 [Information] Job host started ...

Expected behavior

All functions are indexed.

Actual behavior

It looks like the custom extensions are not being loaded.

Known workarounds

Related information

Test project here; https://github.com/stevedenman/azure-functions-v2-di/tree/master/AzureFunctionV2DI

[mlankamp]

I'm having the same problems!

[mlankamp]

If you move your extension to a different project, and reference this project the problem is solved. The IWebJobStartup in your functions assembly is ignored in the published extensions.json. (you can verify this in obj/release/pubTmp/bin)

[stevedenman]

Great, thanks @mlankamp, that fixed it. So that's a workaround - who can escalate this so that IWebJobStartup implementations are detected in the main function app binary?

[djmeck]

This is not limited to custom extensions. I am encountering the same issue with the C# CosmosDB binding here: Microsoft.Azure.WebJobs.Extensions.CosmosDB. Unfortunately, there is nothing to move to another project so there is no workaround.

[dbarentine]

I've been running into this as well. The build generates the correct extensions.json in bin\Release\netstandard2.0\bin. So, I created a build target Directory.Build.targets file that can be dropped into the project directory and will copy the correct file at publishing time.

This will copy the correct extensions.json into the publish directory prior to actual deployment/zipping etc...

[hiraldesai]

Thanks @dbarentine for the workaround. Does anyone know if there a fix in the pipeline?

[maadhureave]

If you move your extension to a different project, and reference this project the problem is solved. The IWebJobStartup in your functions assembly is ignored in the published extensions.json. (you can verify this in obj/release/pubTmp/bin)

this worked for me but will this be fixed ?

[mohammedkamranazam]

I've been running into this as well. The build generates the correct extensions.json in bin\Release\netstandard2.0\bin. So, I created a build target Directory.Build.targets file that can be dropped into the project directory and will copy the correct file at publishing time.

This will copy the correct extensions.json into the publish directory prior to actual deployment/zipping etc...

Awesome. Worked like charm for me after 3 days of struggle in making the Function App run on the server. Was running smoothly in my local system, except on the server. Thanks again.

[rasmuschristensen]

Thanks @dbarentine Fixed my issue

[rikvandenberg]

@fabiocav Any updates or an ETA on this issue?

When developing and running locally in VS2017, the following target frameworks produce same output as specified by @stevedenman

[Error] Error indexing method .....

• netstandard2.0: ✔️
• netcoreapp2.0: ✔️
• netcoreapp2.1: ❌

[gaikovoi]

FYI. With .NET Core 2.1 target, build with the latest VS 2017 doesn't produce proper extensions.json (no "Startup" entry). At the same time, "dotnet build" executed from Ubuntu environment (on the same Windows 10 machine) against the same project folder produces proper extensions.json.

[EliPulsifer]

This fails for me in the emulator as well as when deployed targeting anything above .NET Core 2.0

Has the cause been identified?

[dgaynes]

OK, sorry I'm an AF noob, I apologize if this is not the same topic but it looks like it might be . . .

I have been attempting to implement this package, my function signature / DI:

([HttpTrigger(AuthorizationLevel.Function, "get", Route = null)] HttpRequest req, [Willezone.Azure.WebJobs.Extensions.DependencyInjection.Inject] ILogger logger, [Willezone.Azure.WebJobs.Extensions.DependencyInjection.Inject] DataLakeClientFactorySource dataLakeClientFactory)

works fine in my dev environment but when published is throwing a 500 error. This also works fine when published:

( [HttpTrigger(AuthorizationLevel.Function, "get", Route = null)] HttpRequest req, [Willezone.Azure.WebJobs.Extensions.DependencyInjection.Inject] ILogger logger)

So the code fails on line 3, the custom class injection (line 2 ILogger is MSFT).

Is this the same problem, failing to publish an extension file?

[ElliotWood]

Manually creating the two files in the function root fixed my binding issue, files here:

Directory.Build.targets

<Project>
  <Target Name="CopyExtensionsJson" AfterTargets="_GenerateFunctionsAndCopyContentFiles">
    <Message Importance="High" Text="Overwritting extensions.json file with one from build." />

    <Copy SourceFiles="$(PublishDir)extensions.json"
          DestinationFiles="$(PublishDir)bin\extensions.json"
          OverwriteReadOnlyFiles="true" />
  </Target>
</Project>

extensions.json

{
    "extensions":[
        { "name": "ServiceBus", "typeName":"Microsoft.Azure.WebJobs.ServiceBus.ServiceBusWebJobsStartup, Microsoft.Azure.WebJobs.ServiceBus, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"},
        { "name": "ActiveMQ", "typeName":"Microsoft.Azure.WebJobs.ActiveMQ.ActiveMQWebJobsStartup, ActiveMQ, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"}
    ]
  }

Also, don't forget to copy extensions.json to output dir

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>netcoreapp2.1</TargetFramework>
    <AzureFunctionsVersion>V2</AzureFunctionsVersion>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.Azure.WebJobs.Extensions.ServiceBus" Version="3.0.0" />
    <PackageReference Include="Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator" Version="1.0.1" />
    <PackageReference Include="Microsoft.NET.Sdk.Functions" Version="1.0.24" />
  </ItemGroup>
  <ItemGroup>
    <None Update="host.json">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
    </None>
    <None Include="extensions.json">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
    </None>
    <None Update="local.settings.json">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
      <CopyToPublishDirectory>Never</CopyToPublishDirectory>
    </None>
  </ItemGroup>
</Project>

Then run

func extensions install
func start host

Troubleshooing If you still see the folllowing error with non-custom bindings [yourpath]\.nuget\packages\microsoft.net.sdk.functions\1.0.24\build\netstandard1.0\Microsoft.NET.Sdk.Functions.props(38,3): warning MSB4011: "[yourpath]\.nuget\packages\microsoft.azure.webjobs.script.extensionsmetadatagenerator\1.0.1\build\Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator.props" cannot be imported again. It was already imported at "[yourpath]\obj\ActiveMQFunctionUsageExample.csproj.nuget.g.props (17,5)". This is most likely a build authoring error. This subsequent import will be ignored. [[yourpath]\[yourproject].csproj][yourpath]\.nuget\packages\microsoft.net.sdk.functions\1.0.24\build\netstandard1.0\Microsoft.NET.Sdk.Functions.targets(45,3): warning MSB4011: "[yourpath]\.nuget\packages\microsoft.azure.webjobs.script.extensionsmetadatagenerator\1.0.1\build\Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator.targets" cannot be imported again. It was already imported at "[yourpath]\obj\ActiveMQFunctionUsageExample.csproj.nuget.g.targets (9,5)". This is most likely a build authoring error. This subsequent import will be ignored. [[yourpath]\[yourproject].csproj]

Then you may have to delete the following lines: [yourpath]\obj\ActiveMQFunctionUsageExample.csproj.nuget.g.props <Import Project="$(NuGetPackageRoot)microsoft.azure.webjobs.script.extensionsmetadatagenerator\1.0.1\build\Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator.props" Condition="Exists('$(NuGetPackageRoot)microsoft.azure.webjobs.script.extensionsmetadatagenerator\1.0.1\build\Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator.props')" />

[yourpath]\obj\ActiveMQFunctionUsageExample.csproj.nuget.g.targets Import Project="$(NuGetPackageRoot)microsoft.azure.webjobs.script.extensionsmetadatagenerator\1.0.1\build\Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator.targets" Condition="Exists('$(NuGetPackageRoot)microsoft.azure.webjobs.script.extensionsmetadatagenerator\1.0.1\build\Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator.targets')" />

Then run this again

func extensions install
func start host

[SeriousM]

This is a horrible situation. Any words from Microsoft about this issue state?

[shareonline]

Manually creating the two files in the function root fixed my binding issue, files here:

Directory.Build.targets

<Project>
  <Target Name="CopyExtensionsJson" AfterTargets="_GenerateFunctionsAndCopyContentFiles">
    <Message Importance="High" Text="Overwritting extensions.json file with one from build." />

    <Copy SourceFiles="$(PublishDir)extensions.json"
          DestinationFiles="$(PublishDir)bin\extensions.json"
          OverwriteReadOnlyFiles="true" />
  </Target>
</Project>

extensions.json

{
    "extensions":[
        { "name": "ServiceBus", "typeName":"Microsoft.Azure.WebJobs.ServiceBus.ServiceBusWebJobsStartup, Microsoft.Azure.WebJobs.ServiceBus, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"},
        { "name": "ActiveMQ", "typeName":"Microsoft.Azure.WebJobs.ActiveMQ.ActiveMQWebJobsStartup, ActiveMQ, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"}
    ]
  }

Also, don't forget to copy extensions.json to output dir

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>netcoreapp2.1</TargetFramework>
    <AzureFunctionsVersion>V2</AzureFunctionsVersion>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.Azure.WebJobs.Extensions.ServiceBus" Version="3.0.0" />
    <PackageReference Include="Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator" Version="1.0.1" />
    <PackageReference Include="Microsoft.NET.Sdk.Functions" Version="1.0.24" />
  </ItemGroup>
  <ItemGroup>
    <None Update="host.json">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
    </None>
    <None Include="extensions.json">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
    </None>
    <None Update="local.settings.json">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
      <CopyToPublishDirectory>Never</CopyToPublishDirectory>
    </None>
  </ItemGroup>
</Project>

Then run

func extensions install
func start host

I can't seem to get this to work. I have two other functions, and the only difference i can find is the httptrigger. I'm only using EF Core in all functions, and only in my httptrigger function will the dbcontext not inject. Any idea to this?

Nevermind, When i switched my build agent in Azure DevOps to Hosted Ubuntu it started working.

[tommyk]

i followed what Fowler said at and it started working on my machine (osx). When I build and push to azure, it is not working with my custom extension (its an httpClient getting injected). I checked the extensions.json and its is NOT empty on the server that is failing. thoughts?!

[DaliborTakac]

@tommyk The problem is not that extensions.json is empty, on server it contains only few built in stuff and none of the additional extensions you have in the project (including your custom http client). You can check this by creating a new empty AF project and building it so that extensions.json is created for it and compare against that.

[kemmis]

Still broken for me. Please fix.

[fabiocav]

@kemmis when did you last try this? Can you make sure you're referencing the latest version of the metadata generator package (https://www.nuget.org/packages/Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator/) and retry? An update was just pushed yesterday, and if what you're running into is the same issue with the targets, that change should address it.

[dpakgithub82]

@stevedenman @mlankamp @djmeck @dbarentine @hiraldesai

After making the changes to extension files,we running into new issue "Host Not running issue",its crashing the environment.Its running locally in dev environment and but having the same issue in Azure.

[NicoDvl]

@fabiocav I updated the metadata generator package and it fixes the bindings issue in extensions.json. The Microsoft.NET.Sdk.Functions will also be updated with the latest version of the metadata generator ?

[fabiocav]

@NicoDvl yes, that will eventually be updated to bring in the latest version. We just wanted to get this update out for validation first.

[iamandymcinnes]

I am also getting issues, I've followed the above, by creating a build targets and extensions file. I've add entries for the startup and the metadata generator in there and I'm running the latest versions of everything targeting core 2.2. The issue I see is that WebJobsStartup is not registering my startup class at all.

[iamandymcinnes]

The solution I've found is to ensure you reference your startup in the extension:

{
  "extensions": [
    {
      "name": "Startup",
      "typeName": "MyApp.Startup, MyApp, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"
    },
    {
      "name": "ServiceBus",
      "typeName": "Microsoft.Azure.WebJobs.ServiceBus.ServiceBusWebJobsStartup, Microsoft.Azure.WebJobs.ServiceBus, Version=3.0.3.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"
    }
  ]
}

[DaliborTakac]

@iamandymcinnes The thing you described is the desired outcome, not sollution, it should be up to tooling to generate this file correctly in the first place because you can't and should not need to know what librarys and extension authors have used in order to have their azure function extensions working properly and where do they keep their own initialization code.

[iamandymcinnes]

@DaliborTakac agreed not ideal

[zarusz]

@fabiocav I confirm that referencing Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator in the latest version (1.0.2 as of now) resolved the issue. Tested when publish from VS and Azure DevOps (both using Web Deploy).

My runtime is:

<TargetFramework>netcoreapp2.1</TargetFramework>

And libraries:

    <PackageReference Include="Microsoft.Azure.WebJobs.Extensions" Version="3.0.1" />
    <PackageReference Include="Microsoft.Azure.WebJobs.Extensions.ServiceBus" Version="3.0.3" />
    <PackageReference Include="Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator" Version="1.0.2" />
    <PackageReference Include="Microsoft.NET.Sdk.Functions" Version="1.0.24" />
    <PackageReference Include="Willezone.Azure.WebJobs.Extensions.DependencyInjection" Version="1.0.1" />

Also to clarify, even with the latest metadata geneeator I still had to keep the Directory.Build.targets. Without that target the extensions.json file didn't have the DI extension.

[hiraldesai]

I can confirm this is fixed with latest Sdk 1.0.26.

<TargetFramework>netcoreapp2.1</TargetFramework>

<PackageReference Include="Microsoft.NET.Sdk.Functions" Version="1.0.26" />

My question is - am I allowed to do use target framework netcoreapp2.2 yet?

[hiraldesai]

I got my answer here and the doc will also get updated soon.

[bobvandevijver]

I just updated to the latest Microsoft.NET.Sdk.Functions (version 1.0.26, which includes 1.0.2 of the MetadataGenerator), and that seems to solve the issue for us.

However, I do not need the Directory.Build.targets that @zarusz is referencing to in order to have the DI functional. Might be due to the use of netcoreapp2.2 or maybe it's just the newer Sdk.Function, I'm not sure.

[louisekeegan]

I've added the recommended Build.Targets file, but this didn't fix it for me. I installed Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator v1.1.0 and the latest version of Microsoft.NET.Sdk.Functions (1.0.27). The extensions.json file looks correct and is present in the publish directory. However I'm getting an error when I call my function (Exception binding parameter {dependencyName} -> NullReferenceException). The weird thing is I have a second function (basic hello function) that has a dependency on another service. This function executes successfully. I don't understand why the parameter binding works for one function and not for the other.

[bobvandevijver]

@louisekeegan It sounds like your dependency object fails during construction, probably some uninitialised variable in the constructor?

[fabiocav]

@louisekeegan this looks like a different issue. If you can open a separate issue and point to a repro, it will be easier to investigate

[louisekeegan]

@bobvandevijver yes, you're on the right track. It appears my issue was due to my assumption that my settings.json file was getting published with the app (I'm new to Azure functions). My Startup class needed to read this file to create an instance of a settings class and then register it with the DI container. My guess is I was registering null. Lesson learned :)

[louisekeegan]

@fabiocav looks like it was my mistake so no need to investigate any further. Thanks!

[marshall76963]

I am still facing this issue unfortunately using .Net Core 2.2 and Microsoft.NET.Sdk.Functions v1.0.28

I have a custom startup.cs that is registered and generates correctly when running locally. When I use VS 2017 to deploy to Azure, I can see it generates a extensions.json file with the following in the obj\Release\netcoreapp2.2\PubTmp\Out\bin directory:

{
  "extensions":[
  ]
}

I can see that the correct extensions.json is created in the bin\Release\netcoreapp2.2\bin directory, but this appears to be ignored in the deployment (but would explain why the local execution works fine).

I have tried including Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator (v1.1.1) directly, but this has no noticeable affect one way or the other.

I am loathed to have to follow the suggestion by @ElliotWood as, although it may work, the framework should handle this directly and our build numbers change which would require additional steps to keep the extensions.json file updated.

Can anyone confirm if this is an ongoing issue and if there is either a time-frame for a fix or a viable workaround?

Update I have just tried using .Net Standard 2.0 and .Net Core 2.1 with the rest of the configuration the same and can see that the extensions.json created in obj\Release\\PubTmp\Out\bin is blank as above.

[marshall76963]

I've been running into this as well. The build generates the correct extensions.json in bin\Release\netstandard2.0\bin. So, I created a build target Directory.Build.targets file that can be dropped into the project directory and will copy the correct file at publishing time.

This will copy the correct extensions.json into the publish directory prior to actual deployment/zipping etc...

For some reason I missed this suggestion from @dbarentine earlier, and with a small tweak to the Directory.Build.targets file the correct extensions.json is now getting deployed.

Still interested to know when this workaround will no longer be required.

[brettsam]

@marshall76963 (and others still having issues when using Micrsosoft.Net.Sdk.Functions 1.0.28) -- can you try running a build without your custom build targets, and add the "/bl" flag to the build command line? That will generate an msbuild.binlog file. If you could email that to me (my microsoft email is in my github profile), that'd give us a better picture as to what's going on.

[marshall76963]

@brettsam - I have emailed you with the binlog file - thanks for supporting me on this :)

[brettsam]

For those of you still seeing issues with 1.0.28, can you try:

1. Removing any custom targets files
2. Removing any direct reference to the Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator package

It looks like the direct reference re-orders some things during a publish, which causes the extensions.json generator to miss the functions assembly. That's something we can fix, but removing the package reference should get rid of the ordering problem.

[stofte]

I just randomly started seeing this issue. I think the cause was that I added a AspNet NetFramework package to something, when it was expecting Core stuff (removing obviously does not fix my issue). Now my extension no longer starts up. Suggestions such as re-adding it elsewhere does not resolve my issue.

@brettsam When I peek inside the temp dirs and look in the project.assets.json, I can find references to ExtensionsMetadataGenerator. I don't quite know how to read this:

  "Microsoft.NET.Sdk.Functions/1.0.29": {
    "type": "package",
    "dependencies": {
      "Microsoft.Azure.WebJobs": "[3.0.0, 3.1.0)",
      "Microsoft.Azure.WebJobs.Extensions": "[3.0.0, 3.1.0)",
      "Microsoft.Azure.WebJobs.Extensions.Http": "[3.0.0, 3.1.0)",
      "Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator": "1.1.2",
      "Newtonsoft.Json": "[11.0.2]"
    },
    "build": {
      "build/netstandard1.0/Microsoft.NET.Sdk.Functions.props": {},
      "build/netstandard1.0/Microsoft.NET.Sdk.Functions.targets": {}
    }
  },

But I'd assume it mean it's the functions package which is pulling in your suspect package. I am on 1.0.29 btw. I don't see any extensions.json file either in any output dir which seems to contain the right thing.