Method not found: get_SerializerSettings

[leemorton]

Steps to reproduce

Using the latest released version with .NET Core 3 I am getting a 406 Not Acceptable error on around a third of requests. Debugging locally produces the following error...

@ using (var devops = new VssConnection(eventContext.CollectionUri, clientCredentials))

Method not found: 'Newtonsoft.Json.JsonSerializerSettings System.Net.Http.Formatting.BaseJsonMediaTypeFormatter.get_SerializerSettings()'.
   at Microsoft.VisualStudio.Services.WebApi.VssJsonMediaTypeFormatter.SetSerializerSettings(Boolean bypassSafeArrayWrapping, Boolean enumsAsNumbers, Boolean useMsDateFormat)
   at Microsoft.VisualStudio.Services.WebApi.VssJsonMediaTypeFormatter..ctor(Boolean bypassSafeArrayWrapping, Boolean enumsAsNumbers, Boolean useMsDateFormat)
   at Microsoft.VisualStudio.Services.WebApi.VssJsonMediaTypeFormatter..ctor(Boolean bypassSafeArrayWrapping)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase..ctor(Uri baseUrl, HttpMessageHandler pipeline, Boolean disposeHandler)
   at Microsoft.VisualStudio.Services.Location.Client.LocationHttpClient..ctor(Uri baseUrl, HttpMessageHandler pipeline, Boolean disposeHandler)
   at Microsoft.VisualStudio.Services.WebApi.Location.VssServerDataProvider..ctor(VssConnection connection, HttpMessageHandler pipeline, String fullyQualifiedUrl)
   at Microsoft.VisualStudio.Services.WebApi.VssConnection..ctor(Uri baseUrl, VssHttpMessageHandler innerHandler, IEnumerable`1 delegatingHandlers, Boolean allowUnattributedClients)
   at Microsoft.VisualStudio.Services.WebApi.VssConnection..ctor(Uri baseUrl, VssHttpMessageHandler innerHandler, IEnumerable`1 delegatingHandlers)
   at Microsoft.VisualStudio.Services.WebApi.VssConnection..ctor(Uri baseUrl, VssCredentials credentials, VssHttpRequestSettings settings)
   at Microsoft.VisualStudio.Services.WebApi.VssConnection..ctor(Uri baseUrl, VssCredentials credentials)
   at aggregator.Engine.RuleExecutor.<ExecuteAsync>d__3.MoveNext() in C:\Users\lmorton\Downloads\aggregator-cli\aggregator-cli-6747f6ea131ba7b96105e179f2999c1c0f61830a\src\aggregator-ruleng\RuleExecutor.cs:line 38
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at aggregator.AzureFunctionHandler.<RunAsync>d__3.MoveNext() in C:\Users\lmorton\Downloads\aggregator-cli\aggregator-cli-6747f6ea131ba7b96105e179f2999c1c0f61830a\src\aggregator-function\AzureFunctionHandler.cs:line 76

Expected behavior

Instantiate the VssConnection

Actual behavior

Method not found: 'Newtonsoft.Json.JsonSerializerSettings System.Net.Http.Formatting.BaseJsonMediaTypeFormatter.get_SerializerSettings()'.

[BobSilent]

Just found on stackoverflow https://stackoverflow.com/questions/59888558/method-not-found-get-serializersettings

[leemorton]

Just found on stackoverflow https://stackoverflow.com/questions/59888558/method-not-found-get-serializersettings

Yeah I saw that too, so thought I would try pulling those nuget packages forward to the latest preview versions of the azure devops nuget packages built after the .net core 3 general release. Getting the same error still though :(

[BobSilent]

unfortunately the stackoverflow question was removed.

I played a little bit around and tried to reduce the function to a minimum. So I reused a rule function in azure portal and changed it

#r "../bin/aggregator-function.dll"
#r "../bin/aggregator-shared.dll"
#r "../bin/Microsoft.VisualStudio.Services.Common.dll"
#r "../bin/Microsoft.VisualStudio.Services.WebApi.dll"
#r "../bin/Microsoft.TeamFoundation.WorkItemTracking.WebApi.dll"
#r "../bin/Newtonsoft.Json.dll"

using Microsoft.VisualStudio.Services.Common;
using Microsoft.VisualStudio.Services.WebApi;
using Microsoft.TeamFoundation.WorkItemTracking.WebApi;

using System.Threading;
using System.Reflection;
using aggregator;

public static async Task<HttpResponseMessage> Run(HttpRequestMessage req, ILogger logger, Microsoft.Azure.WebJobs.ExecutionContext context, CancellationToken cancellationToken)
{
    var clientCredentials = new VssBasicCredential("PAT", "<PAT>");
    var devops = new VssConnection(new Uri("<account>"), clientCredentials);

    var client = devops.GetClient<WorkItemTrackingHttpClient>();

    var workitem = await client.GetWorkItemAsync(<ID>);

    logger.LogInformation($"{workitem.Id}");

    var formatter = new VssJsonMediaTypeFormatter(false);
    logger.LogInformation($"{Assembly.GetAssembly(formatter.SerializerSettings.GetType()).FullName}");

    var handler = new AzureFunctionHandler(logger, context);
    var result = await handler.RunAsync(req, cancellationToken);
    return result;
}

but this does not show the error, also locally i did not had the error. And now it's getting hard to analyze.

@leemorton can you also try this minimum example as well.

otherwise I would have to include some more logging. My current assumption from your error is, that the "wrong" Newtonsoft.Json.dll is loaded, like some comments also in #94 mention. And that's also the big difference, in the example above the #r "../bin/Newtonsoft.Json.dll" loads the dll directly from the specified location where v12.0.0.0 should be located.

What I saw on the function console was at least this

[Warning] warning CS1701: Assuming assembly reference 'Newtonsoft.Json, Version=10.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed' used by 'System.Net.Http.Formatting' matches identity 'Newtonsoft.Json, Version=12.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed' of 'Newtonsoft.Json', you may need to supply runtime policy

And this could be an indicator, that depending which dll is loaded at first (if a 10.0.0.0 get's loaded) you sometimes get an error and sometimes not.

But this is currently just an assumption

[leemorton]

It turns out that tfs-aggregator was performing perfectly, but azure functions was creating the 406 error after we pass back the HttpResponseMessage (about half of the time for me).

I've been playing around throughout the day with so many things I have forgotten most of them. But in the end I got it all working stable across a selection of requests by doing the following:

Changing the v3 function signature and return type to use Microsoft.AspNetCore.Http.HttpRequest instead of System.Net.Http.HttpRequestMessage and Microsoft.AspNetCore.Mvc.IActionResult instead of System.Net.Http.HttpResponseMessage

#r "../bin/aggregator-function.dll"
#r "../bin/aggregator-shared.dll"

using System.Threading;
using aggregator;
using Microsoft.AspNetCore.Mvc;

public static async Task<IActionResult> Run(HttpRequest req, ILogger logger, Microsoft.Azure.WebJobs.ExecutionContext context, CancellationToken cancellationToken)
{
    var handler = new AzureFunctionHandler(logger, context);
    return await handler.RunAsync(req, cancellationToken);
}

Changing the following parts in AzureFunctionHandler.cs to consume and return those types.

public async Task<IActionResult> RunAsync(HttpRequest req, CancellationToken cancellationToken)
{
    _log.LogDebug($"Context: {_context.InvocationId} {_context.FunctionName} {_context.FunctionDirectory} {_context.FunctionAppDirectory}");

    var ruleName = _context.FunctionName;
    var aggregatorVersion = GetCustomAttribute<System.Reflection.AssemblyInformationalVersionAttribute>()?.InformationalVersion;
    _log.LogInformation($"Aggregator v{aggregatorVersion} executing rule '{ruleName}'");
    cancellationToken.ThrowIfCancellationRequested();

    // Get request body
    var eventData = await GetWebHookEvent(req);
    if (eventData == null)
    {
        return new BadRequestObjectResult("Request body is empty");
    }

    // sanity check
    if (!DevOpsEvents.IsValidEvent(eventData.EventType)
        || eventData.PublisherId != DevOpsEvents.PublisherId)
    {
        return new BadRequestObjectResult("Not a good Azure DevOps post...");
    }

    var eventContext = CreateContextFromEvent(eventData);
    if (eventContext.IsTestEvent())
    {
        return new OkResult();
    }

    var configContext = GetConfigurationContext();
    var configuration = AggregatorConfiguration.ReadConfiguration(configContext)
                                                .UpdateFromUrl(ruleName, new Uri(req.GetDisplayUrl()));

    var logger = new ForwarderLogger(_log);
    var ruleProvider = new AzureFunctionRuleProvider(logger, _context.FunctionDirectory);
    var ruleExecutor = new RuleExecutor(logger, configuration);
    using (_log.BeginScope($"WorkItem #{eventContext.WorkItemPayload.WorkItem.Id}"))
    {
        try
        {
            var rule = await ruleProvider.GetRule(ruleName);
            var execResult = await ruleExecutor.ExecuteAsync(rule, eventContext, cancellationToken);

            if (string.IsNullOrEmpty(execResult))
            {
                return new OkResult();
            }
            else
            {
                _log.LogInformation($"Returning '{execResult}' from '{rule.Name}'");
                return new OkObjectResult(execResult);
            }
        }
        catch (Exception ex)
        {
            _log.LogWarning($"Rule '{ruleName}' failed: {ex.Message}");
            throw ex;
        }
    }
}
private async Task<WebHookEvent> GetWebHookEvent(HttpRequest req)
{
    using (var sr = new StreamReader(req.Body))
    {
        var jsonContent = await sr.ReadToEndAsync();
        if (string.IsNullOrWhiteSpace(jsonContent))
        {
            _log.LogWarning($"Failed parsing request body: empty");
            return null;
        }

        var data = JsonConvert.DeserializeObject<WebHookEvent>(jsonContent);
        return data;
    }
}

And the corresponding AzureFunctionHandlerExtension.cs

However now that all works nicely live in azure functions, locally I still get the method not found issue on VssConnection. But this time with the expected 500 status. So I think two separate issues in summary, the 406 status resolved by function signature changes above. The VssConnection one only occurring locally (probably specific to my environment maybe).

Thoughts?

[BobSilent]

Ok, I see. I will take a deeper look and add the code to the corresponding branch, either for V3 changes (BobSilent/fix-adaption-for-functions-v3) or to a pull request. Thanks for your investigation, I only checked the aggregator internals.

Concerning your local issues, maybe you can check the debugger modules window and look for the newtonsoft json dll, which version and from where it gets loaded. Is this issues permanent or also sometimes working, sometimes not?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.