Support for not so custom TriggerBinding

[tmasternak]

Question

Azure Functions do not support custom TriggerBindings. There is a good explanation of the reasons provided by Coby and @jeffhollan at the April Azure functions live. As far as I can understand the problem boils down to Scale Controller not having information about custom trigger and not being able to scale the function app when needed. One example might be a function reacting to stream of tweets.

That being said what if the custom trigger would be based on a source that is already supported by the existing trigger e.g. Azure Storage Queue? I've been working on a spike that creates a custom trigger for ASQ and it looks that the Scale Controller handles the auto-scaling properly if I make my trigger pretend it's queueTrigger. In practice it means I provide function.json content as if it was queueTrigger.

Are there any plans to support customs triggers at that level?

Background information

What I'm asking is a potential solution to a problem that I am trying to solve. What I would like to do is provide Azure Function users with capability to receive custom POCO types as input messages and be able to pass message headers between input message and any outgoing message that goes out via output collector. It would be also great if I could handle any custom message serialization behind the scenes. Something like this.

I would be really grateful for any suggestions.

[SeanFeldman]

It's been a week since the question was posted and the issue wasn't even triaged/commented on. Any chance to get an answer to this?

[paulbatum]

You're right that a custom trigger binding that can "piggy back" on existing scale logic could theoretically work. What this would require is a separation between the metadata that the runtime uses and the metadata the scale controller uses so that you could tell the runtime "my trigger type is nservicebus trigger" and tell the scale controller "my trigger type is azure storage queue". Right now, that separation does not exist - they share a source of truth (function.json).

So at a high level I think this is a reasonable feature request. However if we were to rank it against many of our other open bugs or features it would not rank particularly high right now.

That all said, I do wonder whether its possible for you to achieve your main goals without a custom trigger at all. Let me chat with a few folks and I'll reply in a day or so.

[mikhailshilkov]

I believe that's how Durable Function custom triggers (e.g. ActivityTrigger) work. They are based on Storage Queues, so scaling logic is probably reused.

I don't know how that works in details, and whether it's official and encouraged to be used for 3rd party triggers.

[paulbatum]

Hmm, that one is a bit different in that the scale controller knows about durable functions. Yes internally it is able to reuse some of the queue handling logic but that is not the same thing as telling the scale controller "I am just a queue trigger".

[SeanFeldman]

So at a high level I think this is a reasonable feature request. However if we were to rank it against many of our other open bugs or features it would not rank particularly high right now.

@paulbatum you're putting the question in a good perspective. The counter question I'd raise is the value propsitions Functions are bringing. Not being able to create and scale out customer triggers based functions diminishes the value proposition Functions bring to the game.

[mikhailshilkov]

@paulbatum ah, ok, thanks for clarification.

I actually had ideas which were somewhat similar to what @tmasternak did. Not in a sense of NServiceBus support, but more of a custom trigger which is just a wrapper around a standard trigger. I could then change it behavior to model my business domain / architecture in a more precise way.

Say, I were to implement something in line with CQRS pattern. I could then have a CommandTrigger (e.g. Service Bus Queue behind it) and an EventTrigger (e.g. Service Bus Subscription behind it), which would deserialize the messages in the right way and do some other custom actions. I would then create a dozen of functions and be sure a) there's no code duplication b) they work in a uniform way.

Alternatively, in my scenario, I could create the single function with the standard trigger (Service Bus), but then I want to be able to reuse that single function for all 20 command or event handlers in my application. So, to have a way to parameterize that function (make it generic? reflection discovery at startup time? pass parameter from function.json?).

Sorry for diverging from the original question, but maybe you could provide some vision of whether/how similar scenarios will be supported in the future.

[paulbatum]

@SeanFeldman I can agree with your assertion in an abstract sense - every missing feature is a potential missed opportunity. But help me put this feature request in perspective - couldn't the same end user scenario be achieved with some extra lines of code and no custom binding at all? Don't get me wrong - as someone who works on functions I see tremendous value in all the code that functions users don't have to write - its a significant value prop. But in the scheme of things, we're going to lean towards prioritizing features or fixes that unblock scenarios that simply aren't possible, rather than the ones that save a few lines of code. Let me know if I'm miss characterizing this scenario.

@mikhailshilkov this is definitely a divergence from the original question :) Especially when you're getting into "parameterizable" functions.

I made a bit of progress, I think you can write a binding extension that will add the necessary converters, but the trigger type in terms of function.json remains the same. I'll try to share some code tomorrow..

[SeanFeldman]

@paulbatum I would not necessarily categorize custom bindings/triggeres as a few lines of code saved for users. Take for example @tmasternak 's original question. He'd like to enable NServiceBus customers to use Functions. It is more than just a few lines of code.

Another example (ironically, asked by an internal MSFT team) is to use Functions with Azure Service Bus to support messages larget than 1 MB to migrate from MSMQ. There's a plugin for ASB seamlessly implementing claim check pattern. The existing Service Bus trigger doesn't support plugins registration. That would mean duplicating what plugin is doing in all of their functions and owning that implementation rather than focusing on core busines requirement.

I understand the desire to focus on the lower hanging fruits. At the same time, would not understimate the ability to create custom triggers that are first class citizens when it comes to scale, rather than just nice demos for blog posts 🙂

[mikhailshilkov]

@paulbatum So you agree the change is good, but the team has no time. Is there a way community could contribute to this? I guess NServiceBus guys and other geeks like me 😃 could spend their time on features that they find valuable. Embrace the open source, you know.

Obviously some effort from the team would still be required. But isn't that the higher priority work then - helping others to help you?

One obvious problem is lack of scale controller open code, but I'm not sure how many changes are needed there.

[tmasternak]

@paulbatum first of all thank you for your insights and comments.

Let me try sharing context from my end.

But in the scheme of things, we're going to lean towards prioritizing features or fixes that unblock scenarios that simply aren't possible, rather than the ones that save a few lines of code. Let me know if I'm miss characterizing this scenario.

As mentioned in the issue description custom trigger is a solution to a set of problems that I was trying to tackle. Let me try describe them in more detail in context of asq trigger:

• It not support serialization customization - what it means is that any sender has to comply to the serialization format. This is far from perfect especially in scenarios were existing system is extended with new features build on top for Azure Functions. Even if the sender is using Json it can still be incompatible e.g. because of BOM,
• It does not support extending binding data collections based on message content - what it means is that in scenarios when message content contains both control (e.g. headers) data and business data there is not easy way to separate the two,

In general those overlap with scenario mentioned by @mikhailshilkov in which it's hard to provide a business level function signature (business data as input types) and float control data behind the scenes.

[paulbatum]

Let’s figure out if it’s even necessary first. As I mentioned, I think this might be doable without a custom trigger.

You are correct that even community contributions take time to review and sometimes we struggle to find the bandwidth for that. The best fix is no fix at all :)

[paulbatum]

To clarify - my reply was to Mikhail, just saw your post Tomasz, thanks for the info.

[mikhailshilkov]

To the point of @SeanFeldman, custom triggers popularized by companies and frameworks like NServiceBus could be a great channel to onboard new people to Functions, from that different angle.

[paulbatum]

OK I spent a bunch of time playing with this. The TL;DR is that in general, you can use the extensions model to add additional types that user code can bind to, without having to implement a custom trigger itself. The example I used was a wrapping type for EventData (from the event hubs extension). The extension is just:

    public class MyEventHubExtension : IExtensionConfigProvider
    {
        public void Initialize(ExtensionConfigContext context)
        {
            context.AddConverter<EventData, MyEventData>(x => new MyEventData { InnerEventData = x });
        }
    }

    public class MyEventData
    {
        public EventData InnerEventData { get; set; }

        public string GetAsUTF8String()
        {
            return Encoding.UTF8.GetString(InnerEventData.Body.ToArray());
        }
    }

I was able to write and execute an event hub triggered function that used this extension:

    public static class EventHubProcessor
    {
        [FunctionName("EventHubProcessor")]
        public static void Run([EventHubTrigger("extensiontesthub", Connection = "EventHubConnection", ConsumerGroup = "functions")] MyEventData myData, TraceWriter log)
        {
            log.Info($"C# Event Hub trigger function processed a message: {myData.GetAsUTF8String()}");
        }
    }

However, Tomasz is correct with his observations about the queue trigger. It seems like it doesn't participate correctly in the extensibility model so if you tried to do the same as above but for a queue triggered function it would not work. I have filed this issue to track: https://github.com/Azure/azure-webjobs-sdk/issues/1696

I think it makes sense to leave this issue open, because not ALL scenarios can be handled with a custom converter. You might want to change the retry or error semantics for a given trigger type while still relying on the standard scale controller monitoring behavior for activation and scale out. But for all the scenarios where you just want to add the ability to use a new type from the function code, I would suggest using the approach outlined above (which would be viable for queue triggers if that 1696 issue is resolved).

[mikhailshilkov]

@paulbatum What's the way to register your custom extension with type conversion? When I make an extension with a new binding, it gets loaded because the binding is used. Here, there's no new attribute or anything, so runtime just skips your class in my project...

[paulbatum]

After doing a build, does your bin directory contain an extensions.json that looks something like this?

{
  "extensions":[
    { "name": "EventHubConfiguration", "typeName":"Microsoft.Azure.WebJobs.ServiceBus.EventHubConfiguration, Microsoft.Azure.WebJobs.EventHubs, Version=3.0.0.0, Culture=neutral, PublicKeyToken=null"},
    { "name": "MyEventHubExtension", "typeName":"WebJobsExtension.MyEventHubExtension, WebJobsExtension, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null" }
  ]
}

If not, then I think you must be hitting a bug with our build task. Can you try explicitly referencing this package from your function app project? https://www.nuget.org/packages/Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator/1.0.0-beta3

[mikhailshilkov]

@paulbatum There's no extensions.json file, even when my extensions are loaded properly. I referenced this extra package and now the file is created in bin\Debug\netstandard2.0 but it's empty (extensions is an empty array).

Extensions are defined in a class library which is referenced by Function App. Copying it to App itself doesn't seem to change anything.

If input binding is used, all extensions from that library are loaded fine (even ones not used for binding). Otherwise, none are loaded.

[paulbatum]

I just cleaned my sample and rebuilt and now I'm getting an empty extensions.json file. Something is screwed up. Lets try to tease this apart in terms of tooling issues vs runtime issues. Make the extensions.json file manually based on what I shared above and make sure its in your bin folder along with yourapp.dll and yourextension.dll. If that works then we know its just a tooling issue.

[mikhailshilkov]

@paulbatum I can't get estensions.json to work. I've created it manually in bin folder, and indeed now I see the following line in the logs:

Loaded custom extension: MyEventHubExtension from 'C:\bla\FunctionAppV2\bin\Debug\netstandard2.0\bin\MyExt.dll'

but Initialize method is actually never called. As soon as I add an input binding usage, Initialize gets called on all extensions.

[mikhailshilkov]

On top of that, I can't make your example work. I used exactly your converter, but when sending data to Event Hub I get

System.Private.CoreLib: Exception while executing function: GreeterEvent. Microsoft.Azure.WebJobs.Host: Exception binding parameter 'eventData'. Microsoft.Azure.WebJobs.Host: Object reference not set to an instance of an object.

Stack trace on break:

at Microsoft.Azure.WebJobs.Host.SimpleTriggerArgumentBinding`2.d__19.MoveNext() in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Triggers\TriggerArgumentBinding\SimpleTriggerArgumentBinding.cs:line 66

[paulbatum]

I am now getting the same behavior as you. If I switch to an earlier version of V2 this will work. I chatted with @fabiocav and there is a problem in the current release bits where custom extensions are being loaded into the wrong assembly load context and therefore they are not discovered and initialized correctly.

This release (currently in progress) has the fix: https://github.com/Azure/azure-functions-host/releases/tag/v2.0.11843-alpha

[mikhailshilkov]

@paulbatum Ok, I got it working on the latest bits. EventData is converted to MyEventData just fine.

Can the same trick work with HTTP trigger? I tried adding a converter from HttpRequest but it doesn't seem to be triggered.

Looking at code, HttpTriggerAttributeBindingProvider seems to just parse my custom class as DTO from the body. Can I override that with custom code?

I would also like to add output conversion. Are there docs for converters?

[paulbatum]

Thats good to hear, sorry this has been so painful. I have no idea about the status of HTTP - it might have the same issues as queues and not support the converters.

I don't think there are any docs for converters beyond what is in the code itself. We have not really pushed this extensibility very hard yet (we leverage it internally as we add new capabilities to functions, but to create a thriving 3rd party ecosystem requires more investment).

[SeanFeldman]

Any updates on supporting custom triggers? It's been a while, Functions 2.0 is out and would be helpful to know if the stand on this has changed (give the issue is still opened). Thank you.

[paulbatum]

This issue is about supporting custom trigger implementations that piggy-back on existing event sources like azure storage queues, service bus, etc. I suspect you're asking more generally about the ability to specify purely custom triggers?

I will say that this we are treating this topic with higher priority than a year ago and we're considering some work in this area within the next 6 months. I'm being deliberately vague because we haven't worked out any details yet.

[SeanFeldman]

@paulbatum you're absolutely correct. Thank you for distinguishing the two and providing some idea where it is. Not to pollute this thread, is there a separate issue tracking custom (pure) triggers? I couldn't find one and perhaps, if it's something that is under consideration, would be good to have a tracking issue people could subscribe to. Thank you.

[CasperWSchmidt]

Any updates on the custom triggers? I would like to be able to implement a persistent event store triggering functions whenever a new event is added. Pretty much like the CosmosDb change feed, but for other persistence options (we already have a mongoDb running)

[paulbatum]

Lets use https://github.com/Azure/Azure-Functions/issues/1154 as the main place to discuss custom triggers. I will provide a short update there.