Compatibility with netstandard/net core

[mollyporph]

This is a collection of blockers and potential solutions for moving from WindowsAzure.ServiceBus (net45) to Microsoft.Azure.ServiceBus (1.0.0: net451, netstandard1.3. 2.0.0: net461, netstandard2.0)

With Microsoft.Azure.ServiceBus Microsoft has moved the management api to a new project: Microsoft.Azure.Management.Servicebus. This new project is an attempt by Microsoft to align the servicebus sdk to their ARM-everywhere philosophy.

This introduces a couple of issues. The first and foremost is that all management commands are executed via ARM and thus requires Azure Active Directory credentials instead of the much simpler Service Bus connectionstring. Maintainers have received the criticism and have a plan to re-introduce management APIs (follow the issue here: https://github.com/Azure/azure-service-bus-dotnet/issues/65) tl;dr They will gradually re-implement it with a goal of having it all implemented by summer 2018.

Will update with more findings

[hulvei3]

What is the current status on compatibility with netstandard? It's hard to sense if the limitation is still on place or not.

[mookid8000]

As far as I know, Rebus.AzureServiceBus can still not run on .NET Core – not even on .NET Core 2, even though it will readily try, because it will fails due to WindowsAzure.ServiceBus trying to use System.Configuration, which is not available.

[hulvei3]

Okay. This is a shame - we actually planned for migrating to Azure from MSSQL very soon. And we already have dependencies that targets NETStandard. If we would make a temp solution for us, by forking, how what is the culbrit of this problem? (if we have to include Azure AD creds, so be it)

[Meyce]

For whatever the host application is, can you have the host app use a .Net Framework version? I have several web APIs and Windows Services where the underlying project is .Net Core but I'm continuing to target .Net framework 4.7, which has worked out well thus far. This would work fine as long as you were running on Windows, and you wouldn't have to rework the ASB transport.

If you do try and rework the transport, you'll need to use an app registration in Azure AD as a service principal and I think you have to give it the owner role on your subscription. I don't think there are currently any roles targeted at ASB, but I may be wrong on that. The "updated" transport would need to bring in the ARM ASB lib as well to handle all the management operations that the new .Net Standard ASB lib can't do anymore.

There hasn't been much from Microsoft recently on the state of the .Net Standard ABS library. They have talked about adding the necessary features back in, but it'll likely be at least a few more months before this happens.

[hulvei3]

Well, that would not work for us, as we use Rebus in our own slim wrapper lib. This lib is then shared among every components (being services, APIs etc) - some of them being targeted NET Core. So unless we accept that we would remove the NETStandard support in our wrapper lib, I don't see any way for us to consume this right now it seems.

Thanks for the response

[Meyce]

Okay.

If you do look into forking/updating the existing transport the bulk of the work will be in the two transport classes.

I've used the ASB ARM lib a bit. You'll need to get the ((client id/client secret) or cert), authority, app id into the transports so that you can get a bearer token for the Azure Management API. Once you have that the management operations are pretty straightforward. The bearer token will have an expiration, so it will need to have the ability to go out and get another access token in the future. This would have to be done with username/password token flow, so I don't think there will be any possibility of refresh tokens. The ADAL lib would help with the authentication. After that, all the send/receive/etc. methods in the transport will need to be updated, and there will be some differences there with the new library.

If you didn't want to bring in the ARM ASB lib, you may be able to do all of the managment operations via ASB HTTPs endpoint. I think it would work fine without having to use Azure AD credentials. You would need to parse apart the endpoint=sb://{domain/keyname/key} into something that can be used with an HttpClient which I've done with Regex in the past. I think all the management ops can be done over HTTPs, and that "should" allow a straight port to the ASB .Net Standard lib without alot of underlying changes. Then, once Microsoft updates the .Net Standard lib and puts the management features back in, usage of the HTTP endpoint could be updated to use the features in the lib.

[KennethJakobsen]

You should be able to do it with this workaround (https://github.com/Mossharbor/AzureWorkArounds.ServiceBus) and the official .net standard package

[hulvei3]

Thx for the efforts guys. But we actually moving to RabbitMQ instead as SQL itself doesn't scale for us anymore.

[mookid8000]

@KennethJakobsen that's actually pretty interesting! Did you try it?

[KennethJakobsen]

@mookid8000 No not yet, but actually MS is aiming at june to implement management again, due to popular demand. Unless i misunderstood something. https://github.com/Azure/azure-service-bus-dotnet/issues/65#issuecomment-392999678

[KennethJakobsen]

Seems like management is back on the menu boys! https://github.com/Azure/azure-service-bus-dotnet/pull/481

[mookid8000]

Wow – I hope they accept that PR soon 🤞

[frostebite]

To clarify, without the above PR linked there.

Netcore (2.0) cannot work with rebus.azureservicebus?

I'm getting: Rebus.Injection.ResolutionException : Could not resolve Rebus.AzureServiceBus.AzureServiceBusTransport with decorator depth 0 - registrations: Rebus.Injection.Injectionist+Handler ---- System.IO.FileNotFoundException : Could not load file or assembly 'Microsoft.ServiceBus, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35'. The system cannot find the file specified. at Rebus.Injection.Injectionist.ResolutionContext.Get[TService]()

using servicecollection:

var services = new ServiceCollection();
services.AutoRegisterHandlersFromAssemblyOf<handler>();
services.AddRebus(configure => configure
 .Transport(t => t.UseAzureServiceBus("...", "transportQueuething"));
var provider = services.BuildServiceProvider();
provider.UseRebus();

[frostebite]

@KennethJakobsen @mookid8000 is there a workaround with https://github.com/Mossharbor/AzureWorkArounds.ServiceBus that works?

[KennethJakobsen]

@frostebite are you making a new version of Rebus.AzureSeriviceBus or are you trying to make the existing implentation work?

[mookid8000]

If I understand things correctly - in order to make Rebus.AzureServiceBus work on .NET Core 2, it would be necessary to port the code in Rebus.AzureServiceBus to make it work with Microsoft's new ASB driver, and then make it use the AzureWorkarounds library to carry out the management operations.

As far as I know, noone has taken a stab at doing this yet. If I weren't so bogged down with work at the moment, I would do it myself, but unfortunately this is around number 5 on my list of priorities right now.

[KennethJakobsen]

@mookid8000 I have some vacation coming up in a couple of weeks i'll take a stab at it then if noone else beats me to the punch, i'm just too caught up in work to do it right now.

[mookid8000]

Yeah well, me2 😄 let's see who gets to the vacation first then.

We'd probably better coordinate in here, so we don't end up coding the same things at the same time 😁

[frostebite]

I was just trying to assert that my understanding was correct that the exception I was facing was due to it being netcore 2.0 and that it won't be able to work until the issues being discussed here are fixed.

Sounds like that is the case.

I'm super happy to assign one of my team members to work on this, but we've only just discovered rebus so it would be much better to be able to work with one of you or to be able to bother you with questions.

Any suggestions are welcome :).

Also with regard to the workaround, is it worth doing that or waiting for the PR to land that changes the behaviour in the original microsoft package?

[frostebite]

https://github.com/Azure/azure-service-bus-dotnet/pull/481 @mookid8000 @KennethJakobsen

[mookid8000]

@frostebite let's hope they get that merged and release quickly 🤠

[Meyce]

I was able to get the Rebus.AzureServiceBus.Tests working with the branch of the netstandard ASB driver that contains the new management components. I cloned Microsoft's ASB repo and used the branch with the management components contained in https://github.com/Azure/azure-service-bus-dotnet/pull/481, pulled in as a project reference of a fork of Rebus.AzureServiceBus.

It was just a quick and dirty try to update the transport to work with the new driver so it's a bit messy, but I didn't hit any major issues converting to the new driver, unless the problems I hit with net452 can't be resolved see 6. below.

A few things of not while converting it.

1. Exceptions. The Transport uses exceptions types from the driver to determine transient errors. Some of these exception types no longer exist. Further investigation needed.
2. The send and receive operations are split across different types. The old driver had send/receive operations on the entity client(queueClient/topicClient). Now the entity client only contains the Send operations. A MessageReceiver is used to receive messages. I put the mesage receiver into a concurrent dictionary same as the clients, but I don't think this is necessary as there would only ever be 1 for the input address.
3. The Complete/Abandon/Defer methods are all on the MessageReceiver. they used to be on the BrokeredMessage, so it's necessary to get the message receiver to perform operations on the message.
4. It looks like the message receiver can accept a delegate for handling incoming messages. It may be worthwhile to look into what the message receiver is doing here, as it may allow for simplification of the receive code.
5. All the client/receiver methods are async. I didn't look into changing internal methods to async. I just used GetAwaiter().GetResult() where Rebus was already using the synchronous methods on the old driver.
6. I had some issues targeting net452. I've commented it out for now and am targeting net461/netstandard2.0. Needs further research.
7. Version conflicts with newtonsoft between Rebus using v9 and ASB driver using v10.

[KennethJakobsen]

The Management Operations are now merged in.

[frostebite]

@KennethJakobsen @mookid8000 with regards to the issue with netcore in rebus. Is it looking like a few weeks? Or a few months for that to be resolved, now that the ASB management operations are back?

Thanks

[mookid8000]

My ambition is to take a serious look at these things within the next weeks. I am currently on vacation though, so I can guarantee that I will no do anything about it the next 7-8 days, but after that, there's a good chance that I will do it 😄

[Jeern]

I need this too - so I am cheering for you too @mookid8000 :) Please do send updates on the timeline if possible, BTW. I lost access to the Rebus Slack channel has it been discontinued ?

[mookid8000]

I routinely kick out inactive members 😉 also it’s actually for Rebus Pro subscribers, so you can probably easily imagine a way to gain access again 😜

Sendt fra min iPhone

– Mogens Heller Grabe Torsmark 4 8700 Horsens Danmark

+45 29367077 mookid8000@gmail.com

Den 11. jul. 2018 kl. 11.36 skrev Jeern notifications@github.com:

I need this too - so I am cheering for you too @mookid8000 :) Please do send updates on the timeline if possible, BTW. I lost access to the Rebus Slack channel has it been discontinued ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[Jeern]

That's fair :) We have long periods where we do not develop on Rebus related code at all, and then long periods where we do. I hope we will be active again, and perhaps we will need the pro subscription. BTW. you accidentaly added your contact info to the post. You should probably delete it

[frostebite]

@mookid8000 any news on this front?

[mookid8000]

@frostebite I have time to take a stab at it tomorrow evening.

[mookid8000]

I've started working on implementing the new ASB transport for Rebus here: https://github.com/rebus-org/Rebus.AzureServiceBus/tree/feature/net-core-again

It's based on the new preview package, which is pretty great, because it has a ManagementClient.

In many ways it's simpler to use than the old driver, except for the convoluted receive API – it simply doesn't seem to be possible to receive messages without handing a callback to the driver, which is pretty annoying, since Rebus has always been built with a polling receive API in mind.

I'll have to think some more about this 😄 the transport in its current state can do simple send/receive and pub/sub, but deferred messages, prefetch, + more probably is still missing.

[Meyce]

Did you try it with the message receiver type? It has a general receive method that excepts a timeout. I listed it in my notes from a few weeks ago.

I was able to get this fork working prior the the ASB NuGet package getting released.

https://github.com/Meyce/Rebus.AzureServiceBus/blob/master/Rebus.AzureServiceBus/AzureServiceBus/AzureServiceBusTransport.cs

[KennethJakobsen]

@mookid8000 I have also started this, and encountered the same issues but since you are on it, I’ll do no further work! Happy hunting!

[Meyce]

@mookid8000 I just looked at your branch. The QueueClient no longer has a Receive method. You can use the MessageReceiver type which has .Receive().

[mookid8000]

@Meyce w00t! that changes everything!!

[runes83]

Have any status on this?

[mookid8000]

It's almost complete - you can see the work on the net-core-again branch.

Only thing missing seems to be fixing the test that verifies that the peek lock can be automatically renewed. Haven't had time to really sit down and figure it out, so if anyone feels like taking a look at it, it would be awesome.

[mookid8000]

I just pushed Rebus.AzureServiceBus-6.0.0-b01 to NuGet.org – it supports .NET Standard 2.0 as the only platform, because that's what's supported by Microsoft's new driver...

....and then I updated our Fleet Manager DEV environment to use that, so we'll see how it goes when we get it punished some 😈

If anyone feels like trying it, I would be very happy!

[ctcoulter]

thanks for taking the time, man! so much appreciated. will hopefully have a chance to tinker with it over the next few days

[Meyce]

The Microsoft.Azure.ServiceBus 3.1.0 is out of preview.

[Meyce]

@mookid8000 did you get the test for peek lock/auto renew fixed? I pull you latest down and there weren't any failing tests.

[mookid8000]

@Meyce yes!

[runes83]

To recap, can we now run Rebus with Azure servicebus on .net core (even linux) ?

[mookid8000]

Yes 😎

Sendt fra min iPhone

– Mogens Heller Grabe Torsmark 4 8700 Horsens Danmark

+45 29367077 mookid8000@gmail.com

Den 19. aug. 2018 kl. 13.05 skrev Rune Synnevåg notifications@github.com:

To recap, can we now run Rebus with Azure servicebus on .net core (even linux) ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[mookid8000]

Just released Rebus.AzureServiceBus 6.0.0-b02 which now depends on Microsoft.Azure.ServiceBus 3.1.0

[mookid8000]

so... anyone tried it? (besides me, who's currently running two out of three Fleet Manager environments on the new version...)

[Meyce]

Yes, I've done some testing with it, and I am using it in a service deployed as an Azure WebJob to a couple different environments and have had no issues.

[mpaul31]

@Meyce how are you hosting your endpoint within the WebJob? Topper is not compatible with .net core so are you using the WebJobs SDK?

[Meyce]

@mpaul31, In netcoreapp2.1 there is a generic host builder. Its is a generic version of the WebHostBuilder from aspnet core. The generic builder gives some similar facilities as TopShelf service stop/start, etc. I copied the WebShutdownWatcher class from the Azure WebJobs SDK lib. It's the only thing I needed from the SDK. It just watches the file system for a file to show up on the file system that signals the job should stop/shutdown. I tied that classes cancellation token to the stop functionality of the generic host.

This is irrelevant for this thread, so if you need any additional description, I'll put together a repo to demonstrate all the parts.

[mpaul31]

@Meyce thanks i would love to see a repo when you have time!