Local development story?

[jbogard]

Trying to get an understanding of what the local development story is. RabbitMQ for example can be installed on a dev machine, or run as a Docker container. If I have development/build images that need Azure Service Bus, what's the story of getting a runnable instance locally?

[SeanFeldman]

There's no story Jimmy. Either MSDN or another Azure subscription can be used to run on the real thing.

[jbogard]

So what's the story there then? Each developer creates their own namespace? Picks an option that keeps them under the MSDN limit?

Or runs a shared namespace amongst the team? That would run into all the normal problems of shared dev resources.

[SeanFeldman]

Either way works.

Shared namespace is fine if you can "sandbox" development. I always advise to leverage MSDN subscription, unless you plan to use premium. Then you don't want create that namespace and keep around if not used. Depending on your scenario I'd have other options, but you know, "it depends" 🙂

[andrewbadera]

Or, abstract your service bus layer, and run whatever you want locally?

∞ Andy Badera

On Wed, Aug 15, 2018 at 9:39 AM, Sean Feldman notifications@github.com wrote:

Either way works.

Shared namespace is fine if you can "sandbox" development. I always advise to leverage MSDN subscription, unless you plan to use premium. Then you don't want create that namespace and keep around if not used. Depending on your scenario I'd have other options, but you know, "it depends" 🙂

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Azure/azure-service-bus/issues/223#issuecomment-413218586, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAzzIzJgTGFjCOzulz8FJui2UdFLfuvks5uRDKzgaJpZM4V-M3W .

[SeanFeldman]

Layer of abstraction for code, yes. Running code w/o exercising client/broker, no. When working with ASB, you really should experience the "real thing".

[andrewbadera]

How is an abstraction layer any different? Develop locally against whatever you want, run your automated integration tests against ASB. At a developer level, it shouldn't matter.

∞ Andy Badera

On Wed, Aug 15, 2018 at 10:09 AM, Sean Feldman notifications@github.com wrote:

Layer of abstraction for code, yes. Running code w/o exercising client/broker, no. When working with ASB, you really should experience the "real thing".

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Azure/azure-service-bus/issues/223#issuecomment-413227824, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAzzME33C1xxdJ_xe5-hPCdTyDLsCN8ks5uRDmRgaJpZM4V-M3W .

[SeanFeldman]

@andrewbadera and when integration test (if you have one) against ASB is failing, what are you going to do locally? Wouldn't you switch back to ASB from whatever you were running against? And now you have to deal with the "perks" of two different technologies.

[SeanFeldman]

@andrewbadera don't get me wrong. I am not against layer of abstraction. I'm not a fan of replacing ASB with some other technology to emulate it.

[jbogard]

Yeah, there are real differences between different messaging transports that directly affect development. I wouldn't want to swap for example ASB for "messages are files in a folder"

[mahic]

This question is probably related to this feedback where @SeanFeldman also commented.

We do have the ability to run everyting dotnet core, redis, functions, sql server, storage and cosmosdb locally, but not servicebus, hence the number of votes on the feedback page. And the Service Bus On-Prem option suggested in the feedback doesn't really seem to be a in-sync option

[aidapsibr]

Yeah, just gonna put my voice here as well. No option isn't really a great selling point. This has downstream effects too with EventHubs which are incredibly useful, but at least I can simulate them by using Kafka protocol enabled hubs, but that pushes me away from functions. My option here is to have every dev in my org use their MSDN account to setup dev resources? Very tempting reason to not use Service Bus and go to Rabbit.

[ericsampson]

Thoughts @djrosanova ?

[arjunsol]

It seems like utter nonsense to not be able to develop and run integration tests locally. At present I have to strip out and service bus bindings to test.

[PureKrome]

@arjunsol Can you use RabbitMQ for localhost dev and switch over to ASB for staging/prod ? Absolute PITA but ... a workaround.

[ggirard07]

@PureKrome do you mean for mocking purpose? That mean 2 different implementations to maintain, right?

[PureKrome]

@ggirard07 localhost scenario == rabbitmq is like an emulator. It's not a mock as in .. the code does literally interact with it. It's a dependency. staging/prod == ASB. so again, it's a dependency.

code has some abstraction for suscribe/publish and your application decides to leverage either RabbitMQ or ASB for your IEventBus.

That mean 2 different implementations to maintain, right?

Yes. Which is why i'm screaming internally. But when you have to ship something, you gotta work with the tools around you. So to me, switching between the two is far from ideal BUT it works. I got this from MS's NET-Microservices-Architecture-for-Containerized-NET-Applications-(Microsoft-eBook).

Also, I tried asking ClemensV about this on twitter a few days ago. Either I failed to communicate my question or we have a different opinion, on the matter.

... especially when there's emulators and images for Azure Storage / Sql or NoSql db's / Redis / Elastic Search / etc..

SIDE NOTE: also like how Udi Dahan said they use Sql Server as their EB for local/test with NServiceBus.SqlServer as their transport/too.

cc @jbogard - Can you fill us in, with how you/your team handled this scenario, plz?

[ericsampson]

It would probably be a lot closer to use a localhost AMQP1.0 broker for the mock @PureKrome ? Unless you were already thinking of the AMQP1.0 support in RabbitMQ server. There's another twitter thread discussing this topic here: https://twitter.com/jeremydmiller/status/1080496980168712192

[PureKrome]

It would probably be a lot closer to use a localhost AMQP1.0 broker

Um ... er .. all i've researched is using RabbitMQ and have no idea (yet) what this means with resepct to protocols/etc. All this time spent is being wasted (unfortunatly) getting the tooling and abstractions to work ... just to get a localhost dev story up-n-running. Man - this is 2019 and this should be a solved problem already :( These tools have been around for a while now. So much frustration :(

All I know is this:

• We wish to leverage ASB for production. Code to do use ASB is nice and easy.
• Do not wish to use ASB for localhost. No, we're not a plane. Now we need to use RabbitMQ (docker image + does same high level functionality as ASB). Now we also need to abstract over ASB and RMQ.

so - frustrating :(

[arjunsol]

@PureKrome

I had thought of that, we're using FunctionMonkey to cut down on boiler plate. I'm unsure if I can make it play nicely.

https://functionmonkey.azurefromthetrenches.com/

[sampbarber]

It's possible to trigger a service bus trigger via http using the /admin route, (https://docs.microsoft.com/en-us/azure/azure-functions/functions-manually-run-non-http) though I haven't quite got the hang of serializing to a BrokeredMessage, this would be step 1 for me, as I could then just trigger my function via postman when running locally.

The second issue is that if you have a service bus triggered function and you don't provide a connection string the trigger will blow up. If there was a way round this that would be great. Maybe being able to switch out the type of trigger via configuration or suppressing the exception, but not sure if that's possible. Anyone else find a way of doing that?

[atrauzzi]

[My accompanying Twitter thread.](https://twitter.com/Omega/status/1129851475864182785)_

Back in 2016/2017, I looked into a local emulation option for service bus as the company I was working at was in a position to need something like it. Today, I find myself in a similar situation again as we don't have the finances to waste, arbitrarily provisioning cloud resources for each developer and need a consistent and repeatable local development experience.

I didn't really get into it much back then, but feeling a bit more motivated now, I'm sadly seeing things that are a bit off-putting about how badly the message is being missed. I'm going to recount my experience on this, I realize some of it will be reiterating references already mentioned. But by the end, my hope is that you'll understand why some could be frustrated by how this is being handled, how inconsistent it is and how false and even dismissive some of the official responses are coming off...

---

The first result most tend to find is this stackoverflow question which has an answer that's been accepted.

The answer points to this feedback forum question from 2012 with at the time of this comment, 149 votes. What galls me about the response is its sheer combined temerity and inaccuracy:

It has been seven years since 2012. The interest on this uservoice is strong enough to warrant further investigation, yet here's a followup with 68 votes of its own.

---

Today, I decided to ask again after taking the time to do a bit of light research, collating the signals actual users and the community have been trying to send to the service bus team.

The response?

Similar to a prior poster in this thread, when @clemensv was paged into the discussion by @cmatskas, the response was unhelpful and really, seemed to talk past me. It's a little less great in the prior thread from @PureKrome which even seems to get a bit condescending...

While I disagree with this quip on the basis of my own use case, all that can be considered beside the point - it again talks right past the concerns on the basis of a singular ideal.

I'm not sure what the service bus team is waiting for on this one or what would qualify as sufficient. Certainly, a lot has been done to dilute, defer, confuse or even outright dismiss demand for this feature from multiple inquiries fielded seemingly out of hand in multiple forums.

From what I'm reading, I think when it comes to a local service bus emulator, it's definitely at a point that warrants action. If it really and truly is the worst idea possible in this context, do us all a favour and put a clear and official explanation in the service bus documentation as to why we shouldn't want a local emulator and what our alternative should be on Azure, specifically for topics.

Finally, I also think the responses so far are inconsistent both as they pertain to my perception of a modern-Microsoft, but also what many people value as a first class, competitive option.

[glucaci]

"No local dev for anything of serious complexity"... hmm... Kafka can be hosted in a container.

Maybe can someone give us a technical explanation why this is not possible, or maybe they want to be so.

[atrauzzi]

A nice little update today...

Sending good thoughts in the hopes that they find a way to make this happen soon!

[abezverkov]

This is the 3rd time/company have been looking in to this very thing. How to do local development and integration testing with ASB. Every other part of the stack can be run locally in a container, except this. I would be great to get something like Azurite that is API compatible and containerized.

[glucaci]

Until then you can try this https://github.com/SwissLife-OSS/squadron

[open-collar]

It would be very useful to have this when working offline or as part of our unit testing/integration testing systems.

[mgressman]

This just amazes me.

I don't know how many times I have been in a disconnected situation (e.g. 14 hour plane flight overseas) where I would love to get some of my development work done but can't because somebody, somewhere decides to take that kind of a stance.

Can I work around it or on something besides the ASB part? Sure. But why should I have to be told what I can and can't work on based on my connected status.

[gayankanishka]

It would be really really helpful to have an Azure service bus emulator.

[afelipems]

I came across this problem while trying to create a cloud independent test environment. This really gets in our way as a software developer, and I think using RabbitMQ is not a good alternative, considering that the idea of local testing is to simulate a real scenario. Anyway, I hope a solution arrives soon!

[ChristianJensen]

Does anyone know if this has received any traction with MS?

[malandles]

Does anyone know if this has received any traction with MS?

Looks like it's still in the internal backlog of Microsoft. Replied to the linked twitter status in hope to get this revived. Testing locally is a must these days.

Clemens Vasters asked back in August which language and APIs that were primarily used by the community.

[tcfialho]

This is why we don't choose Azure but AWS with localstack.

[open-collar]

This is definitely causing me problems in offline development and automated integration testing. It doesn't need to be fully featured or especially clever, just enough to test that messages are sent a received in the context of a single host.

[n1l]

The profit of Functional/Specification/Integration testing is really REALLY huge, so such way of testing is very important. Sometimes you have no time to do manual testing, you should release quickly and find issues asap, resolving problems on a fly like a ninja!

And this is true for lots of enterprise solutions with serious complexity.

It would be AWESOME to have an approach to test some code which invokes service bus queues and topic without real service bus.

[glucaci]

Dose anyone from Microsoft tracking this issue? @ChristianWolf42 @axisc @clemensv @microsoftopensource

[kfallstrom]

this is a BIG priority for our group.

[PureKrome]

Whatever the outcome of this ends up being, a simple reply from MS here with some type of status update would be nice/respectful to the developer community.

No information is usually interpreted as: nothing happening (for whatever reasons).

So please drop in with a status update please, even if it is "nothing happening, move along.". At least we all know where we stand.

[atrauzzi]

I think that's reasonable. If only because this issue continues to get attention, again, you can't always measure demand directly.

But as we can see, people want this, they're just not always up for the fight to advocate.

[AshleyPoole]

This is a big priority for myself too. This is much needed for development purposes, being able to run this in a container especially is import for local development such as you can do for Cosmos.

Having to setup infrastructure just for local development and testing is far from ideal.

[keithdarragh]

Currently running into this issue as well. I find it unbelievable that their is no solution to run a local/simulated instance of azure service bus locally for testing and development purposes.

Is this currently on any road map to produce?

[vitalybibikov]

For now, we are creating Service Bus queues/topics/subscriptions for each developer , where his own queues/topics start with his machine name: e.g. vbibikov.users.topic. All the elements are located in a common shared dev namespace.

They are created on the fly during Startup (if env == dev), By convention:

• the service that is responsible for writing - creates queue/topic
• the service that is listening a subscription of a topic - creates it.

Also, in order to have buildmachine name in a docker image, we have to set hostsname variable which equals developers machine name.

Thus in a docker-compose.override file we have:

  'HOST_HOSTNAME': ${HOSTNAME}  (which is vbibikov)
  'ServiceBusSettings__ConnectionString': '%%'
  'ServiceBusSettings__SomeQueueName': ${HOSTNAME}.project.queuename.queue

The solution requires: 1) To Create a script that populates ServiceBus namespace if we have a new developer. 2) To setup HOSTNAME variable (as docker machinanme is not readable e.g. 02935a4a5540) 3) Read and setup this variable during docker-compose up.

Kind bulky, those I definitely agree, that having an emulator/local installation will be a thing.

[atrauzzi]

@vitalybibikov - It's probably better for the sake of this topic to not get into specifics about workarounds. Especially ones that go against the main complaints which are that the only current option is to use live cloud resources.

It's just too easy to get derailed and then people joining the discussion will get lost in a thread of people talking about exactly what they didn't come here for...

[vitalybibikov]

I agree, to a point, but I guess, to write a workarounds to solve a problem is what an engineer should do.

Currently, this topic is more like a thread of "we need this feature because rabbit/Kafka have a solution" instead of "why do we need it".

[atrauzzi]

@vitalybibikov - I've had this exact conversation in other threads where they eventually get derailed by people posting the same or similar workarounds over and over. To the point that the original thread is completely lost in a sea of information that is irrelevant to the main point.

The goal here is to make it easy to understand what people want (a first party, official emulator). Not bury the demand with convenient excuses to put it off any longer than it already has.

[PureKrome]

totally what he says, above!

[SeanFeldman]

It is not my intent to derail this thread but to add some food for thought.

The goal here is to make it easy to understand what people want (a first party, official emulator). Not bury the demand with convenient excuses to put it off any longer than it already has.

On every person that is asking for an emulator, there is N number of customers, and I'll defer to the readers to decide what that N is, that work w/o emulator for years. And run in production. And don't feel the need in it. And they are not vocal about it, defending not need emulation. Personally, I don't find the idea appealing not because I want to prove a point, but because I used to think that I absolutely needed it until I learned I didn't.

I'm curious how many of those that thumbed up the request has gone to the product team to understand what it entails to provide an emulation of this specific service? Are you willing to sacrifice features such as JMS integration, cross-namespace forwarding, data plane DR, and many other features collecting dust in the backlog in favour of being able to run an emulator locally? If you are, then perhaps ASB is not the service that you need? Maybe, just maybe, Storage Queues would suffice? Which, BTW, has an emulator.

Long story short, the issue is here because some developers feel the need. It's legitimate.

[PureKrome]

and some potentially more info, from last night:

REF: Twitter link.

[FBryant87]

So have we entered 2020 yet or do we still think local integration tests don't offer much?

[atrauzzi]

@SeanFeldman - Just because you don't want it doesn't mean other people shouldn't.

They aren't choosing the wrong service, there's simply a high value in running services locally during development.

I put in the effort to correlate all of this, many people have spoken, I think there's no point in trying to disprove the merit or characterize people / their choices for wanting it.

[atrauzzi]

@PureKrome - Thanks for the link, I'll echo my reaction on the local dev remarks from twitter here:

I'm a little dismayed at what I heard about local development during the video. It seemed like the dismissive responses I thought we had moved beyond last year were being trotted out again. I worry that all we've managed to accomplish is convincing @clemensv to better hide his disdain for local development.

I don't want this to be tactical or political, so, hopefully for the last time in 2020 - and hoping we don't have to do so again in 2021 this is for you @clemensv:

Everyone knows "how cheap and easy" it is to spin up a cloud resource, this is not the sole driver of the request for a local emulator. Network connectivity isn't the sole driving purpose. No one single factor is the symptom to aim a silver bullet argument at.

Local development is the acceptance that you can't anticipate everything and that there can be no silver bullets. Only the sound principle that technology should be prototype-able from day zero on the developers terms.

No matter what you think, this isn't just about people getting an itch scratched. This can even help you build a better product by giving it a competing/parallel implementation. Lots of good can come of putting a priority on this.