Confirm MSMQ transport works as expected in Windows 2016 Container deployments

[boblangley]

Windows 2016 introduces containers:

They are an isolated, resource controlled, and portable operating environment.

Basically, a container is an isolated place where an application can run without affecting the rest of the system and without the system affecting the application. Containers are the next evolution in virtualization.

If you were inside a container, it would look very much like you were inside a freshly installed physical computer or a virtual machine. And, to Docker, a Windows Server Container can be managed in the same way as any other container.

We should confirm that MSMQ, as we have implemented it using the .NET framework, does not work in a containerized deployment.

[timbussmann]

We should confirm that MSMQ, as we have implemented it using the .NET framework, does not work in a containerized deployment.

@boblangley that seems to contradict with the issue title?

[andreasohlund]

@bording I know that you where interested in this. Assigned it to you (just unassigned if I misunderstood)

[bording]

I've been playing around with windows containers a bit, and so far the answer to the question about will MSMQ work is: No.

The blocker so far is the with the current version of the Microsoft-provided base image, microsoft/windowsservercore, the commands to install MSMQ fail with an error. I've tried both Install-WindowsFeature from powershell and directly calling into DISM as mentioned in the docs.

I have gotten an endpoint using the RabbitMQ transport working just fine in a container, so in general it seems like NSB is going to work.

[MarcinHoppe]

Is this something we can use Microsoft support or personal connections of our MVPs to check if this is by design or growing pains of Windows containers?

[bording]

Is this something we can use Microsoft support or personal connections of our MVPs to check if this is by design or growing pains of Windows containers?

If we have some avenue to talk to Microsoft directly about it, I'd be happy to keep pushing this forward. I'm just not aware of what our options might be.

[MarcinHoppe]

I think each MSDN subscription has some number of support tickets we can use. Also a shout on the engineering Slack channel might help find somebody with a connection within MSFT we can potentially use.

[bording]

I'm in the process of activating my support options that come through the MSDN subscription, so lets see how that goes.

[bording]

It took a while, but I finally got through the Microsoft support process, and the conclusion is that MSMQ is not currently supported inside a windows container. They've suggested posting to UserVoice requesting that it be considered.

Microsoft sent me this link, where it looks like someone using NSB already has requested it: https://windowsserver.uservoice.com/forums/304624-containers/suggestions/15719031-create-base-container-image-with-msmq-server

We could post on there with additional details, an upvote it as well.

BTW, when I asked if there was documentation for what was and wasn't supported inside a container, I was told that there was no such documentation, but he took it as a suggestion for the future, though he said it might be difficult to create it. He'd have to talk to every team to discover if their feature was container-aware apparently.

[SeanFeldman]

Thinking out loud - MSFT strategy is to run on cloud ready server OS. Having container support for MSMQ doesn't seem to align with that strategy.

[timbussmann]

@bording thanks for fighting through the MS support process 👍 I think this should answer the questions raised in this issue. @boblangley are you able to feed back Brandon's results into the pdev issue and close this one?

[boblangley]

Updated POA for the pdev issue based on these findings. Closing this issue. Thanks all!

[TheRealKevykev]

This is totally random and a shot in the dark, but does anyone on this thread have an idea about how to automatically read an MSMQ receiving (InQueue) queue automatically with a WCF Service application hosted in IIS 7 or 8?

[dbelcham]

@TheRealKevykev The closest thing we have for that is this: https://docs.particular.net/nservicebus/msmq/operations-scripting

Since the question you're asking isn't related to NServiceBus any followup questions you have should be directed at StackOverflow.

[fourpastmidnight]

So, I wanted to comment on this, because as of January of this year, MS has released an insiders preview build of Windows Server 2016 Core that now includes support for installing MSMQ in a containerized environment--woot!

[bording]

Thanks for the update @fourpastmidnight! That definitely makes Windows containers viable for the MSMQ transport as well.

[fourpastmidnight]

So the one major question I have with using MSMQ in a container is: what about message storage? It's my understanding that with docker (specifically) that any changes made to the container "filesystem" happen on a separate read/write layer which is layered on top of the container image. Furthermore, if the container is stopped, any changes made in this layer are discarded.

So, going with the "containers are cattle" mentality, ordinarily I shouldn't care if a container were to (catastrophically) fail--I'd just spin up a new one. But with MSMQ, there could be unprocessed messages in the queue and a container all of a sudden shutting down could result in message loss.

Any thoughts or corrections to my thinking?

[dbelcham]

@fourpastmidnight I've not looked at this yet, but here's my guess. You'll first need to make the queues persistent. Then you'll need to set the storage location for those queues (manually or via script if that's possible) to be a folder that has been mapped the host OS's file system using the volume-v parameter when doing docker run ....

This would work great if you were guaranteed to restart the container running MSMQ on the same host OS instance. It does seem to fly in the face of "start and run the container anywhere, anytime" though.

There might be other ways of doing this too, but this jumps out as the most likely for me. Again, I haven't looked at this so while all those configuration options are valid independently, they would need to be tested working together and with NServiceBus.

[fourpastmidnight]

Hmm, yeah, that won't work (changing storage location via script), per a blog article I read (which I meant to link to above but accidentally wiped out during editing). Changing the storage location via script can lead to a broken MSMQ installation. (However, that blog article was from 2009--but then again, I don't think MSMQ has seen that many changes over the years either, so....)

So, this is looking not so container friendly, even if you can get the feature installed on a containerized Windows Server.... Maybe the volume mapping might work, but you would need to make sure there's no storage disruption--but if the volume is located on the container's host, it might work. Hmmm.....

[fourpastmidnight]

Hmm, your comment also made me realize, this would never work in a Kubernetes pod..... Ugh. :) OH wait, forgot about Kubernetes services, so, assuming the storage problem can be resolved, it may work.

Of course, the real answer may be as simple as, don't use MSMQ. ;). Not a small pill to swallow for some, but this may be the best option in terms of container-friendliness, simplicity, and platform independence.

[dbelcham]

Kubernetes: yah, I thought of that while writing my response but forgot to include it. Probably the same problem with AWS ECS too.

You might not have to change the MSMQ storage location in my original idea. I'd have to try it to see if it works, but you could try to volume map the default queue storage location. Not sure how MSMQ would react to that though

[fourpastmidnight]

Just wanted to post that link to the 2009 blog article: Changing the MSMQ Storage location.

[fourpastmidnight]

I've been fooling around with this again lately and have had some limited success. Using the latest Windows Server Core LTSC 2019 image, I have been able to get MSMQ installed in a container, as MSFT promised.

Now, their articles about being able to run MSMQ in a container detail a very limited scope at which they've tested this. Basically, they tested they could get the feature installed and running, and that a sender and receiver in the same container can communicate with one another via MSMQ. So, great, you can send/receive messages in the same container, but this is far from the normal use-case (which they admit).

So, I'm taking it a step further. They also have a Cmdlet: Set-MsmqQueueManager that now allows you to change the storage location for MSMQ via the command line. A few years back, there were many complaints that, while this cmdlet moved the storage, the permissions were setup wrong. Well, this is now resolved.

Also resolved in WSC LTSC2019 is the issue with mapping volumes to Docker images where the destination in the container already has files. BUT, this doesn't help us with MSMQ, as when you do this, the original files in the destination are "hidden" (ok, more technically correct--overlayed) by the volume on the container host. So now, the admin_queue$ queue is gone and MSMQ refuses to start thinking its installation is corrupt.

So it's not as easy as "moving the storage", unfortunately. I have figured out a way around this so that, at least you can map volumes to the container in which to store the MSMQ-related files. It's a bit kludgy, but it works. So, while I've managed to get a Windows Server Core container running MSMQ with persistence across Docker container restarts, I have yet to determine whether or not a message can be sent from one endpoint in one container, to another endpoint in a different container. That's next--well, after I get ServiceControl up and running in a container (using MSMQ as its transport).

[fourpastmidnight]

Today, I successfully started up an NSB Endpoint, a ServiceControl instance, and a SQL Server Database, each in their own container, with the Endpoint and ServiceControl using MSMQ as their transport. I then opened ServiceInsight and put in the URL for the containerized ServiceControl instance and I saw my endpoint in the Endpoint Explorer pane!

So this means that it's totally possible to use NService Bus with MSMQ in Windows Docker Containers--which is great for brown-field projects that aren't necessarily looking to switch transports in the very near future. This was all done using Windows Server 2019 LTSC Windows Container images. (I don't believe it's possible with Windows Server 2016 since MS didn't quite have things working wrt MSMQ and Docker in Windows Server 2016.)