Add Mtom support in WCF runtime

[chris5287]

I am getting the following error when using the Visual Studio WCF Connected Service extension for Visual Studio 2017:

Scaffolding Code ...
Error:Warning: The following Policy Assertions were not Imported:   XPath://wsdl:definitions[@targetNamespace='http://imaging.oracle/']/wsdl:binding[@name='DocumentContentServicePortBinding']
  Assertions:
    <ns1:OptimizedMimeSerialization xmlns:ns1='http://schemas.xmlsoap.org/ws/2004/09/policy/optimizedmimeserialization'>..</ns1:OptimizedMimeSerialization>
Warning: Unsupported binding element type: 'System.ServiceModel.Channels.UnrecognizedAssertionsBindingElement'.
Warning: Unsupported message encoding element type: 'System.ServiceModel.Channels.MtomMessageEncodingBindingElement'. It must be one of the following types: 'System.ServiceModel.Channels.BinaryMessageEncodingBindingElement', 'System.ServiceModel.Channels.TextMessageEncodingBindingElement.'
Warning: Endpoint 'DocumentContentServicePort' at address 'http://localhost/imaging/ws/DocumentContentService' contains one or more bindings not compatible with .Net Core apps, skipping...
Error: No endpoints compatible with .Net Core apps were found.
Failed to generate service reference.

The WSDL/XSD file causing the issue: DocumentContentService.xsd.txt DocumentContentService.wsdl.txt

dotnet --info

.NET Command Line Tools (1.0.0)

Product Information:
 Version:            1.0.0
 Commit SHA-1 hash:  e53429feb4

Runtime Environment:
 OS Name:     Windows
 OS Version:  6.1.7601
 OS Platform: Windows
 RID:         win7-x64
 Base Path:   C:\Program Files\dotnet\sdk\1.0.0

[brux88]

@Lxiamail please any update on the mtom support

[Lxiamail]

Unfortunately, we don't have any further update on this. We also considered to define a spec so that community may be able to help to do the implementation. However, as Matt commented above, the primary complexity is redesign the feature on .NET Core. The investigation cost to define the spec will be high that we can't afford at this point.

[felipecaputo]

@Lxiamail, if we analise and make a spec proposal, the team would take a look and evaluate it?

I think that, if community work togeter and we make a good spec, maybe it could be good for both!

[mconnew]

Hi @felipecaputo. Having the community implement MTOM would be great. In the next couple of days I'll write up a quick summary of my thoughts on how to bring in MTOM support including some potential design ideas I've had for a whiile to improve interop with other stacks, and then you can come back with a spec proposal. Does that sound like it would work for you?

[felipecaputo]

That would be great.

I'll wait for these ideas, and take a look at the source and documentation to get ready.

Thank you @mconnew

[kfrancis]

Adding a specific chat room for us to discuss: https://gitter.im/wcf-mtom/community?utm_source=share-link&utm_medium=link&utm_campaign=share-link

[jducobu]

https://www.w3.org/TR/soap12-mtom/

[mconnew]

So here's my brain dump on the topic of MTOM. I've had two strong goals when porting WCF to .NET Core.

Make code simpler
Internally we've mostly dropped APM implementations and everything is Task based. The sync code path has also been dropped and just calls the Task based API's.

Own less code
WCF on .NET Framework was developed when a lot of what is now taken for granted as existing in the framework just didn't exist so there's a lot of wheels which were invented which have been reinvented and now available generally in the rest of the framework. Nobody went back and removed the WCF implementation and switched to the general one. So the .NET Framework implementation is a lot bigger than it needs to be.

There's some problems with the MTOM implementation which I've become aware of from customers over the years while working on .NET Framework. There are two I'd like to address in a .NET Core implementation. The first one is lack of control in which elements get moved out to separate parts in the HTML multi-part message and which ones get serialized as part of the XML body. The MTOM spec is not opinionated on this and WCF has a heuristic to make this choice. There are some other SOAP stacks which don't like WCF's heuristic. So we should add a mechanism to override the heuristic decision for individual elements to improve interop capability with other stacks. The second problem is the .NET Framework implementation is not extendable through either derivation or encapsulation. The behavior needs to be modifiable by either encapsulation or derivation as there have been multiple issues developers have had over the years which could have been easily addressed if the MTOM encoder behavior could be modified in one of these ways.

The .NET Core implementation of HttpTransportBindingElement is almost a rewrite of the code from .NET Framework. This was needed due to HttpWebRequest not being available (there's a class there now, but it has many undesirable behaviors like forced buffering of the request and isn't the real thing). So the transport was redone using HttpClient. The HttpRequestMessage content is a custom type System.ServiceModel.Channels.MessageContent in order to do things the HttpClient way. Conceptually it lazily does the work of encoding the message when HttpClient is ready to send the body. The reason for this is streamed transfer mode prohibits creating the body up from.
There is a class System.Net.Http.MultipartContent which looks like a good candidate for use with MTOM. This is the principal of "own less code". There are going to be lots of edge cases in the implementation of Multi-part entity bodies and I prefer if those edge cases were somebody else's problem :). Building on top of MultipartContent will simplify the code a lot. The class also has been written in a way to be convenient to derive so we might want to consider making an MtomMessageContent class which derives from MultipartContent. That might not be necessary though as it's a container of a list of innner HttpContent objects and this inner objects can generate their byte stream lazily.

[lennykean]

I developed a workaround to support MTOM in .net core wcf. https://github.com/lennykean/WcfCoreMtomEncoder

This is really a workaround that is only meant to consume existing mtom soap services, but I think it could evolve into a full implementation of mtom in .net core.

[mconnew]

@lennykean, this looks like a great start. I've had a look at the code behind the ReadAsMultipartAsync method and it has one really big flaw. It reads the entire contents into memory. The typical usage I've seen with the MTOM encoder has been when needing to transfer large amounts of data over a SOAP service. Normally SOAP will encode a byte[] as base64 encoded string which increases it's size by ~30%. This is a lot when transferring 200MB of data. MTOM doesn't really gain you much unless you are sending large amounts of data. It might be possible to use the building blocks in the Microsoft.AspNet.WebApi.Client package to build something better. Basically an implementation of ReadAsMultipartAsync needs to lazily create each of the contained HttpContent objects. The HttpContent parts also need to return a stream from ReadAsStreamAsync which isn't backed by an in memory buffer but read from the actual network stream and fake an EOF once the part has been completely read.
What you've provided is great if you aren't returning a large amount of data, and the limitations are mainly in the Microsoft.AspNet.WebApi.Client code.
Edit: I just asked a colleague on the ASP.NET Core team if there was a nicer implementation and he pointed me to this. You can see some sample usage here. It looks like it wouldn't take a lot of work to switch to using that implementation which solves these problems.

[matthiaslischka]

Any updates or reasonable workarounds for big data streaming via mtom? We need to stream-download big files via SOAP mtom in dotnet core.

[RealHabix]

zhenlan commented on Sep 28, 2017 Not in a plan at this point. Please upvote if this is important. It will be helpful for future planning.

How much more upvote and time do you guys need (since it's in backlog for 2+ years)?

[DiegoAndes]

2020 and we still don't have the implementation :'(

[craffael]

I successfully used https://github.com/lennykean/WcfCoreMtomEncoder to read mtom messages. However for writing it doesn't work yet...

[matthiaslischka]

This was already mentioned above. Keep in mind that it does not support streaming, the main advantage of MTOM.

[xtrem14]

I developed a workaround to support MTOM in .net core wcf. https://github.com/lennykean/WcfCoreMtomEncoder

This is really a workaround that is only meant to consume existing mtom soap services, but I think it could evolve into a full implementation of mtom in .net core.

Thank's for this. May not working with HttpsTransportBindingElement :/

when will MTOM support...

[LokiMidgard]

@xtrem14 I have it working with https without problems.

See Client service missmatch issue.

[xtrem14]

@LokiMidgard thanks for your reply.

I initialized the version of the messages in Soap11. I now get a whole other error: A security error was encountered when verifying the message

at System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRuntime operation, ProxyRpc& rpc) at System.ServiceModel.Channels.ServiceChannel.EndCall(String action, Object[] outs, IAsyncResult result) at System.ServiceModel.Channels.ServiceChannelProxy.TaskCreator.<>c__DisplayClass1_0.<CreateGenericTask>b__0(IAsyncResult asyncResult) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter1.GetResult()`

I admit that I don't really understand this error ...

[LokiMidgard]

@xtrem14 Unforuntaly For me it just worked that way and I havn't worked with it in some time. According to a stackoverflow question you may find a more specific error in the event log.

[xtrem14]

@LokiMidgard i found my problem, this was due to a WS security problem. Also not supported by WcfCore.

I solved this one thanks to this solution: https://bbrauns1.wordpress.com/2019/11/07/how-to-add-soap-ws-security-usernametoken-profile-to-net-core-3-0-client/

[mconnew]

@xtrem14, we do support usernametoken profile. If you use WSHttpBinding with security mode TransportWithMessageCredentials and client credential mode of Username, it should all work. You might need to turn off establish security context if the remote endpoint isn't using WS-SecureConversation.

[xtrem14]

@mconnew, i had actually considered this solution, but WSHttpBinding does not seem to be available in .NET Core : https://docs.microsoft.com/fr-fr/dotnet/api/system.servicemodel.wshttpbinding?view=netframework-4.8&viewFallbackFrom=netcore-3.1

[mconnew]

@xtrem14, that's a documentation issue. It's definitely there.

Edit: Actually, there isn't a documentation issue. WCF is part of platform extensions. If you check the docs here: https://docs.microsoft.com/dotnet/api/system.servicemodel.wshttpbinding?view=dotnet-plat-ext-3.1 you should see it's available.

[mjehle82]

Is there a time schedule for the MTOM support? My project will run for the next two years. It would help us decide which way to go.

[yalin78]

Is MTOM supported with .net core now? When is it going to be available?

[ericnewton76]

So I thought I'd mention this here, since I had to spend an hour or so fully understanding whats going on:

WCF for Net Core made an architectural decision without realizing it prevented MTOM encoding efficiently. WCF for Net Core team do not want to rearchitect WCF for Net Core. WCF for Net Core is now recommending gRPC instead.

https://docs.microsoft.com/en-us/dotnet/architecture/grpc-for-wcf-developers/why-grpc

Do I have this right?

[ericnewton76]

Also of note, an external mtom encoder was created: https://github.com/lennykean/WcfCoreMtomEncoder

However, I must stress that due to the architecture limits placed by WCF for Net Core, this WcfCoreMtomEncoder is VERY MEMORY INTENSIVE (according to users of its current implementation, I havent fully examined it myself yet). Meaning, it has to keep the whole contents in memory before it can emit a valid soap message to the client, which is a suboptimal but workable way to accomplish this.

[RealHabix]

So I thought I'd mention this here, since I had to spend an hour or so fully understanding whats going on:

WCF for Net Core made an architectural decision without realizing it prevented MTOM encoding efficiently. WCF for Net Core team do not want to rearchitect WCF for Net Core. WCF for Net Core is now recommending gRPC instead.

https://docs.microsoft.com/en-us/dotnet/architecture/grpc-for-wcf-developers/why-grpc

Do I have this right?

Yes, yes, yes. But what should we do about legacy services that require MTOM - we use plenty and they won't be updated any time soon. Also the WcfCoreMtomEncoder doesn't support streams. Only useful in special cases.

WCF team let us and our customers down big time. We had to switch back to .net framework in many cases.

[mdora7]

I developed a workaround to support MTOM in .net core wcf. https://github.com/lennykean/WcfCoreMtomEncoder

This is really a workaround that is only meant to consume existing mtom soap services, but I think it could evolve into a full implementation of mtom in .net core.

As far as I understand, I am thinking this just manipulate response part therefore it not possible send request with mtom coding. I already tried in code but message still going with soap+xml. You already said but I wanna make notice to those who read this. Also thanks for library

[nalincb]

Recently we wanted to upload files to a WCF service (which used MTOM encoding) from a .NET Core 3.1 client. Below is what worked for us at the end. Hope this will help someone.

var textMessageEncodingBindingElement = new TextMessageEncodingBindingElement
                    {
                        MessageVersion = MessageVersion.Soap11,
                        ReaderQuotas =  { ... }
                    };

var mtomMessageEncoderBindingElement = new MtomMessageEncoderBindingElement(textMessageEncodingBindingElement);
var transportBindingElement = new HttpTransportBindingElement { ... };

var binding = new CustomBinding(mtomMessageEncoderBindingElement, transportBindingElement);
var channelFactory = new ChannelFactory(binding, new EndpointAddress("endpoint"));
var channel = channelFactory.CreateChannel()

[andersonaap]

@nalincb, should it be?

var binding = new CustomBinding(mtomMessageEncoderBindingElement, transportBindingElement);

it works for me.

[nalincb]

@andersonaap , yes that was a mistake and I updated code. Cheers.

[ghost]

Is there ETA on this one? We are trying to do some long term planning and are keen to move to .NET 5, but this is looking like the only impediment. Changing the server is not an option since its a government service that we consume and have no control over. Issue is three years old and made 'Priority 1' nine months ago, but it looks like there has be no traction in that time. Is Microsoft committing to this piece of work or still undecided?

[RealHabix]

Is there ETA on this one? We are trying to do some long term planning and are keen to move to .NET 5, but this is looking like the only impediment. Changing the server is not an option since its a government service that we consume and have no control over. Issue is three years old and made 'Priority 1' nine months ago, but it looks like there has be no traction in that time. Is Microsoft committing to this piece of work or still undecided?

Milestone 6.0

[mconnew]

As @RealHabix pointed out, the plan is to have this work completed for the release which aligns with the .NET 6 release. Note that we still only depend on netstandard 2.0 so we won't have a requirement of .NET 6 to use it. All currently supported versions of .NET/.NET Core support netstandard 2.0 so there shouldn't be forced to update your application to use it. This will continue to be our plan until there's a feature that requires .NET 6.

[NicholasNoise]

6.0 Preview 2 is already here, but XmlDictionaryReader.CreateMtomReader still throws exception. Any news?

[tclopes]

Any news about supporting MTOM?

[mconnew]

Good news, Mtom support has now been posted as a PR!!

[chempro]

This is great, nice wok Is there a rough ETA on release date?

[mconnew]

It will be available in the upcoming RC release for .NET 6. Just a reminder, WCF doesn't require .NET 6 to use our latest packages and until we have a reason to do so, our latest packages will continue to work on all supported version of .NET. We don't have a lot of code churn in the WCF client code, it's mostly adding stuff so you can have a high level of confidence in the stability of pre-release versions.

[bcallaghan-et]

I have installed System.ServiceModel.Http version 4.9.0-rc1.21431.2 to experiment with the new MTOM feature, but I cannot instantiate the MtomMessageEncodingBindingElement to use with my CustomBinding. Instead, I get the following compiler error

The type of namespace name 'MtomMessageEncodingBindingElement' could not be found (are you missing a using directive or an assembly reference?)

I have ensured that all of my NuGet packages are up-to-date (all ServiceModel packages are 4.9.0-rc1, clean, rebuild). Using dotPeek, I can open the System.Private.ServiceModel assembly and see the binding element there, in the namespace I expected. I've double-checked the source code to verify that the class is public. What am I missing?

[mconnew]

@bcallaghan-et, apparently I forgot to add the MtomMessageEncodingBindingElement to the reference assembly itself. You can use it by specifying it's usage with any of the various bindings. For example, BasicHttpBinding has a MessageEncoding property that allows you to specify MTOM. If you need a standalone binding element, you should be able to extract an instance from a binding and set the MessageVersion to be what you need. So for example:

var binding = new BasicHttpBinding { MessageEncoding = WSMessageEncoding.Mtom };
MessageEncodingBindingElement mebe = binding.CreateBindingElements().Find<MessageEncodingBindingElement>();
mebe.MessageVersion = MessageVersion.Soap12;

We'll get it added to the reference assembly before the final release.

[bcallaghan-et]

@mconnew Thank you for the workaround. I need to use MTOM with a custom binding, as the remote service requires both Basic auth and client certificates, and I couldn't find a way to do that with the BasicHttpsBinding. This issue isn't as urgent as I thought, so I can wait for the final release.

[NicholasNoise]

But still:

System.PlatformNotSupportedException
The Message Transmission Optimization Mechanism (MTOM) message encoding is not supported on this platform.
   at System.Xml.XmlDictionaryReader.CreateMtomReader(Stream stream, Encoding[] encodings, String contentType, XmlDictionaryReaderQuotas quotas, Int32 maxBufferSize, OnXmlDictionaryReaderClose onClose)
   at System.Xml.XmlDictionaryReader.CreateMtomReader(Stream stream, Encoding encoding, XmlDictionaryReaderQuotas quotas)

How do I enable mtom encoding?

[mconnew]

@NicholasNoise, because the WCF packages work on any supported .NET Core/.NET release and we don't want to force upgrading to .NET 6, the MtomReader implementation has been built in to the WCF packages themselves instead of calling XmlDictionary[Reader|Writer].CreateMtomReader. This also made sense as MTOM is a SOAP only protocol so including it in the runtime and having everyone pay the cost didn't seem like the correct way to do things. This means XmlDictionaryReader.CreateMtomReader will still continue to throw an exception, but you can configure WCF to use MTOM just fine.
What are you doing to trigger that call stack?

[NicholasNoise]

@mconnew I do not use WCF itself, but to parse MTOM responses. So code that has been valid using netfx is no longer the correct way to do things. Is a workaround here for me?

        public static XmlDocument MtomToXml(Stream stream)
        {
            stream.Seek(0, SeekOrigin.Begin);
            using var reader = XmlDictionaryReader.CreateMtomReader(stream, Encoding.UTF8, XmlDictionaryReaderQuotas.Max);
            var doc = new XmlDocument();
            doc.Load(reader);
            return doc;
        }

[NicholasNoise]

@mconnew any news mate?

[mconnew]

Sorry I forgot to respond to your last comment. The easiest way to do this I believe is to use the MtomMessageEncodingBindingElement. This will output a Message object. Depending on how you want to parse things, I don't know how appropriate this is to your use case. Presuming you want to minimize changes elsewhere in the code, this can be converted to an XmlDocument. Here's some example code on how to do this. You might need to tweak it for your scenario a bit if the SOAP/Envelope version is different than the default. Something like this should work for you:

            var mtomBE = new MtomMessageEncodingBindingElement(MessageVersion.Default, Encoding.UTF8);
            var encoderFactory = mtomBE.CreateMessageEncoderFactory();
            var encoder = encoderFactory.Encoder;
            var message = encoder.ReadMessage(stream, int.MaxValue);
            var messageBufferedCopy = message.CreateBufferedCopy(int.MaxValue);
            var writableMessage = messageBufferedCopy.CreateMessage();
            XmlDocument doc = new XmlDocument();
            using (XmlWriter xmlDocWriter = doc.CreateNavigator().AppendChild())
            {
                using (XmlDictionaryWriter writer = XmlDictionaryWriter.CreateDictionaryWriter(xmlDocWriter))
                {
                    writableMessage.WriteMessage(writer);
                }
            }

Let me know if that works for you or if you hit any problems. I couldn't test this code as I don't have an Mtom stream lying around handily to test this so it might need some tweaks.

I'm curious why you are using the MtomReader directly instead of using WCF to do the work? You can use WCF in a contractless way using the IRequestChannel interface to make a request and get a reply using Message objects. Typically you would do this if the request or reply message if a little too free form for a contract definition. Going low level like this is usually a result of not knowing how to get WCF to go a little lower level. XmlDocument isn't as performant as XmlReader (which is what WCF uses to expose the body) and I wonder if there might be a better way to achieve your end goal. I often find taking a step back and looking at the original problem which is needing to be solved can lead to a simpler/cleaner solution.

[JoaoSaDevdotNet]

Someone Fix it? I got the issue Message Transmission Optimization Mechanism (MTOM) message encoding is not supported on this platform every time I try.

[spodskubka]

@JoaoSaDevdotNet what do you need fixed? It's very hard to help you based on that kind of error description.

Do you have an actual code sample? Maybe with a list of referenced NuGet packages and their versions?

Then somebody may be able to tell you what the problem is. 👍