Implement distributed/promoted transactions in System.Transactions for Windows only

[StephenBonikowsky]

@jimcarley commented on Wed Nov 09 2016

---

@XieJJ99 commented on Tue Jun 11 2019

@ajcvickers @stephentoub The SQL Server team had already provided MSDTC for SQL Server on Linux. Is there any plan to promote the MSDTC for linux as a generic service so that it's possible to support distributed transaction on Linux?

---

@ULSTechology commented on Wed Jun 26 2019

A lack of support for distributed transactions is pretty disastrous for anyone wanting to migrate their old system onto .NET Core.

This puts an end to our hopes that we can start using Blazor - we can't rewrite our entire application to work without distributed transactions, there's just too much of it.

---

@popcatalin81 commented on Wed Jun 26 2019

we can't rewrite our entire application to work without transactions, there's just too much of it.

Surely, if you can rewrite for Blazor, which is a significant rewrite anyway, you can also eliminate distributed transactions.

---

@ULSTechology commented on Wed Jun 26 2019

we can't rewrite our entire application to work without transactions, there's just too much of it.

Surely, if you can rewrite for Blazor, which is a significant rewrite anyway, you can also eliminate distributed transactions.

Using Blazor requires a rewrite of the UI, and we can start small with that.

Rewriting some fairly large chunks of our data access layer, and probably even some of our business layer, is a whole other matter. Unfortunately, it's not an option we could consider at the moment.

---

@mmaderic commented on Thu Jul 18 2019

I would like to vote up for this feature at least for databases on single SQL server instance.

I have case where I would like to enclose two different database commits into single system transaction. (Sort of micro-service architecture).

---

@abbotware commented on Mon Jul 29 2019

@LaterStart - I have the same use case so +1

Creating multiple exe microservices (service fabric) sharing a single SQL database (migration of legacy code / data)

I found it is possible on Net Core to share transactions/connections between ADO. and EF Core within a single process (https://docs.microsoft.com/en-us/ef/core/saving/transactions) ,

However, what options currently exist when 2 separate Net Core exes are involved for implementing a 2 phase-commit like pattern?

1. Can the DTC / Resource Manager be simulated via base classes in Net Standard? (https://docs.microsoft.com/en-us/previous-versions/ms229975(v=vs.90))

2. Call the DTC directly via Win32 Api and some how enlist?

1-2 seem like a lot of work, but from what I have read (somewhere) I could just target Net Standard 2.0 and run within a Net Framework host - That will allow System.Transactions to enlist / promote into a DTC transaction, (I can settle for Netstandard + 4.7.2 + new sdk project format if that is my only option)

Even if I use a Net Framework Host (or Net Core if this open issue ever gets implemented), does that mean it should it is possible to obtain the transaction token / correlation id from process A so it can be sent/serialized to process B for manual enlistment?

---

@pappasa commented on Tue Sep 03 2019

How do you use .NET Core with IBM WebSphere MQ queue? For example, I do not want to remove the message from the queue, and the put to the database to fail.

---

@aajmot commented on Fri Sep 20 2019

can someone tell me if the issue has been fix or still there? It's almost 2020 now? Is there any other way around, please tell me.

---

@pi3k14 commented on Fri Oct 11 2019

@dasetser wtf - you moved this into the unforeseeable future? Don't you want to on-board enterprise solutions to .Net Core, you want to push us all over to the dark side (java)?

---

@am11 commented on Fri Oct 11 2019

Going by https://referencesource.microsoft.com/#System.Transactions/System/Transactions/Oletx/DTCTransactionManager.cs,2c1154236b6b7acf, seems like there are some win32 specific components involved which probably is the main hurdle. Could someone shed light on what are those and what would require to bring them cross platform?

---

@dasetser commented on Tue Oct 15 2019

This got moved to future because after investigating it we determined it's going to be a significant rewrite of the feature rather than just a port. We're looking into options for this, but we aren't able to commit to getting this done in the upcoming release.

The main hurdle is that System.Transactions depends on MSDTC when it has to promote transactions, but MSDTC is only available on Windows. In order to get this to work cross platform we would need to rewrite the feature to allow it to work with other distributed transaction coordinators that are available on other platforms, or look into doing something similar to what the SQL Server team did where they brought the DTC components inside the SQL component. Either approach would be a significant change to this feature.

---

@Peperud commented on Tue Oct 15 2019

Where do I vote for this?

---

@pi3k14 commented on Tue Oct 15 2019

@dasetser - I appreciate your clarification. But, us current users of Distributed transactions don't care about cross platform. Just support it on Windows, and put this cross platform paradigm that we haven't request into the future.

Thank you.

---

@am11 commented on Tue Oct 15 2019

Thanks for the explanation @dasetser. If the target linux machine has MSDTC component from SQL Server installed, and its C API is the same as Windows, can the implementation take advantage -- as in, SQL Server as a prerequisite (for now)? We can separately request and poll SQL Server team to ship MSDTC component as a standalone library. :)

---

@randyburden commented on Thu Dec 05 2019

We were planning to begin porting many of our current applications to .NET Core starting next week as well as start writing our new apps on .NET Core but we just ran into this issue of distributed transactions not working which feels like a deal breaker. Almost every application we write or will write in the future requires distributed transactions.

It is very odd to have experienced a platform not supported exception with this particular feature when we all know that it already exists in the full .NET Framework. It's hard to understand why a Windows platform specific implementation wasn't implemented either in CoreFX or as a shim NuGet package.

Is there a workaround?

---

@pi3k14 commented on Fri Dec 06 2019

@dasetser - Wouldn't implementing WS-AtomicTransaction be the way to go. And since MSDTC supports it and WCF has an implementation of it https://docs.microsoft.com/en-us/dotnet/framework/wcf/feature-details/using-ws-atomictransaction this shouldn't take that much time. Those wanting distributed transactions on Unix can then get a third party solution.

[popcatalin81]

Why not guarantee single-platform distributed transactions first? If even a single platform cannot use this, It's really disappointing

You're opening a political can of worms. It's very common to switch .Net Core from Windows to say Linux Docker. This would become an impediment towards that goal. However if instead you design your services to not use Distributed Transactions, the switch becomes seamless.

[lscorcia]

It's not political at all. The use case for DTC is much larger than an hypotetical switch to docker containers. Especially in large, business applications, DTC is not a workaround for a messy codebase but a business requirement. I have yet to see a business requirement that implies switching to a different OS, but I have seen plenty of them that require writing in a coordinate manner to - say - a database and a message queue.

I am starting to think that this feature is unwanted as it does not fit well on an Azure workloads, and .net core is supposed to embrace Azure as much as possible.

[rcollette]

You're opening a political can of worms. It's very common to switch .Net Core from Windows to say Linux Docker. This would become an impediment towards that goal. However if instead you design your services to not use Distributed Transactions, the switch becomes seamless.

I'm not advocating a single platform solution, but wasn't the entire Windows Forms capability ported over to .NET core but dependent on Windows?

[precision-sean]

It's a challenging problem. Rcollette's comment about Windows Forms is well-taken.

What bothers me most about this is the lack of clarity and feeling a bit burned by following Microsoft's recommendations.

From Writing a Transactional Application: "You can utilize the explicit programming model by using the Transaction class, or the implicit programming model in which transactions are automatically managed by the infrastructure, by using the TransactionScope class. We recommend that you use the implicit transaction model for development."

So we followed the recommendation and used TransactionScope, which automatically promoted to a Distributed Transaction when necessary -- it just worked.

Years ago, since TransactionScopes were supported in .NET Core/Standard, we started to move our libraries to .NET Standard to begin the move to .NET Core. It wasn't until after we moved them that we found that distributed transactions weren't supported on .NET Core and only broke at runtime. We engineered some workarounds to keep things running, but put the brakes on moving our existing data libraries to .NET Standard.

Help appeared to be on the way. In December 2019, this was added to the 5.0 milestone, telling us that the lack of distributed transactions was temporary and giving us a timeline for the ability to move these data libraries to .NET Standard.

However, in July 2020, this was removed from the 5.0 milestone.

And now it's July 2021, and there has been no movement on this.

So it would be nice to have some clarity: should developers that relied on this functionality wait for previously-promised support, or start to re-engineer their code?

[jzs11]

I am keen to see some progress for this topic.

[rvdginste]

... However if instead you design your services to not use Distributed Transactions, the switch becomes seamless.

For new applications, I agree that this can be taken into account from the beginning and that you can create a design that does not rely on distributed transactions.

For upgrading .net full applications to .net core, you cannot go over this so lightly. If the use of distributed transactions was a choice in the original design, it means that the application relies on it to work correctly. If distributed transactions are no longer available, every flow in the application that relies on it must be found and needs to be redesigned; in a lot of cases this will result in rewrites of big parts of these applications. As other people have mentioned, it is not only about pure database transactions, but also for example msmq operations that enlist in transactions. WCF made it really easy to propagate transaction flow (and thus rely on distributed transactions). I think a lot of enterprise applications will be relying on this.

I've been following this issue for almost 2 years and it's very disappointing that there is no clear answer from Microsoft. It seemed to be planned for .net 5, and now there's no indication whether it is being worked on, or whether it will be worked on. Some feedback would be appreciated, even if the feedback is that it also won't be available in .net 7/8. It is currently not clear what upgrade paths are possible for .net full applications that use this.

The missing support for distributed transactions is known from 2016 after the release of .net core 1.0 (see the original issue https://github.com/dotnet/runtime/issues/19318). In the mean time, we're 5 years further, .net core and .net full have been "unified" in .net 5 and the .net 6 LTS version and there seems to be no plan regarding distributed transactions... is there a real intention to support it or not?

[fazleskhan]

This is a critical feature at enterprise scale and is desperately needed

[mwt-jyskebank-dk]

Supporting distributed transactions (implicit transactions from System.Transaction namespace) is an absolute must-have for all enterprise-scale applications. You cannot be serious that 5 years into the development of the replacement for .Net Framework, you don't support this. An absolute deal-breaker for .Net 5+.

Question: With the advent of the net5.0-windows OS-specific TFM, wouldn't it be fairly straight-forward to support DTC just for Windows, while you guys at Microsoft tangle for some more years with the cross-platform issues you're facing?

[mrtarkhan]

any update?!

[StruninIhor]

any update?!

Yes, it was planned for EF7, 😃
https://docs.microsoft.com/uk-ua/ef/core/what-is-new/ef-core-7.0/plan#distributed-transactions

[ajcvickers]

From https://docs.microsoft.com/en-us/ef/core/what-is-new/ef-core-7.0/plan#distributed-transactions

The System.Transactions library in .NET Framework contains native code that makes use of the Windows Distributed Transactions Coordinator (DTC) to support distributed transactions. This code was never ported to .NET Core. In the .NET 7 timeframe, we plan to investigate and begin the process of bringing this functionality to modern .NET. This will be initially for Windows only, and will support only database scenarios where the ADO.NET provider also supports distributed transactions. Other uses of distributed transactions, such as in WCF, will not be supported in .NET 7. Based on feedback and cost, we may implement support for other scenarios and/or non-Windows platforms in a future release.

[joergmetzler]

I <3 U

[jimmyzimms]

From https://docs.microsoft.com/en-us/ef/core/what-is-new/ef-core-7.0/plan#distributed-transactions

The System.Transactions library in .NET Framework contains native code that makes use of the Windows Distributed Transactions Coordinator (DTC) to support distributed transactions. This code was never ported to .NET Core. In the .NET 7 timeframe, we plan to investigate and begin the process of bringing this functionality to modern .NET. This will be initially for Windows only, and will support only database scenarios where the ADO.NET provider also supports distributed transactions. Other uses of distributed transactions, such as in WCF, will not be supported in .NET 7. Based on feedback and cost, we may implement support for other scenarios and/or non-Windows platforms in a future release.

Thank you for providing some concrete information on this. There's 0 issue with this initially being a windows specific library / component during the first rollout provided that the namespaces and packaging reflect that (assuming there's some type of bootstrapping required to plug it into the pipeline at app start). This is a scenario fully covered in other platforms such as JTA; It's not a technical limitation but a design limitation, one that is looking like will finally be addressed.

[torgashov]

I am so sorry for my English. We have many applications work with financial transactions with rich business logic. Each time needs to commit changes in several systems. Those applications must have cansist transactions. Transaction timeout does not matter. Now developers have houl stack applications, that need distributed transactions. Now we need general interface for decide distributed transaction tasks. I think we should have dtc cross platform.

[jzs11]

I am so sorry for my English. We have many applications work with financial transactions with rich business logic. Each time needs to commit changes in several systems. Those applications must have cansist transactions. Transaction timeout does not matter. Now developers have houl stack applications, that need distributed transactions. Now we need general interface for decide distributed transaction tasks. I think we should have dtc cross platform.

haha, good try, one step at a time.

[John0King]

DTC will have bad performance if the transaction commit too slow (for example , there are slow async task before commit) , the default isolation level will cause whole system waiting for the transaction to complete.

I notice there a TCC(Try Confirm Cancel) very popular in Java eco-system, and is also being ported to .net with some limitation (it require object proxy) .

the reason that TCC have better performance is because it make an application level transaction wrap database transactions , the database transaction complete as quick as possible and do not care about other operation is success or fail , and when an operation is fail , then the application level transaction (eg. Cancel) will rollback the successed data (eg. delete / update back the successed data) , thus the database transaction will not block other read operation (ReadCommited)

Maybe the new System.Transactions should consider this pattern for distributed transaction.

[torgashov]

DTC will have bad performance if the transaction commit too slow (for example , there are slow async task before commit) , the default isolation level will cause whole system waiting for the transaction to complete.

I notice there a TCC(Try Confirm Cancel) very popular in Java eco-system, and is also being ported to .net with some limitation (it require object proxy) .

the reason that TCC have better performance is because it make an application level transaction wrap database transactions , the database transaction complete as quick as possible and do not care about other operation is success or fail , and when an operation is fail , then the application level transaction (eg. Cancel) will rollback the successed data (eg. delete / update back the successed data) , thus the database transaction will not block other read operation (ReadCommited)

Maybe the new System.Transactions should consider this pattern for distributed transaction.

DTC will have bad performance if the transaction commit too slow (for example , there are slow async task before commit) , the default isolation level will cause whole system waiting for the transaction to complete.

I notice there a TCC(Try Confirm Cancel) very popular in Java eco-system, and is also being ported to .net with some limitation (it require object proxy) .

the reason that TCC have better performance is because it make an application level transaction wrap database transactions , the database transaction complete as quick as possible and do not care about other operation is success or fail , and when an operation is fail , then the application level transaction (eg. Cancel) will rollback the successed data (eg. delete / update back the successed data) , thus the database transaction will not block other read operation (ReadCommited)

Maybe the new System.Transactions should consider this pattern for distributed transaction.

You are right but not in all cases. We have many applications that work as background process and not need best performance. Very important to have data cansistance. Need use acid principle. And need to have clear application architecture. And need block database rows, while transaction not complete. DTC need to use only in certain applications class, in certain tasks class.

[cyanite]

I am wondering if there is any timeline update on this? Specifically I mean: How likely is it looking to make it into .NET 7 final?

[roji]

@cyanite this is one of the high-priority items on my list fo .NET 7, so I'd say there's a good chance. But as there are a lot of unknowns around this and many other tasks, it's always possible it would be punted out of the release.

[swh-cb]

For those who want to run MSDTC distributed transactions already under .NET 6.0 with multiple Microsoft SQL Servers on the Windows platform, our library might be of interest:

https://github.com/softwarehelden-com/Softwarehelden.Transactions.Oletx

The library is now open source. We have been using the library internally in our deployment processes for a few months now as a workaround to run distributed transactions on multiple MSSQL servers. At runtime we still use .NET Framework 4.8 because we need to support distributed transactions with Oracle database servers as well. Please read the readme in the repo to learn how it works under the hood.

Update (Version 1.1.0): Applications targeting .NET Core 3.1 or .NET 5.0 can now use the library to perform distributed transactions for MSSQL servers.

Update (Version 1.2.0): It is now possible to enlist Oracle database servers and MSSQL servers in the same distributed MSDTC transaction. This means we can finally target .NET 6.0 for our runtime environments. :-)

Update (Version 1.3.0): Data providers can now be loaded in an isolated compatibility assembly load context if they are not 100% compatible with .NET Core. For example, the unmanaged ODP.NET provider uses types and methods from the .NET Framework unknown to the .NET Core runtime to coordinate the distributed transaction between MSDTC and Oracle.

[yedf2]

Here is a commonly used distributed transaction framework which also support dotnet. Transaction coordinator: https://github.com/dtm-labs/dtm Dotnet SDK: https://github.com/dtm-labs/dtmcli-csharp A blog about how to design a distributed transaction: https://www.c-sharpcorner.com/article/distributed-transaction-in-c-sharp-microservices-using-saga-pattern/

[hosseinitabar]

Our code was working just fine under .NET 3.1. We were using this code:

            var transactionOptions = new TransactionOptions
            {
                IsolationLevel = isolationLevel,
                Timeout = TransactionManager.MaximumTimeout
            };

            return new TransactionScope(TransactionScopeOption.Required, transactionOptions);

And after we upgraded to .NET 6.0, it broke.

What is the problem? How 3.1 supported this?

[roji]

@hosseinitabar .NET Core and .NET 5+ do support TransactionScope on a single database - but they don't support distributed transaction, which is when the same TransactionScope encompasses more than one database.

As for your specific problem, please open a separate issue with a full, runnable code sample showing the code that works in 3.1 but fails in 6.0.

[roji]

FYI work is ongoing to port the distributed transaction support from .NET Framework, to .NET 7.0. Unfortunately, that support is tightly coupled to the Windows MSDTC service (and the OleTx protocol), so this will work on Windows only (on other platforms, starting a distributed transaction will throw as today). Enabling distributed transactions in a cross-platform way is a significantly bigger task, and is now being tracked in #71769.

Since it's late in the .NET 7.0 release, we're not fully sure that work will be done in time - but we're doing the best we can. Note that some specific functionality that worked in .NET Framework may be excluded from this port (we'll provide more details later).

[rwasef1830]

@roji Is there any prominent possible alternative 3rd party library you are aware of that cross platform can use in the meantime, and is there an integration point for such API into the .NET API ? such as a provider model ? or any pluggable mechanism ?

[roji]

@rwasef1830 not to my knowledge.

The problem with 2PC is that it requires coordinated support across a transaction manager (currently MSDTC only), the multiple database drivers out there, and a user-facing API for managing the transaction (currently System.Transactions). Because of this I don't think it's possible for some 3rd-party library to just "handle" all of it; at minimum, database drivers which are currently written against the Sys.Tx support would have to be adjusted to work against a another more cross-platform one.

I've spent some time researching what this would would mean and how it would look like; it's not trivial, and we're not sure we're seeing lots of user demand for it (#71769 is for gauging that). In the meantime, see my comment for alternatives to 2PC which may be more appropriate in modern cloud settings.

[SimonCropp]

@roji awesome. thanks