Why is DataTable/View/Set absent?

[FransBouma]

I looked here: https://github.com/dotnet/corefx-progress and saw DataTable/View/Set isn't ported to .NET Core. IMHO this is a mistake, as DataTable and friends are a convenient way to handle untyped data at runtime, and is e.g. often used when fetching data through stored procedures as it's easy to e.g. fetch a dataset with the result of a proc which returns 3 cursors on e.g. Oracle.

I fail to see why this cornerstone of many people's data-access code is absent in a framework that targets servers.

[jonat3n1]

One important note about Dataset/DataTable/DataView is that these classes have been available in Mono for several years now and are also available in Xamarin's products for IOS and Android. I believe this has made it easier for many .Net developers to create apps for Linux, Mac and IOS products while actually avoiding other key Microsoft technologies such as Silverlight, Windows Phone, RT, etc. I don't know if Microsoft realizes how many .net applications have Datasets/DataTables embedded in their core libraries that don't need to change, would be too expensive to change and would add unnecessary risk. On top of all of this EF 7 is still much slower than core ado.net. I think the adoption rate for universal windows apps or asp.net 5,6,7 etc using .net core would take off if these classes were added to .net core.

[chadbr]

I'll do a +1 on DataSet (/Table/View) -

I've never understood the distaste for these objects by the general population. I get the feeling we have many many more developers now that can't think at that level of abstraction? I don't see how anyone builds a general data framework (i.e. non code-based entities / models / whatever) without them. A sad step back IMHO.

I'd be happy to join any effort to port them as an external library on top of .net core...

[mgravell]

Well, "general data framework" isn't something that most people need. Historically, such frameworks have been abused and misused when model-based / DTO-based frameworks would have been far preferable. There are a few limited cases where genuine "I don't know the layout of the data at all" applies, but that tends to be for things like reporting frameworks via some DSL and query generation. And in those cases too: DataTable can be unnecessarily expensive (with original/current value tracking, etc).

The scenarios where they are genuinely good fits do exist, but are much rarer in practice than the books, IDE tooling, and common patterns of 10 years ago might suggest. For 95%+ of the scenarios that they have been used historically: the world has moved on. Dismissing people who are using appropriate tools for their needs as being unable to "think at that level of abstraction" is disingenuous.

[atifaziz]

Well, "general data framework" isn't something that most people need. For 95%+ of the scenarios that they have been used historically: the world has moved on.

@mgravell In your very own words, isn't it “disingenuous” to assume that those people using appropriate tools (like DataTable) for their needs are some niche & shrinking group that hasn't moved on?

DataTable can be unnecessarily expensive (with original/current value tracking, etc).

Agree so then let's make that a feature request, that change tracking can be disabled for the many times it's not needed (even have it off way by default).

good fits do exist, but are much rarer in practice than the books, IDE tooling, and common patterns of 10 years ago might suggest

That's a re-education problem that's (hopefully) not going to need addressing. But yes, we definitely don't need as much IDE tooling as we had in the past, especially around DataSet & company being components you can interact with and connect on a design surface. That's where their weight needs to be shed.

[NickCraver]

@YoungGah @saurabh500 I'm seeing the new APIs popup, which is great. For everyone to get on the same page though, can you please update us on the plans here? What's the current plan for classes coming to .Net Core? What's coming? What's not? What's in RC2 & RTM? What needs design or implementation help? We're more than happy to help.

[FransBouma]

I'm closing this issue, as it's continued in other threads 10 times over.

[terrajobst]

@FransBouma

Thanks, that seems like a good idea. This thread has become a bit unmanageable due to sheer size.

[MelbourneDeveloper]

I noticed today that the DataTable class exists in System.Data in .NET Core, but it also seems that there is no SqlDataAdapter, so there is no way to read the data in to the DataTable

[atifaziz]

I noticed today that the DataTable class exists in System.Data in .NET Core

@MelbourneDeveloper Except it is empty.

[zhanzushun]

Frustrated and astonished finding DataTable/View/Set is missing from NET CORE. Have to stop migration... :-(

[MelbourneDeveloper]

It's not hard to implement them. I implemented them in Silverlight. The annoying one is going to be reading the data with a data adapter.

How do I submit code?

[MihaMarkic]

AFAIK ADO.NET (and much of the assemblies we know in "classic" .net) will be added in next releases.

[MelbourneDeveloper]

Can I take it that we will be able to load a table of data in to a datatable using an sqldataadapter?

[MihaMarkic]

@MelbourneDeveloper I'd say yes.

[MelbourneDeveloper]

Is very nice.

[andrew-vdb]

@MelbourneDeveloper Unless @Eilon or @terrajobst state that DataTable will be ported as is in current .Net Framework, don't get your hopes up

[VitaliyMF]

For everyone who is looking for old DataTable/DataRow/DbDataAdapter/DbCommandBuilder in .net core: take a look to my NReco.Data library, maybe it is exactly what you need. It provides similar API (RecordSet model) and offers great flexibility over database agnostic SQL generation.

[redowl3]

As Azure now only supports Asp.Net Core RTM how can bits of the framework still be missing and waiting to be ported across? In our case we use an external library to load up Excel files which produces a DataSet, not having the ability to easily manipulate this data is an absolute show stopper for us.

[benaadams]

@redowl3 you can run Asp.Net Core on full framework which will be present on the Windows VMs

[redowl3]

Thanks Ben - do you know what references are needed though to access DataSet/DataRows etc...?

[FransBouma]

@redowl3 Just reference System.Data, from the full framework, as you run on full framework anyway :)

[redowl3]

We've tried adding net45 framework and adding System.Data to the supported framework assemblies but that caused an issue with other dependencies that don't support that framework?

[benaadams]

Try net451 its the earliest framework aspnet core supports

[redowl3]

Thanks

[MelbourneDeveloper]

We shouldn't have to do this, but instead of us all howling to get this done. Why doesn't someone who has the time, export the code from System.Data using ILSpy, and then bring the necessary code for SqlDataAdapter to compile? The code is all there. You can export the whole lot with ILSpy. It wouldn't take much to get it compiling in .NET Core. Just exclude all the unnecessary code for now.

Maybe someone wants to set up a repo on GitHub somewhere and others could contribute until it's done?

[MelbourneDeveloper]

Well, actually, DataTable is even simpler. It would take only a couple of hours to pull in to Core and then you could write your own adapters...

[benaadams]

Why doesn't someone who has the time, export the code from System.Data using ILSpy

Well you might want to use the MIT licenced reference source instead; just saying...

Also its already on github https://github.com/Microsoft/referencesource/blob/master/System.Data/System/Data/DataTable.cs

[MelbourneDeveloper]

In that case, what are we even worried about?

[FransBouma]

@MelbourneDeveloper It might be a good idea to read this whole thread to know why it's not just the datatable class itself, but more importantly the ADO.NET api which exposes DataTable. As these methods are absent on .NET Core, you can port the datatable class all you want, but it won't get you anywhere. :) (and porting it is more time consuming than you might think as it comes with dataview, which contains a lot of databinding code, all of which won't work on .net core)

[MelbourneDeveloper]

?

The code is all there. If it's not in the repo mentioned above, you can pull it out with ILSpy. That includes DataViews and database specific adapters like SqlDataAdapter.

It's not a solution in and of itself. Someone will need to go through the process of getting the code to compile for .NET Core, and then test it. But, it is possible.

There's no reason legal or otherwise, as far as I can tell why someone can't just start a repo and go for it. Of course, I'm not a lawyer and am not offering legal advice.

[chadbr]

Mono? Or does it have the binding bits also?

[MelbourneDeveloper]

Mono? Binding Bits?

Look! Here is the code for DataView. https://github.com/Microsoft/referencesource/blob/master/System.Data/System/Data/DataView.cs

It's all there.

[FransBouma]

The code is all there. If it's not in the repo mentioned above, you can pull it out with ILSpy. That includes DataViews and database specific adapters like SqlDataAdapter.

Yeah, but that's against the EULA.

It's not a solution in and of itself. Someone will need to go through the process of getting the code to compile for .NET Core, and then test it. But, it is possible.

The question isn't whether it's theoretically possible, but whether it's doable. 'Someone will need' yeah, that's always the case, isn't it? 

There's no reason legal or otherwise, as far as I can tell why someone can't just start a repo and go for it. Of course, I'm not a lawyer and am not offering legal advice.

Heh, yeah that's clear ;)

Mono? Binding Bits?

Look! Here is the code for DataView. https://github.com/Microsoft/referencesource/blob/master/System.Data/System/Data/DataView.cs

It's all there.

yeah, but it's not compilable on .net core because it depends on apis which aren't in .NET core at the moment. 

And even if you spent all that effort to port these too, you'll run into the issue where DbDataReader doesn't expose a datatable with GetSchemaTable(). 

If you want to port all that, fine by me, but it will take more than just a couple of days. And I doubt MS will merge the PR. 

    FB

[redowl3]

Which (I think?) brings us back to my question of how can MSFT only support Core RTM apps on Azure when not everything has been ported?

[MelbourneDeveloper]

"The question isn't whether it's theoretically possible, but whether it's doable. 'Someone will need' yeah, that's always the case, isn't it?"

Of course it's doable. Most people here only need certain bits and pieces to get their code compiling. Why doesn't someone who's got some time get their bit compiling and then stick it up in a repo somewhere? Then, others can contribute as they get their bits compiling. This is better than sitting around an complaining.

yeah, but it's not compilable on .net core because it depends on apis which aren't in .NET core at the moment.

Some will compile, some won't. In the majority of cases it will be a matter of finding the equivalent API call in .NET Core.

And even if you spent all that effort to port these too, you'll run into the issue where DbDataReader doesn't expose a datatable with GetSchemaTable().

It's exposed. The method is there in the public repo:

virtual public DataTable GetSchemaTable() { throw new NotSupportedException(); }

That method needs to be overridden by the inheriting class.

[FransBouma]

@MelbourneDeveloper I'd suggest you read this whole thread, you're trying to rehash things which have already been discussed at length here and in other threads. The TL;DR: it's not as simple as you seem to think, not by far. Don't you think we would otherwise already have implemented a workaround?

[VitaliyMF]

@MelbourneDeveloper , @redowl3 DataSet/DataTable/DataRow implementations are over-complicated and it is good that they are not ported to .NET Core "as is". The structure is too heavy; it implements too many functions and in overall this "monolith" component is not the best choice in every concrete case.

Instead of just asking for old DataTable in .NET Core, could anyone list the functionality/usage scenarios that still needed and not covered by existing .NET Core-compatible libraries?

[MihaMarkic]

Instead of just asking for old DataTable in .NET Core, could anyone list the functionality/usage scenarios that still needed and not covered by existing .NET Core-compatible libraries?

Oh, no, please, not again.

[mahara]

Of course it's doable. Most people here only need certain bits and pieces to get their code compiling. Why doesn't someone who's got some time get their bit compiling and then stick it up in a repo somewhere? Then, others can contribute as they get their bits compiling. This is better than sitting around an complaining.

@MelbourneDeveloper You talk like it is something easy to be done. Why don't you start doing what you said, and let others benefit from the work you'd done? Piece of cake, isn't it?

[roji]

@VitaliyMF please read this entire thread (as painful as that might be).

Note that since this was debated quite a bit of time has gone by, so here's a quick summary of the current state of affairs as I see it. .NET Core RTM has been released and it does include an alternate resultset metadata API which does not rely on DataTable/DataSet (#5915), addressing one of the main concerns originally raised. A database/table metadata API (as opposed to a resultset metadata API) is still missing (#5024), but it could be argued that users that really need this can query schema information manually (e.g. INFORMATION_SCHEMA or some database-specific structures), even if that's not database-independent and generally sucks.

IMHO this means that at this point there doesn't seem to be any missing functionality depending on DataTable/DataSet - you can do (almost?) anything without DataTable/DataSet (e.g. access dynamic results via DbDataReader APIs). So the remaining reason for DataTable/DataSet is for porting across code that already uses them, or a need for an untyped data access API that also includes the DataTable/DataSet extras (e.g. optimistic concurrency). These are of course valid requests/requirements.

Note that Microsoft seems to have also shifted their general strategy, and intend to make .NET Core more backwards-compatible with .NET Framework, to help porting (see this article). This means that DataTable/DataSet may come back, or they may not. Either way, IMHO it's great to have them as a totally separate package without any core aspects of ADO.NET depending on them (i.e. metadata).

[VitaliyMF]

@roji Thank you for the detailed explanation. As I know that IDbColumnSchemaGenerator was specially introduced to cover schema aspect (not very helpful in practice - for example, Microsoft.Data.Sqlite.SqliteDataReader doesn't implement this interface).

Am I understand correctly, that only reason to have DataTable in .NET Core is backward compatibility?.. Well, not everything can be easily ported to .NET Core 1.0 and a lot of other incompatibilities will prevent old .NET projects from migration anyway (you don't say "System.Web" and WebForms should come back, right?). But new .NET Core projects will not use/depend on old DataTable!

Regarding untyped data access API: when needed, it may be provided by 3rd party libraries (I already mentioned my NReco.Data lib that implements command builder/data adapter and RecordSet with API similar to DataTable/DataRow). For me, it is still unclear, why old DataTable should come back into NETStandard ADO.NET API.

[chadbr]

I thought this thread was closed :)

Could someone from Microsoft simply clarify if it's on the roadmap or not?

[roji]

As I know that IDbColumnSchemaGenerator was specially introduced to cover schema aspect (not very helpful in practice - for example, Microsoft.Data.Sqlite.SqliteDataReader doesn't implement this interface).

Are you sure about that? It's documented on SqliteDataReader. The Npgsql provider also implements it.

Am I understand correctly, that only reason to have DataTable in .NET Core is backward compatibility?.. Well, not everything can be easily ported to .NET Core 1.0 and a lot of other incompatibilities will prevent old .NET projects from migration anyway (you don't say "System.Web" and WebForms should come back, right?). But new .NET Core projects will not use/depend on old DataTable!

Well, it's true that originally Microsoft seemed to want to remove legacy/unwanted APIs in .NET Core. But at some point (again, see this blog) the decision seems to have made to really prioritize making .NET Core as compatible as possible with .NET Framework, to help people port. Whatever we think of DataTable/DataSet, there are definitely tons of programs out there relying on it.

Especially with .NET Core being broken down into nugets, there doesn't seem to be any harm in making DataTable/DataSet available in .NET Core, as a separate package (as long as it isn't needed for any core operation, e.g. metadata). I'm guessing it might be somewhat low on the internal priority list though, with everything else going on.

[MelbourneDeveloper]

@MelbourneDeveloper You talk like it is something easy to be done. Why don't you start doing what you said, and let others benefit from the work you'd done? Piece of cake, isn't it?

I already did some work around this in Silverlight by simply looking at the metadata of the System.Data namespace in Visual Studio (ILDasm like functionality) - not with any use of ILSpy, or with any code from the reference source. It's not as hard as people make out.

Here's an example of production code Fill method for a DataAdapter:

            for (int i = 0; i < reader.FieldCount; i++)
            {
                var columnName = reader.GetName(i);
                if (!dataTable.Columns.Contains(columnName))
                {
                    dataTable.Columns.Add(new DataColumn(columnName));
                }
            }

            while (reader.Read())
            {
                var dataRow = new DataRow(dataTable);

                for (int i = 0; i < reader.FieldCount; i++)
                {
                    var columnName = reader.GetName(i);
                    dataRow[columnName] = reader.GetValue(i);
                }

                dataTable.Rows.Add(dataRow);
            }

Originally, @benaadams mentioned that this code was using the MIT license

Well you might want to use the MIT licenced reference source instead; just saying... Also its already on github https://github.com/Microsoft/referencesource/blob/master/System.Data/System/Data/DataTable.cs

But, after looking at the license agreement, it only allows us to use the code as a "reference": http://referencesource.microsoft.com/license.html http://referencesource.microsoft.com

The source is actually released under the MS-RSL license. Which according to Wikipedia "is the most restrictive of the Microsoft Shared Source licenses" (https://en.wikipedia.org/wiki/Shared_source#Microsoft_Reference_Source_License_.28Ms-RSL.29).

Are we able to get clarification on this? Are we allowed to take the System.Data code, slightly modify it for the purpose of compilation and use on .NET Core, and then stick that code up in a GitHub repo? The code may, or may not be used by Microsoft at a later point as a basis for building more functionality in to the System.Data namespace in .NET Core.

Just to clarify, I have no intention of breaking the license agreements put forward by Microsoft, or breaking the spirit of the agreements. I am merely raising this as a point that if Microsoft were to allow us to do it, we could recreate this library ourselves. If 10 programmers spent a small amount of time working on just the area of the functionality they need, and those areas were different, we'd be able to knock this over in a very short time span. But, if Microsoft doesn't want us to do this, then I withdraw any comments about attempting this, and will be happy to wait for the team to go ahead with this.

[MelbourneDeveloper]

it could be argued that users that really need this can query schema information manually (e.g. INFORMATION_SCHEMA or some database-specific structures)

This is what I have done in the past, but it's not ideal. You will need to write a layer over the top if you are implementing for multiple database platforms.

with .NET Core being broken down into nugets, there doesn't seem to be any harm in making DataTable/DataSet available in .NET Core, as a separate package (as long as it isn't needed for any core operation, e.g. metadata).

Exactly. And, why not let the community build it?

[MelbourneDeveloper]

@VitaliyMF

@MelbourneDeveloper , @redowl3 DataSet/DataTable/DataRow implementations are over-complicated and it is good that they are not ported to .NET Core "as is". The structure is too heavy; it implements too many functions and in overall this "monolith" component is not the best choice in every concrete case.

Instead of just asking for old DataTable in .NET Core, could anyone list the functionality/usage scenarios that still needed and not covered by existing .NET Core-compatible libraries?

On a basic level you are right. These classes are bloated and unnecessary. People should rewrite their code so as to avoid them.

That doesn't change the fact that legacy code does exist, and in order to embrace the new .NET Core technology, legacy code must be made to compile and work with the new technology. With the flick of a switch, large swathes of legacy code could be made to compile, and since this library does not need to be deployed with every instance of the .NET Core architecture, it's a mute point to say that it's too heavy. Any library could be too heavy. But, just taking some existing code and compiling it for .NET Core won't slow .NET Core down. Every developer still needs to make conscious decisions about performance no matter what.

[svick]

@MelbourneDeveloper The source code at http://referencesource.microsoft.com is released under the MS-RSL license. But the code at https://github.com/Microsoft/referencesource/ is released under the MIT license.

[MelbourneDeveloper]

Boom!

Off we go!

[bharatmangal04]

Can anyone suggest what would be the recommended approach to pass data ( User Defined Table type or paramenters) to stored procedure in ASp.NET Core if using ADO.NET

[saurabh500]

@bharatmangal04 Are you referring to SQL CLR UDTs on Sql Server? Can you open a new issue for this question?