Why is DataTable/View/Set absent?

[FransBouma]

I looked here: https://github.com/dotnet/corefx-progress and saw DataTable/View/Set isn't ported to .NET Core. IMHO this is a mistake, as DataTable and friends are a convenient way to handle untyped data at runtime, and is e.g. often used when fetching data through stored procedures as it's easy to e.g. fetch a dataset with the result of a proc which returns 3 cursors on e.g. Oracle.

I fail to see why this cornerstone of many people's data-access code is absent in a framework that targets servers.

[tthiery]

@shanselman we are still here and curiously waiting ;)

[davkean]

Sorry for the late reply, I thought I already replied to this issue.

First of all I should clear things up, just because something hasn't been ported yet, does not mean it won't be ported. These things take time, design[1] and resources, and we're right now focusing on getting .NET Core working for two different app-models: Universal Windows Apps and modern MVC/WebAPI web app across Mac, Linux and Windows. Right now we don't even have a networking stack across these three OSs and we haven't yet open sourced the data stack.. :) After that we'll start looking at other scenarios that we're missing with the new stack.

To summarize some conversations we've already had internally about this:

• We want DataSet/DataTable, et al, in some form going forward (but separate to the base policy-free abstractions)
• We want GetSchemaTable in some form (we should get a separate bug filed on that, however, as you could imagine bringing that back with a different design that doesn't pull in DataSet/DataTable).

[1] For example, as you've noticed GetSchemaTable pulls in DataTable, which itself pulls in DataSet, a large policy-heavy type that we do want to bring back in some form but does not belong in the same assembly as our policy-free data abstractions.

[FransBouma]

Thanks for the reply!

Sorry for the late reply, I thought I already replied to this issue.

First of all I should clear things up, just because something hasn't been ported yet, does not mean it won't be ported. These things take time, design[1] and resources, and we're right now focusing on getting .NET Core working for two different app-models: Universal Windows Apps and modern MVC/WebAPI web app across Mac, Linux and Windows. Right now we don't even have a networking stack across these three OSs and we haven't yet open sourced the data stack.. :) After that we'll start looking at other scenarios that we're missing with the new stack.

:)

To summarize some conversations we've already had internally about this:

• We want DataSet/DataTable, et al, in some form going forward (but separate to the base policy- free abstractions)

Is this something the community could help with? As in, if someone surgically cuts out the DataTable and friends from reference source into a repo, would that be something MS would be interested in if the community would come up with, say, an assembly which contains those classes? Or is it better not to spend time on that at the moment because you guys/girls don't have a plan set in stone yet with what to do with these classes?

• We want GetSchemaTable in some form (we should get a separate bug filed on that, however, as you could imagine bringing that back with a different design that doesn't pull in DataSet/DataTable.

I think if there's a GetSchemaTable which returns something we can use at runtime, whatever the form, (as well as GetSchema()), it is fine by all of us. DataTable for fetching is then a separate issue, as that can be added later on, the GetSchema/GetSchemaTable methods on the other hand are important to have from the get-go as they're in the interface for connection/reader, so adding them later is a serious breaking change and a dependency hell, so if that's avoidable, all the better. 

[1] For example, as you've noticed GetSchemaTable pulls in DataTable, which itself pulls in DataSet, a large policy-heavy type that we do want to bring back in some form but does not belong in the same assembly as our policy-free data abstractions.

Yeah, I can imagine you'd want to avoid pulling in that behemoth if you can avoid it. :)

Cheers,

    Frans

[william-gross]

Thank you @davkean for the information. I agree with you and @FransBouma that GetSchemaTable's functionality could be brought back in a different form that doesn't rely on DataSet/DataTable.

[dmitry-a-morozov]

To me the most important advantage of DataTable is ability to push data back to a database. It comes in 2 flavors:

• Updates via SqlDataAdater with optimistic concurrency, batching and merging support.
• Bulk copy via SqlBulkCopy which also relies heavily on DataTable. This is very useful for analytical workloads (big data story). In a company I work for we use it heavily in data cleansing processes.

Being exclusively F# developer I don't use DataTable very often but sometime it's really handy. It will tedious to roll out similar functionality on my own. Also, I must say DataTable is fairly straightforward type in it's core as oppose to obscure objects models used by EF .

[Dean-NC]

I use ASP.Net MVC 5, but still prefer ADO.Net and Stored-procs. over Entity Framework (I use a personal library that abstracts ADO.Net so ADO.Net code is not scattered in my apps).

With that said, I use DataTable when calling procs with Table-valued parameters. I just looked, and there's a very limited amount of ways to use TVP's, with DataTable being the only way for me. DBDataReader can also be used, but the data I'm sending to the proc is never something I've queried...it's data I have in a IList<> or an array.

Without DataTable, is there any reasonable way to use TVP's ?? I can understand not putting DataSets in the new Core, but DataTable is a lighter-weight object and has many uses...there are some good examples posted here. PLEASE put DataTable in Core.

[Dean-NC]

ping

[terrajobst]

@saurabh500 might have some thoughts here.

[Dean-NC]

@terrajobst I'm fairly new to github, is there a way I can contact @saurabh500 ?

[YoungGah]

DataSet/DataTable: We are evaluating to see if this is something we want to bring in a future release or whether it will be in the same form or not. Please let us know if you have opinions on whether it will be acceptable to see the feature in different form. It will be also good to get examples of what scenarios people are using DataSet/DataTable for from community.

GetSchema: We are going add the support in the future although its return value might be in DataTable form.

[Dean-NC]

@YoungGah, Thanks for the reply, but you can get plenty of examples of why it's needed by looking at the comments in this thread. The one I posted about was passing a table-valued parameter to stored-procs. I wouldn't care if you changed the "form", just as long as it still worked for sending table-valued-parameters to stored procs.

Please strongly consider getting this added. Table-valued parameters are very beneficial in several scenarios.

I just did some testing, and it's possible to use a List (of SqlDataRecord) instead of DataTable for using the table-parameters, but it's not as clean since each SqlDataRecord takes a little more to setup, and with each record you have to define the MetaData (column info), whereas the DataTable is defined once and you just add rows to it. Of course I could abstract that into a helper method.

[FransBouma]

DataSet/DataTable: We are evaluating to see if this is something we want to bring in a future release or whether it will be in the same form or not. Please let us know if you have opinions on whether it will be acceptable to see the feature in different form. It will be also good to get examples of what scenarios people are using DataSet/DataTable for from community.

Just look at this thread, there are plenty of examples given: dynamic, untyped resultsets and e.g. getschema results. 

GetSchema: We are going add the support in the future although its return value might be in DataTable form.

'the future', as in post RTM? how are people going to obtain resultset information at RTM then? You do know that it is a bit hard (read: impossible) to determine a column's type without getschema on the datareader, when e.g. the value is NULL. 

Without getschema on the datareader at release your data-access API will be unusable unless the resultset is known, e.g. the query is generated from meta-data obtained through other means. This means that any stored proc call issued as-is which fetches data is dead in the water. I truly don't understand why you'd cripple your API like that. 

    FB 

[GiorgioG]

+1 what FransBouma said.

[anpete]

@FransBouma Can you use GetFieldType, GetName etc?

[FransBouma]

@FransBouma https://github.com/FransBouma Can you use GetFieldType, GetName etc?

For the .NET type alone it might work, but GetSchemaTable returns other information as well like (max)length and e.g. whether the field is a PK value. This is info which is usable for the object the data is used in e.g. when it is persisted again. the problem with the lack of this kind of information at runtime is that you need to obtain it through other means, and people using stored procedures will then need to use special tooling. I of course won't mind that, as I build these tools for a living, but not everyone wants/can use my tooling or tooling from my competitors. 

Another problem is when one of the columns is a BLOB or equivalent and it has to be read by a stream reader. You can't determine that from the interfaces provided by the dbdatareader, it's available in the schema table though. 

This all might sound like edge cases but for people using stored procedures these are the building blocks they work with and therefore aren't edge cases at all. Again, it's not required to obtain this information through the datatable type, if there's an equivalent, just offer it. But you'll see that e.g. IDataReader.GetSchemaTable offers different schemas depending on the underlying ado.net provider. So it's not that easy: the datatable offered the dynamic nature needed for this method. 

I understand it's a bit of a problem as choosing to keep the existing api requires the datatable/view/set family jump on board which is a lot of work and choosing to use an alternative means it has to be designed well and flexible enough for all ado.net providers and that's not simple either and costs time and that's likely not there. However simply saying 'we'll add it in the future' has no real value: not having this API in whatever form that works is equivalent to making your new and shiny API stumble and not really an alternative to the people who left .NET for other platforms and who you all so eagerly want to come back. 

    FB

[NickCraver]

Have there been any new thoughts or progress on this? It took me a while to find this issue, but I've also hit a blocker here on getting column sizes from a DataReader here due to DataTable and as a result GetSchemaTable() missing. I can get nvarchar from the DataReader via .GetDataTypeName(), but (as far as I can tell), there's no way to get the column width. Since I want to take the data and create tables elsewhere with it, this is particularly problematic. The best I can do is guess at column widths based on ym widest data from the reader, and that has down-the-road issues of future inserts handling resizes, etc. It's not a good position to be in.

This is a problem specific to core. Is there another way to get this data I'm unaware of? If not, then +1 for adding these types back in, they are needed.

[NickCraver]

@terrajobst Can you please comment if any way to get a result set schema is coming to .Net Core? If it's not then I need to abandon .Net core for features and entire projects on my end. Not knowing what is or isn't on the roadmap is extremely frustrating from the user end of things. There are a large number of use cases here and there's been no Microsoft response on these pretty fundamental features needed in many apps for over 3 months. If they're not something you guys have any interest in adding back for whatever reason, that's fine. But it'd be very helpful to let the community know so we can avoid .Net Core. Earlier in this issue these features were called "legacy," but they are not. They are the only way to do things. They have not been superseded because no other method to get this information currently exists.

To be clear: I don't care what method Microsoft wants to provide to get this schema information, but there needs to be a method.

A simple concrete use case: I want to select data via a query and create a table with that data elsewhere, I currently can't. There's no way to tell how wide any field is or if it's nullable.

[Petermarcu]

@YoungGah to help provide more clarity on the roadmap for this component.

[GiorgioG]

It's short-sighted on Microsoft's part to brush aside the serious concerns of some pretty smart people in the .Net community. If Microsoft is serious about cross-platform .Net, then it can't leave out vital functionality. I've been considering leaving .Net altogether for some time and Microsoft's mishandling of issues like this only serve to alienate people like me further. Sad, given that I've been using it since .Net 1.0 beta became available.

[benaadams]

@NickCraver there is a SqlMetaData class, not sure where/how it fits in though.

[FransBouma]

@benaadams Anything in the SqlClient namespace is not usable to solve this issue. It has to be a Data.Common class so it is usable by all ADO.NET providers that will be running on .net core. That's what makes this issue so important: DbDataReader, the base class, has to have an API for this (as it does now in full .net, but it's using DataTable at the moment, hence the issue! :)) so ADO.NET providers can fill in the blanks in their provider to provide the actual data at runtime.

The situation currently is that one can execute ExecuteReader on a DbDataReader but there's no way to know what the resultset looks like (column specifics), as DbDataReader misses an essential part of the puzzle. Why there's no-one within Microsoft concerned about this is beyond me. It looks like there's internal chaos/turmoil around .NET core as there's no real leadership expressed in their actions. Making things OSS is nice, but if there's no vision implemented it's just marketing.

[schotime]

I use List<Dictionary<string, object>> for all dynamic queries. Done!

[FransBouma]

@Petermarcu There's no need for a roadmap, it has to be in the API at day RTM as all ADO.NET providers have to provide the same API to produce this essential piece of information. It's useless to have some SqlClient class which can perhaps do it, as users working on e.g. PostgreSql with Npgsql can't use that.

Lessons from ADO.NET in .NET 1.0/1.1 clearly dictate if you don't put it in the API on day one you'll regret it later on, a clear example of that is the way errors were reported in ADO.NET in .net 1.x or how the lack of DbProviderFactory was making using multiple databases in a .NET application a nightmare.

Also, has anyone within Microsoft ever thought of the reason why there is DbProviderFactory in .NET core, but not a universal way to obtain metadata like in the full .net api? I.o.w.: Why have a DB agnostic way to create and use ADO.NET provider classes, yet to do anything with them you have to fall back to ADO.NET provider specific API classes, making the whole DB agnostic factory system totally useless?

So is there anyone designing this API, or in charge of a vision of it? Or are you people simply looking at the sky and thinking "Today is a good day to port that random class over to .NET core"?

@schotime don't pollute this discussion with random bollocks.

[schotime]

@FransBouma Good point. You've done a pretty good job of that yourself ;)

[FransBouma]

@FransBouma Good point. You've done a pretty good job of that yourself ;)

Of course. https://github.com/schotime/NPoco/blob/8334a03ff06d2bbe8dcd4caf0c342cd4730202c7/src/NPoco/SqlBulkCopyHelper.cs#L28

Have fun porting that to .NET core, Adam. Or closer to home (which I now maintain Massive), I'd like to port Massive to .NET core some day but that's difficult too, as a lack of schema info retrieval (the core issue) is a bit of a hurdle. If there's no dynamic resultset retrieval done then everything is fine, but the world isn't that. I'm happy for you that you can get away with it, others don't.

[schotime]

So it won't be in the first version of .NET Core. That's why its the first version. Neither is SignalR and other staple libraries.

The .NET core version of NPoco will be pretty easy. It just won't have that method in it. You can only support what the platform libraries support when they support it.

[FransBouma]

@schotime So you're not affected by it, that's great. Others unfortunately are. Your post suggested those people are talking out of their ass. I can assure you, we're not. It's not that we want that method, but a method which allows developers to obtain meta-data of a resultset. If it's not in v1.0, every ADO.NET provider out there will provide their own solution which is meaningless as it's not db agnostic, and which forces developers to make a choice: either not use the method (which will be hard in some cases, not in your world, but in other people's worlds) or take a dependency on the ADO.NET provider assembly (which is a pain, as we all (not you, but the people who were around in the .NET 1.x era writing ORMs) have learned the hard way).

Having DB agnostic code in non-DB agnostic libraries is a pain, and this is a core example of it. DataTable is the vehicle the meta-data is currently provided in .NET full. If DataTable is too big to port, I can understand that (as I've said months ago already) but provide an alternative for the missing API methods that aren't ported because of DataTable not being ported.

That you don't understand what the fuss is all about and are fine with it how it is now: great. Not everyone lives in your world, some people have code to write which is affected by this. And before you think you're clever: yes we have looked at alternatives, but there aren't any, because, how odd, the only way to obtain the information today is through the API that's not ported!

(to elaborate: even if I use my meta-data retrieval queries in my drivers, I can't deal with the information, as I don't know the resultset layout per column of the resultset that's returned by the database! So I have a datareader, and I have to guess that column "DataType" in my resultset is an int and not nullable. Perhaps on this version of Oracle it's an Int64, because ODP.NET thought it would be funny to return a NUMBER(10,0) as Int64, and on another version of Oracle it's an int32. Oops... I guess your micro ORM simply assumes what it gets back from the DB fits in the shapes it has to project to, which is fine, but some people like to have more robustness so they can deal with a variety of situations which might not match exactly or require extra code to check whether they're dealing with the right type of data.)

[nvivo]

So you're not affected by it, that's great. Others unfortunately are. Your post suggested those people are talking out of their ass.

@FransBouma This is definitely not what I got from his post. To me @schotime was simply being reasonable. You said you have a product that depends on these features and you make a living out of it, so you have a lot at stake, we get it. But this can't be the only side the team sees to make a decision.

That you don't understand what the fuss is all about and are fine with it how it is now: great. Not everyone lives in your world, some people have code to write which is affected by this.

How you're not the one being a jerk here? You're the one who thinks people with a different view are here just talking out of their ass.

The reason why this has not been added yet seems pretty clear to me:

1. Schema discovery is a hard problem, the current abstraction was not goog enough to be universal (as you said)
2. It depends on DataTables which are considered legacy and most people don't use them anymore
3. Schema discovery is useful, but is not required. 99% of the cases you can do just fine without it, and that's what people have been doing.
4. Most new projects (which is the focus of CoreFx) are moving to nosql/schemaless solutions
5. Most old projects never had the need for schema discovery

I am not talking out of my ass here. As someone who uses ADO.NET a lot too, I'd really like the provider model to be simplified.

Schema discovery can be done by looking at the INFORMATION_SCHEMA and if that's not standard enough, it won't get better by creating another abstraction and moving the problem to the provider level. It didn't work universally before, it won't work now.

I say let's keep the provider dead simple as it is, and see if it really impacts people. DataTables/DataViews can be added as a separate packages. Don't make DbDataReader depend on it.

I say let the team focus on more important stuff and let the community figure out better ways to do it. If there is a winner model, I'm sure it can be included later.

[ryanbnl]

@FransBouma don't you use INFORMATION_SCHEMA? It's cross platform, cross database and cross language. It's THE API for database metadata.

I use it for all of my database-related code generation needs, from generating types to automated migrations.

Can't you replace your existing metadata subsystem with an implementation based on INFORMATION_SCHEMA? Or do you have architectural issues blocking you from making the changes?

[NickCraver]

@nvivo I think there are several problems and bad assumptions with the view here. The mere fact that someone was labeled Legacy which has not been superseded immediately indicates to me that the .Net team isn't aware of many use cases altogether, not that they have been consciously recognized and not prioritized.

As far as specific points:

Schema discovery is a hard problem, the current abstraction was not good enough to be universal (as you said)

Saying that the only method wasn't good isn't justification for removing the only method available.

It depends on DataTables which are considered legacy and most people don't use them anymore Schema discovery is useful, but is not required. 99% of the cases you can do just fine without it, and that's what people have been doing.

The current method does, yes. No one here is advocating for restoring that method exactly, just having a method. We're after the functionality without specifying implementation.

Most new projects (which is the focus of CoreFx) are moving to nosql/schemaless solutions

Citation needed. If that was true, why even bother to include SqlClient?

Most old projects never had the need for schema discovery

Sure, that's a fair statement. But some do. It's mostly libraries underneath that transparently used this before, which is mostly taken for granted.

I say let's keep the provider dead simple as it is, and see if it really impacts people.

It is impacting people, and we're letting the .Net Core team know. Right here, right now.

Schema discovery can be done by looking at the INFORMATION_SCHEMA

No, it can't. You're assuming the only schema someone needs is user tables. I'm working with DMVs currently. INFORMATION_SCHEMA doesn't include any of the system views. It also doesn't cover someone going SELECT * From MYTABLE - how do you propose I translate that to a known schema given the result set? That's a extremely non-trivial task. If you have a more complex query it becomes an exercise in re-implementing each platforms query parser to figure out which column came from where.

If there's an alternative way to get schema from a result set, please: I am all ears. As far as I know, it doesn't exist in .Net core. We need this functionality; it doesn't exist. These points can't be debated, only the priority (or not) or bringing functionality back is an open question. We really need to hear from Microsoft on that front.

[FransBouma]

@ryanbnl Not all databases implement all INFORMATION_SCHEMA schemas, and not all of them do it correctly (I've written meta-data retrieval systems for many RDBMSs and there isn't a way to obtain deep meta-data (like which FK fields are used in which FK constraint) for all DBs). But even if they all do, the problem is still that the information inside the schemas isn't of the same type in each DB, so you still need to know which type is which in which DB. But even that is perhaps retrievable up front and storable in a big mapping blob that's used every time.

The problem is of a different kind: you execute a query Q on database D and you get a resultset back. What are the types and characteristics of each column in that resultset? As the query is perhaps a select on a complex join with aggregates and derived tables, you can't simply say which type is what and how big the string typed column is and whether it's nullable or not.

So in that situation you need the information of the resultset. This information is available in the datareader, it's exposed in a generic API in .net full, and essential for some operations. this is missing currently in .net core and MS isn't willing to add it soon, if at all. That's the issue. :)

[ryanbnl]

Ahh, that makes a lot of sense. I've not (yet) hit issues of incomplete INFORMATION_SCHEMA metadata. That is indeed an massive advantage to using the .net metadata.

Do you have to offer 100% support or .Net Core targets? People targeting .Net Core next year will all be the early adopter types, they (we) can accept pragmatic limitations..

[nvivo]

@NickCraver,

It seems we have 3 different discussions here:

1) having datatable/dataview available in CoreFX somehow 2) having some way to extract schema information for objects and dynamic queries 3) keeping the old api for returning a DataTable with a schema from DataReader

I'm all in favor of the first two. (1) can be done in separate packages and it probably will and (2) can be added easily, but probably won't make it into 1.0, which is no big deal. I'm really against (3).

Saying that the only method wasn't good isn't justification for removing the only method available.

It is not the only method available. You can query INFORMATION_SCHEMA or any other system tables as you always could. That is what the provider does under the hood. The point that some small number of people want the provider to abstract it for them just to avoid some queries is something else.

I know INFORMATION_SCHEMA doesn't have all the data. The point is that this the most standard we will get. If the people creating databases couldn't come up with a standard way to represent this information across all of them, a .NET api won't either. Also the schema tables provided by .NET were never standard, they have some standard columns and a lot of custom ones depending on the provider. If you're doing something more advanced, you already need to decide per provider what to do.

How is that different from just calling select * from sys.something on SQL Server or select * from mysql.something on mysql if you need? This discussion sounds like there is no way to do it, or that it's too hard to get this information while it's very easy. Libraries will have to do it instead of the provider, but that's it. As someone who implements these kinds of libraries, I think we can do a better job abstracting this for the specific usage of the library than trying to make .NET come up with a general abstraction for the entire universe of all databases as it has tried before.

As for @FransBouma point on dynamic queries, I agree. But this can be done in a new API without dependency on DataTables:

interface IColumnSchema {
    int ColumnOrder;
    string ColumnName;
    string ProviderDataType;
    Type ClrMappedType;
    int MaxLength;
    bool CanBeNull;
    Dictionary<string, object> CustomProviderData;
}

IReadOnlyList<IColumnSchema> schema = reader.GetResultSchema();

We could also have a very simple interface to get the mappings for datatypes:

Dictionary<string, Type> mappings = providerFactory.GetTypeMappings();
// bigint = Int64,
// varchar = String
// etc

It's not the end of the world people, and it's not hard to come up with a good enough api for this. We could focus on discussing what we actually need instead of complaining that the old .net api is not available.

[NickCraver]

It is not the only method available. You can query INFORMATION_SCHEMA or any other system tables as you always could. That is what the provider does under the hood.

No...it doesn't. That is, to be blunt, a gross misunderstanding of what it does. It shows you the shape and schema of the result set. Let's take a very simple example:

Select 'bob' as Name

Show me how to get the column type and length for Name. INFORMATION_SCHEMA is related but not at all a viable solution for what we're talking about. Nor is it readily queryable since we're talking about the result of queries, not raw tables. That's assuming it even had the data in the first place, which it doesn't for DMVs, etc. And let's not forget stored procedures with dynamic output depending on the code paths and options passed.

Look back in this thread, no one is complaining directly about the API. We're pointing out missing, essential functionality. If it's restored via a new API that's fine, no one's debating that. But before going down the forking-from-CLR road, it'd be extremely nice for Microsoft to chime in with better reasoning. The reasoning provided thus far ("legacy") doesn't show awareness of the current use cases or the lack of any alternatives. That's fine, I'm not blaming anyone for that - but we've now pointed these things out and it's time for a more informed decision and prioritization I think.

[nvivo]

I just proposed a solution for that, it seems you didn't read my post. I'm here since the fisrt post, I'm aware of what is being discussed.

[FransBouma]

@ryanbnl

Do you have to offer 100% support or .Net Core targets? People targeting .Net Core next year will all be the early adopter types, they (we) can accept pragmatic limitations..

The thing is that they do need some sort of API to obtain meta-data of a resultset. And the only way to obtain the information in the first place is the ADO.NET provider itself, so they'll provide the code, and that means, as there's no generic interface, that developers have to hard-reference the ADO.NET provider, which is a pain, as .NET 1.x has learned us. So that might evolve in some sort of generic interface down the road but that will mean that code which utilizes that (ORMs, micro-orms etc.) will have to deal with the legacy crap that's left behind before that new API, i.e. the APIs of the ADO.NET providers offered themselves. We already have that with error reporting in ADO.NET today and it's stupid design and above all a burden (I know, I have to maintain code on that). This is a new framework, MS should invest in defining the generic API first, and then implement it in ADO.NET providers like SqlClient. But they're doing it the other way around: first SqlClient and if someone yells hard enough they'll perhaps implement it in a generic way.

@nvivo

I know INFORMATION_SCHEMA doesn't have all the data. The point is that this the most standard we will get. If the people creating databases couldn't come up with a standard way to represent this information across all of them, a .NET api won't either.

INFORMATION_SCHEMA's are standarized. SQL is too. Both are not implemented in full by all databases and by all versions of these databases.

Resultsets have nothing to do with these schemas, but with the actual query executed. Inside ADO.NET providers (the datareader implementations which produce data per row) this information is already available (as they have to know e.g. the type aspects of a column so they have to know how to convert a set of bytes into a .NET typed object).

Anyway, I've spent enough time debating your derailing attempts.

@terrajobst It's been long enough. Who inside Microsoft do I have to contact to get support for this added to the ADO.NET api? As the people from MS involved in this thread are incapable of getting anything going. Mind you, I started this issue on February 27th. That's almost 7 months ago (!).

[nvivo]

@FransBouma You obviously didn't read anything, so there is no point in replying.

Also, MS won't do anything with complaints. They didn't forget, they decided to remove the API. The only way anyone will get any results is to start a new issue and propose a solution that doesn't involve undoing exactly what they decided to do.

[dmitry-a-morozov]

@nvivo

It depends on DataTables which are considered legacy and most people don't use them anymore

This is slight overstatement. Look at what I've done in F#. http://fsprojects.github.io/FSharp.Data.SqlClient/data%20modification.html

You can of course hand-write all those INSERT, UPDATE and DELETE statements but it's a lot of boilerplate code. Also optimistic concurrency, in-memory merge and bulk insert support will be missed too.

[terrajobst]

@nvivo

They didn't forget, they decided to remove the API.

That's a bit of an overstatement. We simply decided not to port it yet.

[nvivo]

@terrajobst

We simply decided not to port it yet.

I was talking specifically about DataReader.GetSchemaTable(). It seems this was intentionally removed from the API, wasn't it?

[YoungGah]

Thanks for all the input and feedbacks. We will consider and evaluate bring in DataSet (might be in the same form as .NET Framework or in modified form) in the next version of .NET Core. For now, we are fully focused in delivering SqlClient including open sourcing and making it available on Linux and Mac. Once we achieve the right level of quality for SqlClient, we will start planning and evaluation for the next set of components/features to brining in. When we have more concrete plan, we will share out to the community. Regarding the feedback on SqlDataReader.GetSchemaTable, we are planning to add the functionality back to SqlClient in V1 although most likely it won’t in the form of DataTable. Please stay tuned.

Meantime, if you prefer not to wait for us to evaluate DataSet in the future version, the source for the .NET Framework implementation of DataTable and DataSet is licensed under MIT and is available on Reference Source. We are happy to provide guidance to anyone who is willing to port the implementation as a standalone library for .NET Core. The right way to start this would be to file an issue so we can lay down how a port could look like and what this means for libraries that want to target both, .NET Framework as well as .NET Core.

[FransBouma]

For now, we are fully focused in delivering SqlClient including open sourcing and making it available on Linux and Mac. Once we achieve the right level of quality for SqlClient, we will start planning and evaluation for the next set of components/features to brining in. When we have more concrete plan, we will share out to the community.

Does this mean you're focused on System.Data.Common and as a specialization implementation SqlClient, or are you just focused on SqlClient and is System.Data.Common the ugly stepchild which gets code only after SqlClient has proven that it works? Because you state:

Regarding the feedback on SqlDataReader.GetSchemaTable, we are planning to add the functionality back to SqlClient in V1 although most likely it won’t in the form of DataTable.

The functionality should be in DbDataReader, with provider specific customization in SqlClient. Only then we can be sure other ADO.NET providers will have the same interface and not, as I described earlier, each their own specific method which is only usable through either reflection or a hard reference to the ADO.NET provider (which is a STUPID thing to have in generic database code.) Sorry for the caps, but I'm starting to get a little fed up after 7 months of waiting for an answer and then reading about Microsoft only focusing on SqlClient.

Asking people to do your job by asking them to port the code is only going to work if Microsoft stops focusing on their own products first: as it then never will become a community effort but will stay a Microsoft product.

[YoungGah]

@FransBouma

System.Data.Common is an important investment to us as well especially as it will provide the common interfaces for all types of providers. We will continue to invest in System.Data.Common and extend them as we add features just as we have done on .NET Framework. We just believe that System.Data.Common is in better shape than System.Data.SqlClient at this point although System.Data.Common has bugs we need fix. And, yes, you are correct that we will have to add GetSchemaTable on DataReader as well. I didn't mean to imply that we will only add it to SqlDataReader.

[FransBouma]

@YoungGah Thanks for the clarification! then I have no further questions and hope you and your team can pull this off to everyone's satisfaction :)

[nvivo]

@YoungGah. This is great information, pretty much what I was expecting. Will there be an issue to track the design of the data provider interface? It would be great if the community could provide some feedback before v1 on this core component.

I had issues trying to mock the original providers, as they do a lot of internal calls. Would be very useful to have real interfaces formalized and updated instead of removed, so we can write code these interfaces and mock them using standard mocking frameworks.

[YoungGah]

@nvivo Absolutely! We are tossing around ideas of the API returning schema in JSON format. Any opinions? The issue is track at dotnet/corefx#3423.

[andrew-vdb]

@youngGah I need DataTable as it is part of my data structure, just like anyone use list or stack. You can put WriteXml or WriteJson as extension method right?

[lstratman]

@YoungGah mentioned previously that the best way to get DataTable/Set/View into CoreCLR in the immediate future would be to implement it in a standalone library. I may be able to take that on, but is there an idea on when those classes will be officially implemented, even if it's extremely vague, like fairly soon after the initial release, or the end of next year? The project that I'm going to be converting will be depending on DataTable and I just want to make sure that it's going to be worth my time to do the implementation.

[FransBouma]

A separate library won't help, there's an essential aspect missing: the meta-data api in DbConnection and DbDataReader which you need to create datatables prior to filling them with data. No meta-data api: no untyped resultsets and no datatable usage as it is used today.

What's so mindboggling is the feet dragging over at Redmond about designing this meta-data API. As if it's something so new and so weird, it takes ages to come up with something.

[sanjoych]

So System.Data.Common is feature complete now for V1 of CoreCLR?