ESENT database provider

[Opiumtm]

Provide Entity Framework 7 provider for a ESENT NOSQL database. This database is officially supported on UWP platform, but now there is only ISAM imperative access to ESENT DB in C#.

ESENT is proven as very fast, stable and reliable NOSQL database, and it is supported in UWP apps. It's a good alternative to SQLite on UWP platform. But ESENT lacks any high-level API, it have only low-level imperative C++ style APIs for C#.

[ErikEJ]

Could you link to the current C# api?

[Opiumtm]

Current ESENT C# api:

https://managedesent.codeplex.com/

[Opiumtm]

And here documentation of ESENT C# on MSDN: https://msdn.microsoft.com/en-us/library/dn375980(v=exchg.10).aspx

[Opiumtm]

ErikEJ, could you answer if EF Core team planning to support ESENT DB in future? To be honest, I'm not satisfied with SQLite on UWP platform. I have used SQLite many times in a past and it certainliy not a best option. But today UWP platform have not any other options, only SQLite DB. It's very weird, because UWP platform (and Microsoft Windows as well) natively support ESENT embeddable database, known for decades as a good, stable, fast and reliable database engine. The only problem with ESENT its out of date APIs, lack of support from any major data access frameworks. In the server or desktop worlds it's not a problem - there is many other supported database engines, SQL or NoSQL. But on the UWP platform officially supported database engines is scarce. SQLite is recommended by Microsoft as a primary solution. And it's very weird! Microsoft have it's own well tested and proven reliable embeddable database engine, ESENT. And Microsoft is totally silent about ESENT. It's officially supported on UWP, but to productively use ESENT in UWP you must be a die hard C++ developer, or at least have this hardcore experience in the past. Newbie developers have not any chances to use ESENT in their UWP applications, because APIs is very hardcore and aged, no sane C# developer would use ESENT when there is much easy to use SQLite option. But, I must write it again, SQLite is not a best option technically speaking. ESENT is a much more technically advanced database engine. For an embeddable database engine ESENT is certainly a piece of art.ESENT supports snapshot isolation (feature supported not by all server-scale full blown SQL databases!) for a transactions, multi-value index keys, pre-joined tables, an much more... And same time ESENT have very small memory and CPU footprint, it's truly embeddable database, suited for use even on a smartphones (and internally Microsoft use ESENT DB in many parts of Windows OS itself, ESENT was an internal OS related project made publically available some time ago).

[ErikEJ]

@Opiumtm I do work for the EF team, so I am not aware of any plans. One major issue I see with ESENT is that it does not have a query language and a query processor as in a RDBMs system. But I think it would be a good candidate for a useful NoSQL provider, and I would maybe be interested in taking a stab at it, but the main issue with that is, that there are no NoSQL EF providers implemented based on the current codebase. The plans as I understand them are to revive the Azure Table Storage provider, which exposes a key/value storage quite similar to the PersistentDictionary.

[Opiumtm]

@ErikEJ

major issue I see with ESENT is that it does not have a query language and a query processor as in a RDBMs system

It will not be a problem because ESENT does not have its own query language, but it anyway a relational DB and ESENT API supports querying, indexes, tables, joins via its own ISAM imperative style (open table, have an open handle, add filters and indexes, enumerate filtered records via method calls and so on) API. It does not differ fundamentally from any relational SQL database (ESENT is a perfect example of a relational database) and ESENT APIs can be mapped to a LINQ language without a SQL intermediate language. In fact, this direct mapping to a LINQ queries will be much faster, because it will be mapped directly, without intermediate text based language, without query processing overhead.

So, ESENT is a relational NoSQL database (NoSQL in a sense that it does not support SQL queries, but support relational queries and operations directly via low level API). ESENT is well suited for a direct LINQ-to-"low level relational API" mapping!

[ErikEJ]

@Opiumtm Sounds great!

[Opiumtm]

@ErikEJ yes! And think about ESENT is an embeddable database, designed to use in a resource constrained environments (but it as well used at an enterprise level too - just think about a fact that ESENT used as a data storage solution in such products as Miscrosoft Exchage Server and Active Directory!). Its embeddable nature matches well an opportunity to direct map its relational API to LINQ query language without any intermediate layers. It will be a perfect solution for an UWP universal platform, it will make (honestly speaking) mediocre SQLite out of competition easily.

[Opiumtm]

@ErikEJ Also I personally have some experience in ESENT C# use on an UWP platform (and some past experience long time ago in a native C++). Major negative factor for its use is not a technical issues (it works good technically) but an out of date low level API and lack of support from any data access frameworks. It require a big amount of code to use in production. When you familiar with ESENT hardcore low level API it is not a problem to write a workable code, but a problem with a big amount of a boilerplate error prone code still persists, or you must invent your own homebrew framework for a personal ESENT use.

[rowanmiller]

@Opiumtm this is interesting because SQLite is very limited in what it supports, and this flows on to a bunch of limitations in EF functionality (see http://docs.efproject.net/en/latest/providers/sqlite/limitations.html). I have never looked at ESENT, do you think it would support a number of the things that SQLite does not?

[Opiumtm]

@rowanmiller Not sure, what you mean as "schemas". If you mean a queryable database structure scheme - it's supported. If you mean "schema" as a SQL logical construct - it's not supported by ESENT because ESENT is not a SQL database, ESENT does not support access rights (it is supposed to do by an application layer). ESENT is a relationanal transactional (it supports snapshot isolation) database, but it does not have any text based universal query language, all queries, sorts and joins are performed via direct ESENT API calls, ISAM style. So, ESENT is perfectly suited for a direct mapping of relational API method calls into a LINQ queries without any text based query language intermediate layer. ESENT is fundamentally designed for this approach as a fast small footprint embeddable database engine - omit any intermediate logical layers. ESENT is a perfect candidate for a LINQ query language support.. LINQ by design is not a text based language but a direct mapping to a method calls represented in a compiled code or in an abstract query tree (in case of query expressions). To be honest - SQL is an useless excess entity here. SQL is not needed at all. LINQ fundamentally is a declarative style method mapping to a relational operators - and ESENT supports exactly this, relational operators without useless (for an embeddable database) intermediate text based language.

[Opiumtm]

@ErikEJ @rowanmiller guys, could you share more info about planning (or not planning...) a support for ESENT DB in EF Core?

[ErikEJ]

@Opiumtm I do not work for the team, I am just working on the SQL Server Compact provider. I might consider having a look at this when the Azure Table Storage provider has been released, after RTM of 1,0. I do however see limited scope in the Esent provider, as it is only available for desktop and UWP tablet apps, not Windows Phone (or any other platforms). One advantage of sqlite is that it is available on all platforms

[Opiumtm]

@ErikEJ You are not correct. ESENT is officially supported on Windows 10 Mobile for ARM and I already use ESENT in one my UWP application which targets both desktop and mobile. So, Windows 10 for mobile support ESENT and it's tested.

[ErikEJ]

@Opiumtm That is great, did not realize that!

[Opiumtm]

@ErikEJ I successfully deployed application that use ESENT on a Lumia 920 smartphone and it works nice. ESENT supported on smartphones as of Windows 10 version of Windows Phone (now Mobile).

[rowanmiller]

@Opiumtm this probably isn't a provider that our team would develop, we typically work with database vendors or third parties to develop the vast majority of providers. This allows us to push forward with new features in the core of EF.

[Opiumtm]

@rowanmiller And it's weird because ESENT is a native Microsoft embeddable database known for decades for its speed and reliability, it's an official part of classic native WinAPI (and part of UWP API as well). I do not understand why ESENT is not considered as a first-class database provider by a Microsoft "recommended API" team. As I described above, ESENT is perfectly suited for LINQ. If EF Core does not suited for ESENT - it's a major issue. Maybe modern cross-platform minded developers does not care about this brilliant Windows classics, but for an aged veteran windows developer like me it looks like a betrayal and spit directly in a soul. Sorry for an emotions, but it's exactly what I feel. And it certainly not a wise decision. You must not discard things that works perfectly and promises strong technical advantages. ESENT simply is far better than SQLite and its lack of SQL query language support is not a weakness for an embeddable database suited for even a smartphone use. SQL language is not needed at all when you can directly map ESENT relational API to LINQ queries. SQLite is a mediocre and limited database that suffer from a performance issues when database size is slightly larger. ESENT on the other hand is very well known for its speed and scalability. And ESENT supports snapshot isolation for a transactions, have normal strong-typed data model instead of SQLite lame weak-typed data model. And it all we got in a embeddable database. It's certainly a CRIME to does not support ESENT for an embedded database scenario for an UWP applications!

[ErikEJ]

@Opiumtm I am sure everyone would be happy for a open source contribution to enable this provider

[Opiumtm]

@rowanmiller Well, and at EF 3-6 era I understands why ESENT is not supported by Entity Framework. In the past EF was a strictly SQL language based framework. As ESENT does not support SQL language queries it was not a subject for an EF support. But now it is claimed that Entity Framework Core does not tightly binded to a SQL language and it will be NoSQL providers for an EF. When I read this on a technical news feed I think - time have come. I think Microsoft would certainly support its own ESENT as a part of a first-class EF providers, especially when there is claimed that EF Core is suited for an UWP platform too. SQLite only support for an UWP platform is a lame. I understand that EF goal is primarily to support enterprise-scale server scenarios involving Microsoft SQL Server, PostgreSQL, Oracle and so on. But please, does not forget an embedded and client-oriented world! Until now there was very few options for a client-oriented .NET developers. SQLite is a well known cross-platform client oriented embeddable database, but certainly is not a better one. It will be nice for a UWP developer to have an alternatives to mediocre and not best at all SQLite database (which personally I consider a lame and mostly unprofessional database engine flourishing on the client-side because of lack of an adequate and well-known alternatives, and ESENT is certainly an adequate alternative for a Windows Platform and it here for a decades). And after all ESENT is a Microsoft native database and must be respected at least by a Microsoft staff. It certainly deserves more respect. As I see, knowledgeable 3-rd party developers have more respect for an ESENT DB than Microsoft themselves.

[Opiumtm]

@ErikEJ It's bad that ESENT DB have not much attention in a developer circles. Known not by all, even not well known by a many Microsoft teams. It was a shocking fact for me that ESENT is not well known even to a Entity Framework Core team! It looks like a sadomasochistic perversion when Microsoft officially recommends SQLite for a client UWP applications and totally silent about anyway officially supported ESENT. What? Microsoft staff are totally replaced by an iOS hipsters or what?

[divega]

@Opiumtm ESENT is actually one among a few data stores that were discussed when we started prototyping non-relational providers at the very beginning of the development of EF Core. Personally I agree that an EF Core provider for it would be a very good addition and potentially a rich and unique source of challenges that could help us improve our designs.

Plans and priorities can change at any time, however as @rowanmiller explained we currently intend to only focus on building providers for a very small set of stores. For the rest we prefer to partner with people from the OSS community, from database vendors and other teams at Microsoft that already have the necessary expertise on the specific data store technologies and that are willing to go through the exercise of learning how to create an EF Core provider with us.

I have been following this thread with attention hoping that you would be interested in contributing to that effort. As @ErikEJ mentioned, after EF Core 1.0 RTM and after we have made more progress on some of the non-relational providers using the current codebase would be a much better time to start with an effort like this.

[Opiumtm]

@divega

ESENT is actually one among a few data stores that were discussed when we started prototyping non-relational providers at the very beginning of the development of EF Core

Well, ESENT is a relational transactional ACID-compliant database provider. "SQL language" is not equal to a "relational", it's just a relational database ANSI standard for a text-mode API and a declarative programming language. Just point to this again because there is some theoretical misunderstanding on this thread.

however as @rowanmiller explained we currently intend to only focus on building providers for a very small set of stores

For a server enterprise-level tier ESENT provider certainly is not a top priority. But it's vital for a mobile client-oriented UWP platform. And EF Core is designed for an UWP platform too. So, it's very different set of real life requirements and challenges on a different platforms. As Joel Spolsky said in his famous Law of a leaky abstractions article, it's an exact situation. Abstraction of "support of a most needed data providers" leaks when we see that on a UWP client-side platform there is only mediocre and arguably lame SQLite provider is supported and there is no clear plans for much better alternatives like ESENT (which itself is violently kicks a leaky "SQL language" abstraction, that certainly leaks on a mobile and embedded platforms and become such abomination as SQLite database engine is).

I have been following this thread with attention hoping that you would be interested in contributing to that effort

Personally I thinked about this. But as for now I am already writing a some open-source client oriented project (not on a GitHub, on a CodePlex). Maybe later I can do some contributions to a ESENT DB provider, and in near future I plan to upgrade one of my Windows Phone applications to a Windows 10 universal UWP platform and migrate it from a SQLite to ESENT (which is supported starting with UWP Windows 10 and was not supported on a Windows Phone 8.1 runtime). This certainly will give me a live measurements of an ESENT usage in a wild. As for SQLite, I already got a serious performance degradation issue with it when application is actively used and data in a database updated many times after a each-day usage pattern and grows to a small but many indexed records. In my somewhat distant developement past for a native desktop Win32 platform I already got this SQLite issue when it used for a client-side database cache. It works well and fine first time but as time pass and cache grow (not to terabytes or even not to a gigabytes, I must say!) SQLite seriously suffer from an enormous performance degradation to a point of a total uselessnes. Maybe I need more SQLite expertise and must invoke some magical SQL commands to optimize and defragment SQLite database... But it certainly a discussed above point to a law of leaky abstractions. SQLite abstraction of a "full featured SQL database on a smartphone and embedded system" is certainly leaks. SQL query processing on an embedded system is have its overhead, is not complete, is not optimal (SQLite certainly does not have such a powerful query optimizer as Microsoft SQL Server or PostgreSQL have) and come with many compromises and limitations - and it's known that SQL query language itself is very abstract and good performance is up to a database query optimizer, not a SQL query itself which does not represent much of a detail. It's how abstractions leaks and it is not a SQLite code base issue, it's an issue with a fundamentally lame abstraction of a "full featured SQL language support on an embedded system" which naturally leaks. LINQ on the other hand is more specific and less abstract than SQL query language and it naturally represents a relational operators call tree. LINQ may be optimized by a programmer in a more straightforward way than SQL and it is optimized by a developers when queries are made not to a relational data sources but to a XML in-memory datasources or a to a in-memory object tree and this LINQ query mechanics is not hard to understand.

After all, I am not against some contributions (at least theoretical) to an embedded scenarios of EF Core including ESENT.

[divega]

ESENT is a relational transactional ACID-compliant database provider

Thanks for clarifying that. I had read you mention that before, so sorry if I used the term inaccurately anyway :smile: BTW, when I say "non-relational" in the context of EF Core providers I am actually using it at a shortcut to refer to a category of providers that won't inherit from our base relational provider. The latter makes lots of assumptions about SQL being supported, an ADO.NET provider being used to execute commands, etc.

Maybe later I can do some contributions to a ESENT DB provider, and in near future I plan to upgrade one of my Windows Phone applications to a Windows 10 universal UWP platform and migrate it from a SQLite to ESENT ... This certainly will give me a live measurements of an ESENT usage in a wild.

Cool! We also need to do some groundwork to enable this kind of non-relational provider again on the EF Core 1.0 codebase. I hope the timing will work well and to hear back from you.

[Opiumtm]

@divega if it's not too late before release consider rename your inside EF Core terminology from a "relational" to a more correct "SQL based" :-) It may be a big benefit for a better theoretical understanding of a EF Core codebase and architecture. "SQL based" in practical terms means only that LINQ queries is translated into a SQL intermediate language and SQL representation of a query is a central point of a "SQL based" provider, it's all about generation of such a SQL representation for any raltional query and operation, even for a database tables construction at a run time. "Relational" is a more general term and does not require SQL language (or any standartized intermediate text based language at all). Tables, indexes, rows, primary keys, relational operators ("select" or "join" for example) - it's a "relational" and it's supported by many non-SQL databases, especially by any ISAM database engines (on top of which SQL database abstraction can be naturally built, as it did by MySQL database on top of MyISAM storage engine). So, any classical ISAM engine is relational but not SQL.And as ISAM engine can be upgraded into SQL engine naturally. Historically most well known SQL databases (starting with IBM DB2) are indeed built on top of classical ISAM or some ISAM variation - it is correct to generalize "relational" term into an ISAM data source providers, which are anyway historically a foundation for SQL databases, on top of which SQL query processor and SQL query optimizer are built.

[Opiumtm]

@divega So, "NoSQL" term is largely incorrectly used buzz-word and much abused in a modern developer circles. Non-relational databases are hierarchical databases (every day used OS file system is a perfect example of a hierarchical database and most FS are built on top of typical database B+ tree data structure and typical database journaling for data integrity - just think about it), document oriented databases, "key-value" databases and so on. It's all examples of a non-relational DB. "NoSQL" term as it used by a most developers is based on a false assumption that "SQL" an "relational" terms are equal and it drives most discussion into a theoretical misunderstanding and confusion. They are not equal. SQL databases are most used implementation of relational databases, but "SQL" is a subset of "relational", not equivalent.

[10Dev]

"ESENT Data Provider" and #329 "Migrations: Table rebuilds" are boring issue titles that mask the emergency situation that there is still no practical working EF on UWP over 7 months after Windows 10 was generally available.

Should we be making a specific new issue for that?

I have listed an 8 step plan in #329 which was written in a light-hearted style to encourage creative thinking, but one of them could perhaps be the spark for a solution.

Maybe I need to back up 1 step and confirm with anyone on the EF team that UWP is actually an important target platform for Microsoft and why 7 months into Windows 10 there is no sense of emergency priority about what appears to an outsider as an absurd situation.

[rowanmiller]

@10Dev @Opiumtm (also addressing comments from https://github.com/aspnet/EntityFramework/issues/329#issuecomment-187429902)

It seems like there are two main concerns/questions that underpin your comments:

• Why are you focusing on SQLite when it has the limitations discussed? Why not something else, like ESSENT.
• Why did we not have EF support for UWP earlier (like at RTM of UWP) and with support for more database providers.

Here are a few thoughts that hopefully provide some insight into where our team is coming from.

EF Core is a long term strategy. There are lots of things (features, database providers, etc.) that we want to have, however we are still laying the foundations of this product to ensure we have the right core to build on for the coming years. To that end, we need to focus on a subset of the problem space to get to an initial v1 of the product. We are also very much aware that EF Core will not be the right choice for many folks even when we reach v1 RTM. We know that the missing features and/or database providers will mean many applications can’t use it. That said, the other option is to build all those things now and delay the v1 RTM for everybody.

I think it’s generally accepted these days that assigning more people to a software project doesn’t automatically result in a reduced timeline or more features in the same timeline. Sometimes it does, but when it comes to v1 products where you are building an architecture and components that will underpin future releases, adding more people usually just creates more chaos and churn – thus pushing out the date at which you have a stable core to build on. That was actually what drove our decision to shelve out Azure Table Storage and Redis provider work so that we could get to a stable v1 release.

The feedback we heard from the developer community before starting work on EF Core was overwhelmingly that SQLite is the database of choice for local data. There are obviously reasons that SQLite isn’t amazing, but it is widely used and available on the vast majority of devices. I totally hear that you would love to see ESSENT supported too (or with a higher priority than SQLite) – and that is 100% valid. I just want to get across that we didn’t make the SQLite decision in a vacuum – we decided to start there because that’s what our customers wanted. Obviously no decision will satisfy everyone, and if EF Core doesn’t have what you want at v1 then I am going to be the last person to tell you that you should be using it anyway.

Regarding the concern about EF being “Microsoft's recommended data access technology” but having limited providers available on UWP (raised by @10Dev in #329). We are not applying that statement to EF Core now, or when we reach v1 RTM. EF Core is very much a v1 and it needs time to mature before it should be the defacto choice over EF6.x – this was the primary driving force behind the change from EF7 to EF Core. Of course, EF6.x doesn’t work everywhere, so if you really want to use EF on UWP then you will have to use EF Core – but I am not going to push you to do that over another technology that is working for you today.

Regarding timing compared to the UWP RTM. We started EF Core a couple of years ago, and building a solid data access stack just takes time. UWP didn’t even exist when we started (we originally used wpa8.1 as a proving ground). I agree it would have been great to have EF Core ready to go for when UWP shipped… but we missed that train long before we even knew it existed.

Hopefully that provides some insight. I realize I’m unlikely to change your mind, but I wanted to at least give an honest answer to your questions.

[rowanmiller]

@10Dev @Opiumtm

I want to be clear that the following comments are from me personally, not my team or company…

I understand that online interaction with someone you have never met lacks the personal connection you get from sitting face to face. That said, don’t forget that comments on social media, GitHub projects, blogs, etc. are still ultimately directed to a person (or set of people). I realize from your end that you may feel like you are directing a passionate objection toward Microsoft, but in reality we are a team of individual developers. Individuals who enjoy being torn to shreds about as much as anyone does. We care deeply about data access, we care about our product, and we want to build something that helps our customers get their jobs done. We try to make the right decisions, sometimes we get it wrong. Even when we get it right in the eyes of the majority, with over 16 million downloads of our NuGet package, at least some folks that own those projects are going to disagree with what we do.

If I was discussing my product face to face with someone and they called me “sadomasochistic”, an “iOS hipster”, or seriously suggested that I go picket my CEO’s office… then I would politely excuse myself from the conversation and walk away. A different outcome should not be expected if that interaction is online.

Today at work I was in a good discussion about the need to assume the best in people – something I do not always do. I am going to apply that to our interaction and look forward to collaborating with you in the future.

Peace, Rowan

[divega]

Just :+1: to everything @rowanmiller said.

[ajcvickers]

+1 to @rowanmiller's comments

[10Dev]

Hey Rowan,

I didn't say anything I wouldn't say straight to your face in a meeting, but you would have seen my smile as I said it. I did point out that the intent of my "8 step plan" was a light hearted way to get some creative ideas started.

[10Dev]

I don't know Opiumtm but I suspect he also was making some attempt to lighten things up but who knows.

The one thing that is certain and you might have fogotten here is that we wouldn't be making these somewhat passionate crys for help if we didn't deeply admire the design of EF and desparately want to be using it with UWP. Within that context there are absolutely zero alternative choices for UWP. When you say "it's ready when it's ready and we aren't forcing you to use it" you are forgetting that as our drug dealer you have already addicted us to the Magic Unicorn crack cocaine of EF Code First Data Migrations.

You already have my eternal respect.

[10Dev]

Although surprsingly I run into developers all the time unfamiliar with Brook's Mythical Man Month, as I said previously I was hoping some sort of creative thought or brainstorming could uncover options to consider that would not derail the core effort.

No matter how it came to pass, the lack of a workable EF target for UWP this long after the Windows 10 launch is going to end up being seen as an absurd situation. The SQlite target might help tick off a checkbox but as soon as you hit some real world complexity, the Unicorn dies a very painful death, and that's really sad for something as beautiful as the Unicorn but developers will stop dead in their tracks in a giant WTF moment, evaluate their options with a sinking feeling in their hearts and realize they will have to choose from a palette of unapetizing options. As an act of kindness, you could perhaps make an even larger callout in your documentation asking people to avoid using SQLite until #329 is completed.

And I suspect the effort to handle #329 will be even worse than it looks as SQlite edge cases will come out of the dark closet to haunt that approach but if no alternative presents itself then I hope it can be made to happen. And it's easy to see that if you resourced an additional team from somewhere the bandwith of interaction with the current team would be high.

But it seems that perhaps a project for a Data Provider might be able to be comparmentalized in a way that doesn't violate Brooks. Whether that might be ESENT or some other DB that runs on UWP, it might be the right approach for EF to take ownership on a UWP solution even it it's a bit out of band for the types of things the EF team might normally undertake?

Or maybe the SQL Server team or the Azure guys could get dragged into it?

[rowanmiller]

I didn't say anything I wouldn't say straight to your face in a meeting, but you would have seen my smile as I said it.

Understood, the subtleties of humor, sarcasm, etc. do get lost in the online world.

And I suspect the effort to handle #329 will be even worse than it looks as SQlite edge cases will come out of the dark closet to haunt that approach.

Absolutely agree, that is one of the reasons we are hesitant to head down that path. We've actually been discussing it since the EF4.1 days... but it does have the makings of a Pandora's Box.

But it seems that perhaps a project for a Data Provider might be able to be comparmentalized in a way that doesn't violate Brooks.

This will be absolutely true once we reach RTM (or get closer to it). Each provider we add requires some changes/improvements in the core and uncovers bugs that are not hit by other providers - which at the moment works against our efforts to stabilize for the providers we have. But yes, soon spinning up small teams to work on a provider (internal and external) will be productive.

[Opiumtm]

@rowanmiller I do not mean any personal offence and when I pointed to a UWP problems and overall platform inconsistence (after all, ESENT and SQL CE are native Microsoft products and SQLite is a not Microsoft solution and it have first-class support instead of native Microsoft technology) I described just how it looks from an outside world. I do not feel personal hatred against any member of EF Team. I feel very disturbed with general direction of things and I think if no one points to a problem - it does not mean problem is not exists. When I say about "iOS hipsters" and "sadomasochistic" - it is exactly how it looks from an outside. I know it is not true, but it is how it looks and it leave bitter taste in a mouth. It may be somewhat rude, but it at least honest and personally I wrote it with good intentions and with love for Microsoft and its technology. If all people will be too polite and politically correct, no major problem in our society will be noticed (and personally I think, political correctness is already ruining society). Sometimes it is good to be straight (in all meanings of this word, he-he) and not be quiet when problem exists. I can certainly tell why majority of people wanted SQLite support first place and why they voted for it. It is not because SQLite is best solution. It is because SQLite is a most well known solution in a mobile developing world. When you google for an "embeddable database" you will see a SQLite in the top and no links to ESENT at all. Did you think about WHY there is so much links to SQLite? It's because SQLite is heavily advertised, for now it is advertised even by a Microsoft itself on a MSDN. When you see it from this point of view, such a poetic metaphors as "sadomasochistic" and "iOS hipsters" are not looks much rude, it looks like a bitter truth which must be spoken openly by a loyal Microsoft developer veteran. Microsoft does not paid much attention to their own technologies in a past (ESENT itself is not well known at all, people who know about ESENT can be counted by a fingers on a hand metaphorically speaking), and there is perfectly expectable outcome - Microsoft already lose developer acknowledgement on most of its outright brilliant technology. Even when Microsoft deliver a better technology most of the modern developers are totally ignorant and do not want to play with Microsoft at all. I'm not joking. I work in a developer firm as a senior C# developer and architect and I feel myself as a (sorry for a rude speach again...) sexual minority in a modern developer environment. Everybody speaking about Linux and cross-platform. Everybody fap on a Apple devices, even a people who work as Windows Server system administrators. Everybody tell that Microsoft is utterly a legacy platform. Did you hear this? Did you understand this? As a senior Windows developer with also a heavy UNIX-like C/C++ developing background I do not agree with that. I do not love Linux and have many reasons to this. I do not love Java, Python and so on. But if things with Microsoft commitment to its own intellectual property is not changed (if a course to abandon many good things) remain - I think my next work will be again a C++ development for a NIX operating system and it's matter of some years before vast majority of developers will abandon Microsoft platform once and for all. If you (not you personally, I speak about Microsoft as whole) try to adopt "most adopted by developers cross platform technologies" such as SQLite as first priority thing for a flagship UWP universal platform - it's just a next stage of acceptance of failure. When you must fight bravely you accept yourself as loser and leave yourself on a mercy of victor. As history tell us, victory is never come with a mercy. It always come with death of a loser, fast and painless death if you are lucky. So I told you some emotional background about my poetic metaphors on this thread and provided some practical knowledge about why people asked you to support SQLite on a UWP platform. It's because people see no alternatives and because people are totally ignorant about alternative technology on Windows platform. As I see, this ignorance about own intellectual property is poisoning even a Microsoft itself. Just open MSDN in a browser and read SQLite is a recommended data access solution for an UWP platform. Next step will be to abandon UWP at all and deliver an Android distributive as a recommended Microsoft Mobile Platform. Why not? It's the direction in which Microsoft is moved last years. Maybe customers will not care (they already are very ignorant about Microsoft technology and see Microsoft products as legacy at best), but for a veteran Windows developer to see that is like to see a slow death of own wife from a cancer and be unable to help.

Well, and when we cry about support of ESENT and anything else for a UWP we cry in desperation. There is basically no good local database solution for UWP. It's our everyday reality and we must face it outright. There was SQL CE local database for a classic desktop .NET apps. On UWP platform there is no choice but already well known for a years as mediocre and despised by a hardline developers SQLite nonsense.

So, excuse me if I was too straight and not too politically correct on some points. I just wanted to speak truth as I personally feel it deeply in my heart. I mean no offence and do not want to attack anyone personally. I speak this not so beautiful looking truth in an attempt to make things better and with an admiration of a Microsoft developer platform.

[10Dev]

If we can choose to ignore individual words here and there, Opiumtm is expressing a heart felt desire for the success of EF.

When looking at the entire Microsoft development stack, there appear the a few points where missteps in execution or communication have led to massive developer issues and abandonment of the platform. There are probably a huge number of developers sitting on the fence post right now to see if UWP platform will be another dissappointment or will spread some wings since the architecture provides a way for all organizations large and small to move forward into the future in a flexible manner covering a huge range of target devices. One of the potential roadblocks is EF not showing up. For people like Opiumtm who care, it makes them fearful of how this will work out.

And Yoda says fear leads to anger and anger leads to the dark side of the Force.

I don't have a crystal ball to know if the trigger event of killing Silverlight which led to huge adbandonnment ended up producing a "Dead Man Walking" situation or if the recovery is just around the corner with huge uptake of Windows 10 leading to universal UWP but the concerns are real.

If the EF team has been trying to figure out this stuff since 4.1, then I am doubting I can come up with any great ideas but can only think that clear communication by the EF team who are both indviduals and also part of some nebulous concept called "Microsoft" will be important.

For your daily mental gymnastics, just like electrons are both particles and waves at the same time, if you hear bad things direct that to the Microsoft "entity" and if you hear good things, well that's the individual team members...

[10Dev]

As a minor idea, if it doesn't degrade your workflow, it would be sensitive to your "customers" if you could re-tag this issue to "bug" (or nothing) since having a working EF on UWP just can't be seen as an "enhancement" to the outside world. In that context, "up-for-grabs" is like a slap in the face and "under review" might be a better choice?

And #329 is a "bug" which maybe also needs "can't fix" but #329 is surely not an enhancement in the context of a fully working EF on UWP.

[Opiumtm]

@10Dev I think there is never too late to recover. Yes, I am very sceptical about future of Microsoft developer platform. Not because I like this skepticism. I'm certainly not like it. It's because I can not deny it anymore. Yes, rapid abandonment of own technology is not help to make things clear. There is no point to invest in a new platform/technology when its future is utterly uncertain. No matter how brave and determined Microsoft looks at developer conferences when speak about some technology. As we have learned already, bravado on a scheduled developer conference essentially mean nothing. Last years Microsoft is too accustomed to abandon things even before it fully mature. And most terrifying of all Microsoft is too eager to accept "actual developer trends". Not dictate such trends, merely accept. For example, I see JSON as very weird format for a project config format in VS 2015 .NET Core projects. JSON essentially is a JavaScript bastard child, widely (mis)used because so much people are Html/JavaScript monkey coders (just name things honestly) and they are too well accustomed with such a ad hoc format as JSON is. Well, it looks good in a browser, but in a VS solution folder it looks like a homeless miserable at a aristocracy banquet. It's just a wrong place for him. It's just one example, but it good in illustration of how Microsoft essentially shifted from a major influential force in a developer world to a merely "accepting the trend" position even within its own environment. It is utterly a passive position. Well, any extremes are harmful. But today we see a very passive Microsoft, passive even on its own platform. Microsoft is so passive it could abandon any technology any time because some girlishly looking hipsters say "this technology is out of trend pheeeee".

[10Dev]

Opiumtm, I enjoy considering your thoughts and hopefully with the huge wealth of focused code level issues on the EF list, the team will be kind enough to tolerate a few tangental comments related to this rather key issue.

I always feel a bit weird when people refer to a large company with many thousands of people as if it is some sort of pseudo-person with its own thoughts and desires... I'd rather think of Microsoft as a kind of process flow diagram for a nuclear power plant.

Another point is that people in that "plant" have access to huge amounts of metric data about sales and usage although I suspect some of the large blunders come from how such data is compressed and "sanitized" to upper management. But they have a lot of market data that nobody outside the organization has acess to and it is part of their often perplexing decision making.

It is possible that what you are seeing as "passive" is an attempt to be "inclusive" and there is also the horrible list of communications and marketing firms they have used over the last decade. Most of the Silverlight mess which did huge mindset damage was just horrible communication. (with a possible dash of typical large corporation absurdity - rumour has it that Balmer killed Silverlight as part of a secret deal in CEO to CEO meeting he had with Adobe becuase he saw the Silverlight effort as a "Flash Compete" and didn't realize that a large numer of enterprises were planning to adopt the technology for in-house application deployment via the OBE option) and then they disbanded the WPF team and it just looked like a complete slash and burn to the outside world.

But all that wonderful technology is still available, the WPF team is operational again and XAML has been embedded into the Windows API to yield both performance benefits and universal availability to other languages and the things you can get done in the Microsoft ecosystem is better than it ever has been (well, except the EF UWP thing) and somehow even now the positive steps are not being well communicated. It's not "pasive", just failure to communicate.

The EF team has developed an excellent world leading product that has also been under-promoted. At the least they should have a dedicated web site leading out with "As a developer is your time worth anything at all to you? The EF Team gives you SuperPowers in a Bottle"

[Opiumtm]

@10Dev I agree with you on that now Microsoft doing many truly wonderful things. For example, .NET Core project and .NET foundation activity as whole are things that make .NET what it intended to be from the start - truly cross platform not in a cross-OS meaning, cross platform in a meaning that it can run on a wide spectrum of different hardware platforms, existing and future. But, all right and wonderful long-term movements does not answer a sharp questions of present day. On present day there is very much amount of uncertainty, unfilled gaps, lack of public relation (for example, there is no understandable roadmap to Microsoft platform and UWP future), lack of integration and lack of backward compatibility with a lot of .NET classics and lack of feeling of that Microsoft is truly dedicated and put enough resource to make developers certain in their future. After all, our future is tightly coupled with a future of Microsoft development platform. It's our wages, work places, business opportunities, our sales.

[rowanmiller]

I'm going to wrap this one up with a few comments and then step away from the conversation for a while.

We hear you on the fact that SQLite has some serious limitations that do surface heavily in the EF workflow. We are not ruling out the idea of table-rebuilds (#329) as a way to solve these - we're kicking around some ideas at the moment about how to achieve it without too much hidden magic.

We also hear you that SQLite is not a good fit for everyone - the concerns you raise are valid. We do want to have other providers available for UWP. It's just a matter of timing - I know you disagree with this, but our focus for 1.0.0 is SQLite.

Somewhere you had asked about calling out exactly what is supported in UWP rather than just saying "EF Core works on UWP" (I can't find it now though). This is a fair point and until we have other providers we can adjust the message to be more like "EF Core can be used in UWP with SQLite for local data access (note some limitations of SQLite)".

As a minor idea, if it doesn't degrade your workflow, it would be sensitive to your "customers" if you could re-tag this issue to "bug" (or nothing) since having a working EF on UWP just can't be seen as an "enhancement" to the outside world. In that context, "up-for-grabs" is like a slap in the face and "under review" might be a better choice?

Here is how we use the tags, and I think this is consistent with generally accepted use:

• An enhancement is a feature that has not yet been worked on.
• A bug is a piece of functionality that is already implemented but does not behave as expected.
• up-for-grabs doesn't have any impact on priority or whether we will do the work on our team, it just means we are flagging the issue as something that is a good candidate for someone outside of the EF team to work on.

[rowanmiller]

And for my last personal comment on this thread too…

So, excuse me if I was too straight and not too politically correct on some points.

I am all for direct speech and addressing things head on rather than skirting around issues. That said, being direct and to the point is different than name-calling and using minority/less-privileged people groups in a derogatory manner.

When you must fight bravely you accept yourself as loser and leave yourself on a mercy of victor. As history tell us, victory is never come with a mercy. It always come with death of a loser, fast and painless death if you are lucky.

Mercy may not be a common tactic to win a war… but it sure goes a long way to preventing one :wink:.

[10Dev]

I would guess from the style of speech that English is not Opiumtm's native language. On the interenet, good idea to think about the information content and leave the rest of it to "lost in translation"

I can't agree that taking something that is not working and making it work is a "feature" although in some abstract kind of Kafkaesque way I can see the intent.

And although it feels like quibbling, from your previous description of how work on providers might easily lead to required changes in the core project (in the current state of affairs) then it only follows that "up-for-grabs" should not even be a practical option at this point.

In any case, I for one appreciate the feedback the EF has provided and to the extent that they are individuals completely appreciate their efforts on our behalf and to the extent that they are "oracles" of the entity known as MIcrosoft, there is the sense of some sort of management roadbloack in the flow chart of the organization that has impeded severely the delivery of an EF solution into the WInRT environment for 3 years now. Taken from a broader perspective this is clearly a dysfunctional situation that has caused some damage abstractly to the reputation of Microsoft and practically in the form of the cost of lost opportunity. Reading between the lines, this is completely beyond the control of the EF team to deal with and they are doing the best they can with the resources they have. They probably feel as much pain about the issue as we do on the outside.

Opiumtm: I think the EF team has provided as much guidence as they can and from that I would guess it will be about 6 months before we see something workable for production use and even that feels optimistic. It is what it is. Everything else about the UWP ecosystem seems very positive as all the previous roadblocks such as needing "sideload keys" to deploy apps have been removed. The new Composition API removes the previous DX interop roadblocks and other perf roadblocks that were previously hampering large complex enterprise WPF applications. Of course none of this has been communicated properly and that apears to be an eternal "feature" of Microsoft. Keep an eye on my and I will try to document all of the data options for UWP but nothing can compare to EF.

[Opiumtm]

@10Dev You are right, English is not my native language, I am from Russia. And political correctness here does not have much impact on a society, jokes about minorities quite popular even among minorities themselves (but outright aggression and hate speech against individual people or groups is seen as crime and heavy violation of social rules). After all, very popular, commercially successful and award winning modern Russian writer Viktor Pelevin (successful not only in Russia, his works are translated in many languages and Germany creating a film based on his works "Buddha's Little Finger") write in a genre that can be described as intellectual, sarcastic, philosophical fiction with hard social criticism, which is very funny but extremely political incorrect (not sure how such a political incorrectness is translated to German language and how film is created based on such a sharp sarcastic political incorrect material including black humor on drug abuse, alcoholism, mental instability, social exploitation, racial, national and gender stereotypes, violence, humor on nazi, communism, liberalism and capitalism ideologies alike, sarcasm on the actual political and cultural icons such a Barrack Obama, George W Bush, Vladimir Putin).

And for UWP platform, it is very promising in long perspective, but not generally polished, lacks vital functionality and certainly is not suited for now as a substitution for a classic desktop.NET/WPF applications. WCF support is limited, database support is even worse, essentially non existent.

[Opiumtm]

@rowanmiller as for war, I am not against open source movement and accept Microsoft determination to make many of its technology accessible as open source. It's great, because to look into reference code of Microsoft tools and frameworks is a good school and helps understand technology which I as a developer use. Microsoft is going into a right direction with its open source initiative. But when we look from a commercial perspective (I am not a big fan of a capitalist system, because capitalism is well known for its ability to kill brilliant technology things in a commercial wars, bankruptcies, cost reduce process and so on), there are Apple, IBM and Google IT giants. They are heavily invest to drive customers and developers from a Microsoft platform and it is a dire reality of our days, in many aspects they are already succeeded. It's what I mean when I said "war". They openly abuse open source movement, bribe IT journalists and gadget dealers, heavily invest in a post-PC and non-Microsoft platforms and so on. Spirit of a collaboration and peace is a good thing, but war for a survival is already here and Google is a worst technology abusing domination hungry monster I ever seen in my professional life, no matter how Google is painting itself as an Empire of Good in a mass media (not unlike from a USA government strategy, USA government and Google corporation share too much similar traits, well it's just another political incorrect question). If Google will be a totally dominant winner in a platform wars we will see a much worse toxic IT world. So be brave to fight back.

[Opiumtm]

@10Dev on your page

ESENT Managed Interop it is not clear if the .NET wrapper is UWP compatible

It is clear. ESENT Managed Interop is compatible with UWP. Not all nuget package are (for example, PersistentDictionary built on top of ESENT interop package is not compatible and it is certainly a problem with package manifest), but core Interop package on nuget is compatible with UWP, I tested it myself and it work.

[10Dev]

Joseph Conrad was a native Polish speaker that somehow ended up writing some of the greatest works of English literature. Such things are rare and Optiumtm, your English is very eloquent and humbles me in that I couldn't write a paragraph of anything in another language.

Cultural differences can be quite large and form a communication gap perhaps larger than language. I would say this discussion has proceeded rather smoothly despite such gaps and is a testimony to the kindness and patience of all the participants.

I think the UWP platform is the next generation of both Silverlight and WPF and fixes many of the serious blocking issues in those previous platforms. It should be possible to harmonize the XAML and API a bit better between then and now that Microsoft has purchased Xamarin, there might be some motivation to create common platform that is more of a "superset" than the current "subset"

As it exists today UWP is mature, performant, reliable and very deployable in large applications within a few significant limitations. As we all know, the largest limitation is in the data area which is why we are here. In fact I finally got around to making a GitHub account for the sole purpose of determining the EF situation on UWP. And so I find out it is dismal. And that's frustrating and almost physically painful actually. For now it hurts Microsoft more than us since we can make work-arounds but all the enterprise project starts that switch to other technologies are lost forever. That's opportunity cost and the EF team has to juggle that as best they can for reasons we may never know. But for me, I choose not to let this apparent absurdity of a situation affect my view of the overal platform. Like the internet, we just need to route around the damaged part of the network!

[Opiumtm]

@10Dev well, not all thing with UWP is such good as you described. For now UWP have one maybe corner case but very fundamental problem. In Silverlight and desktop .NET frameworks underlying operational system calls was carefully wrapped and internally optimized (just study how elaborated platform Interop in a classical .NET is). On the UWP platform things are much worse. Windows Runtime essentially is a COM with fancy winmd metadata on top. And here we got significant performance penalty when we deal with Windows Runtime calls. Even .NET Native is not good with Windows Runtime Interop from a performance point of view. And on UWP such Interop is a fundamental nature of things because all Windows Runtime API surface is exposed via COM ABI. COM/.NET Interop was never good, but in a traditional .NET vast majority of operational system calls was done via plain win32 platform invoke, not COM. And for now we have something similar to Frankenstein monster full of Interop between .NET/COM boundary suffering from a terrible performance penalty. It's not matter if you can avoid too many direct Windows Runtime calls, but File I/O and manual XAML UI manipulation in code (not rely on data templating engine and bindings, go more low level - which is unavoidable if you write very non trivial UI, custom markup rendering for a social app for example) can be painfully slow because it involve massive Interop between .NET managed code and native COM Windows Runtime API.

So, .NET Native toolchain must be improved to eliminate (or at least significantly reduce) .NET/COM Interop performance penalty. After all, Windows Runtime provide most basic and vital services, and in non trivial scenarios massive system calls are unavoidable. For now, .NET Native do not help much to reduce COM calls marshalling costs, on the other hand plain C++/CX code does not suffer from such an issue. Even on MSDN it is clearly suggested to avoid excessive marshalled calls to a Windows Runtime API. And C++/CX code is fundamentally COM based and its calls are marshalled only in a rare cross Thread Apartment scenario. So, now it is reasonable to write performance critical relatively low level code in C++ and split application into a C++ and C# parts. C# is not suited well for all scenarios and so we have no solid developer platform experience.

[Opiumtm]

@10Dev to make things more clear, on a x86 desktop .NET/COM calls marshalling cost is tolerable. But on a ARM based smartphones it become a complete disaster. ARM processors are very resource constrained if compared with powerful x86 bulldozers. On ARM .NET/COM Interop penalty become a major performance issue. Such a subtle and not self obvious issue (one must inderstand internal Windows Runtime architecture and be familiar with not well known to a typical .NET developer details) must be known to all UWP library authors.