praeclarum / sqlite-net

Simple, powerful, cross-platform SQLite client and ORM for .NET
MIT License
4.04k stars 1.42k forks source link

Had to call SqlitePCL.Batteries.Init() to Fix missing e_sqlite3 error. Am I doing this correctly? #801

Open pha3z opened 5 years ago

pha3z commented 5 years ago

I was getting an intermittent error when I run my application saying "Unable to load DLL 'e_sqlite3' followed by module not found. I looked all over the place and couldn't find a solution that sounded like it applied to my environment.

I noticed in the description of Nuget pkg "SQLitePCLRaw.bundle_green", it says "Call SQLitePCL.Batteries.Init()"

So I added that to the entry point of my application and the problem seems to have been fixed. But that doesn't seem correct to me. Sqlite-net should already be making this same call shouldn't it?

Could be related to this issue: https://github.com/praeclarum/sqlite-net/issues/643

EDIT: Crap.. Because the issue is intermittent, it had only "seemed" to fix the issue. Now I'm getting the load DLL error on the batteries.Init() line I added. LOL

Why could this be intermittent anyway? Sometimes my app runs fine and sometimes it halts with this error.

NOTE: I should add a lot of people are citing this problem on Windows 8. I'm on Windows 10 with VS 2017. My project is .NET Framework 4.7.2

ericsink commented 5 years ago

I haven't seen this before, at least not an intermittent version of this.

Who are "a lot of people"? Do you have evidence that others are seeing intermittent failures of this kind?

Diagnostic questions:

What version of SQLitePCL.raw are you using?

Does the behavior change if you clean your tree completely and do a full rebuild?

Is there any chance another e_sqlite3.dll exists somewhere?

The you linked is a unit test project. Is yours?

pha3z commented 5 years ago

Who are "a lot of people"? Do you have evidence that others are seeing intermittent failures of this kind?

I wasn't clear. I just meant that when I searched for the ""Unable to load DLL 'e_sqlite3'", I find a lot of issues on github and stackoverflow where people discuss that general error on Windows 8.1 or android or iOS. I'm not targetting any of those. I just have a WPF application on Windows 10.

What version of SQLitePCL.raw are you using?

v1.1.11 according to Nuget package manager.

Is there any chance another e_sqlite3.dll exists somewhere?

Not sure how to answer this question simply. See further down in this post. I apparently broke things even worse by trying to uninstall the nuget packages an reinstall them.

The you linked is a unit test project. Is yours?

Negative. Its just a issue I read when I was desperately seeking answers before posting for help.

New Status (Condition: Worse )

1) I uninstalled all of the SqlitePCLRaw references and sqlite-net-pcl itself. 2) I reinstalled sqlite-net-pcl and let it bring in all the dependencies 3) I did a build and got this error: Found conflicts between different versions of "SQLitePCLRaw.batteries_v2" that could not be resolved. These reference conflicts are listed in the build log when log verbosity is set to detailed. C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\15.0\Bin\Microsoft.Common.CurrentVersion.targets 2110

That came along with similar errors about "SQLitePCLRaw.Core" and "SQLitePCLRaw.provider.e_sqlite3"

4) I uninstalled all of the nuget packages again 5) I searched the entire project folder for any files containing "sqlite". I deleted every one of them, except for one that appeared to be Visual Studio's own file for managing projects (am I mistaken that such a file exists?). I didn't want to break VS so I left that one alone. 6) I reinstalled sqlite-net-pcl with nuget and let it bring in dependencies. 7) I did a build and got the same errors as above.

I've never had to deal with conflicts like this before. Deleting all of the files in the project folder was my best guess at a solution. Do I need to delete old references in some config files or something?

Addendum: I just remembered that yesterday, I had Updated the various SQLitePCLRaw packages because Nuget was suggesting I do so. It presently says the newest is v1.1.13 but I'm on 1.1.11 after trying to reinstall.

Still, I don't think that necessarily invalidates my concern about the conflicts. What should I be looking for to resolve these conflicts from previous installed files that I thought I removed?

Aside: I read some of your blog. I love the material on there! 👍

ericsink commented 5 years ago

This looks like you've got leftovers somewhere.

You're on .NET Framework with WPF, so I assume your project might be using packages.config. So for example, when you uninstall all those nuget packages, I wonder if all trace of them is gone from csproj and packages.config. Perhaps take a look at those files in a text editor and see if anything looks weird.

pha3z commented 5 years ago

Ok. I used Sublime Find-in-Files. Sure enough, there were various sqlite references in App.config of some of my projects for my solution. After removing all that and using Nuget to add sqllite-net-pcl again, the errors about "Conflicting versions" are gone. yay \o/

Unfortunately, I'm back to my original problem with this error:

Exception thrown: 'System.DllNotFoundException' in SQLitePCLRaw.provider.e_sqlite3.dll
Unable to load DLL 'e_sqlite3': The specified module could not be found. (Exception from HRESULT: 0x8007007E)

But now that I've relaxed and got a better handle on things, I think I may know what problem pertains to. I will have to explain a lot more detail about my overall environment/solutions.

I have 2 solutions. Solution 1) Contains a project called SqliteNetHelper.DLL which is my own wrapper for sqlite-net. I also have a SqliteNetHelper.Test project that executes tests on a dummy database file using sqlitenethelper.dll. This solution is working perfectly. I installed the Nuget package for sqlite-net-pcl for those two projects in this solution and never had any issues.

Solution 2) This is my WPF application. It contains four projects. Three projects are DLLs and one is the WPF app. I'm not actually using sqlite in the WPF app itself. I've abstracted all my logic into the DLLs. Each DLL has its own internal database and corresponding disk file. As such, each DLL needs a reference to my own SqliteNetHelper.DLL and also the Nuget Packages for Sqlite-net-pcl.

Since Solution 1) Unit Tests work fine, but Solution 2) throws this sqlite3.dll error, I've begun wondering if somehow the conflict is due to the fact that I've got dependencies on my own wrapper DLL which also depends on sqlite-net-pcl. I verified that all Nuget package installations are using the same version and files through the Visual Studio Nuget GUI. I don't see any problems here.

I'm wondering if I've created DLL hell with the standalone wrapper. Unfortunately, I can't easily remove it and stick to the dependencies on Sqlite-net-PCL, because the wrapper is what my actual code uses. I think I'm going to try this:

1) Create brand new solution with a DLL and a WPF app 2) Install Nuget sqlite-net-pcl in the DLL. Call some methods for sqlite-net 3) reference the DLL into WPF and call an entry point 4) Run it and see if it works 5) if it works, then tie in my own wrapper into the dll and see if that causes things to break.

Not sure what else to do. Any other ideas?

If it proves to be the case that somehow my wrapper dll's dependency on sqlite-net is somehow conflicting with my primary dlls depending on sqlite-net, I'll probably have to just abandon sqlite-net and roll my own ORM. I've done that in the past for full server relational databases with great success. I picked sqlite-net because I thought it would get me off the ground with sqlite quicker (never having used sqlite), but now I'm beginning to wonder if there are too many moving parts.

Of course, the problem there is it still doesn't take sqliteRawPCL out of the equation, which, from my understanding, I'll need no matter what if I want crossplatform support. At the present time, I only anticipate targeting OS X and Windows. Maybe there's something else I should be doing since I am only targeting those two desktops?

ericsink commented 5 years ago

I don't think you should have to abandon sqlite-net and write your own ORM just because of a quirky tooling problem. This can get worked out.

My current guess is that the problem is caused by having two solutions. When the stuff in solution 2 references SqliteNetHelper.dll, exactly HOW does it do that? Is the SqliteNetHelper project a member of both solutions? Does building solution 2 cause a build of SqliteNetHelper?

I wonder what would happen if you have one solution instead of two.

Also, do the DLLs in solution 2 reference sqlite-net stuff directly, or do they only call stuff in SqliteNetHelper. In other words, why do they need references to the sqlite-net package? Oh, wait. As I said earlier, you're probably still using packages.config (instead of PackageReference in csproj), so transitive dependencies work the old way.

Another thing to consider:

In some scenarios, it is easier to structure the sqlite-net and SQLitePCL.raw references differently, along these lines:

In library projects, instead of using the sqlite-net-pcl package, use the one called sqlite-net-base. Same maintainer, same code. The only difference is that sqlite-net-base takes its dependency on the core SQLitePCLRaw.core library instead of on the SQLitePCLRaw.bundle_green.

In the top-level (executable, app, etc) projects, add a dependency on SQLitePCLRaw.bundle_green, and add a call to SQLitePCL.Batteries.Init() in the app startup.

The thing about dealing with SQLite in the .NET world is that SQLite is a native library written in C. And the build issues around such things can be complicated. But a library project doesn't need to know about any of that, so it only needs to depend on the SQLitePCLRaw.core library. It is only when building an executable for a given platform that the dependencies on the native SQLite library need to come into play.

pha3z commented 5 years ago

My current guess is that the problem is caused by having two solutions. When the stuff in solution 2 references SqliteNetHelper.dll, exactly HOW does it do that? Is the SqliteNetHelper project a member of both solutions? Does building solution 2 cause a build of SqliteNetHelper?

Because I was trying to keep my solution 2 lightweight and sanitized, I've been copying the release build of the SqliteNetHelper.DLL into my solution 2 directory and adding it as a direct reference to any other DLLs that need it. The project from solution 1 isn't referenced at all. It's just a plain-old copy-and-paste DLL setup.

Also, do the DLLs in solution 2 reference sqlite-net stuff directly, or do they only call stuff in SqliteNetHelper. In other words, why do they need references to the sqlite-net package? Oh, wait. As I said earlier, you're probably still using packages.config (instead of PackageReference in csproj), so transitive dependencies work the old way.

My DLLs only call methods in SqliteNetHelper. They don't directly call anything in Sqlite-Net. If its possible to completely get rid of the sqlite-net and sqlitePCLRaw references, that might fix the problems. Would this "old way" and "new way" difference offer me a way to only reference sqlitenethelper and nothing else? I'm not familiar with the variations you're describing.

In some scenarios, it is easier to structure the sqlite-net and SQLitePCL.raw references differently, along these lines:

Your explanation is thorough and tremendously helpful. I didn't know about the EXE vs DLL reference variation when using a native library. I'll try the setup you described using sqlite-net-base and only add sqlitePCLraw.bundle_green to the exe app.

pha3z commented 5 years ago

Ok, here's what I tried. (Bear in mind this is all in Solution 2. As explained above, Solution 1 projects are not referenced by Solution 2. Solution 1 is only the source for creating my NetWrapper DLL and the Unit tests still work fine in it.)

1) Uninstall all nuget packages for sqlite-anything. Search all files for "sqlite" and clear any leftover references 2) Install Sqlite-net-base nuget pkg to my library projects 3) Install SQLitePCL.bundle_green to my WPF exe project 4) Add SQLitePCL.Batteries.Init(); in the bootstrapper entry point of my WPF app 5) Build all and run

I got this error: Exception thrown: 'System.Exception' in SQLitePCLRaw.core.dll You need to call SQLitePCL.raw.SetProvider(). If you are using a bundle package, this is done by calling SQLitePCL.Batteries.Init().

I set a breakpoint after the Batteries.Init() call and verified that yes, Batteries.Init() is being called before any of my DLLs are entered.

Here is a snapshot of my installed nuget packages for WPF EXE: image

And here is a snapshot of installed packages for my library DLL projects: image

As you can see, the bundle_green installation in hte EXE claims the versions are a bit older. I'm not sure if that's having any bearing here.

:(

ericsink commented 5 years ago

Oh dear. :-(

The 0.9.x versions of those packages (the ones with names of the form SQLitePCL.*) are all deprecated.

Only use the ones with >1.x version numbers, with names of the form SQLitePCLRaw.*

Er, sorry for the confusion. Maybe it's time to remove all those old packages from nuget. :-(

pha3z commented 5 years ago

HALLELUAH! \O/

I uninstalled that package and found the correct one and installed it. And now everything works!!!!

By the way, per your comment about library method references, I checked and realized that the Wrapper wasn't 100% hiding sqlite-net because one method was being allowed to return a sqlite-net specific object. I abstracted this away, and was able to completely remove all nuget packages having anything to do with sqlite from my libraries. All I have to do is reference my own sqlitenetwrapper in my libraries now. I'm such a newbie for not having set this up right. :)

Your suggestion of only referencing bundle_green in the EXE works perfectly. Cleaned up all my references beautifully. Unless you think it's counter-recommended, I'd like to write a wiki page devoted to common configuration scenarios and explain the difference between sqlite-net-pcl and sqlite-net-base. It's the least I can do to try to give back... and hopefully help someone else in the future.

Thank you so much for your help! Now I can finally get back to building my application!

Conclusion:

Works perfectly.

Apparently if it can be broken, I'll find a way to break it twenty different ways before it works. :P

ericsink commented 5 years ago

Glad it's working for you!

As for the wiki page idea, if you're talking about here in this repo, that's up to @praeclarum . This is his repo. I just try to help with support issues here and there. :-)