Library e_sqlite3 not found when using Mono on Linux

[Crono1981]

What version of SQLitePCLRaw are you using?

2.1.2

If you are using one of the SQLitePCLRaw bundle packages, which one?

bundle_green

What platform are you running on? What operating system? Which version? What CPU?

Ubuntu Linux 22.04 on x64 processor. More specifically, it is a GitHub provided runner for GitHub Actions.

What target framework are you building for?

4.8.1

Are you on .NET Framework or the newer stuff (.NET Core, .NET 5+, etc)?

.NET Framework (mono)

Are you using the command line, or an IDE? Which IDE? Which version of that IDE?

I'm running dotnet test from the command line:

dotnet test REDACTED.sln -c Release --no-build --logger trx --settings .runsettings --results-directory "Test/Reports"

In case you want to see it, here's the build command that was ran prior to the test command:

dotnet build REDACTED.sln -c Release  -restoreProperty:RestoreLockedMode=true -p:"Version=1.14.0,AssemblyVersion=1.0.0.0,AssemblyFileVersion=1.14.8342.35452,InformationalVersion=1.14.0+14.Branch.main.Sha.fc1288a079ac393b426c39c72d6a6c74d93b8646,TreatWarningsAsErrors=true,GenerateDocumentationFile=false,BuildDocFx=false,RepositoryUrl=https://github.com/REDACTED,RepositoryBranch=main,RepositoryCommit=fc1288a079ac393b426c39c72d6a6c74d93b8646,ContinuousIntegrationBuild=true,DisableImplicitNuGetFallbackFolder=true" /WarnAsError

Is this problem something that just started happening? For example, were things working well for you and then you updated something and then the problem showed up? What changed?

It's the first time I ever try building on Mono for Linux.

What is the exact error message you are seeing when the problem happens?

Here's the complete error output:

  System.TypeInitializationException : The type initializer for 'Microsoft.Data.Sqlite.SqliteConnection' threw an exception.
  ---- System.Reflection.TargetInvocationException : Exception has been thrown by the target of an invocation.
  -------- System.Exception : Library e_sqlite3 not found
  plat: dlopen
  suffix: SO
  possibilities (2):
      1) /home/runner/work/***REDACTED***/bin/Release/net481/runtimes/linux-x64/native/libe_sqlite3.so
      2) /home/runner/work/***REDACTED***/bin/Release/net481/libe_sqlite3.so
  dlopen TryLoad: /home/runner/work/***REDACTED***/bin/Release/net481/runtimes/linux-x64/native/libe_sqlite3.so
  thrown: System.DllNotFoundException: dl assembly:<unknown assembly> type:<unknown type> member:(null)
    at (wrapper managed-to-native) SQLitePCL.NativeLibrary+NativeLib_dlopen.dlopen(string,int)
    at SQLitePCL.NativeLibrary.TryLoad (System.String name, SQLitePCL.NativeLibrary+Loader plat, System.Action`1[T] log, System.IntPtr& h) [0x00069] in <8a9cf65dcc6542b59d3abc28e58f9815>:0 
  dlopen TryLoad: /home/runner/work/***REDACTED***/bin/Release/net481/libe_sqlite3.so
  thrown: System.DllNotFoundException: dl assembly:<unknown assembly> type:<unknown type> member:(null)
    at (wrapper managed-to-native) SQLitePCL.NativeLibrary+NativeLib_dlopen.dlopen(string,int)
    at SQLitePCL.NativeLibrary.TryLoad (System.String name, SQLitePCL.NativeLibrary+Loader plat, System.Action`1[T] log, System.IntPtr& h) [0x00069] in <8a9cf65dcc6542b59d3abc28e58f9815>:0 
  NOT FOUND

Are you using PackageReference or packages.config?

PackageReference

If you are using mobile platforms, are you on classic/legacy Xamarin or on .NET 6 and higher?

Not using mobile platform.

Sometimes other packages using SQLitePCLRaw cause problems when they are mixed together. What other packages are you including in your project?

• coverlet.collector 3.2.0
• Microsoft.NET.Test.Sdk 17.3.2
• Microsoft.NETFramework.ReferenceAssemblies
• xunit 2.4.2
• xunit.runner.visualstudio 2.4.5

How can we reproduce the problem you are seeing? Your issue will get attention much faster if you attach a minimal reproduction sample project.

I don't have a repro sample project, but I can tell you this: project is a Xunit test project, with the following MSBuild properties:

  <PropertyGroup>
    <TargetFrameworks>net6.0;net481</TargetFrameworks>
    <RuntimeIdentifiers>linux-x64;osx-x64</RuntimeIdentifiers>
    <Platform>x64</Platform>
  </PropertyGroup>

I've also taken care of adding a xunit.runner.json file within the project's directory:

{
  "$schema": "https://xunit.net/schema/current/xunit.runner.schema.json",
  "shadowCopy": false,
  "diagnosticMessages": true
}

I tried running the same commands on both Linux and MacOS runners in GitHub Actions. Much to my surprise, on the Mac runner, the whole thing works. It looks like there's something particular going on with Mono on Linux.

I did notice that on the Ubuntu runner the following output file was produced after build:

bin/Release/net481/runtimes/linux-x64/native/libe_sqlite3.so

On the Mac runner, I had this one instead:

bin/Release/net481/runtimes/osx-x64/native/libe_sqlite3.dylib

Judging from the message in the stack trace, I'm assuming that there's something wrong with libe_sqlite3.so when it is accessed from Mono on Linux, whereas its dylib counterpart works okay on Mono for Mac. 🤷‍♂️

If I think of anything else, I'll comment below. In the meantime, if you have any pointers you can give me, that would be much appreciated.

Thanks for all your incredible efforts.

[Crono1981]

Another thing that might be of interest:

It is reported here that Linux runners on GitHub have SQLite v3.37.2 preinstalled.

I did not find anything about SQLite for MacOS runner.

I have no idea how / if that has any impact on what SQLitePCLRaw does, though.

[ericsink]

I think the last time I worked in that area of the code I broke certain cases involving Mono. Now that .NET Core (and its descendants) is mature and cross-platform, Mono is becoming an even less common case. That said, if you're using net481 with actual RuntimeIdentifiers like linux-x64, then I think SQLitePCLRaw should do the right thing.

[mattjohnsonpint]

Hi. We have the same issue. After GitHub Actions moved the ubuntu-latest image tag from 20.04 to 22.04, some of our tests for Entity Framework Core that use SQLite during testing started failing. We multi-target to run those tests on net7.0;net6.0;netcoreapp3.1;net48 and matrix in GitHub Actions to run on Ubuntu, macOS, and Windows. Only the net48 target fails, and only on Ubuntu 22.04.

Switching to specifically use the ubuntu-20.04 GitHub Actions runner image appears to have mitigated the problem for now. I haven't investigated further. See https://github.com/getsentry/sentry-dotnet/pull/2083 for details.

[ericsink]

@mattjohnsonpint Interesting. What changes does ubuntu 22.04 imply? For example, is it bringing in a different version of mono?

[mattjohnsonpint]

Nope. Same version of Mono, as far as I am aware. 6.12.0.182. It was rather surprising to me as well.

I'll investigate further when I can find some time.

[mattjohnsonpint]

Comparing the GitHub Actions runner images package lists (20.04 vs 22.04), I see that 20.04 has sqlite3 3.31.1 preinstalled, and 22.04 has sqlite3 3.37.2 preinstalled. Not sure why that would matter for this though.

[ericsink]

I agree that the pre installed versions of SQLite should not matter.

Different versions of the dotnet SDK?

[mattjohnsonpint]

Shouldn't matter for tests that execute on Mono. The error happens at run time, not at build time. They are both using the latest .NET 7 SDK to run the tests.

My gut feeling is it's some lower-level native dependency that is the culprit.

[ericsink]

The error happens at run time, but it could be due to a failure to copy the native library dependency into place during the build. In fact, that's my guess of what's going on here. There's an msbuild targets file that does this copying, and in the past its behavior has been affected by a lot of different build-related issues.

If that supposition is correct, a key difference is which shared libraries are getting copied into the build output.

[Crono1981]

@ericsink

but it could be due to a failure to copy the native library dependency into place during the build. In fact, that's my guess of what's going on here.

I did find the following file on the Ubuntu runner after building the code:

bin/Release/net481/runtimes/linux-x64/native/libe_sqlite3.so

Is it what you're talking about?

[ericsink]

@Crono1981 Yes, but is that from the one that succeeded or failed?

And you might find build outputs in that form (in a runtimes directory, with a RID directory under that, etc). Or you might find libe_sqlite3.so in the top level of the output directory, and the latter case might depend on whether you are building with a RuntimeIdentifier property or not.

I'm being vague here because your situation may be a part of a larger related issue, but I don't fully understand what's going on yet.

[Crono1981]

@ericsink the one that failed.

[ericsink]

My gut feeling is it's some lower-level native dependency that is the culprit.

Maybe my e_sqlite3 builds are somehow incompatible with ubuntu 22.04. Like, for example, they are looking for the wrong libc, or something.

But if that were the case, why would it affect only net48/Mono?

We need a more minimal repro.

[mcrossley]

I'm having the same issue with Framework 4.8.0 running on Raspberry Pi OS (32bit) (basically Debian), and mono 6.12.0.182.

dlopen TryLoad: /usr/share/CumulusMX/runtimes/linux-arm/native/libe_sqlite3.so
thrown: System.DllNotFoundException: dl assembly:<unknown assembly> type:<unknown type> member:(null)
  at (wrapper managed-to-native) SQLitePCL.NativeLibrary+NativeLib_dlopen.dlopen(string,int)
  at SQLitePCL.NativeLibrary.TryLoad (System.String name, SQLitePCL.NativeLibrary+Loader plat, System.Action`1[T] log, System.IntPtr& h) [0x00069] in <7ea8f9164048497c988e9906e742dabe>:0
dlopen TryLoad: /usr/share/CumulusMX/libe_sqlite3.so
thrown: System.DllNotFoundException: dl assembly:<unknown assembly> type:<unknown type> member:(null)
  at (wrapper managed-to-native) SQLitePCL.NativeLibrary+NativeLib_dlopen.dlopen(string,int)
  at SQLitePCL.NativeLibrary.TryLoad (System.String name, SQLitePCL.NativeLibrary+Loader plat, System.Action`1[T] log, System.IntPtr& h) [0x00069] in <7ea8f9164048497c988e9906e742dabe>:0
NOT FOUND

  at SQLitePCL.NativeLibrary.Load (System.String libraryName, System.Reflection.Assembly assy, System.Int32 flags) [0x00058] in <7ea8f9164048497c988e9906e742dabe>:0
  at SQLitePCL.Batteries_V2.MakeDynamic (System.String name, System.Int32 flags) [0x00010] in <7ea8f9164048497c988e9906e742dabe>:0
  at SQLitePCL.Batteries_V2.DoDynamic_cdecl (System.String name, System.Int32 flags) [0x00000] in <7ea8f9164048497c988e9906e742dabe>:0
  at SQLitePCL.Batteries_V2.Init () [0x00000] in <7ea8f9164048497c988e9906e742dabe>:0

The libe_sqlite3.so file is in the runtime folder it is searching.

Minimal test, also fails: https://github.com/mcrossley/SQLiteTest1 Using PCLraw v2.1.4

[mcrossley]

Prior to trying these packages, I just used the sqlite3.dll (x86 for Windows - download direct from www.sqlite.org), with the sqlite-net wrapper file SQLite.cs

This limited me to 32bit only which is what I wanted to get away from. But that Windows x86 DLL works fine on Raspberry Pi OS Linux when running under mono on an ARM processor.