Linux binaries in 2.0.5 release are incompatible with ubuntu 18.04

MDzyga commented 3 years ago

Hi, I'm not sure if it's cconnected with this task https://github.com/ericsink/SQLitePCL.raw/issues/444. So, I create new one. The latest version is working fine locally but during executing unit tests on jenkins, it throws exception below:

System.TypeInitializationException : The type initializer for 'Microsoft.Data.Sqlite.SqliteConnection' threw an exception.
14:28:36  ---- System.Reflection.TargetInvocationException : Exception has been thrown by the target of an invocation.
14:28:36  -------- System.DllNotFoundException : Unable to load shared library 'e_sqlcipher' or one of its dependencies. In order to help diagnose loading problems, consider setting the LD_DEBUG environment variable: libe_sqlcipher: cannot open shared object file: No such file or directory

ericsink commented 3 years ago

I need more details. What is different about the jenkins environment? Is it .NET Core or .NET Framework? What unit test framework are you using? Did this previously work with 2.0.4?

MDzyga commented 3 years ago

Jenkins environment is linux virtual machine with docker. Locallly we work on Windows with docker. We execute each steps using bash scripts to build, test and publish our application.

Application: .NET Core 5 Unit test framework: xunit

Yes, it's working with 2.0.4. Currently I downgraded to this version.

Do you need something else?

ericsink commented 3 years ago

Do you need something else?

Probably, but I'm not sure yet what to ask for. :-)

Let me think about this a bit.

ericsink commented 3 years ago

Well I guess the obvious thing to ask for is a minimal docker setup that reproduces the problem. :-)

Any chance you could make one?

MDzyga commented 3 years ago

This is our Dockerfile:

FROM ubuntu:18.04
RUN apt-get update
RUN apt-get install -y ca-certificates

# https://packages.microsoft.com/config/ubuntu/18.04/packages-microsoft-prod.deb
COPY packages-microsoft-prod.deb /tmp
RUN dpkg -i /tmp/packages-microsoft-prod.deb
RUN apt-get update

RUN apt-get install -y \
      apt-transport-https \
      dotnet-sdk-5.0 \
      dotnet-runtime-5.0
# because microsoft - that's why...
ENV DOTNET_CLI_TELEMETRY_OPTOUT=true
# needed for ~/.dotnet/
ENV HOME=/mnt/src/.home

# need newer NPM, than the one in Ubuntu 18.04 repos
RUN apt-get install -y \
      lsb-release \
      gnupg
COPY nodesource.list /etc/apt/sources.list.d/
COPY nodesource.gpg.key /tmp/
RUN apt-key add /tmp/nodesource.gpg.key
RUN apt-get update
RUN apt-get install -y nodejs # for npm
RUN npm install -g newman

RUN apt-get update && apt-get install -y openjdk-11-jdk
RUN dotnet tool install dotnet-sonarscanner --version 5.0.4 --tool-path /usr/local/tools/sonarscanner
ENV PATH="${PATH}:/usr/local/tools/sonarscanner"
RUN chmod 777 -R /usr/local/tools/sonarscanner/

RUN chmod 777 -R /tmp/NuGetScratch/

nodesource.list

deb     https://deb.nodesource.com/node_12.x bionic main
deb-src https://deb.nodesource.com/node_12.x bionic main

nodesource.gpg.key

MDzyga commented 3 years ago

Installed packages in Tests project:

xunit: 2.4.1
xunit.runner.visualstudio: 2.4.3
moq: 4.16.1
microsoft.net.test.sdk: 16.11.0
coverlet.collector: 3.1.0

Open connection: var connection = new SqliteConnection("DataSource=:memory:");

ericsink commented 3 years ago

Hmmm. What I was looking for is a minimal way to reproduce the problem so I can try to diagnose what's going on.

That Dockerfile doesn't exactly fit my definition of "minimal". I mean, it installs big things like node and the JDK. Can this be pared down to something smaller and still reproduce the problem?

Furthermore, CAN I use this Dockerfile to actually reproduce the problem? I see your follow-on comment about the dependencies of the Tests project, but that project doesn't seem to be mentioned in the Dockerfile?

MDzyga commented 3 years ago

I will prepare simple solution to reproduce this problem. In my opinion it can require only test project which creates Sqliteconnection with connection string like above. Probably we can skip node and JDK packages in Dockerfile because we need only dotnet sdk there.

FROM ubuntu:18.04
RUN apt-get update
RUN apt-get install -y ca-certificates

# https://packages.microsoft.com/config/ubuntu/18.04/packages-microsoft-prod.deb
COPY packages-microsoft-prod.deb /tmp
RUN dpkg -i /tmp/packages-microsoft-prod.deb
RUN apt-get update

RUN apt-get install -y \
      apt-transport-https \
      dotnet-sdk-5.0 \
      dotnet-runtime-5.0
# because microsoft - that's why...
ENV DOTNET_CLI_TELEMETRY_OPTOUT=true
# needed for ~/.dotnet/
ENV HOME=/mnt/src/.home

RUN chmod 777 -R /tmp/NuGetScratch/

ericsink commented 3 years ago

It seems likely that something subtle is going on, but I'm not sure what it is.

You report that things work for you in one environment not the other.

My own test suite does use xunit with both .NET core and .NET framework, and everything passes.

Something about your jenkins environment is different.

MDzyga commented 3 years ago

Ok, I prepared simple project which reproduces the problem :) Maybe the problem is related to Microsoft.EntityFrameworkCore.Sqlite.Core or version of ubuntu. https://github.com/MDzyga/SQLiteTest.git

ericsink commented 3 years ago

Thanks for posting the repro sample. Yes, it looks like a problem with the ubuntu version. Your sample works fine for me on Windows, and on Ubuntu 20.04, but NOT on Ubuntu 18.04.

MDzyga commented 3 years ago

Unfortunately, we can't change version of ubuntu because our application will be hosted on a device with this one. Do you think that you will be able to fix it?

ericsink commented 3 years ago

Yes. I think this problem slipped in when we changed the native builds to use GitHub Actions. I'm working on it.

ericsink commented 3 years ago

OK, I think I've confirmed the problem. I changed the script over in the cb repo to build the binaries on ubuntu-18.04 instead of ubuntu-latest. Unsurprisingly, the resulting libe_sqlite3.so works fine on ubuntu-18.04 (since that's where it was built).

@bricelam I suspect this issue is relevant to your release cycle, as you probably don't want e_sqlite3 binaries that only work on ubuntu-20.04. How far back in ubuntu releases do you need to support?

ericsink commented 3 years ago

The repro project posted by @MDzyga fails for me on ubuntu-18.04 with 2.0.5 but it works with 2.0.6-pre20210902144138.

ericsink commented 3 years ago

@bricelam I haven't found anything I would call definitive, but it kinda looks like .NET 6 intends to support ubuntu-16.04.

And technically, 14.04 LTS doesn't go end of life until April 2022.

https://wiki.ubuntu.com/Releases

bricelam commented 3 years ago

Yes, 16.04. Here is the official list of supported OSes.

ericsink commented 3 years ago

I just checked, and 2.0.6-pre20210902144138 (which is on nuget) does work for me on 16.04, even though it was built on 18.04.

ericsink commented 3 years ago

Fixed in 2.0.6

MDzyga commented 3 years ago

@ericsink Thank you :)

ericsink / SQLitePCL.raw

Linux binaries in 2.0.5 release are incompatible with ubuntu 18.04 #445