Libloader patch

[ChrisMcKee]

Adds libload support for net core on nix and binaries required to work.

[ChrisMcKee]

@bp74 I've tested (run docker build . in the root to run the tests x-platform) this and built binaries for debian + alpine to match. I believe what is in there is a viable and relatively clean solution now.

Contains

https://github.com/bp74/Zstandard.Net/pull/7

And would resolve https://github.com/bp74/Zstandard.Net/pull/8

As well as #6 #1

But would make #9 a non-working scenario. All depends how much you want / need to support out of maintenance release?

[ChrisMcKee]

the pathing needs jiggling

[ChrisMcKee]

I'll probably have little remaining sanity after this. The windows version loaded via nuget now works fine again (the native loader seemed to bork it)

The tests pass on alpine/bionic/stretch

IF I copy the zsf lib to a sane location (/usr/lib) I can run my small test app (compiled for alpine) in alpine.

If I rely on dotnet to do the right thing...

Unhandled Exception: System.DllNotFoundException: Unable to load shared library 'libzstd' or one of its dependencies. In order to help diagnose loading problems, consider setting the LD_DEBUG environment variable: Error loading shared library liblibzstd: No such file or directory
   at Zstandard.Net.ZstandardInterop.ZSTD_versionNumber()
   at Zstandard.Net.ZstandardStream.get_Version()
   at TestZstdLib.Program.Main(String[] args) in I:\ZSTDTEST\TestZstdLib\TestZstdLib\Program.cs:line 17
Aborted

It's like it wants a slap.

So ...

• The tests run in docker (all envs) without fudging where the file lives
• The windows side is fine, no problems with that hunky dory (both in the sln and from a nuget file)
• The linux side (in alpine at least) errors unless you copy the lib to the lib folder. (Running from a Nuget file)

note:

Compiling a test for Nuget testing

• Copy nupkg file from bin folder to a local path (c:\nugetlocal)
• Add that path to your nuget install on a sln
• Install from that location, nick some code from the sandbox
• Run

To compile the test sln for alpine you have to pass a different param to normal

dotnet publish -c Release -r linux-musl-x64 --self-contained=true TestZstdLib.csproj

[ChrisMcKee]

@bp74 @tom-corwin

At the current state of the branch it works. To use on linux you have to

• Install LibC (this is for the lib loader, not really an option) [libc6-compat / libc6-dev]
• Copy the binary [cp linuxalpine/libzstd.so /usr/lib/ || cp linuxdebian/libzstd.so /usr/lib/]

Which all seems reasonable tbh.

[simplexidev]

I might have a solution to copying the alpine/debian binaries, with some code in one of my projects. I'll can work on that here in a bit, if you're interested.

If I rely on dotnet to do the right thing...

Unhandled Exception: System.DllNotFoundException: Unable to load shared library 'libzstd' or one of its dependencies. In order to help diagnose loading problems, consider setting the LD_DEBUG environment variable: Error loading shared library liblibzstd: No such file or directory
   at Zstandard.Net.ZstandardInterop.ZSTD_versionNumber()
   at Zstandard.Net.ZstandardStream.get_Version()
   at TestZstdLib.Program.Main(String[] args) in I:\ZSTDTEST\TestZstdLib\TestZstdLib\Program.cs:line 17
Aborted

I'm not sure if that was a previous typo in the code or just the comment in the first line of the code block above: Error loading shared library liblibzstd: No such file or directory.

[ChrisMcKee]

Yeah the liblib outputs weird but as i believe it's saying lib:libzstd as in lib path module

[simplexidev]

Ahh, that makes sense.

[ChrisMcKee]

Seems alright. It's not a great ask for people on nix to have the libs needed. I'd rather say 'use these lines' than ask people to use apt/APK (more so as they both carry did versions)

[simplexidev]

If you wanted to have the binaries copied (as needed) inside the code, rather than making another step for the end user, you could determine the Platform/OS/Version/RID using the code from (this gist)[https://gist.github.com/tom-corwin/763cc4af64a2dad295c7758d6a9df8e4]

[ChrisMcKee]

@tom-corwin i'd be inclined to agree but for the security issue; if the containers running --read-only the process would crap out. Alternatively could hook up travis to push docker images out with it prebaked. In an ideal world it would just load bloody locally 🤣

[krs43]

Heads up - switching to the NativeLibrary w/ function calls, causes windows to evaluate permissions on the dll on every single call. This adds a kernel call and disk touch on every single call into the api. The built in PInvoke style with a zstdlib.so in the Path should get things working nicely.

[GreenIreland]

Hey! At first I want to say thank @ChrisMcKee for this fork ZStandard.Net.

There is my problem. I have netcore 3.1 application (developed in VS 2019) with referenced ZStandard.Net package from current branch.

I run app in linux docker container (Docker Desktop for Windows) and got runtime exception when decompress data: Unable to load shared library 'libzstd' or one of its dependencies. In order to help diagnose loading problems, consider setting the LD_DEBUG environment variable: liblibzstd: cannot open shared object file: No such file or directory

Note: Interop static ctor is ok. libzstd.so is found by path and Libdl.dlopen(path, Libdl.RTLD_NOW) seems works.

Steps I did:

1. apt-get install libc6-dev (before this I got error "Unable to load shared library 'libdl.so'")
2. cp bin/Debug/netcoreapp3.1/libzstd.so /usr/bin/
3. Tried ln -s /lib64/libzstd.so.1 libzstd.so

Lib is actually exists:

What I am doing wrong? Why library still not found?