Support for Nim dynlib loading

[PMunch]

What would be really nice is if we could write modules for min in Nim and compile them as dynlibs (.so/.dll) and have min load them on runtime. If you would like I can have a look at implementing this.

[h3rald]

Yes that would be nice!

I would like to keep min... minimal, but I guess that optional external dynamic libraries would definitely provide ways to plug in additional functionalities into the language.

It would be nice if you could look into it of course, any pull request is welcome!

[PMunch]

Okay, I've got a POC working now. I'll try to polish it up a bit more and start a PR later today. Currently it works by passing procedures to the dynlib, which is a bit problematic as you lose the ability to have two procs named the same but with different arguments. I think it should be possible to work around this with some templates though.

[PMunch]

Got a nice working version now: https://github.com/h3rald/min/pull/8

[h3rald]

WOW! That’s simply fantastic! Plenty there release this weekend... with this and the other pull requests there’s definitely enough for a new minor version!

I’ll also add some docs and probably also a nice “contributors” page somewhere ;)

Now that we have dynlibs supporti guess the big question is: what should be compiled statically and what should be available as dynlib? I was thinking of more optional modules like crypto or even functionalities like zipping/unzipping (which also rely on third-party libraries) or even your new trig module... they could maybe be good candidates to be moved out to dynlibs instead... what do you think?

[PMunch]

Yeah this still needs some documentation. And currently the loading part is running a couple of echo statements to indicate the status of the loading, which could probably have been done a bit different (could quickly become annoying if you have many libraries). Where to put the dynamic libraries is also a bit weird right now, it's relative to the application directory which is where the min binary is. This means that if you move min to say /etc/bin then the libraries would be loaded from /etc/bin/dynlibs. We should probably find a better path to put this in. On linux there is a dedicated place for this in /usr/lib/<program name> but not sure where it would go on Windows. Speaking of Windows I haven't actually tried it on a Windows machine, so you might want to do that before you strip too much functionality out of the main binary.

When it comes to what to move to dynamic loading crypto is definitely a good candidate since you already have a compile-time switch to disable it. And zipping/unzipping as well since it relies on a third-party library. Not really sure about what else though. The trig module could certainly go, and it should probably be expanded to include more of the Nim math library as well.

[h3rald]

The one thing I can suggest would be allowing min libraries to be placed in a .minlibs directory under $HOME (or %HOMEPATH% on windows), much in the same was as I do for .minrc and so on.

I actually tried to compile your changes on windows (with MinGW) and I got a compilation error...

gcc.exe: error: unrecognized command line option '-rdynamic'

Basically it looks like it would only work on Linux (maybe not even on macOS...) -- for info see online, e.g.:

https://stackoverflow.com/questions/31212393/unrecognized-command-line-option-rdynamic-on-gcc-v4-9-2

It would be nice to be able to manage this in a cross-platform fashion before making this public really... I can include it and release it "undocumented" and try it out and experiment a little bit.

[PMunch]

Yeah I was afraid that was going to be an issue. The problem is that each platform does dynamic libraries a little different. I'm pretty sure they all support what I've done, so it should just be a matter of finding the correct compiler flags to pass for each platform.

And .minlibs would probably be a nice place to put them

[PMunch]

For reference, the second and third solution here is what I used to get the current solution. So if you are able to do the same for Windows or Mac then it should be simple to figure out how to do it in Nim.

[h3rald]

Got it working on macOS with Clang!

Min compiles properly as clang supports the -rdynamic flag (at least on macOS) with no issues.

When I tried to compile your sample library with the following command

nim c --app:lib --noMain -d:release dynamicadd.nim

...linking failed complaining for missing symbols. After some research I discovered that -undefined dynamic_lookup needs to be passed to clang when linking to manage dynamic libraries properly.

So the above on macOS becomes:

nim c --app:lib --noMain -d:release -l:"-undefined dynamic_lookup" dynamicadd.nim

That's great. Now I have a libdynamicadd.dylib file, but unfortunately, when placing it in the right folder and running min, I always get the following error:

Unable to load lib from /Users/h3rald/Development/min/dynlibs/libdynamicadd.dylib

...which means that loadLib() failed to load the library :sad:

I'll keep trying to see if I can fix it.

[PMunch]

Yeah that's how far I've come as well. It seems like Nim is unable to load the dylib on a Mac, which is a bit weird. Maybe try some super simple examples first, to check that those work? Then try to mix in these extra things.

As for Windows I've asked around a bit and it seems like Windows binaries don't have the concept of a symbols table. So automatically loading symbols this way is not going to work. What we can do is create all the calls as templates that will look up the symbol in the main binary. Would probably be a bit slower, but since this is only done once during startup I don't think that would be much of a problem.

[h3rald]

I actually got a tiny little bit further than that.

I modified the dynamicadd.nim file a bit simplifying the module and changing pragmas a bit and I was able to load the library, retrieve the setup symbol but I get:

SIGSEGV: Illegal storage access. (Attempt to read from nil?)

...when I try to call setup(). Sigh.

Here's my current dynamicadd.nim:

include mindyn

proc setup*(): DynInfo {.dynlib, exportc.} =
  result = DynInfo(moduleName: "dyntest", dynlibVersion: 1) 

proc dyntest*(i: In) {.dynlib, exportc.} =
  echo 1

[h3rald]

Got it to work... well, not really, but I simplified things until I was able to load symbols and call them.

So now I am able to load symbols if the don't return anything 😮 -- better than nothing I guess.

I'll try to rewrite the loading mechanism to load a single symbol that defines the module by naming convention and see if that works.

[h3rald]

I finally managed to get it working! It turns out that the extern value for isFloat was wrong.

I am now able to import the test module and use it properly. I’ll tidy things up and modify the loading mechanism to point to a common directory for dynamic libraries, and maybe add install/uninstall commands as well.

I don’t think I’ll document this until I can get it to work on Windows and test it properly though. Still, it’s great to have it working on macOS and Linux so far!

[PMunch]

Ah, I was afraid that that could be an issue. The extern values are the mangled name of the Nim procs, which are meant to be deterministic. That last part which you changed is some kind of counter, so it seems compilation differences might make them quite not so deterministic after all, at least not across platforms. The mindyn.nim file was created by using a script similar to this https://raw.githubusercontent.com/PMunch/min/3e00bd82d4c2a566292ae593e2dee73f10c7328b/dynparser.nim which relies on going into core and compiling each file there with nim c --debuginfo to create the .ndi file that it loads. This way you know all the symbols are the same as what they will be in the binary.

I'll try your changes on Linux and see if they run, but I think I'll have to use my mindyn file for it to work. I also noticed you had added the clang flag to pass to the Nim compiler, it should probably be noted that this is clang specific as GCC would likely spit out an error if you tried that.

[h3rald]

I see... what about using the extern pragma on the original procs to determine exactly what name to use when mangling? Would that work?

Concerning the extra flag... that would be necessary only when compiling dynamic libraries with clang yes, I’ll probably include a few compilation examples in the docs maybe instead of having it in the examples folder.

I also removed the {.passL -rdynamic.} pragma you had at the start of min.nim: it seems to work just the same without it, on macOS at least.

BTW: I mentioned your username and link to Github profile on the about page... let me know if you want me to mention your real name and web site instead!

[PMunch]

Yes, that is probably the best solution but I wasn't sure you were going to like it since it meant adding an extern to everything you might want to use in a library definition. I just checked and I get a "/home/peter/.minlibs/libdyntest.so: undefined symbol: isFloat_BKcj9aQlJC73fcAZURf0pHw_8" when I run it here (after adding some debug information from dlerror) so either we need two separate library files, or we need mark everything with extern (or rather only the parts you would use, I just went with everything since it was easier).

Yeah, -rdynamic is a flag that expands to a --export-dynamic on platforms that support it, so on macOS it will not do anything. But it took me a while to realise that you had removed it when I tried to compile it here as the .so wasn't able to see the symbols in the main executable without it.

Thanks for the mention. You could change it to "Peter Munch-Ellingsen" and "peterme.net" if you'd like, but I'm fine with it either way!

[h3rald]

OK, I explicitly specified extern pragmas for all procs you may want to use in library definitions and it still works (on macOS). Hopefully it should also be fine on your linux machine now.

Tomorrow I'll also try on a Linux machine and also try cross-compiling min for windows & linux arm. Then... well, gotta try out windows at some point when I get a chance.

[h3rald]

I am closing this for now -- I'll release this in version 0.12.0 in a few minutes as an experimental feature. If anything doesn't work (e.g. Windows support) we can manage it with bugs.

[PMunch]

Yeah, I know for sure that this approach won't work on Windows. It simply doesn't link things that way. But I have an idea on how we might be able to create something similar that should work on Windows.