Add the ability to create a dynamic library

[sheosi]

As of now, crystal lacks any possibility of making dynamic libraries, which is a bump for this language as it would open a great number of possibilities, theoritically, it's possible I've made myself Crystal code called by C code, and as much as I've tried (not that much, anyway) there's no problem, so far the only show-stopper I found was the libraries were compiled without pic, making dynamic libraries impossible. Maybe, by compiling the .bc/.ll by hand there's a possiblity this can be workarounded, but I couldn't get there myself (version from git failed to build).

[bcardiff]

It would be really nice. There have been some efforts in that direction, but Crystal is not there yet

Maybe #241 and this Makefile could help a bit. But have in mind this last comment

Again, I agree it would be really nice.

[sheosi]

Sorry, it seems I didn't dig enough, a shame anyways.

[bcardiff]

@sheosi Not at all! It wasn't an easy search.

Also gave in mind that fun functions are like C externals, they can be referenced by name when loading. Other def have some mangling due to template like expansion. If you deal with the GC you might be able to come with a solution.

It will be really good to expose Crystal code to the outside world. There is just no clear roadmap for this yet.

[sheosi]

If this helps anyone all that I did was to copy crystal's main inside a initialize function,also removed yield keyword from the main( it gave some strange issues, I couldn't print from crystal).

[asterite]

@sheosi Crystal requires a GC, does it really make sense to use it for dynamic libraries?

[sheosi]

While I don't know much about GC side-effects, I guess it should be usable anywhere where Lua is usable, particularly I'm interested in plugins, as thanks to crystal's power they can be written in a declarative-style.

[ozra]

It wouldn't be impossible despite GC. But you'd get the reverse problem of manual memory managment (remembering to free memory), namely ensuring that memory is kept, keeping lists of pointers to everything that might still be alive in the libs using code, so that the GC still has references to them. Or, doing manual allocations for everything that is passed over to using code.

[fabianloewe]

I'm working on this and got a shared library by emitting an object file and linking it to such a library. My only problem is all Crystal symbols are hidden/local so one can't load them dynamically or link to them. What in the compiler causes this?

BTW, I don't think the GC is a problem. We could just let all libraries use the executable's GC so all objects are tracked at one place. But I haven't got so far...

[ysbaddaden]

@hyronx You need to write an interface between the Crystal code and the outside. That is you must write a C API, much like you would to expose a C++ library. Thankfully you can write it in Crystal using fun instead of def, taking care to Box references to Crystal objects.

Since I was interested, I put together a Proof of Concept that shows that this is possible and that it does work! But it requires a bunch of boilerplate to be manually written (maybe macros could help): https://github.com/ysbaddaden/crystal_library

It only requires a small hack to avoid loading the original main.cr and expose an alternative method to manually initialize the GC and the Crystal world (constants, ...). Maybe it could interesting to have a -Dlibrary flag in the original main.cr @asterite? Even though we don't support this explicitly, there are some usage of embedding Crystal into another language (Ruby extension, plugins, etc) and it's a tiny change.

[fabianloewe]

@ysabbaden That's really nice work. I would like to make linking Crystal with Crystal code possible without the C layer between it. Maybe I get so far in the next days with the help of your PoC. It would help me if somebody could explain why all Crystal symbols are local.

Shared library support and linking to all kinds of existing code could help making Crystal more popular I suppose.

[ysbaddaden]

No, creating libraries for Crystal code isn't possible without hacking the Crystal compiler, and this is probably impossible to achieve:

• It would miss all the class/module/method signatures, and it would need a mechanism to read them from (like a header file).
• Furthermore, method argument types are dynamic, and creating a library would fail, unless all method arguments are fully typed beforehand (killing most of the pleasure of using Crystal).

If you want dynamically loadable plugins written in crystal into a crystal application, you'll need to have a C API exposed and a lib binding to this exposed API. This isn't pretty, but it could work. Bonus: plugins could be written in C, Rust or C++ now.

[fabianloewe]

The compiler could get its definitions from the library .cr files and then look for the binary containing this code in standard paths. If it finds one it could instead of compiling the code into the executable, link to this shared library. Otherwise it just compiles the library and links then to it (possibly also installs it). So far my idea. A bit more research and compiler modding could show what's possible but I believe that is achievable.

Anyway having a C API between it is definitely better than nothing and I'll go this way at the moment for my stuff.

Update: I found an answer to my question myself. Switching context.fun.linkage = LLVM::Linkage::Internal to External makes all symbols visible. This breaks a lot of other things but a step forward for me. :dancer:

[fabianloewe]

Some more progress: I made my first Crystal library which links to a Crystal application. The GC's are still seperated and I haven't tested any parameters but puts works. :smile: I will further play with the compiler in the next days when I have some time to spend.

If somebody is interested: https://github.com/hyronx/crystal-shared-lib

[ozra]

I wish dynamic linking was dead already B-) And also, that everyone shared their source, and everyone's source was in Crystal B-) (Pipe dream disintegrates and I'm suddenly back in the bitter reality.. hehe)

No but seriously: it's definitely cool PoC to fiddle with, but why on earth not take advantage of using shards and letting Crystal do it's whole-program-magic - if usage in Crystal was your aim? I'd get it if it was to expose a great crystal lib to C / insert-esoteric-language-here users...

Anyways, keep hacking :-) You'll probably invent something cool.

[ysbaddaden]

Plugins are really the single use case I can think of. Build support for countless providers but allow to only load a few of them; allowing external plugins to be developed without having to recompile the whole application (and all the plugins you want).

[fabianloewe]

Okay, imagine Crystal has grown in the last year/s. There are already multiple applications written in Crystal pre-installed. Do you really want to have every application include in its executable the same parts of code from the std library? Furthermore there will be some often used shards and their code will be included in every executable that's using them.

This is a huge storage wasting so the decision to separate code into shared libraries were already made lots of years ago. Why should we know do the opposite by not gaining anything from my point of view?

And what I try to reach: Exactlly that. Naturally compiling libraries downloaded by shards to shared libraries. Not for dynamic loading but linking as we are linking against pthread but everything more automatically through the compiler.

[ozra]

Do you really want to have every application include in its executable the same parts of code from the std library?

Yes, and no. Yes: I'd prefer all applications on my entire system statically linked. I think that's a much safer and more sustainable way of keeping a system intact: Decentralize, decouple, distribute. No: Not the same parts - with today's linking methods only the used parts are included in the binary.

This is a huge storage wasting...

Storage is cheap, and cold pages of code (unused...) will be released from memory (the kernel knows where to find them).

...so the decision to separate code into shared libraries were already made lots of years ago.

The decision to separate code in to shared libraries was made yeeeeears ago when storage space was expensive and hard to cram in to an architecture at all. When there wasn't such a multitude of desktops and frameworks. If all apps used libc x.y.z and xlib u.v.w - then fine. When one app use KDE, one QT4 alone, one QT5 alone, one Gnome 3, one GTK3 alone, one GTK2... ad nauseam - the overhead outweighs having compiled them statically and having only the functionality actually used in the apps take space in mem, instead of entire chunks of dead code paths of the entire lib loaded as dynlib (which of course to be fair, would also be paged out though).

Why should we know do the opposite by not gaining anything from my point of view?

Maintaining ever changing dynamic dependencies is expensive, especially when they break. As a fresh anecdote I upgraded one of my distros just two days ago, and the new version of libc refused to install, but the distro-upgrade continued with just a warning. At the end app after app fucked up. When doing some hell-holish gymnastics to get that package in manually after and salvage it, it turns out it had refused to install because "/lib" was in my LD_LIBRARY_PATH and that was a "don't-like-wont-install" prerequisite. That's brittle system integrity in my view. (Note my opinions are not based on this incident, I've held them for a while)

And on a flip side, comparisons for Linux distros aiming for everything statically compiled (in this case with musl as libc) indicate (aside from the obvious faster startups) that memory usage is actually lower. That's probably only true for core utils and not for all cases, (and perhaps because the comparison is dyn-linked glibc ;-) ). But this is not of immense weight to me.

There's always the ASLR argument, but that can easily be torn into shreds.

I don't mean to discourage you at all - these are just my two cents, and of course a lot of studies on this, on a wide variety of setups, would be needed to get a truely clear picture. I guess my main point is: not being able to dynlib cr-code shouldn't be a worry even if you don't share my view that it's actually a "feature" ;-)

[asterite]

I agree with what @ozra says. For example Mac OSX apps are mostly self-contained, and these apps contain duplicate dependencies between each other. This is fine, storage is cheap. And the good thing is that when you uninstall an app all of its dependencies go away too. And there's no conflict with versions and so on. So it's much simpler and you don't get the version messes you have in linux.

[ozra]

The version messes in Linux distros not using all statically linked binaries you mean ;-)

[fabianloewe]

Yes, storage is cheap but we also have Raspberry Pis (we can connect an external device, so not such a huge problem here) and smartphones today which only have 32GB of storage. I would really like to see Crystal applications running on them, too. (at least on rpi)

For desktops and laptops you are completely right. With shared libraries I also had in mind to link to existing code written in other languages in the future so we don't have to reinvent the wheel.

But maybe I'm on the wrong way... Currently there is no native static library support, right?

[ozra]

I was about to mention in my rant that even RPi's and hand helds wouldn't be a problem - but you might build enormously bigger applications than me B-)

[fabianloewe]

Not really. :smiley:

Okay okay, I drop this stuff. At least I've learned much about the compiler.

[miketheman]

Go 1.8 released a plugins package - we might look at how they implemented a shared object approach and see if any ideas surface? https://beta.golang.org/pkg/plugin/

[bew]

I kinda like the RPC-based plugin system (made for go here: https://github.com/hashicorp/go-plugin)

With more information here : https://en.m.wikipedia.org/wiki/JSON-RPC

In Crystal we won't need any hard modification, I think this would ship as a little library on the host process, which coordinate RPC's I/O with a running "plugin", which would be another crystal program (or any other language) that can get input from the host and gives back some output through a specific transportation layer for the RPC I/O.. Which could be over UNIX sockets for example.

[ozra]

Agree with @Bew78LesellB - Messaging, whether with help by, say, ZMQ, straight pipes, TCP or just about anything, with MsgPack, JSON, or just about anything, again - is a better choice for most cases. Less upgrade hell, rpc api's tend to upgrade with older versions still available.

And for performance - well then you should have the source - not a blob of binary tacked on with gaffer-tape, to get the benefits of FPO and LTO and avoid the expensive calls in to dynamic hell. I don't really see any step in between. It's all-in-one performance or messaging (well, both have their place in the same app obviously) And, still then, there's the always-possible option of exposing a non-mangled-C-function interface... And I think llvm can LTO objs from C, Crystal, etc. in the same binary (don't shoot me if I'm wrong).

Just saying, dynamic libs feels like bottom of the list of priorities, that's all.

[refi64]

Just out of curiosity, has anyone considered just using objects as the interface? (IIRC this is how Felix does plugins.)

The main program defines an abstract class specifying the plugin interface:

class Interface < PluginInterface
  abstract def foo
end

The plugin implements and returns this interface:

class MyPlugin < Interface
  def foo
    puts "Hi!!"
  end
end

# magic macro or something
export_plugin MyPlugin

export_plugin would define a C function that returns a pointer to the plugin implementation, along with a magic hash (not SHA-1, of course! ;) to make sure the interface and implementation versions match.

Then, the main program could just dlopen the shared library, and call the exporter function. Maybe there cou even be a way to pass arguments to the class constructor.

[doughsay]

Sorry for jumping in on an old thread here, but I think there's maybe something that wasn't called out specifically. (But my understanding of these things is far from complete)

I agree with the general feelings towards dynamic linking above, the shards system for downloading and including source code into your project is better than dynamically linking to shared libraries at compile time. But that doesn't say anything about the possibility of dynamically calling into a library at runtime (using dlopen, as I understand it).

Having some way of being able to write crystal code that exposes an interface and compiles to a library, and then being able to dynamically load that library at runtime, would allow for building crystal apps with plugin systems. This would be really useful for a lot of applications. To extend an existing app with extra (optional) functionality by having to add extra source files and recompiling would be much more tedious then, for instance, dropping some libraries into a folder that the crystal app could dynically load...

[wied03]

@ozra - You make good points above. What about security/updates though? If a library has a problem, it's nice to update the 1 copy of the library on the system and know you're covered rather than track down/rebuild every executable, no?

[ozra]

@wied03 - well that argument could really be swung both ways without being a stretch, imo: If a library is updated in a haste because of a security problem, that perhaps only affect a sub-set of applications using it, then you're potentially introducing bugs into all the others (as well as those needing it, but: the lesser evil).

[edit] Dependencies should still be tracked by a good OS / environment anyway, so that all applications built with lib X, Y or Z can be updated / show up in "security updates". That wouldn't be hard to script for, based on online package information resources. But now I'm digressing far from the issue into utopia ;-)

[wied03]

@ozra - Yep. It sort of depends on key lib developers following semantic versioning well.

[Wulfklaue]

Dynamic runtime loading is one of those positive elements that i look in languages. Unfortunately, a lot of languages resort to a very crude way of dlopen and forcing c -> d/go/... conversion for each call. While i understand the reasoning why they want the shared dynamic libraries to be in C style output ( can be reused by other programs ), it simply adds a layer of complexity to it. I never liked it and it also opens up issues as stated above.

Now, i will put my foot where my mouth is and simply state: If one of the crystal core developers can create a dynamic loading library system that is easy on the end user ( Mark a file as "library", compile, link and load in the main file, no boilerplate, inline functions )... then i am willing to put some financial support toward this goal.

[wied03]

@ozra - Thinking about it more. Yes if the packaging systems tracked it, that would be nice. You at least know about the problem then. That doesn't solve the problem of how to get all the different vendors to update with the new library.

I think there are pros/cons either way. Just making the point that there is a noticeable downside to making everything statically linked.

[larubujo]

dynamic libraries plain impossible in crystal. on compile time every type gets an id. new compilation, new ids, might (will) use same ones for different types. same for symbols. interoperability impossible. crystal was thought just for one compile all source code.

[pannous]

Nothing is impossible. it 'just' needs a binary format to store the computed type graph

On 17. Dec 2017, at 01:33, larubujo notifications@github.com wrote:

dynamic libraries plain impossible in crystal. on compile time every type gets an id. new compilation, new ids, might (will) use same ones for different types. same for symbols. interoperability impossible. crystal was thought just for one compile all source code.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[fabianloewe]

@Wulfklaue https://github.com/hyronx/dynamic-library https://github.com/hyronx/crystal-shared-lib https://github.com/hyronx/plugal

[doughsay]

The shared lib stuff @hyronx just posted looks really cool. I assume the modified branch of crystal is woefully out of date at this point though; can any crystal-devs chime in on the approach taken above and why it might not be an official addition to crystal?

Even so though, the approach taken still doesn't satisfy what I had originally wanted (I don't think). The feature I was looking for is a way to dynamically extend an existing (already compiled) crystal application, with plugins (written in crystal and compiled to shared library files), and have them be picked up and initialized at runtime.

This could be used in many ways, but my use-case was that I was trying to develop a game that could be extended via plugins. Having to recompile the entire game every time you added a plugin would not have been desirable (especially if you wanted to close-source the game, but have community extensibility)

[larubujo]

example doesnt have arguments, just self.test class method. does it work with arguments? does it work with instance methods? otherwise its just very very limited and generally not useful.

[fabianloewe]

@doughsay How you can achieve that is explained by ysbaddaden in this thread. Summary: Write a plugin interface and plugin loader which hide internal C code. The problem is compiling Crystal code to a shared library because the compiler isn't capable of this right now. (This is why I started playing with the compiler internals)

What I've mentioned was never meant to be fully usable at this point but just to be a small PoC. The Crystal community wasn't really interested in this stuff back than so I stopped working on it.

[doughsay]

Thanks @hyronx, I went back and re-read some of the posts; this is a long and old thread!

I realize what you did was only a proof of concept, but I support the idea. I will be watching crystal closely in 2018, and am looking forward to what everyone comes up with.

Good work everyone! I look forward to playing with crystal again in the future.

[fabianloewe]

@doughsay No problem. Let me know if you dig deeper into this some time or other.

[Proximyst]

I managed to load and call .so files written in pure C from Crystal, and loading Crystal from C, which should then also show the shared objects could be written in Crystal instead. This was also done with only gcc, crystal and static linking, which I was quite surprised was all needed, but it does have quite a bit of different components. I can probably set up a repo if anyone wants it as of now, else I'll most likely write a plugin library using my findings.

[RX14]

Loading a plugin written in crystal from C isn't too hard. Loading TWO plugins written in Crystal from C will get you segfaults. And for the same reason, loading a plugin written in crystal from crystal should also get you segfaults.

Although, it's likely possible to modify crystal's compiler and stdlib to allow you to load multiple crystal "runtimes" in one process, they'll never play nice together.

And even after you've solved that you'll have to work to define a consistent binary marshal format for serializing objects between plugins. You don't really want to be stuck with C structs and integers for plugins method arguments.

[asterite]

Plus each compilation will assign different type ids to every class, so is_a? can never work well in plugins... at least not now.

[RX14]

@asterite that would be part of the "consistent binary marshal". You'd encode a class Foo into a binary format then decode it back. Creating the class in the normal way and changing out the object IDs.

Of course you could never have shared objects/memory between plugins, only RPC.

[bew]

I was just reading about Kotlin, and saw it can be used as a dynamic library: https://kotlinlang.org/docs/tutorials/native/dynamic-libraries.html It's interesting how the C header file is generated and how the library is used from C, maybe we can take inspiration from them

I'm just putting this here

[jensb1]

Though I would share this, was playing around to create a simple Node JS api with Crystal. It works. https://github.com/jensb1/crystaljack2js

Anyway, I understand the risks with GC cleaning up allocated memory in a shared library context (so it become invalid). But I guess that can be mitigated by allocating the memory manually using Pointer(T).malloc(1) if required.

Any other thoughts on this?

[bew]

Allocating memory is not the problem, freeing it is!

[jensb1]

Allocating memory is not the problem, freeing it is!

Quite new to this but do you mean that if the Crystal process allocate memory that is handled within the GC then freeing that memory in another context (not aware of the GC) would set the GC out of sync with what is allocated and what is not?

If so, couldn't that be solved by implementing a allocation function outside the GC and then that could be freed without the problem above?

[asterite]

@jensb1 @bew I think you are discussing an entirely different thing that's what being discussed here (it has nothing to do with the ability to compile crystal code to a library that can be loaded by some other crystal code)

[j8r]

Wouldn't shared/static libraries be also a solution to the infamous incremental compilation?