Add the ability to create a dynamic library

[sheosi]

As of now, crystal lacks any possibility of making dynamic libraries, which is a bump for this language as it would open a great number of possibilities, theoritically, it's possible I've made myself Crystal code called by C code, and as much as I've tried (not that much, anyway) there's no problem, so far the only show-stopper I found was the libraries were compiled without pic, making dynamic libraries impossible. Maybe, by compiling the .bc/.ll by hand there's a possiblity this can be workarounded, but I couldn't get there myself (version from git failed to build).

[bcardiff]

@j8r only if you have well-defined boundaries of the libraries. The binary representation can't be determined unless you have the whole codebase.

[meltheadorable]

For what it's worth, my use case would be writing a libretro core, I need to be able to compile a c-compatible library with a set of already-defined functions. It'd be amazing for this to someday be possible in crystal, and the alternative options (like using the source code/shards or defining an RPC interface) don't apply because it's got to be compatible with the existing API.

I get that this isn't really seen as a priority, but there are valid use cases here with no valid workarounds where crystal would otherwise be a great fit, so it'd be great if this was at least seen as a worthwhile use case to keep in mind for the language in the future (or if i've misunderstood, and this case is possible somehow, it'd be great to be pointed at the workaround)

[turbo]

It will be really good to expose Crystal code to the outside world.

Another use case from my side. I'd specifically like to write Lua modules in Crystal (not embedding Lua in Crystal - the other way around). To me, Crystal is a nice fast and typed alternative to MoonScript, which is what I'm using for nginx development. But I'd rather write most of the code in Crystal and use moon to glue it to the runtime (in this case OpenResty's LuaJIT).

[asterite]

I'm closing this issue. There's no way Crystal can be used to generate dynamic libraries. It has a GC. It has no modular compilation model. There's simply no use trying to invest effort in this. It's simply not Crystal's target. It's like trying to creating a dynamic library with Ruby.

[bew]

Maybe we could push something like go's pie plugin system: https://github.com/natefinch/pie ? This would work even with a GC, and should work quite easily with crystal!

[jemc]

Is there any appetite for making the Crystal-callable-from-C approach from @ysbaddaden (https://github.com/crystal-lang/crystal/issues/921#issuecomment-226157633) an officially supported case?

It sounded like there was only a small change needed to fix the need for the main.cr hack. And I'm willing to do that work and submit a PR if there is appetite for accepting it.

Should we open a new ticket to track the Crystal-library-callable-from-C use case, separately from the Crystal-library-callable-from-Crystal use case that took most of the discussion here, and was deemed unfeasible?

[rdp]

With a new GC it could...possibly work. I don't see it as being totally outside the scope of crystal. I'd see this ticket as a "feature request" so maybe worth leaving open. Cheers!

[jemc]

I don't understand why you say it needs a new GC - @ysbaddaden only said in his comment that the GC needs to be manually initialized.

[rdp]

Some people still write static libraries so they can be consumed by other programs (ruby, C etc.). Anyway I assume if you link against two different crystal libraries then "boom" with conflicting GC's but good point, if it's only one, bdwgc might still work except...how does it know where all the roots are? If there are crystal object owned by some calling C client. I assumed each library would need its own "separate" GC somehow or other...but who knows. Maybe it could only pass back native types, never objects, but that doesn't seem quite right either...anyway it's more of a curiosity to me than anything, probably a low priority, but possibly fun for "some day"? :)

[jemc]

Yeah, personally I'm not really interested in having multiple Crystal libraries linked to a C program, or any interactions between such.

My interest would basically be to embed Crystal-written functionality (and presumably, the runtime) into a C program, using native/FFI/C types for the interface between the two. It seems like @ysbaddaden's example does that in a minimally invasive way that just needs a small patch or two to crystal proper, so it seems like an easy win to enable that kind of use case.

[RX14]

My interest would basically be to embed Crystal-written functionality (and presumably, the runtime) into a C program

Then why not embed the C program into the Crystal one? If you're only ever going to be using one crystal codebase in a given executable, this is equivalent and supported today.

[jemc]

It's not really equivalent because it doesn't meet the use case of distributing the functionality as a shared-library to be used in arbitrary C programs that I may or may not personally control or have access to the source code of.

[RX14]

that I may or may not personally control or have access to the source code of

Then you cannot guarantee they do not use another Crystal shared library.

[jemc]

Does the Crystal runtime rely on static/global state that pollutes the global namespace? Otherwise, I wouldn't expect the mere presence of another Crystal shared library to be a problem.

[rdp]

Yeah it might work...there's some confusion for me in terms of the threading model when it's called as embedded...

[RX14]

Does the Crystal runtime rely on static/global state that pollutes the global namespace? Otherwise, I wouldn't expect the mere presence of another Crystal shared library to be a problem.

Yes. It has a runtime, and that runtime doesn't expect to be loaded twice.

[jemc]

Yes. It has a runtime and [...]

Obviously I know that Crystal has a runtime, it's right there in my question about "the Crystal runtime". There's no need for this part of the comment, unless you are trying to imply that "doesn't expect to be loaded twice" is a universal feature of language runtimes. But that isn't the case. Not all language runtimes rely on static/global state, and some are properly isolated for such uses.

[rdp]

It's definitely possible. Just will take quite some work :) And probably not a high prio for the core guys, but PR's welcome! I still think this ticket should be reopened as a feature request :)

On Tue, Feb 18, 2020 at 1:27 PM Joe Eli McIlvain notifications@github.com wrote:

Yes. It has a runtime and [...]

Obviously I know that Crystal has a runtime, it's right there in my question about "the Crystal runtime". There's no need for this part of the comment, unless you are trying to imply that "doesn't expect to be loaded twice" is a universal feature of language runtimes. But that isn't the case. Not all language runtimes rely on static/global state, and some are properly isolated for such uses.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/crystal-lang/crystal/issues/921?email_source=notifications&email_token=AAADBUD2GKFVOBKIW44FO23RDRAC3A5CNFSM4BKEO2UKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMEP5PA#issuecomment-587792060, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAADBUCDCQTI3GKWQ7BZOTDRDRAC3ANCNFSM4BKEO2UA .

[konovod]

Yes. It has a runtime and [...]

There's no need for this part of the comment, unless you are trying to imply that "doesn't expect to be loaded twice" is a universal feature of language runtimes.

It is not a universal feature, but it is a feature of Crystal runtime. Thing isn't only in polluting global namespace - i haven't looked in details, but at least two problems come in mind - crystal runtime sets signals handler (so second one will override first one) and boehm gc have global state that won't work nice with two gc users. Of course it is solvable.

[watzon]

Definitely should be reopened and probably tagged with "feature request" and "help wanted". Being able to export a C interface would be nice for a number of reasons, not the least of which is making something like a metacall loader really easy.

And with something like that we'd have full Crystal/Ruby interoperability.

[infoman]

I'd prefer all applications on my entire system statically linked.

IMO the entire concept of bundling/static-linking everything is horribly broken, because if one tiny library has a critical security issue, then suddenly all projects that depends on it need to recompile. And if it's a proprietary project and the author just don't care — we now have an application that is vulnerable by design.

[bellaz89]

@asterite Hi! First of all, thank you very much for your effort! The language seems very promising and nice :) Sorry if I ask dumb questions. Just curious..

I'm closing this issue. There's no way Crystal can be used to generate dynamic libraries. It has a GC.

I don't get the point here. boehm was originally made for C. Plus it has functions to arbitrarly remove/add roots and from the boehm gc.h header multiple invocations of GC_INIT are harmless.

There's simply no use trying to invest effort in this. It's simply not Crystal's target. It's like trying to creating a dynamic library with Ruby.

Well, Ruby modules can be and are used by C/C++ programs by linking the runtime (as for Lua/CPython and other GCed interpreters). Having the possibility to generate Crystal dynamic libraries would open the possibility to generate plugins and modules for other languages.

I know that the directiony you want to give to the language is different but I would e.g. totally love to use Crystal to speed up python code without relying on 'heavy' languages such as C/C++/Rust. This IMO would expose Crystal to a much bigger audience that is stuck with other languages for a variety of reasons. For example: Why just not using only Crystal in the first place instead of Python? Because Python has still a much-much bigger ecosystem.

[RX14]

The tricky part is not GC, as far as I can see, it's that crystal programs, or any dynamic library which you make in Crystal, includes the standard library and all other libraries. Loading a single crystal dynamic library (in its own thread so it can own the event loop) is not that difficult to achieve. However it's inadvisable to do this, since linking the executable would break as soon as another dynamic library written in crystal is added, because there would be symbol clashes.

The core team isn't looking at this near-term, but as far as I know there have been a few proof of concepts for crystal dynamic libraries. There are clearly remaining problems to be solved, and I think they can be, but the effort will most likely come from the community and anyone interested and not the core team.

Some ideas to look into for anyone who wants to look into this:

• dynamic library mode which prefixes all symbols with a nonce, to allow multiple crystal programs to link together
• support wrapper for dynamic libraries which handles spawning the event loop in a new thread, and "portalling" calls from exported symbols in a calling thread to crystal functions running in a crystal thread (is this neccesary?)

[shayneoneill]

I realise this is probably beating on a dead horse, but it would be nice to have this facility. I write music software, and the primary market for independents is VST/AU/LV2 plugins, usually in the form of a .dll , .so or dylib. Still , the problem sounds like its a bit hard to really solve so oh well. But just wanna point out that the arguments philosophically against dylibs really arent generally applicable, there really are use cases for dlls that cant be solved other ways.

[rdp]

So you want to create dylib's is that right?

On Wed, Apr 20, 2022 at 4:37 AM shayneoneill @.***> wrote:

I realise this is probably beating on a dead horse, but it would be nice to have this facility. I write music software, and the primary market for independents is VST/AU/LV2 plugins, usually in the form of a .dll , .so or dylib. Still , the problem sounds like its a bit hard to really solve so oh well. But just wanna point out that the arguments philosophically against dylibs really arent generally applicable, there really are use cases for dlls that cant be solved other ways.

— Reply to this email directly, view it on GitHub https://github.com/crystal-lang/crystal/issues/921#issuecomment-1103783126, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAADBUGBKZSPDOW2FJEEOC3VF7M77ANCNFSM4BKEO2UA . You are receiving this because you commented.Message ID: @.***>

[axvm]

I want to vote for this feature. And I can't figure out why crystal is apparently trying to become a golang for Rubyists but can't have the same features as golang. I.e., golang has a build mode for dynamic libs. And I would love to do some computations in crystal and call it in ruby.

I believe this is an essential feature in the context of mass adoption.

[HertzDevil]

Today I have tried to build a true Android app written in pure Crystal from my Termux / Android NDK branch. The entry point must be defined through a shared library, and it basically boils down to this:

fun on_create = ANativeActivity_onCreate(activity : LibAndroid::ANativeActivity*, savedState : Void*, savedStateSize : LibC::SizeT)
  GC.init

  # we must pass something here otherwise `PROGRAM_NAME` will fail to initialize
  # this is _not_ `--prelude=empty`! we already have virtually the entire stdlib working
  args = ["(???)"]
  LibCrystalMain.__crystal_main(args.size, args.map(&.to_unsafe).to_unsafe)

  activity.value.callbacks.value.onStart = ->(activity : LibAndroid::ANativeActivity*) do
    # ...
  end
end

Crystal's runtime libraries (GC, libevent etc.) are then linked statically. This and the VST example are perfectly valid use cases where the shared library containing the Crystal runtime is effectively the only running Crystal instance, as the library still manages the whole lifecycle of an application, except that the entry point is no longer that of the usual command line. Thus building a shared library does not necessarily imply that we are consuming Crystal code from other Crystal code via this mechanism.

So I'd vote to reopen this issue.

(Now this scenario is solved by cross-compiling, but it won't be the case for VSTs, nor if someone decides to build the Android app on Termux itself.)