Has crystal ever considered adding dynamic require

[femto]

Discussion

if 1 require "foo" end or str = "sth" require "#{str}adapter" --> that can loads different db gem in web framework or different llm connector based on config value. (It's common needs app that needs to connect to llm now)

ruby can dynamic require, Has crystal ever considered adding dynamic require.

[Blacksmoke16]

It's not really my area of expertise, but I don't really see this being a thing. Just because Ruby has something doesn't mean Crystal needs/wants it.

It's unclear how this would work given Crystal is compiled. I.e. if you can only know if code should be required at runtime, then the compiler has no way to know if that code is compatible with the rest of the program until you run it. This essentially negates all the benefits of compile time type checking.

Crystal also comes with tree shaking, so unused code will not be included in the final binary. I.e. it's not really that big of a deal to explicitly require potentially optional code. If you want to be explicit, you can use macros to conditionally add require into the program at compile time however.

[femto]

this relates to

https://github.com/crystal-lang/crystal/issues/14179 , I want to add marshal support

Generally ruby web framework will let you specify db type to connect and dynamic require the related gem. how does Athena handles it? (I just bought your book, but haven't have time to read it:)

[Blacksmoke16]

What does this have to do with marshaling support? Why would you need to dynamically required the related DB shard? If you know what DB you're using just add a require "pg" or require "mysql" and be done with it.

[femto]

how does Athena handles it? Or does Athena doesn't have db component? I'm assuming it has like rails does, which is a big whole full stack framework.

Then what about llm? In MetaGPT I'm contributing to, https://github.com/geekan/MetaGPT the framework using config to connects to different llm provider,
which makes sense not to load all llm provider eggs.

[Blacksmoke16]

how does Athena handles it?

There is no dedicated ORM component or anything at the moment. The user installs/requires whatever DB they want to use, and generally follows https://crystal-lang.org/reference/1.10/database/index.html. Likely also leveraging https://crystal-lang.github.io/crystal-db/api/latest/DB/Serializable.html. Even if it did however, it wouldn't need dynamic requires. Like how none of the other frameworks that do have a DB/ORM integration need it either.

Then what about llm? which makes sense not to load all llm provider eggs.

The user would just have to require which ones they want to use? I'm not familiar with LLMs, but that seems reasonable.

EDIT: Unless you have an actionable proposal, the forums may be a better place for this disucssion.

[femto]

The user would just have to require which ones they want to us? I'm not familiar with LLMs, but that seems reasonable.

the user, that depends, if it's fixed user, a company, that can do. but if it's some general provider, then that's not good. like https://huggingface.co/spaces/deepwisdom/MetaGPT here huggingface is a generally platform where you can host your machine learning model, so MetaGPT created a space here to let others from the world try MetaGPT. assume there will be different configuration to fill in the form. To huggingface it wouldn't know beforehand which file to require.

I need to know Core Team's attitude towards this, if they want to go for a more dynamic path or not. to support dynamic require, we need to split the compilation unit, which means every file compiles to bytecode first, (which better needs Marshal support, dump bytecode into file) , Marshal.dump(RubyVM::InstructionSequence.compile(code)), also that can help incremental compile(But I guess incremental compile is still a static thing)

[femto]

How does interpreter implement require(currently interpreter can require files that unknown to it beforehand)

[straight-shoota]

In contrast to Ruby, Crystal is a compiled language, so the concept of dynamic require doesn't translate that well. The interpreter is a bit special, it can require code at runtime, but that's only possible because of its natural overlap of run time and compile time.

In Crystal we need to know all code at compile time. If you know which code you need at compile time, you can use compile time macros to make requires conditional. If you don't know which code you need at compile time (e.g. your program needs to support multiple db backends and it's only decided at runtime which one is used), then you need to compile all the code and only use what you need at run time.

In case it's impossible to know all the code that might need to be executed at compile time, this is quite difficult in Crystal. You could have some kind of pluggable components at runtime via dynamic libraries which you load as necessary. Crystal doesn't work very well for dynamic libraries though, so you would most likely need to write the libraries in a different language (or abstain from using almost anything from Crystal's standard library).