Support building native modules from other languages with jpm

[ghost]

jpm is able to compile native modules for use in artifacts, but currently only supports those modules being written in one language: C. There are other languages, such as Zig or Rust, that people may also want to use, but integrating them into the jpm build system requires hacks. I propose we remove the assumptions that impose this restriction.

I see two ways this could be done:

• Rather than passing arguments of declare-native :source directly to cc, first check the file extension, and pass to a user-definable command; so for instance .c and .cpp (or whatever alternative C++ extension the author uses) could be configured to route to cc -o {name}, .zig to zig build-lib -dynamic -o {name}, and so on. This would require the least intervention from the author, but would be ambiguous in the case of multiple input files with different extensions.
• Add another argument to declare-native: :command (or :compiler, :build, etc.), which takes a command string to pass to the shell, i.e. (declare-native :name "mymod" :command “zig build-lib -dynamic -o {name}” :source @["mymod.zig"]). This is much more flexible and covers all cases, although the naïve implementation is more verbose. (This can be partially mitigated by defining the command string earlier in project.janet.)

[andrewchambers]

I'm not sure i would classify other language support as hacks, it's more like some languages come with builtin 'make' rules and others don't. Since both rust and zig do their own fine grained dependency tracking, simply making jpm call out to those tools seems more reasonable than trying to reverse engineer how they build things.

Perhaps jpm needs something like an 'externally built native' and the tool just needs to arrange for a .a and .so to be put in a specific place.

It's a good thing to think about, these languages have lots of good libraries that janet can leverage.

[sogaiu]

A side note -- perhaps it's well-understood, but IIUC there is C++ support (in addition to the aforementioned C). See: https://github.com/janet-lang/janet/blob/master/CHANGELOG.md#1155---2021-04-25

[bakpakin]

So jpm's general rules system is quite general and language agnostic, but yes all of the built in declare-* functions (any function that generates rules) are tailored for C and C++. The question is whether or not things like declare-native should accept/auto-detect language, or should we just have functions like declare-zig-native. I can see arguments for both.

I can perhaps see the desire to mix languages in one declare call to generate a single .so/.a/binary file, but I don't think it is as simple as it is made out to be above - the linker needs to be aware of all of the languages and how they would interact. C and C++ work together because we can just say "if any of the files are C++ files, just use c++ instead of cc." I image other languages wouldn't be that much harder, but it would be a case-by-case basis.

A function like declare-zig-native is much easier to implement because we wouldn't have to worry about how to handle other C and C++ source files unless we wanted to.

[uvtc]

Another language that I could imagine being of interest for this kind of thing: Drew Devault's new not-yet-released (or even named?) systems language: https://drewdevault.com/2021/03/19/A-new-systems-language.html (more sample code here).

[GrayJack]

I was thinking the same, implementing declare-{lang}-native would be easier

[tionis]

Could this be solved by the changes done/proposed in #1444 (bundle/* and related)?

[bakpakin]

Yeah, for now closing this. The bundle/ doesn't directly solve this, but makes it clear that interfacing with arbitrary compiler toolchains is outside the scope of what Janet can do out of the box.