various issues with linking modes

[CosmicToast]

When linking a native module that includes external linkage (such as jurl), the ldflags (i.e libs) must be specified after the -shared flag, else the shared object will not hold the library references in its dynamic section.

For example, when building jurl on my server. With jpm as it is, the command line is: cc -std=c99 -I/usr/local/include/janet -I/usr/local/lib/janet -O2 -lcurl -o build/jurl/native.so build/src...o -shared -lpthread The correct command line is: cc -std=c99 -I/usr/local/include/janet -I/usr/local/lib/janet -O2 -o build/jurl/native.so build/src...o -shared -lpthread -lcurl

Testing is trivial using readelf -d build/jurl/native.so. When building with jpm as it is, output starts as such:

 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]

When built with the correct command line, output starts as such:

 0x0000000000000001 (NEEDED)             Shared library: [libcurl.so.4]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]

The consequence of this is that when later linking against jurl/native.so, the former will not imply -lcurl, and thus it will have to be inserted manually - something jpm does not do. When linking dynamically, it's generally ok for the symbols to only become available later, so dynamic-linking-related invocations in link-cc should likely specify ;ldflags at the end of the command line rather than near the start.

Furthermore, this is also related to a separate issue making static linking difficult even when it is available. When both a dynamic and a static version of a given library is accessible, most (so including gcc and clang) C compilers will default to preferring the dynamic version of the library (which is likely why jpm currently specifies static libraries for janet and static native modules using their paths rather than -l flags). The correct way of resolving this is passing a -static flag in front of any -ls that are available statically, and then a -dynamic flag in front of any -ls that are not. Alternatively, jpm could take a :static argument (on quickbin and on declare-executable) and prefix ;ldflags with -static when it is enabled (hard fail if any library is unavailable).

While I could try patching jpm and opening a PR at this stage (I have some rough patches around), it's probably important to see what the actual maintainers think (especially given the relatively low level of activity around the tooling and janet itself, e.g I've had #janet/1097 open for about a week; which is not an issue but pushes me in the direction of discussing first, patching later).

[pepe]

I have been bitten by this too. I guess? Cause my expertise in this area could be much higher.

Thanks for your patience. There is only one maintainer on this level.

[CosmicToast]

I have been bitten by this too. I guess? Cause my expertise in this area could be much higher.

I've got some experience developing distributions, which does lead to somewhat better familiarity with the whole process. I'll have to think about ways of improving the general Janet and JPM experience... My main concern will necessarily be windows compatibility because I don't really ever touch that stuff.

Thanks for your patience. There is only one maintainer on this level.

No worries! I'm quite aware of what it's like to have limited output (on top of the usual internet-related ADD and similar, I am also chronically ill) and have no complaints on the overall, so long as things do get dealt with eventually. I'm primarily mentioning it to explain the change in approach (i.e why I'm opting for a lengthier discussion rather than a "just write a patch" approach), especially in a case complex such as this one.

If there's anything else I can do to help (albeit in my limited capacity as it is, and given the skills I have (and more importantly those I don't)), please do tell me.

[bakpakin]

It's been a while since I have looked at this, and also a while since this was filed, but better late than never - thanks for looking at this.

For legacy reason we have both lflags and ldflags - we should probably get rid of lflags or make it just an alias for ldflags - this likely caused some confusion in the implementation where lflags are where the ldflags should go. While I don't want to break existing software, It does seem people more often than I would like need to work around the default flags (myself included).

[CosmicToast]

Would a good way to improve the situation definitively be to write a jpm alternative for POSIX-compatible platforms (I can't reliably test or predict windows behavior, I don't really work with it), aiming for proper handling of everything? That way, the code could be looked at and compared for bringing into mainline later. Really, what I want is relatively simple, I would want better handling of compiler tests, being able to do true static linking (while keeping dynamic linking for runtime/REPL use), and pkg-config integration. It's just that this is a known pain point for many languages, and one that feels possible to resolve in Janet.

[bakpakin]

Jpm really does need a bit of a rewrite, especially for posix compatible stuff. Ideally, it would be part of spork as well for easier setup. I had somewhat planned this for a while, as there are other issues with JPM's interface that are confusing and hard to reuse. JPM began as an opinionated build tool more than a full on package manager, so it doesn't deal with dependencies ideally - issues with pkg-config and such aren't as much of an issue if you are building everything in one project.janet.

A number of issues that would be good to change:

• Not easy to use from inside a Janet repl - interface should be repl first.
• Some functionality, like interfacing with C compilers, isn't exposed well for reuse.
• Custom configuration system that is confusing to people. Instead, we should use normal dynamic variables for configuration
• pkg-config functionality (there were some scripts to do this scattered in various places)
• Too many configuration environment variables. We should just use JANET_PATH and infer everything else. This may reduce possible configurations, but I think that is a win.
• Stronger validation of project.janet. A lot of issues with jpm come from broken packages, circular/over-specified dependencies, and broken flags.
• Lots of duplicated functionality in spork - building jpm into spork would get us lots of things for free.
• Building spork as part of the bootstrap process would be a good self-check that your build system was setup correctly, rather that installing jpm and finding out it is not well configured.

Another idea is that project.janet should be executable as normal scripts - no implicit imports. Hypothetical project.janet:

# JPM analog in spork with aforementioned changes.
(import spork/project :as pj)

(pj/defproject
  :name "..."
  ...)

(pj/declare-executable
  ...)

# Generate a main function that has subcommands for build, test, deps, etc.
# Also validate the built up project state.
(pj/generate-main)

I think once posix works, it would be doable to make MSVC work as well as most of the general procedures are fairly similar, although Windows dynamic linking works differently than linux/posix.

However, there are a number of functionalities in popular package managers that I don't think are very good, including support for multiple versions of the same packages, and semantic versioning.

[sogaiu]

@bakpakin Would you mind elaborating on this point:

in popular package managers that I don't think are very good, including support for multiple versions of the same packages

Is this about the theoretical possibiliity of multiple versions of the same packages or specifically the way it has been done elsewhere?

In general, I'm not a fan of regularly using multiple versions of the same package and realize there can be issues if different versions of a particular package are not written to coexist with each other, but I have found that being able to do so in certain situations has been helpful in getting things to work out.

[bakpakin]

My point is that practically, it doesn't work very well because of native dependencies. Of course it is possible to implement, but you add complexity for questionable value. This "feature" from the npm world seems great if you never lift the curtain and see the waste, complexity, and fragility, but there are plenty of mainstream package managers that take a different approach - separate versions live in separate trees.

[sogaiu]

Thanks for the point about the native dependencies.

(I don't really have experience with using multiple versions from npm (I'm not at all a fan having suffered from the related churn multiple times). I had issues with not being able to use multple versions of things elsewhere and having to resort to unfortunate renaming hacks.)

The type of situation I have concerns with was described by @ianthehenry in the "Modules and Packages" chapter of "Janet for Mortals":

you can depend on the Spork [...]library and only use a small part of it, but by doing so you’re locked into a single version of Spork for all of its components. If you want to upgrade to a newer Spork to pick up changes to the spork/zip module, but you want to keep running an old version of spork/argparse, then… you can’t.

I think what's meant here is using the current jpm dependencies mechanism.

What I've started to do recently is to use git-subrepo to vendor my dependencies. This doesn't use the dependencies mechanism and protects against dependency repository disappearance, mutation, and some breakage as well, but it does seem somewhat extreme for all projects [1]. I think it can handle multiple versions of things depending on specifically what those pieces are -- the developer needs to arrange things manually of course. Granted I'm not doing this with any native stuff.

Perhaps this kind of situation could be handled using this idea you mentioned elsewhere?

What might make sense is the ability to install packages to a different location like installing spork to subtree/spork and being able to import that separately, no need for tags or redoing the import mechanism.

---

[1] I think it can make a lot of sense for developer tools though.