search

janet-lang / janet

A dynamic language and bytecode vm
https://janet-lang.org
MIT License
3.54k stars 228 forks source link

Re-integrate jpm into janet #1444

Open tionis opened 6 months ago

tionis commented 6 months ago

Many janet installs out there are broken or only partially correct, simplifying the setup might help improve this. To quote @bakpakin :

Really just making the installation easier would be a good start. One major issue is that non standard, broken installs are pretty common - the easiest way to make that not possible is to have just one binary - Janet

Any approach bringing a tighter integration with the runtime and package management should reduce possible misconfigurations and make it simpler for people to get started with janet.

There are still a few open questions like:

sogaiu commented 6 months ago

I think having something like jpm be bundled with janet has merits.

I'm not so sure about trying to bring jpm itself back in though. Possibly a new thing might be better? Haven't thought too deeply about it though.

pepe commented 6 months ago

Once mentioned idea of moving jpm into spork and making spork part of the janet build and release sounds the best to me.

bakpakin commented 6 months ago

I'm not so sure about trying to bring jpm itself back in though. Possibly a new thing might be better? Haven't thought too deeply about it though.

I agree with that. jpm has usablility problems that make that not ideal.

So current jpm issues as I see it:

I guess I don't think the entirety of jpm should be in Janet, but I think it would be interesting if janet itself had some simple functions for modifying the contents of it's own syspath. The flow could be something like this:

$ janet
Janet 1.34.0-0da640ef linux/x64/gcc - '(doc)' for help
repl:1:> (bundle/list) # traverse (dyn *syspath*) and show install packages
nil
repl:2:> (bundle/install-dir "/home/crose/my-local-project") # install from a directory
Doing local install...
...
done!
nil
repl:3:> (bundle/install-git "http://github.com/janet-lang/spork.git") # download spork using CLI git
Doing git install...
...
done!
repl:4:> (bundle/list)
@["spork", "my-local-project"]
repl:5:>

This kind of flow is pretty common in repl-first languages, and avoids some of the annoying and complicated setup of jpm, where it needs to be installed system wide in a compatible way to Janet.

Configuration would be done solely by where (dyn *syspath*) pointed to. Basically, Janet would just get more functions for dealing with (dyn *syspath*)

I also don't think "installation" into a tree should be that complicated by default - a git project that was installable this way could just be a project that had a bundle/setup.janet, bundle/test.janet, bundle/morestuff.janet, etc.

Lastly, the make-style build system built into jpm can be nice but I think is something that could just be left in spork if people needed it - most projects I've seen really don't need it.

Curious on peoples thought

bakpakin commented 6 months ago

The hardest part is of course an opinionated C build system (which is actually what jpm started out as, and then morphed into a bad package manager), but can come later and be a package itself (probably part of spork to make things simpler) as a module that would just help users find compilers, linkers, libraries and headers on their system.

tionis commented 6 months ago

I like the ideas of having a few simple primitives for module management etc. in janet itself and keeping more complicated parts in spork to import as needed.
Perhaps more complicated projects could import from spork/build in some setup script to compile deps? That begs the question though if spork should be kept as is or made part of janet releases (by, e.g., prepopulating the syspath with it)

sogaiu commented 6 months ago

I guess I don't think the entirety of jpm should be in Janet

...

the make-style build system built into jpm can be nice but I think is something that could just be left in spork if people needed it - most projects I've seen really don't need it.

Interested in this kind of decoupling along with having some key feature(s) make it into janet.

Having build system bits be in spork or elsewhere might be nice for extensibility and experimentation by 3rd parties.

sogaiu commented 6 months ago

An observation about the specific example repl session above...

IIUC, it demos the idea of being able to install spork with just an installation of janet. ATM, I think a C compiler is required in the case of spork because it has native modules.

Possibly the janet-built-in stuff can use a C compiler if one is detected and bail otherwise?

bakpakin commented 6 months ago

An observation about the specific example repl session above...

IIUC, it demos the idea of being able to install spork with just an installation of janet. ATM, I think a C compiler is required in the case of spork because it has native modules.

Possibly the janet-built-in stuff can use a C compiler if one is detected and bail otherwise?

So spork already actually contains some of the needed functionality for building C modules (see spork/cc) - my idea is that we would eventually have sport be able to self bootstrap with it's own utilities.

EDIT: as far as installing jpm without a C compiler, also a possibility but a number of modules depend on other native modules like json.

bakpakin commented 6 months ago

So I was able to a working version of the bundle/ proposal with spork such that spork could build and install itself to (dyn *syspath*). This initial version of this bundle module lives currently in https://github.com/janet-lang/janet/tree/bundle-tools

An example of what the spork hooks look like can be found here: https://github.com/janet-lang/spork/tree/bundle-hooks/hooks

Usage would be something like the following:

$ git clone --single-branch --branch bundle-hooks https://github.com/janet-lang/spork.git
...
$ janet
Janet 1.34.0-60e22d97 linux/x64/gcc - '(doc)' for help
repl:1:> (bundle/install "/home/crose/code/spork") # directory where git clone lives
running /home/crose/.hooks/spork/build.janet for bundle spork
compiling src/crc.c...
linking build/crc.so...
compiling src/json.c...
linking build/json.so...
compiling src/cmath.c...
linking build/cmath.so...
compiling src/utf8.c...
linking build/utf8.so...
compiling src/rawterm.c...
linking build/rawterm.so...
compiling src/tarray.c...
linking build/tarray.so...
compiling src/zip.c...
compiling deps/miniz/miniz.c...
linking build/zip.so...
running /home/crose/.hooks/spork/install.janet for bundle spork
adding /home/crose/spork
adding /home/crose/tarray.h
adding /home/crose/spork/schema.janet
adding /home/crose/spork/base64.janet
adding /home/crose/spork/test.janet
adding /home/crose/spork/argparse.janet
adding /home/crose/spork/pgp.janet
adding /home/crose/spork/path.janet
adding /home/crose/spork/cron.janet
adding /home/crose/spork/misc.janet
adding /home/crose/spork/msg.janet
adding /home/crose/spork/mdz.janet
adding /home/crose/spork/data.janet
adding /home/crose/spork/randgen.janet
adding /home/crose/spork/tasker.janet
adding /home/crose/spork/getline.janet
adding /home/crose/spork/cjanet.janet
adding /home/crose/spork/rpc.janet
adding /home/crose/spork/htmlgen.janet
adding /home/crose/spork/build-rules.janet
adding /home/crose/spork/regex.janet
adding /home/crose/spork/stream.janet
adding /home/crose/spork/math.janet
adding /home/crose/spork/init.janet
adding /home/crose/spork/ev-utils.janet
adding /home/crose/spork/temple.janet
adding /home/crose/spork/cc.janet
adding /home/crose/spork/netrepl.janet
adding /home/crose/spork/http.janet
adding /home/crose/spork/fmt.janet
adding /home/crose/spork/channel.janet
adding /home/crose/spork/httpf.janet
adding /home/crose/spork/sh.janet
adding /home/crose/spork/generators.janet
adding /home/crose/spork/services.janet
adding /home/crose/spork/cmath.so
adding /home/crose/spork/rawterm.so
adding /home/crose/spork/crc.so
adding /home/crose/spork/utf8.so
adding /home/crose/spork/zip.so
adding /home/crose/spork/tarray.so
adding /home/crose/spork/json.so
nil
repl:2:>

Curious on feedback here - I know this does not include all of the features that jpm has currently, by design.

This could also be put into spork rather than Janet as there really isn't anything here that needs integration with janet itself, but keeping this interface for installing packages (bundles) inside libjanet means we can more easily install packages without requiring a big, external library like spork or jpm.

tionis commented 6 months ago

I just tried it, and it seems to be working well for me. I do wonder however how updates should be handled as it currently refuses to override an existing module.

sogaiu commented 6 months ago

bakpakin wrote:

keeping this interface for installing packages (bundles) inside libjanet means we can more easily install packages without requiring a big, external library like spork or jpm.

This sounds good to me.

tionis wrote:

I do wonder however how updates should be handled as it currently refuses to override an existing module.

You mean this sort of thing?

repl:3:> (bundle/install "./spork")
error: bundle is already installed
  in bundle/install [boot.janet] on line 4064, column 5
  in _thunk [repl] (tailcall) on line 3, column 1

On a side note, bundle/uninstall and bundle/list seemed to work for me too.

bakpakin commented 6 months ago

Did some more work on the branch, and added bundle/reinstall and bundle/pack for updating bundles from local disk, as well as changed the way files are stored. Also taking some care to make sure that installation will properly rollback if there is an issue during the installation. None of the format are interface are final.

All that said, I can see functionality here growing and growing and growing, and at some point a line needs to be drawn. Not only should the functionality be simple but genuinely useful, the implementation needs to be fairly simple.

sogaiu commented 6 months ago

Did some more work on the branch, and added bundle/reinstall and bundle/pack for updating bundles from local disk, as well as changed the way files are stored.

Tried out briefly -- seemed to work here.

I can see functionality here growing and growing and growing, and at some point a line needs to be drawn. Not only should the functionality be simple but genuinely useful, the implementation needs to be fairly simple.

I agree about these endpoint characteristics -- wonder how to get there though (^^;

sogaiu commented 6 months ago

Tried it briefly on Windows.

Encountered at least one issue with bundle/install -- it looked path-related (sorry, I don't have the error output ATM).

bakpakin commented 6 months ago

I haven't tested any of this on windows but I wouldn't be surprised if there are path issues.

pyrmont commented 6 months ago

@bakpakin Is the idea that module authors will include hooks files in the module to run on build and install? Or is this just an aspect of the proof of concept that will have improved UX later on? I will say that I like the way modules all currently have a project.janet file and you have one place to look to understand a module.

bakpakin commented 6 months ago

This is all proof of concept, having each "hook" be a function inside a single "project" file in some ways makes more sense. I will likely change over to this way, and was playing with this locally. However, one of the issues we ran into with JPM was how to evaluate such files before dependencies were installed - separate files avoids that problem.

I'm hesitant to commit even to this whole idea since JPM does already work, but I do want to explore it.

bakpakin commented 6 months ago

@bakpakin Is the idea that module authors will include hooks files in the module to run on build and install? Or is this just an aspect of the proof of concept that will have improved UX later on? I will say that I like the way modules all currently have a project.janet file and you have one place to look to understand a module.

The idea was that authors would include their own hooks, but UX would certainly be improved. However, not sure the same level of "declarativeness" as is in JPM is likely.

pyrmont commented 6 months ago

@bakpakin I realise it's annoying when people compare Janet to Clojure and so at the risk of doing that in a way that turns you off completely, you could do something similar to what Clojure did when they introduced tools.deps: support a 'data only' file (deps.edn) that contains the information necessary for the built-in dependency installation tool.

Unlike Clojure, Janet's data-only file (project.jdn?) could also contain module metadata (similar to declare-project) as well as paths for build and install hook files.

Something like:

{
 :name "Foo"
 :description "The best module"
 :author "Janet Doe"
 :license "MIT"
 :url "https://git.example.org/foo"
 :repo "git+https://git.example.org/foo"
 :deps [“https://git.example.org/bar”]
 :src [“foo/a.janet”
       “foo/b.janet”]
 :build-file "./build.janet" # potentially this is only required if you want to use a different file from the default
 :install-file "./install.janet" # same as above
}

JPM could then be updated to look for this file if there is no project.janet file (or perhaps look for it first and then fall back to a project.janet). A 'data only' file like this could easily be evaluated without needing any other dependencies to be loaded first.

CFiggers commented 6 months ago

I like the idea of a project.jdn file. On top of project info, dependencies, and build recipes, it sounds pretty useful for tooling to tap into as well—if arbitrary metadata can be added to the project.jdn file as long as it doesn't collide with anybody else's stuff, then I can very easily use that as a canonical location to store small stateful per-project config options for CLI tools, Janet LSP, or editor plugins like Janet++.

If we went that route, we might want to settle on some clear conventions (maybe enforced by the Janet binary to one extent or other?) about how to include "third party" project metadata consistently.

bakpakin commented 6 months ago

I realise it's annoying when people compare Janet to Clojure and so at the risk of doing that in a way that turns you off completely, you could do something similar to what Clojure did when they introduced tools.deps: support a 'data only' file (deps.edn) that contains the information necessary for the built-in dependency installation tool.

Not at all, that is a good comparison and a valid point. The data only file would allow more universal processing of bundles, and is relatively low effort to implement. I don't expect we would do much processing of this data in the bundle/ module besides move it around uncorrupted - not even sure we want to have the capacity to fetch remote dependencies over a network.

Currently there really isn't even the concept of dependencies - the user would be responsible for providing them. There are a lot of ways to package and distribute software versions and dependencies that are not quite orthogonal, (npm/cargo/go style language package managers, system package managers, version control (git submodules, monorepos), containers, VMs, tarballs, disk images, linux file system conventions, etc. The longer I do this, the more I prefer version control for dependency management (sometimes called "vendoring") so I can get perfect control over code I am building/running, but people have different wants here. Frankly, git is way better at tracking versions than npm, cargo, etc. I realize this is not exactly the most mainstream opinion.

One thing though that might be feasible and useful though is a way to check all installed bundles and make sure that they are working with all other installed bundles, basically a "check-installation" hook where authors of packages could enforce any constraints they wanted on the environment, and raise a (nice) error if there is a problem. For example, if a software author chooses to use a dependency that releases using semver, incremental version numbers, date ranges, etc., then they could do something like:

# check if sporky installed with compatible version
(check-semver-for-installed-bundle "sporky" ">1.2.3,<1.2.8")
# check if sparky is installed with the correct git hash
(require-exact-git-hash "sparky" "abc1234")
# check if spot is installed and built within the specified dates
(require-build-date-range "spot" "2023-05-06" "2024-01-28")
# or use feature checking
(require-import-and-binding "spork/zig" zig-compile-and-link-v2)
# or only warn on mismatched bundle
(warn-exact-git-hash "spark" "abc1234" "Only git hash abc1234 has been tested and verified as dependency for this version")

# And finally run a simple smoke test
(import spot)
(spot/smoke-test)

inside a hook. However, trying to automatically "resolve" versioning conflicts and search for the best version of packages for compatibility is something that should be up to users. Or at least, outside the scope of bundle/.

pyrmont commented 6 months ago

I don't expect we would do much processing of this data in the bundle/ module besides move it around uncorrupted - not even sure we want to have the capacity to fetch remote dependencies over a network.

Even if you're using vendoring as a way to distribute code together with its dependencies, you still need some way to get the dependencies into the project tree in the first place and if you want to make that something you can do from the REPL, downloading remote dependencies is still an important piece of that, isn't it?

tionis commented 6 months ago

Well in the case of vendoring I would think that the dependencies would be managed using git-subtree, git-subrepo, git submodules or maybe just by extracting an archive and commiting it.

amano-kenji commented 6 months ago

Vendoring increases repository size. Even though rust statically links a binary, it doesn't force repositories to carry dependencies. Haskell allows a program to depend on functional packages installed onto the system or ~/.cabal.

I prefer functional packages that you can see in haskell, guix, and nix.

I think haskell dependency model is a role model for janet.

amano-kenji commented 6 months ago

Git submodules increase size of archives.

bakpakin commented 6 months ago

I don't expect we would do much processing of this data in the bundle/ module besides move it around uncorrupted - not even sure we want to have the capacity to fetch remote dependencies over a network.

Even if you're using vendoring as a way to distribute code together with its dependencies, you still need some way to get the dependencies into the project tree in the first place and if you want to make that something you can do from the REPL, downloading remote dependencies is still an important piece of that, isn't it?

Yes, but doing so is actually fairly simple. E.g. with git:

(defn bundle/install-git
   "Install from git"
   [repo-url name]
   (def tempdir (string (dyn *syspath*) "/" (os/cryptorand 16) ".git"))
   (os/mkdir tempdir)
   (defer (os/execute ["rm" "-rf" "--" tempdir] :px)
     (os/execute ["git" "clone" "--depth=1" repo-url tempdir] :px)
     (bundle/install tempdir name))
   name)

You could do similar with mercurial, curl + tar, svn, cvs, perforce, etc. However, the only lightweight way to embed into janet is to shell out, so I'm hesitant to include it in Janet itself.

bakpakin commented 6 months ago

Vendoring increases repoisitory size. Even though rust statically links a binary, it doesn't force repositories to carry dependencies. Haskell allows a program to depend on functional packages installed onto the system or ~/.cabal.

I prefer functional packages that you can see in haskell, guix, and nix.

I think haskell dependency model is a role model for janet.

So Haskell + Cabal seems to use something called PVP, which seems like a slightly different flavor of semantic versioning.

As for vendoring, it will not be the only way to do things, but I would encourage it and it needs to be something that could be officially well supported.

bundle/install doesn't really do much besides optionally build things like shared objects then copy a minimal set of files to (dyn syspath). Notably, we don't just blindly copy the entire repository into the syspath. You should also be able to delete the original source code, uninstall bundles, prevent collisions of installed files and bad states, and be resistant to messing up the syspath.

The idea being that dependency resolution and fetching sources/binaries over a network sits at a layer above this.

amano-kenji commented 6 months ago

I'm not particularly fond of PvP. I prefer semantic versioning for its simplicity.

llmII commented 6 months ago

@bakpakin wrote:

The idea being that dependency resolution and fetching sources/binaries over a network sits at a layer above this.

I agree regarding those should be at a separate layer.

I tend to think of package installation as a flow of steps:

  1. Fetch
  2. Extract (optional, sort of a tarball only thing)
  3. Build
  4. Install to path

Fetch should likely be something along the lines of a dispatcher that sees an URI and finds a URI handler that exists for it and calls it to execute a fetch. Build should probably have some nice "shortcuts" available (maybe in spork or something) but is entirely package defined. Extract probably needs a "what's that file end with" dispatcher.

I think having bundle/install or whatever the end name is handle steps 3 & 4 (Build, Install) makes great sense, and that perhaps in the future there would be a bundle/fetch with dispatching support as well as a bundle/extract that is ran when what is fetched happens to be an archive. There is also the issue of if the package decides to find the install directory and copy stuff over to that location manually. Protecting against malicious use of the bundle/install call is difficult, no idea how that'd be done aside from chroot or making sure the install directory is owned by a certain user that isn't what the Build step is ran under.

Dependency and conflict resolution is a difficult thing and it'd be nice if there was something like a bundle/process that read a file and did all the steps listed above for each dependency with each thing having the correct versions... recursively. Unwinding that in the event of failure is difficult unless the "Build" step happens in a "staged" directory, in which case it just deletes the directory.

Having Janet able to manage installing packages out of the box sounds like a pretty good idea, but once someone needs more than that, perhaps a library (spork, install-utils, whatever) to provide for managing such at a higher level, handling all the fetching, dependencies, conflicts, and so forth would be the best of both, especially if that library is self contained with no other dependencies other than Janet so it'd be as simple as fetching it manually and doing (bundle/install package).

The one issue I see is, if Janet has a bundle/uninstall and it doesn't do dependency and conflict resolution (as well as bundle/upgrade or similar) then it'll be quite easy to screw up dependencies or versions. The only thing I can think of regarding this is that all the bundle/* are "low-level" calls, and once someone starts using something like the bundle/process thing above I mentioned, then it's up to the user to remember to continue using such for all further tasks? That sounds pretty bad, maybe someone will have a better idea. In general, the idea of what goes on in a lower level and a higher level might need a lot more fleshing out.

On another note... something like bundle/installed? could be useful, a user could have a script that isn't really a package itself and do stuff like 

(when (not (bundle/installed? "package")) 
  (bundle/install package))

Having all this easily available from within Janet, REPL or Script or Project/Package, would likely feel more fluid and more easily approachable.

tionis commented 6 months ago

I think the main idea would be to have the base module handling (steps 3 & 4) in janet itself and keep the other parts in spork. So janet releases could be supplied with spork preinstalled, so that it can update itself over the network and also handle the other protocols you suggested. This would keep janet leaner and keep frequently changing code in spork.

sogaiu commented 6 months ago

I'm not too keen on all of spork being preinstalled with janet.

It now has a fair number of pieces by various authors at various levels of "completeness" / "stability".

I wonder if there is a way to get "just the necessary bits"...or perhaps there could be some other arrangement.

Edit 2: Perhaps a jpm-like tool could depend on code in spork but not cause spork to be installed as a library (so like what is possible now with declare-executable in jpm -- the tool could be a standalone binary).

So the tool could still end up as part of an installation of janet but it could remain unaffected for its basic functionality by what happens to live in janet's syspath. I hope I'm making sense here (^^;

Perhaps I didn't quite understand what was meant by "janet releases" though...

Edit 1: It occurs to me that there may be projects that depend on specific versions of spork. It seems possible that some of these might break if a newer version of spork gets installed when an upgrade to janet occurs. Not sure if this is a valid concern, but FWIW.

tionis commented 6 months ago

Hm, these are some valid concerns, I'm not sure what the best solution would be regarding those. Perhaps @bakpakin has some opinion on these problems?

llmII commented 6 months ago

I'm unsure if spork should be an automatically included "batteries" library that essentially achieves a status as a de-facto part of the Janet standard library.

It's starting to sound like package management shouldn't be part of spork, if one's goal is to in essence say "use latest version of package management and/or Janet". Which almost sounds like keeping a jpm of sorts somewhere though it might be different than the jpm of today, being that it relies on things built in to the Janet standard library.

@sogaiu wrote:

Edit 2: Perhaps a jpm-like tool could depend on code in spork but not cause spork to be installed as a library (so like what is possible now with declare-executable in jpm -- the tool could be a standalone binary).

So the tool could still end up as part of an installation of janet but it could remain unaffected for its basic functionality by what happens to live in janet's syspath. I hope I'm making sense here (^^;

This would be nice but then package management wouldn't be very REPL-ish aside from low level concerns of uninstall/remove/reinstall.

If it makes sense for package management to be done via REPL and for the higher level package management tool depend on spork then spork might need a split between "stable" and "evolving", that or the REPL higher level package management library should intern spork basically (privatize all the symbols or namespace them behind a prefix like pm/spork/*).

Having wrote all this it almost seems like we need many small libraries instead of a few conglomerates but I don't really like that idea either (that's Javascript-esque... there's a library for that for everything). So now we have a better and more difficult idea that might satisfy all the things in the next sections.

Versioned imports. If we have a directory structure something like:

$syspath/spork/@latest
$syspath/spork/@$version

Then we could let spork be installed by default, the higher level package management tool could pin to a version, a user could request the latest, and unversioned imports from within Janet default to the latest. This means one could do something like (import spork :version 'version) or whatnot.

With that idea, spork could be included as well as a higher level package management library for the REPL, without clashing with what users need. Whether or not it's a good idea is another question, and even if it's a good idea, it's questionable if it's worth the extra complexity outside of the fact that this saga we're currently participating in will not be the only case where such a need is likely to come up wherein one needs 2 versions of the same library to service tools and service user code.

bakpakin commented 6 months ago

I don't really plan on spork being released with Janet. If that was the goal, it would just be merged into Janet. However, spork contains a lot of the core functionality of JPM, just better.

On windows, it might make sense to have a spork setup option in the installer but ultimately it should be pretty easy to setup manually. Currently we have a Makefile target to try and setup JPM, so something like that might make it in.

However, the key idea is that you could have different versions of spork per project because it's just another pacakage in the tree

amano-kenji commented 6 months ago

Is this going to create breaking changes?

bakpakin commented 6 months ago

Is this going to create breaking changes?

No, this should be purely additive. JPM will still work for the foreseeable future, and won't require any changes to existing projects. Existing packages that use JPM will still work as well, and I see no reason why a project could not support both besides a bit of extra work. There are various levels of integration that we could go for here (should jpm installed packages be visible to bundle/install and bundle/uninstall), but the addition of plain functions won't mess things up here.

As per usual though, to avoid breaking changes, stick with release versions of Janet rather than latest master.

bakpakin commented 6 months ago

Versioned imports. If we have a directory structure something like: $syspath/spork/@latest $syspath/spork/@$version

Then we could let spork be installed by default, the higher level package management tool could pin to a version, a user could request the latest, and unversioned imports from within Janet default to the latest. This means one could do something like (import spork :version 'version) or whatnot.

So I'm open to ideas, but at first glance versioned imports seems like something I'm very much not going to implement. Rather, it would be a versioned install and the import would be something like

(bundle/install "/downloads/spork"  "spork-1.2.3") # install script dynamically chooses what root directory to install to based on bundle name

(import spork-1.2.3 :as spork)
(require (string "spork-" (calculate-needed-version)))

This looks awful to me in most cases, but its implementable via the usual mechanisms.

amano-kenji commented 6 months ago

Perhaps, can project.janet specify something like this?

(declare-project
  :dependencies ["spork >= 1.3 && < 2"])

This is the haskell way of specifying dependencies. In ~/.cabal, various versions of spork are available.

sogaiu commented 6 months ago

it would be a versioned install

IIUC, at least this could help mitigate the problem of installation of one project clobbering the dependencies of another.

As an example, with how things work with jpm currently, suppose:

Assume further that X < Y [1] and Y has changes that are not compatible with some functionality in X.

If someone has project A installed and then installs project B, project A may break because version X of spork has been replaced with version Y of spork.

With the proposed "versioned install" idea, if various projects make use of the mechanism, it seems like the aforementioned problem could be avoided (or at least lessened) -- at least by projects that make use of the mechanism.

[1] Strictly speaking, I don't think one needs any kind of version "number" ordering to express the problem mentioned above. IIUC, all that is necessary is that there are two different versions with some incompatibilities (e.g. imagine the case of two different commits from a particular repository where there are no version "numbers" associated with either commit).

As a side note, vendoring can also protect against the issue of disappearing dependencies (e.g. this recent PR). It looks like other communities have "immutable" stores that can address this.

May be that's not so important at this stage.

llmII commented 6 months ago

@bakpakin wrote:

So I'm open to ideas, but at first glance versioned imports seems like something I'm very much not going to implement. Rather, it would be a versioned install and the import would be something like

(bundle/install "/downloads/spork"  "spork-1.2.3") # install script dynamically chooses what root directory to install to based on bundle name

(import spork-1.2.3 :as spork)
(require (string "spork-" (calculate-needed-version)))

This looks awful to me in most cases, but its implementable via the usual mechanisms.

It does look awful, but versioned install solves the issue I had in mind. Someone could potentially "monkey-patch" import and friends to support "versioned import" to make it look nicer (pretend there's a canonical way as defined by the higher level package management library to version installs and someone's likely to have a little library to make versioned imports look nicer eventually).

You are pretty spot on that built-in low-level import and package management need not necessarily support versioning directly though.

@sogaiu wrote:

With the proposed "versioned install" idea, if various projects make use of the mechanism, it seems like the aforementioned problem could be avoided (or at least lessened) -- at least by projects that make use of the mechanism.

If various projects make use of it soon it'll be the canonical thing, and somewhere in there someone's going to make it look much nicer probably.

@sogaiu wrote:

As a side note, vendoring can also protect against the issue of disappearing dependencies (e.g. this recent PR). It looks like other communities have "immutable" stores that can address this.

May be that's not so important at this stage.

The immutable store thing sounds awesome, until going about the implementation thereof. Vendoring is a lot easier at than making an immutable store that supports $favorite-vcs, $archive, $something-phenomenally-ridiculous-but -somehow-necessary-to-support.

sogaiu commented 6 months ago

The immutable store thing sounds awesome,

I messed up a bit with terminology above -- probably I should have written something like (mostly) "append-only". A specific example of this might be maven central.

Git repositories on GitHub with the default branch being protected are somewhat close but IIUC these can just "go away" if a user removes the repository (or like in the recent xz incident, MS decides to just make something unavailable).

May be there was no confusion, but FWIW.

until going about the implementation thereof. Vendoring is a lot easier at than making an immutable store that supports $favorite-vcs, $archive, $something-phenomenally-ridiculous-but -somehow-necessary-to-support.

Yeah, probably it is not a simple matter, but I suppose it depends on the specifics a bit regarding how complicated things can get. Over time, depending on popularity, "append only" things seem like they could get costly though because the stored content just keeps growing...and growing (^^;

Edit: Sorry if this is getting a bit off-topic. May be it wouldn't be bad to make a list somewhere of related "concerns" (e.g. installed dependency clobbering, dependency disappearance, etc.) though. Possibly it would be helpful for present and future discussions?

bakpakin commented 6 months ago

Been doing some thinking about dependencies and bundle/install and bundle/uninstall, and I do think we should allow users to prevent breaking dependencies. Essentially, this is something that bundle/uninstall and bundle/install would check about what's currently installed before doing anything.

bundle/install would check that all required dependent files were present on the system before build and install, and bundle/uninstall would somehow check that no other bundles depend on any of its files. If either case fails, we also need to give a nice, instructive error message or result.

This does some what require that we distinguish between "incidentally" installed bundles and "directly" installed bundles.

(defn bundle/install
  ...
  (def deps (read-deps-data-jdn))
  (def missing (seq [d :in deps :unless (bundle/installed? d)] d))
  (when (next missing) (errorf "missing required dependencies: %s" (string/join missing ", ")))
  ...)

(defn bundle/uninstall
  ...
  (def deps (read-installed-deps-data-jdn))
  (def dependents (seq [d :in deps :if (bundle/installed-directly? d)] d))
  (when (next dependents) (errorf "cannot uninstall due to dependent bundles: %s" (string/join dependents ", ")))
  # uninstall my files, then uninstall dependent files
  (each d deps
    (if (= 1 (bundle-dep-ref-count d))
      (bundle/uninstall d)))

This works without needing to know anything about how to fetch dependencies or versions - the constraint is a function of what's in the syspath so we could definitely put it into the bundle/ module.

EDIT: The concept of directly installed bundles vs. incidentally installed bundles is basically "autoremove" in apt-get, or "orphaned packages" in pacman.

sogaiu commented 6 months ago

I've been trying to figure out how this:

bundle/install would check that all required dependent files were present on the system before build and install

would be done.

Is this a matter of checking existing manifest files?

bakpakin commented 6 months ago

I've been trying to figure out how this:

bundle/install would check that all required dependent files were present on the system before build and install

would be done.

Is this a matter of checking existing manifest files?

I was thinking so, just check that existing bundles are installed first. It could also include a manual "deps" hook where you could also search for system dependencies or even do feature testing a la autoconf.

sogaiu commented 6 months ago

Thanks for the clarification.

It could also include a manual "deps" hook where you could also search for system dependencies or even do feature testing a la autoconf.

That sounds quite interesting.

stutonk commented 5 months ago

As a user of Janet and someone who thinks a lot about package managers, I've got some opinions/feedback about this. Since some of what I have to say is critical, I would like to preface this with a statement of my gratitude for all the thought and work that's gone into Janet including that of its community members helping answer questions on the forums as well as publishing excellent-quality information about various specific Janet topics. Having used almost every parens language, this one is absolutely the best and easiest to use/understand except for one enormous papercut: the relation between and usage of janet, jpm and spork. As it stands, it's way more difficult than it should be to get all three of these things up and running. The standard operation of jpm also has some serious issues.

My apologies for a bit of an aside here about spork since this thread is supposed to be about jpm. These three pieces of software are treated as separate but, due to their entanglement and some of the functionality in spork, I would argue are all actually de facto a part of the "standard Janet distribution", if you will.

I'm not too keen on all of spork being preinstalled with janet.

I'm unsure if spork should be an automatically included "batteries" library that essentially achieves a status as a de-facto part of the Janet standard library.

I don't really plan on spork being released with Janet. If that was the goal, it would just be merged into Janet. However, spork contains a lot of the core functionality of JPM, just better.

I primarily use Arch and Alpine. Currently, Alpine packages the most recent release of Janet but without jpm. In order to get spork in an Alpine container you have to install git and an entire C toolchain, build jpm and install it to your root tree not-under-package-management, and then use it to install spork. Arch does not package janet in its official repos. In the Arch AUR, there are several incompatible packages that either do or do not include jpm and thus some attempts to individually package spork will fail to build because of cross-confusion between their respective maintainers about the relationship between janet, jpm, and spork.

Just look how many times spork is mentioned in this thread. Spork documentation is included in the standard library documentation. Some of the functionality (such as netrepl) is critical editor integration tooling. And spork depends on jpm for delivery. Since some of the talk here is about functionality varying from commit to commit in spork. If nothing else changes, spork needs releases. That way it can be packaged by external packagers and installed without jpm or a C compiler as well as having better-defined information about what's in what version. But my opinion is that it should officially be a part of the stdlib (or have parts of it moved into the stdlib) and distributed with Janet or else have its documentation moved elsewhere. Having confusion that creates friction getting this stuff up and running is likely turning people away from the language. It's certainly frustrating to me.

In addition to what's mentioned above about hidden state via env variables creating problems, my primary issue with jpm is that the default workflow is to use superuser privileges to install packages to my root file tree. For me this absolutely non-negotiably does not fly (hence running it in a container) nor should it for anyone. Even in the absence of malicious intent, the potential for a critical mistake somewhere is enormous. Installing and uninstalling jpm packages will also not necessarily clean up all the files, allowing pollution to accumulate in the root tree. The other two installation options are not much better. They are to either specify some environment variables (which the original go package manager proved to be extremely brittle) or to use jpm -l which combines node_modules-style pollution with python-style environments. None of these three options are necessary. On Linux (and potentially other Unix-like systems), there's a (somewhat) well-defined list of standard locations to install things that don't require elevated permissions such as XDG_CACHE_HOME, $HOME/.cache, etc. Even ~/.janet, while not ideal, would be better. On Windows I believe there are also user-specific data/cache directories. I believe that some of the ideas presented here already such as the bundle family of functions have the potential to address some of these concerns but I have some additional ideas on how that might be made better.

At the top level, the interface to the whole thing could be along the lines of 

(import-dep '{:git-url "https://github.com/janet-lang/spork :commit "ec40c7a"} :as spork)

Under the hood it would, upon first run/build use perhaps bundle to fetch, build in a tmpdir, and install the dependency to wherever in the user's tree. The specification would just be some struct whose content would vary depending on the specifics. For example, you could specify a :tag for git instead of a commit. The source doesn't have to be git. And the user would potentially be able to supply their own. With a central package repo it could look like (import-dep lib@version :as lib). Subsequent runs would just resolve to something like (import @the_user_install_dir/spork-ec40c7a :as spork). Specifying deps this ways has numerous advantages:

As for building and installing, having a just-data package.jdn or whatever with some metadata in a struct (as mentioned above by pyrmont) to be possibly consumed by a central package repo along with a build.janet which ultimately just a script that can perhaps leverage some standard tooling seems sensible. The build.janet would be able to use whatever specific tools are necessary to build the project such as the full on C compiler apparatus for native modules or just a list of which files to install in the simple case of a pure Janet module. The build.janet itself could even have it's own import-deps.

I realize that some or all of these ideas may not be that great. Having deps build/install lazily and recursively on first run/build might not be ideal. Maybe it's too different of a workflow for existing projects to want to adopt. The functionality of jpm is currently covering two separate concerns (building janet projects and getting other people's code) which may be better left separate. However, Janet is still young enough to do something better without an enormous amount of friction. Many other languages have failed miserably at this admittedly extremely difficult problem and are locked in with their pile of crap. I implore you to please not reproduce the faults of luarocks, pip, gem, npm, or go's first package managers. For lack of better ideas, just copy cargo. Cargo only needs one metadata file per crate and doesn't clobber differently-versioned deps. It needs no env variables, virtual environments, per package dependency folders, or superuser privileges and has such sane defaults that it rarely requires configuration. It uses lock files and requires packages to be versioned so that builds are reproducible.

If not any of that, a robust and modular build system that's part of the stdlib like zig that lets distro packagers or CI easily package/build releases and so that vendoring is viable (which seems where this thread is tending) would be most welcome.

Once again, thank you all for all the hard work, excellent programming language, and effort around its community.

bakpakin commented 5 months ago

@stutonk, agree with most of what is said here. It's not that well documented yet, but the bundle-tools branch has most of what you asked for already. Unlike JPM, packages are basically a set of plain Janet scripts + some extra static metadata for dependencies and whatever else you want, with not requirement on a C toolchain or multiple paths. There is also no requirement for any external tools or even networking - it operates simply on local paths.

Show and tell: The bundle-tools branch adds the following functions to Janet core:

For managing your local bundles (packages):

For inspecting current bundles:

For building a manifest:

There is also example usage of the bundle/* functions in test/suite-bundle.janet as well as in https://github.com/janet-lang/spork/tree/bundle-hooks

Lately I've been working more on the compile-opt branch, but I think I will likely merge the bundle-tools branch to master soon, as I don't know if there is much more to take away that wouldn't really hinder usability.

bakpakin commented 5 months ago

My apologies for a bit of an aside here about spork since this thread is supposed to be about jpm. These three pieces of software are treated as separate but, due to their entanglement and some of the functionality in spork, I would argue are all actually de facto a part of the "standard Janet distribution", if you will.

That would be nice, but I think that is the step after integrating JPM, as spork has C dependencies and I want it to be able to be built and developed independently of Janet. The bundle-hooks branch of spork can build itself without JPM. Currently, we don't distribute anything with Janet that isn't baked into the binary.

Originally jpm and spork were all separate so I didn't need to make so many frequent releases, but that is less of an issue now.

pyrmont commented 5 months ago

I thought there might be some benefit in summarising where things are. It looks like the following:

  1. In the near future, releases of Janet will include a collection of functions with the bundle/ 'namespace' that allows for the installation and uninstallation of bundles of code (including transitive dependencies) that are located on the local filesystem.

  2. In theory, users will be able to write code in their own projects to relatively easily bring in dependencies without relying on JPM. If these dependencies in turn support installation using the bundle/ functions, this would mean JPM is not required even for code that requires compilation. (To be clear, this does require the dependency authors to update their code to support this kind of installation.)

Additionally:

  1. JPM is not presently being changed to use the bundle/ functions. This could happen later but is not part of what @bakpakin is working on right now.

  2. The default module loading functions in Janet are not presently being changed. This means that if a user wants to do something like use different versions of the same module, they could do this but would have to do something like use different directory names for different versions (which they can already do).

This issue, as originally opened by @tionis, read to me as being about whether JPM should be integrated into Janet. The resolution of the above is that JPM won't be integrated into Janet. Rather, a collection of functions will be built-in to core. These functions could be used by users to install dependencies on their own (thereby avoiding any use of JPM) or to create their own package manager (thereby obviating JPM entirely).

noncom commented 1 month ago

Bump.

What is the current status of this? I'm looking into organizing my workflow, and maybe automating some of the builds and project mainenance, and therefore it would be good to know if I should base it on the jpm, or if there's going to be some redesigned tooling soon.