Closed amano-kenji closed 2 months ago
sogaiu
commented
2 months ago Please see this discussion for another method that was considered.
Update: Here something that points out the specific idea (though it's just an extract of the discussion mentioned).
BTW, thanks for posting the link.
amano-kenji
commented
2 months ago I just read jpm man page. If JANET_PATH took a (colon-separated) list of paths, I think nix and guix would be able to take jpm modules from various packages.
But, if this change takes place, JANET_MODPATH won't be able to default to JANET_PATH.
sogaiu
commented
2 months ago I don't know why no one seems to be commenting on the proposal in this discussion:
For this use case, I don't think setting JANET_PATH would be sufficient since it only contains one directory. I think what I need is a search path that supports a colon separated list of directories where modules can be found. In order to get that, I'm thinking of adding a Guix specific patch to Janet in our package that add such a search path via a GUIX_JANET_PATH environment variable. We've used similar strategies with other software before to good effect.
AFAICT, the idea there was proposed by someone who had knowledge of both guix and nix. The quote above even has a link to a case where a similar approach was tried with alleged success.
There was code discussed and both andrewchambers [1] and @bakpakin reviewed the idea and seemed to think it could work:
andrewchambers:
Looks pretty good to me in that case. Another approach might be to make a symlink farm to create a virtual JANET_PATH.
I think the 'Nix' way would be to have a nix function that takes in a list of janet packages and returns a janet with a wrapper that sets JANET_PATH to these paths.
bakpakin:
The proposed patch looks pretty good to me. However, ...
(elided modification suggestion that appeared to be accepted by proposer)
I think it would be better for people to consider that idea first (or the additional alternative -- see [1] below), before suggesting a change to janet like this.
IMO, there ought to be clearly spelled out reasons why the earlier proposal (or its alternative) won't / doesn't work, before asking for changes on janet's side.
[1] andrewchambers also suggested another approach which AFAIU does not involve modification to janet.
amano-kenji
commented
2 months ago I just ran a thought experiment in my head. Having a long colon-separated list of paths in JANET_PATH will be slow because every module will go through $JANET_PATH once.
For transitive dependencies, transitive symlinks are natural. Transitive symlinks would be still faster than a long list of search paths. However, if guix just copies symlinks intead of making symlinks to symlinks to symlinks to symlinks, we can replace transitive symlinks with direct symlinks. Direct symlinks are the fastest as long as they point to the right paths.
I don't know whether symlinks are problematic in practice. One potential problem of direct symlink farms is disk usage.
amano-kenji
commented
2 months ago Adam Faiz wrote
Search paths are preferred because their purpose is to provide a list of places to look for extensions/plugins/mods of a package. Essentially, search paths are a way to provide runtime dependencies that are optional.
Packages usually link directly to their dependencies at compile time, except for scripting programming languages(which aren't expected to have linking functionality). Propagated inputs are used for the transitive dependencies you mention, but they should only be used for required dependencies that have to be installed with the package for it to detect them.
Packages shouldn't need to have symlinks in their output to another package when it could just directly reference it instead. More information on how native-inputs, inputs and propagated-inputs are different is explained in the Guix manual: https://guix.gnu.org/manual/devel/en/html_node/package-Reference.html#index-inputs_002c-of-packages
amano-kenji
commented
2 months ago IMO, there ought to be clearly spelled out reasons why the earlier proposal (or its alternative) won't / doesn't work, before asking for changes on janet's side.
What proposal? You need to be clear. List them out in clear language.
sogaiu
commented
2 months ago Did you digest the content of this comment?
bakpakin
commented
2 months ago I just ran a thought experiment in my head. Having a long colon-separated list of paths in JANET_PATH will be slow because every module will go through $JANET_PATH once.
I have a local change that actually does this, but there are some caveats. Mostly a little extra complexity, but I don't think the performance cost is really that much of an issue. It does mean that, if you use a separate path for N modules, loading all modules spends O(N^2) checking for files, but on modern systems that isn't that slow anyway. My main concern is that we have a concept of a single (dyn *syspath*), and that is pretty useful for package management, searching for things, etc.
However, if guix just copies symlinks intead of making symlinks to symlinks to symlinks to symlinks, we can replace transitive symlinks with direct symlinks. Direct symlinks are the fastest as long as they point to the right paths.
I think this is probably the best approach. The module code use stat, not lstat to check for existence so this will work fine.
That said, for multiple paths, here is my local diff. The first path in a colon separated JANET_PATH is the syspath, while subsequent paths are manually added to module/paths.
diff --git a/src/boot/boot.janet b/src/boot/boot.janet
index d1aab5d8..d87aad3a 100644
--- a/src/boot/boot.janet
+++ b/src/boot/boot.janet
@@ -2827,6 +2827,23 @@
(array/insert mp curall-index [(string ":cur:/:all:" ext) loader check-relative])
mp)
+(defn module/add-syspath
+ ```
+ Add a custom syspath to `module/paths` by duplicating all entries that being with `:sys:` and
+ adding duplicates with a specific path prefix instead.
+ ```
+ [path]
+ (def copies @[])
+ (var last-index 0)
+ (def mp (dyn *module-paths* module/paths))
+ (eachp [index entry] mp
+ (def pattern (first entry))
+ (when (and (string? pattern) (string/has-prefix? ":sys:/" pattern))
+ (set last-index index)
+ (array/push copies [(string/replace ":sys:" path pattern) ;(drop 1 entry)])))
+ (array/insert mp (+ 1 last-index) ;copies)
+ mp)
+
(module/add-paths ":native:" :native)
(module/add-paths "/init.janet" :source)
(module/add-paths ".janet" :source)
@@ -4488,7 +4505,11 @@
(var error-level nil)
(var expect-image false)
- (if-let [jp (getenv-alias "JANET_PATH")] (setdyn *syspath* jp))
+ (when-let [jp (getenv-alias "JANET_PATH")]
+ (def paths (string/split ":" jp))
+ (for i 1 (length paths)
+ (module/add-syspath (get paths i)))
+ (setdyn *syspath* (first paths)))
(if-let [jprofile (getenv-alias "JANET_PROFILE")] (setdyn *profilepath* jprofile))
(set colorize (and
(not (getenv-alias "NO_COLOR"))Did you digest the content of https://github.com/janet-lang/janet/issues/1505#issuecomment-2356013382?
I tried and failed to understand it because it reads like an IQ puzzle.
As far as I know, there are three options.
If you know a better option, write it here. My brain is not optimized to solve IQ puzzles especially when I'm sleepy. If I really want, I can solve IQ puzzles, but I'm too tired. My english tends to be brutally direct because indirection increases cognitive burden, english is my second language, and my brain is not optimized.
amano-kenji
commented
2 months ago Somehow, guix seems to prefer search paths to symlinks.
https://guix.gnu.org/manual/devel/en/guix.html#Search-Paths
But, I still don't understand why they use search paths instead of symlinks.
amano-kenji
commented
2 months ago My current conclusion is that janet build system for guix can either
There is no real fundamental obstacle that prevents either option in gnu guix. Which option is better?
amano-kenji
commented
2 months ago I think that even if I try to use direct symlinks as much as possible, it actually makes sense for janet to support search paths.
When I compile a janet executable out of direct dependencies that contain symlinks to (transitive) dependencies, it doesn't make sense for the janet executable guix package to contain symlinks to all files in its direct dependencies because the janet executable already contains all the modules.
Search paths make sense for building compiled executables.
Another scenario where search paths make sense is guix shell. In a guix shell environment, I specify a list of direct janet dependencies I want to use. Guix shell doesn't construct a directory that has symlinks to all direct dependencies.
I guess things will become complex without search paths on purely functional linux distros.
sogaiu
commented
2 months ago @amano-kenji
I tried and failed to understand it
If you didn't find the summary to be palatable, I suggest you read the original discussion. Other folks seemed to be capable of following the content, I doubt that you cannot manage.
There are even patches there for one of the approaches you can cross-check your understanding against.
Two potential drawbacks to changing Janet include:
(dyn :syspath)If it's possible for Janet to not change, I think it shows more respect to the existing Janet users and code.
Guix and Nix are neat systems, but it was their choice to change how they do things at a fundamental level (with the store idea, patching elf binaries, immutability, etc.), I think for the most part it should mostly be up to them to adjust to existing software. Why should end software change to accomodate them if that could be a problem to existing users?
In the discussion of the mostly discussed proposal, there was even a link given to a similar case in Guix where a similar approach was taken, allegedly with success:
We've used similar strategies with other software before to good effect.
That seems to suggest that it is technically possible for Guix (and I would guess Nix as well) to make a suitable adjustment.
amano-kenji
commented
2 months ago If you didn't find the summary to be palatable, I suggest you read the https://github.com/janet-lang/janet/discussions/695. Other folks seemed to be capable of following the content, I doubt that you cannot manage.
As far as I know, only two possibilities are known.
The third posssibility, symlink, has been knocked out by guix shell and nix shell and jpm executables.
amano-kenji
commented
2 months ago With a single path there is no issue of there being multiple things with the same "name" being available because this doesn't happen.
Not having multiple paths in janet is not an option now. Either, guix adds its own patch, or janet internally supports multiple paths. Package maintainers will have to manually prevent file conflicts unless janet starts supporting importing different versions of jpm packages.
sogaiu
commented
2 months ago Just because we do not imagine scenarios that could cause breakage / problems, it does not imply such things do not exist. It may be more that we lack imagination.
What one sees clearly in one's mind is just one view. Clarity does not equal correctness.
If Guix adds a patch to make things work for Guix, then upstream Janet does not change and still uses the original (dyn :syspath) mechanism. For upstream Janet in this situation, it would count as not having multiple paths. So, it is an option for upstream Janet.
Existing users and code does not incur additional risk of breakage:
Unforeseen consequences to existing Janet code people have written that may be relying on a single (dyn :syspath)
nor potential maintenance / debugging issues:
When there is a set of paths, multiple things can exist with the same "name" -- they just reside at different directories on the path. As I understand it, this can lead to complications when uninstalling, updating, debugging, etc. Why should existing and future users shoulder these additional burdens?
amano-kenji
commented
2 months ago Even if guix adds its own patch to janet, unforseen issues caused by multiple paths will probably need to be fixed in upstream. So, it's better for janet to support multiple paths in JANET_PATH in a way that doesn't break existing users.
Existing users aren't forced to use multiple paths in JANET_PATH just because JANET_PATH supports multiple paths.
sogaiu
commented
2 months ago None of that addresses:
When there is a set of paths, multiple things can exist with the same "name" -- they just reside at different directories on the path. As I understand it, this can lead to complications when uninstalling, updating, debugging, etc. Why should existing and future users shoulder these additional burdens?
As mentioned multiple times above, there may be code that people have written that depends on there being a single (dyn :syspath). We just don't know.
amano-kenji
commented
2 months ago Change can lead to bugs. That's unavoidable. So, change nothing because there can be bugs?
When there is a set of paths, multiple things can exist with the same "name" -- they just reside at different directories on the path.
Jpm itself doesn't account for file conflicts. So, people will have to manually manage file conflicts anyway with or without nix and guix. With or without nix and guix, package maintainers are forced to manually manage file conflicts.
bakpakin
commented
2 months ago So for the meantime, (dyn :syspath) is definitely going to remain a single string (one syspath). That said, allowing multiple path strings seems common enough and I can add support for it - currently working on updating docs. There are some other details to get right like what path separators should we accept on windows? mingw? and should we expose the helper function that sets all this up.
But not really opposed to the idea in general, but I would probably not use it. Module import failures could get quite verbose...
EDIT: Go also set some precedent here with GOPATH - it has supported multiple paths for a long time, but certain tools and packages would designate special significance to the first one, i.e. when installing new packages.
sogaiu
commented
2 months ago So for the meantime, (dyn :syspath) is definitely going to remain a single string (one syspath). That said, allowing multiple path strings seems common enough and I can add support for it - currently working on updating docs.
So does this mean existing code that relies on (dyn :syspath) being a single path must now doing splitting?
bakpakin
commented
2 months ago So does this mean existing code that relies on (dyn :syspath) being a single path must now doing splitting?
Nope! That wouldn't change - only parsing JANET_PATH on startup and adding some extra stuff to module/paths if there are subsequent paths
bakpakin
commented
2 months ago Example:
$ JANET_PATH=/home/calvin/a:/home/calvin/b janet
Janet 1.37.0-dev-88e60c30 linux/x64/gcc - '(doc)' for help
repl:1:> (setdyn *pretty-format* "%.99P")
"%.99P"
repl:2:> (dyn *syspath*)
"/home/calvin/a"
repl:3:> module/paths
@[(<function is-cached> :preload <function check-not-relative>)
(":cur:/:all:.jimage" :image <function check-relative>)
(":cur:/:all:.janet" :source <function check-relative>)
(":cur:/:all:/init.janet" :source <function check-relative>)
(":cur:/:all::native:" :native <function check-relative>)
(":sys:/:all:.jimage" :image <function check-is-dep>)
(":sys:/:all:.janet" :source <function check-is-dep>)
(":sys:/:all:/init.janet" :source <function check-is-dep>)
(":sys:/:all::native:" :native <function check-is-dep>)
("/home/calvin/b/:all:.jimage" :image <function check-is-dep>)
("/home/calvin/b/:all:.janet" :source <function check-is-dep>)
("/home/calvin/b/:all:/init.janet" :source <function check-is-dep>)
("/home/calvin/b/:all::native:" :native <function check-is-dep>)
(".:all:.jimage" :image <function check-project-relative>)
(".:all:.janet" :source <function check-project-relative>)
(".:all:/init.janet" :source <function check-project-relative>)
(".:all::native:" :native <function check-project-relative>)
(":@all:.jimage" :image <function check-dyn-relative>)
(":@all:.janet" :source <function check-dyn-relative>)
(":@all:/init.janet" :source <function check-dyn-relative>)
(":@all::native:" :native <function check-dyn-relative>)]
repl:4:> Thanks for the clarification and example.
On the surface it seems similar to what you mentioned about Go:
Go ... has supported multiple paths for a long time, but certain tools and packages would designate special significance to the first one, i.e. when installing new packages.
sogaiu
commented
2 months ago What about if some code that is installed to a directory that is not the first one on JANET_PATH [1] wants to access non-code resources [2]?
Currently, I think this can be done by constructing a path via (dyn :syspath), but once the proposed change is made, it seems that one will need to process JANET_PATH and search through the contained directories.
[1] So not (dyn :syspath), but listed among the other ones in JANET_PATH.
[2] So not using import and friends.
amano-kenji
commented
2 months ago janet hasn't been designed to offer resource files.
sogaiu
commented
2 months ago It is used in the wild.
See here for example.
Thanks to @pyrmont for finding this one.
Update: similar sort of thing in the same project.
sogaiu
commented
2 months ago I think here is an example where (dyn :syspath) may get set from JANET_PATH.
If JANET_PATH can be a colon/semicolon-separated list of paths, perhaps this code ought to get updated to accomodate.
Probably @subsetpark would know better though.
amano-kenji
commented
2 months ago I think janet can provide things like
But, this is kind of inefficient if people use module/find-raw to find every little file instead of one directory that contains all the relevant files.
These things are band-aids. They work, but they are not efficient. I think purely functional packages should account for efficiency in seeking resources, and language designers should account for purely functional packages in some ways.
amano-kenji
commented
2 months ago I wish there was something like
JANET_JPM_PATH={
"j3blocks" "/gnu/store/xxxxxx-j3blocks-1.1.0/lib/janet"
"j3blocks-extra" "/gnu/store/xxxxx-j3blocks-extra-1.1.0/lib/janet"
...
}This is a struct.
Then, import can look like
(import "j3blocks" "j3blocks/module-1")
(import "j3blocks" "j3blocks/module-2")
(import "j3blocks-extra" "j3blocks-extra/module-1")
(import "j3blocks-extra" "j3blocks-extra/module-2")module/find-raw can look like
(module/find-raw "j3blocks" "path/to/config.txt")
(module/find-raw "j3blocks-extra" "path/to/bluetooth.txt")These statements are extremely efficient because they are based on hash map. These work with purely functional packages. Even if your janet script or application requires 200,000 jpm modules, import and module/find-raw won't be slow at all.
bakpakin
commented
2 months ago @amano-kenji The above can be done by adding custom module/path entries. I would suggest looking at the default first entry in the module/paths array that allows "preloading" modules - something like that could do exactly what you have above, or you could use preloading as it is. Perhaps a patch to boot.janet is best, or maybe Nix/Guix could use JANET_PROFILE to set up extra paths for imports.
Really though, this is a Nix/Guix problem and not something we can address for all operating systems, distributions, and OS-like software. There is Nix, Guix, Fedora SilverBlue, NixOS, flatpak, appimage, snaps, etc. that all have their own arbitrary dogmatic way of arranging the filesystem to try and isolate packages. There are many (probably too many) ways to make this work in Janet using some scripting, and it's probably not a good idea to try and build them all into the binary.
bakpakin
commented
2 months ago janet hasn't been designed to offer resource files.
Yes it has. Import can be extended via module/loaders to load any kind of file.
bakpakin
commented
2 months ago What about if some code that is installed to a directory that is not the first one on JANET_PATH [1] wants to access non-code resources [2]?
Yeah, this is admittedly an issue. You can work around this with custom module/loaders, which is the intended way to do this, but that is definitely a headache compared to slurp.
At this point, I'm still on the fence on this feature and given the examples from Joy, maybe we shouldn't add it to default Janet.
amano-kenji
commented
2 months ago As far as I know, fedora silverblue, flatpak, snap, and appimage create container environments which should work fine with janet as it is now.
nix and guix will be supported if there is something like JANET_EXTRA_PATHS. If janet supports multiple module paths, gobo linux will also be supported although gobo linux doesn't have purely functional packages. Gobo linux has /Program/program-name/version/... for each package version.
So, the only divide you need to care about is between one module path and multiple module paths.
sogaiu
commented
2 months ago I wasn't sure whether the following would be affected, but I noticed these lines in spork/netrepl:
(def syspath (dyn :syspath))
(net/server
host port
(fn repl-handler [stream]
# Setup closures and state
(setdyn :syspath syspath)Is the (setdyn :syspath syspath) necessary because the net/server handler starts with a "blank slate"?
Would module/paths already be appropriately set like in your example?
May be in this case everything works fine.
sogaiu
commented
2 months ago I doubt there is much of the following sort of thing, but I also came across these lines in a project.janet:
(phony "install" []
(copy "globals.janet" (os/getenv "JANET_PATH"))
)Probably that kind of thing is pretty easy to fix though?
amano-kenji
commented
2 months ago What is phony install? That's not documented by jpm?
bakpakin
commented
2 months ago I doubt there is much of the following sort of thing, but I also came across these lines in a
project.janet:(phony "install" [] (copy "globals.janet" (os/getenv "JANET_PATH")) )Probably that kind of thing is pretty easy to fix though?
Yeah that's just not a good thing to do - JANET_PATH doesn't even need to be set.
sogaiu
commented
2 months ago Re: phony - I see some docs here:
run rule
Run a rule. Can also run custom rules added via `(phony "task"
[deps...] ...)` or `(rule "ouput.file" [deps...] ...)`.Alternatively, one may get a sense by jpm | grep phony.
bakpakin
commented
2 months ago phony is just an alias for a task in jpm, I don't encourage it since the name is strange.
amano-kenji
commented
2 months ago Import can be extended via module/loaders to load any kind of file.
It's hard to imagine how import can be used to read a text file. I'm not good at make things do circus tricks.
sogaiu
commented
2 months ago Perhaps this is the sort of thing that was meant:
(defn add-loader
"Adds the custom template loader to Janet's module/loaders."
[]
(put module/loaders :mendoza-template template-loader)
(array/insert module/paths 0 ["./templates/:all:" :mendoza-template ".html"])
(array/insert module/paths 1 ["./mendoza/templates/:all:" :mendoza-template ".html"])
(array/insert module/paths 2 [":sys:/mendoza/templates/:all:" :mendoza-template ".html"])
(array/insert module/paths 3 ["./templates/:all:" :mendoza-template ".tmpl"])
(array/insert module/paths 4 ["./mendoza/templates/:all:" :mendoza-template ".tmpl"])
(array/insert module/paths 5 [":sys:/mendoza/templates/:all:" :mendoza-template ".tmpl"])
(array/insert module/paths 0 ["./templates/:all:" :mendoza-template ".xml"])
(array/insert module/paths 1 ["./mendoza/templates/:all:" :mendoza-template ".xml"])
(array/insert module/paths 2 [":sys:/mendoza/templates/:all:" :mendoza-template ".xml"]))Here is a simple example.
# Add a loader for text files - puts the contents of the file in a binding 'text.
(defn load-txt [path &] (def module @{}) (put module 'text @{:value (slurp path)}) module)
(put module/loaders :text load-txt)
# Add paths on how to search for text files
(module/add-paths ".txt" :text)
# Now assume we have a file resource.txt with the contents "Tadah!"
(import ./resource)
(print resource/text) # prints "Tadah!"Okay. I can see now.
Here's my conclusion.
import can load joy/init.janet when it tries to import the path to joy, I suggest a separate function named import-path that can import the path to a file or a directory. Sometimes, people may want to import the path to a directory.Summary
import-path that can import the path to a file or a directory since import cannot know whether somebody wants to import dir/init.janet or the path to dir.import-path instead of syspath.amano-kenji
commented
2 months ago It seems import-path should be a separate issue.
On nix and guix, each package lives in its own tree.
For example,
Without multiple paths in janet's environment variables, janet-j3blocks would have many symlinks to jpm, and janet-j3blocks-extra would have many symlinks to janet-j3blocks which in turn has many symlinks to jpm.
Practical Guix: Packaging Janet's jpm (Part 2) shows the difficulty of making jpm packages for nix and guix.