Closed tionis closed 1 year ago
I also ran into this bug.
I looked into this a little bit.
For jeff's case [1], jpm --verbose build
shows:
$ jpm --verbose build
cc -c fzy.c -DJANET_BUILD_TYPE=release -std=c99 -I/home/user/.local/include/janet -I/home/user/.local/lib/janet -O2 -fPIC -o build/fzy.o
generating meta file build/fzy.meta.janet...
generating executable c source build/jeff.c from jeff/cli.janet...
error: error: could not find module ../build/fzy:
build/fzy.jimage
build/fzy.janet
build/fzy/init.janet
build/fzy.so
in require-1 [boot.janet] on line 2934, column 20
in import* [boot.janet] on line 2973, column 15
in _thunk [jeff/scorer.janet] (tailcall) on line 1, column 1
in dofile [boot.janet] (tailcall) on line 2911, column 7
in source-loader [boot.janet] on line 2922, column 15
in require-1 [boot.janet] on line 2942, column 18
in import* [boot.janet] (tailcall) on line 2973, column 15
in dofile [boot.janet] (tailcall) on line 2911, column 7
in source-loader [boot.janet] on line 2922, column 15
in require-1 [boot.janet] on line 2942, column 18
in import* [boot.janet] (tailcall) on line 2973, column 15
in dofile [boot.janet] on line 2911, column 7
in <anonymous> [/home/user/.local/lib/janet/jpm/cc.janet] on line 293, column 24
in <anonymous> [/home/user/.local/lib/janet/jpm/rules.janet] on line 20, column 22
So at this stage there is no fzy.so
in the build
directory. But creating the executable is attempted and I guess as part of that, the import form for fzy in this file doesn't work out.
Perhaps the order of execution is not quite right?
For reference, a subsequent invocation of jpm --verbose build
does produce fzy.so
(not sure why):
$ jpm --verbose build
cc -c fzy.c -DJANET_BUILD_TYPE=release -DJANET_ENTRY_NAME=janet_module_entry_fzy -std=c99 -I/home/user/.local/include/janet -I/home/user/.local/lib/janet -O2 -o build/fzy.static.o
generating executable c source build/jeff.c from jeff/cli.janet...
error: error: could not find module fzy:
/home/user/.local/lib/janet/fzy.jimage
/home/user/.local/lib/janet/fzy.janet
/home/user/.local/lib/janet/fzy/init.janet
/home/user/.local/lib/janet/fzy.so
in require-1 [boot.janet] on line 2934, column 20
in import* [boot.janet] on line 2973, column 15
in _thunk [jeff/scorer.janet] (tailcall) on line 1, column 1
in dofile [boot.janet] (tailcall) on line 2911, column 7
in source-loader [boot.janet] on line 2922, column 15
in require-1 [boot.janet] on line 2942, column 18
in import* [boot.janet] (tailcall) on line 2973, column 15
in dofile [boot.janet] (tailcall) on line 2911, column 7
in source-loader [boot.janet] on line 2922, column 15
in require-1 [boot.janet] on line 2942, column 18
in import* [boot.janet] (tailcall) on line 2973, column 15
in dofile [boot.janet] on line 2911, column 7
in <anonymous> [/home/user/.local/lib/janet/jpm/cc.janet] on line 293, column 24
in <anonymous> [/home/user/.local/lib/janet/jpm/rules.janet] on line 20, column 22
cc -std=c99 -I/home/user/.local/include/janet -I/home/user/.local/lib/janet -O2 -o build/fzy.so build/fzy.o -shared -lpthread
However, although there is now a fzy.so
, a further jpm --verbose build
doesn't succeed:
$ jpm --verbose build
generating executable c source build/jeff.c from jeff/cli.janet...
error: error: could not find module fzy:
/home/user/.local/lib/janet/fzy.jimage
/home/user/.local/lib/janet/fzy.janet
/home/user/.local/lib/janet/fzy/init.janet
/home/user/.local/lib/janet/fzy.so
in require-1 [boot.janet] on line 2934, column 20
in import* [boot.janet] on line 2973, column 15
in _thunk [jeff/scorer.janet] (tailcall) on line 1, column 1
in dofile [boot.janet] (tailcall) on line 2911, column 7
in source-loader [boot.janet] on line 2922, column 15
in require-1 [boot.janet] on line 2942, column 18
in import* [boot.janet] (tailcall) on line 2973, column 15
in dofile [boot.janet] (tailcall) on line 2911, column 7
in source-loader [boot.janet] on line 2922, column 15
in require-1 [boot.janet] on line 2942, column 18
in import* [boot.janet] (tailcall) on line 2973, column 15
in dofile [boot.janet] on line 2911, column 7
in <anonymous> [/home/user/.local/lib/janet/jpm/cc.janet] on line 293, column 24
in <anonymous> [/home/user/.local/lib/janet/jpm/rules.janet] on line 20, column 22
I tried modifying the import
form like: (import ../build/fzy :prefix "" :export true)
and then invoked jpm --verbose build
again. This time it did succeed in building jeff
(though if one tries after jpm clean
, it still seems necessary to run jpm build
3 times). Not really a work-around, but just reporting.
[1] Modified the project.janet
file to re-enable the declare-executable
bits.
If the import
form issue could be figured out (e.g. have something that installs fzy.so
somewhere?), the following bits added to project.janet
, might be a work-around:
(rule "jeff/cli.janet" [])
(add-dep "jeff/cli.janet" "build/fzy.a")
(add-dep "jeff/cli.janet" "build/fzy.so")
Locally, with (import ../build/fzy :prefix "" :export true)
and the above changes to project.janet
, jpm build
succeeds (and the resulting executable does at least start up).
Actually, the .so
add-dep
doesn't appear necessary, as:
(rule "jeff/cli.janet" [])
(add-dep "jeff/cli.janet" "build/fzy.a")
seems to be enough along with the modified import form.
Ok, it looks like a similar scenario is covered in Chapter Nine: Xenofunctions [1] of the Janet for Mortals book by @ianthehenry.
If I understand the method described there correctly, what would work here is to make the import
form in scorer.janet
:
(import /build/fzy :prefix "" :export true)
and adapt a portion of project.janet
to have the following bits:
(def native-module
(declare-native
:name "fzy"
:source ["fzy.c"]))
(declare-source
:source ["jeff"])
(declare-executable
:name "jeff"
:entry "jeff/cli.janet"
:deps [(native-module :static)]
:install true)
That worked for me.
@tionis If you try this out, please let me know how it goes for you.
[1] Search for the text:
But let’s try it anyway. I’m going to add an executable with a native dependency
Sorry, I don't know how to link to the relevant part of the chapter.
This is indeed a working hotfix. But this is still a roundabout way to solve this. So maybe the right way to solve this would be to make some changes in the way jpm orchestrates builds, so that it either detects such cases or simply always builds the native-module artifacts first?
Never mind, using (import /build/fzy)
breaks the uses of jeff as library
Thanks for reporting back.
Too bad about the not being usable as a library :(
I think there's something in the Janet Guide about this type of situation but I haven't quite grokked that part yet.
The bit of text I had in mind is this:
Write a native Janet module that imports and re-exports the environment of a private native module, then declare-source it in our project.janet.
I think this is the easiest thing to do, but note that we will no longer be able to mix and match an executable with a library written in this way.
For a really dumb reason: in order to refer to a locally built native module from an executable, you need to use something like (import /build/set), while to import an installed native module you need to use (import set). You can sort of hack this by symlinking your jpm_tree/lib to the build/ directory but… is it worth it?
I'm not sure if the above text has a solution in it, but it seemed close.
This does seem relevant. The main issues seems to be in the way native modules are imported during (executable) compile time. (Maybe a hotfix could be to have separate sources for executables and libraries or do some comptime magic around the import, but event if this works, it would be better to solve the root of the problem)
So the problem is, don't use (import /build/fzy)
in the scripts, use (import fzy)
. And yes, if you do use a native dependency in an executable, you need to explicitly tell jpm to add it as a dependency to the script.
Ok, so that's the intended behavior? In that case, this should be mentioned in the documentation.
Yes, there should be some environment set up so that you can import things in the build
directory as if they were installed. However, there may be some issues here when building native executables.
Hm, when using
(def fzy
(declare-native
:name "fzy"
:source ["fzy.c"]))
(declare-source
:source ["jeff"])
(declare-executable
:name "jeff"
:entry "jeff/cli.janet"
:deps [(fzy :static)]
:install true)
in the project.janet
And doing a (import fzy)
in the scorer.janet i still get a compile error:
$ jpm install
compiling fzy.c to build/fzy.o...
generating meta file build/fzy.meta.janet...
generating /home/tionis/.local/lib/janet/.manifests/jeff.jdn...
compiling fzy.c to build/fzy.static.o...
creating native module build/fzy.so...
creating static library build/fzy.a...
generating executable c source build/jeff.c from jeff/cli.janet...
error: could not find module fzy:
/home/tionis/.local/lib/janet/fzy.jimage
/home/tionis/.local/lib/janet/fzy.janet
/home/tionis/.local/lib/janet/fzy/init.janet
/home/tionis/.local/lib/janet/fzy.so
error: build fail
in pdag [/home/tionis/.local/lib/janet/jpm/dagbuild.janet] (tailcall) on line 79, column 23
in run-main [boot.janet] on line 3838, column 16
in cli-main [boot.janet] on line 3986, column 17
So a current workaround is to use (import @build-dir/fzy)
instead - currently (declare-executable ...)
doesn't patch that environment like running tests.
Should (import @build-dir/fzy)
in (in this example) scorer.janet expand to the correct path and import the build artifact? Because on my machine it tries to import from the root of the filesystem:
removing /home/tionis/.local/lib/janet//fzy.so
removing /home/tionis/.local/lib/janet//fzy.meta.janet
removing /home/tionis/.local/lib/janet//fzy.a
removing /home/tionis/.local/lib/janet/jeff
removing /home/tionis/.local/bin/jeff
removing manifest /home/tionis/.local/lib/janet/.manifests/jeff.jdn
Uninstalled.
generating executable c source build/jeff.c from jeff/cli.janet...
error: could not find module @build-dir/fzy:
/fzy.jimage
/fzy.janet
/fzy/init.janet
/fzy.so
generating /home/tionis/.local/lib/janet/.manifests/jeff.jdn...
error: build fail
in pdag [/home/tionis/.local/lib/janet/jpm/dagbuild.janet] (tailcall) on line 79, column 23
in run-main [boot.janet] on line 3838, column 16
in cli-main [boot.janet] on line 3986, column 17
How up to date is your jpm? I was able to use this pattern locally to add an executable to the spork project that imported json via (import @build-dir/spork/json)
.
EDIT: This will also only work on Janet 1.26.0 and later
@tionis It works for me if I use a full path with the @
construct, e.g.
(import @/home/user/src/jeff/build/fzy :prefix "" :export true)
Then jpm --verbose build
gives:
$ jpm --verbose build
cc -c fzy.c -DJANET_BUILD_TYPE=release -std=c99 -I/home/user/.local/include/janet -I/home/user/.local/lib/janet -O2 -fPIC -o build/fzy.o
generating meta file build/fzy.meta.janet...
cc -c fzy.c -DJANET_BUILD_TYPE=release -DJANET_ENTRY_NAME=janet_module_entry_fzy -std=c99 -I/home/user/.local/include/janet -I/home/user/.local/lib/janet -O2 -o build/fzy.static.o
cc -std=c99 -I/home/user/.local/include/janet -I/home/user/.local/lib/janet -O2 -o build/fzy.so build/fzy.o -shared -lpthread
ar rcs build/fzy.a build/fzy.static.o
generating executable c source build/jeff.c from jeff/cli.janet...
found native /home/user/src/jeff/build/fzy.so...
found native /home/user/.local/lib/janet/jermbox.so...
found native /home/user/.local/lib/janet/spork/utf8.so...
compiling build/jeff.c to build/build___jeff.o...
cc -c build/jeff.c -DJANET_BUILD_TYPE=release -std=c99 -I/home/user/.local/include/janet -I/home/user/.local/lib/janet -O2 -o build/build___jeff.o
linking build/jeff...
cc -std=c99 -I/home/user/.local/include/janet -I/home/user/.local/lib/janet -O2 -o build/jeff build/build___jeff.o /home/user/src/jeff/build/fzy.a /home/user/.local/lib/janet/jermbox.a /home/user/.local/lib/janet/spork/utf8.a /home/user/.local/lib/libjanet.a -lm -ldl -lrt -pthread -rdynamic
tionis and I both seem to have $HOME/.local
-based setups. Is it possible that's a problem?
I'm on janet 1.28.0-dev-57b751b9
and latest jpm with following env:
JANET_BINPATH=~/.local/bin
JANET_HEADERPATH=~/.local/include/janet
JANET_LIBPATH=~/.local/lib/janet
JANET_PATH=~/.local/lib/janet
JANET_PROFILE=~/.config/janet/profile.janet
Maybe @bakpakin means with @build-dir to insert a hard-coded build path like you did? If I hardcode it like you did it compiles for me
If I import with (import @/home/tionis/dev/tionis/jeff/build/fzy)
(the build dir of the project dir) it compiles fine.
With hardcoded paths use as a library works fine, too
For reference, based on bakpakin's hint about janet 1.26.0, I looked at items under Janet's CHANGELOG:
Allow importing modules from custom directories more easily with the @ prefix to module paths. For example, if there is a dynamic binding :custom-modules that is a file system path to a directory of modules, import from that directory with (import @custom-modules/mymod).
Yeah, the full path makes the leading at sign redundant.
Try putting (print "build-dir: " :build-dir)
before your imports in the entry file of the executable - it should display the current directory with /build
appended, not nothing.
Edit: (print "build-dir: " (dyn :build-dir))
Do you mean (print "build-dir: " (dyn :build-dir))
?
It seems the dynamic binding is empty.
(pp [:build-dir (dyn :build-dir)]
gives me (:build-dir nil)
With tionis' code, I get:
$ jpm build
compiling fzy.c to build/fzy.o...
generating meta file build/fzy.meta.janet...
compiling fzy.c to build/fzy.static.o...
creating native module build/fzy.so...
creating static library build/fzy.a...
generating executable c source build/jeff.c from jeff/cli.janet...
build-dir: build/
error: could not find module @build-dir/fzy:
/fzy.jimage
/fzy.janet
/fzy/init.janet
/fzy.so
Note, the code was inserted in jeff/cli.janet
(at the top of the file):
(print "build-dir: " (dyn :build-dir))
(use spork/argparse)
(use ./ui)
Ah yes if I insert it in jeff/cli.janet
I also get build-dir: build/
So, if I understand the problem correctly, the best way to tackle this is to add a note in the documentation to add the module as :dep
to declare-executable
.
And also add some environment monkey patching to jpm during executable compilation so that janet can find the native module?
When I add a similar form to jeff/scorer.janet
as well, I see a blank value like tionis did for that file:
$ jpm build
compiling fzy.c to build/fzy.o...
generating meta file build/fzy.meta.janet...
compiling fzy.c to build/fzy.static.o...
creating native module build/fzy.so...
creating static library build/fzy.a...
generating executable c source build/jeff.c from jeff/cli.janet...
build-dir: build/
build-dir 2:
error: could not find module @build-dir/fzy:
/fzy.jimage
/fzy.janet
/fzy/init.janet
/fzy.so
May be that's expected?
Ok, I see what is happening. The monkey patch env
table does not persist when modules are recursively imported, it is only available in the entry file. The real fix should be patching module/paths so the built files in the build/
directory are available for import.
So (import @build-dir/fzy)
will work in cli.janet
, but not in scorer.janet
.
The real fix should be patching module/paths so the built files in the build/ directory are available for import.
Is this an easy fix?
Will you implement this, or should I work on a Pull Request?
I don't know if this an easy fix, but I will work on it. The general issue is the declare-executable does not spawn a new Janet process that runs in a consistent, monkey-patched environment - rather, imports the module into the same environment as process that jpm is running in to do analysis on the image.
I tried the changes in af002e6e.
Worked better :+1:
The fix with (import @build-dir/fzy)
works for me now, thanks.
Should I leave this open until the import paths are fixed, too?
Oh nevermind, it compiles, but when using (import @build-dir/fzy)
using it as a library is broken
Possibly a new issue might make sense.
Not sure.
@tionis Would you mind spelling out what you tried?
FWIW, after jpm install
, via janet
, I get:
$ janet
Janet 1.28.0-dev-57b751b9 linux/x64/gcc - '(doc)' for help
repl:1:> (import fzy)
@{_ @{:value <cycle 0>} fzy/has-match @{:private true} fzy/positions @{:private true} fzy/score @{:private true} fzy/score-max @{:private true} fzy/score-min @{:private true} :macro-lints @[]}
repl:2:>
May be you meant something different?
FWIW, here, the fzy
bits live at:
$ ls ~/.local/lib/janet/fzy.*
/home/user/.local/lib/janet/fzy.a /home/user/.local/lib/janet/fzy.so
/home/user/.local/lib/janet/fzy.meta.janet
Oh I already reverted the commit and added a bin script as a workaround.
So really Jeff ist currently only used as library and as such the import statement in scorer is currently just (import fzy)
So, the (import @build-dir/module-name)
problem was fixed during compilation, but fails when used as library
So, may be you mean this sort of thing?
$ janet
Janet 1.28.0-dev-176e816b linux/x64/gcc - '(doc)' for help
repl:1:> (import jeff)
error: could not find module @build-dir/fzy:
/fzy.jimage
/fzy.janet
/fzy/init.janet
/fzy.so
in require-1 [boot.janet] on line 2934, column 20
in import* [boot.janet] on line 2973, column 15
in _thunk [/home/user/.local/lib/janet/jeff/scorer.janet] (tailcall) on line 2, column 1
in dofile [boot.janet] (tailcall) on line 2911, column 7
in source-loader [boot.janet] on line 2922, column 15
in require-1 [boot.janet] on line 2942, column 18
in import* [boot.janet] (tailcall) on line 2973, column 15
in dofile [boot.janet] (tailcall) on line 2911, column 7
in source-loader [boot.janet] on line 2922, column 15
in require-1 [boot.janet] on line 2942, column 18
in import* [boot.janet] on line 2973, column 15
in _thunk [/home/user/.local/lib/janet/jeff/init.janet] (tailcall) on line 1, column 1
in dofile [boot.janet] (tailcall) on line 2911, column 7
in source-loader [boot.janet] on line 2922, column 15
in require-1 [boot.janet] on line 2942, column 18
in import* [boot.janet] on line 2973, column 15
in _thunk [repl] (tailcall) on line 1, column 1
Yes exactly
@tionis I'm not sure if the following is appropriate, but I have a patch for jpm that gave seemingly good results here:
diff --git a/jpm/cc.janet b/jpm/cc.janet
index 9617754..dc702e2 100644
--- a/jpm/cc.janet
+++ b/jpm/cc.janet
@@ -290,7 +290,7 @@ int main(int argc, const char **argv) {
(create-dirs dest)
# Monkey patch stuff
- (def token (do-monkeypatch bd))
+ (def token (do-monkeypatch (string bd ":all::native:")))
(defer (undo-monkeypatch token)
# Load entry environment and get main function.
That seems to work [1] if I've got the following for jeff's project.janet
:
(declare-project
:name "jeff"
:author "Josef Pospíšil <josef.pospisil@laststar.eu>, tionis <janet@tionis.dev>"
:description "Janet Extended Fuzzy Finder"
:license "MIT"
:url "https://tasadar.net/tionis/jeff"
:repo "git+https://tasadar.net/tionis/jeff"
:dependencies ["spork"
"https://github.com/MorganPeterson/jermbox.git"])
(def fzy
(declare-native
:name "fzy"
:source ["fzy.c"]))
(declare-source
:source ["jeff"])
(declare-executable
:name "jeff"
:entry "jeff/cli.janet"
:deps [(fzy :static)]
:install true)
Would you mind trying out this arrangement?
Edit: I failed to mention changing the import
form in jeff/scorer.janet
to be:
(import fzy :prefix "" :export true)
[1] By "works", I mean that:
jpm build
works (ofc after jpm deps
succeeds)jpm install
works (even after jpm clean
)(import jeff)
from a janet repl works(import fzy)
from a janet repl worksjeff --help
worksIt does work! In all my tests, it worked exactly as expected. My other projects that use it as dependency also still compile fine. Thanks for looking into this!
Thanks for trying it out and sharing your results :)
There is a slightly related point I think is worth considering and it has to do with collision of things in JANET_PATH
.
It's true that a number of other projects have already placed .a
, .so
, and .maeta.janet
files at the root of JANET_PATH
, but I think this is asking for trouble eventually because two different projects may end up choosing the same name for "native" bits.
I think it's also a bit on the cluttered side AND when manually having to repair problems (hopefully not so often) it is more error-prone to have to pick out individual files from a sea of others :)
What some folks have done instead is to place all of their bits in a single directory. Some examples of this include:
It may be a bit more work to arrange and it's also possible there will be a collision, but I think it's less likely and it's tidier.
To examine a specific case, at the time of this writing, for stx, once installed, there is a single directory under JANET_PATH
named stx
that contains:
init.janet
native.a
native.meta.janet
native.so
init.janet
makes use of native.so
like:
(import "stx/native" :prefix "" :export true)
IIUC, there's no technical reason everything has to be in one directory though. I think having the "native" parts live in a separate sibling directory should work too. I don't know if you think that might apply for fzy.so
and friends but I thought I'd point out the possibility.
If it turns out the potential collision concerns of the sort mentioned above are warranted, perhaps something approrpiate could eventually be recommended via janet-lang.org docs.
@iarebatman If this issue is still relevant for you, perhaps you can try bdf59999a to see if things work better for your situation.
A summary of what was changed in tionis' jeff project is here [1].
In particular, the tip from @ianthehenry about :deps
for declare-executable
(also echoed by bakpakin here) might be relevant [2].
If you do try things out, please let us know how it goes for you :)
[1] The patch to jpm mentioned in the comment was not what made it into jpm -- it was a bit different (hopefully an improvement).
[2] For import
, we're using (import fzy :prefix "" :export true)
and not (import /build/fzy :prefix "" :export true)
as was suggested by adapting the tip here originally. Though see here for some thoughts about where to put native bits.
@tionis If this has still been working for you may be it's ok to close this. What do you think?
It's working fine for me, I do think however that we should document somewhere that something like :deps [(fzy :static)]
is required to compile executables with native deps.
I'm not sure where, though, as we probably don't want to explain every special case in the main docs?
I do think however that we should document somewhere that something like :deps [(fzy :static)] is required to compile executables with native deps.
Yes, I agree.
I'm not sure either, but I do find the current text at the official docs a bit misleading:
jpm is smart enough to figure out from the one entry file what libraries and other code your executable depends on, and bundles them into the final application for you.
One idea is to link to an suitable example as is done in other parts of the docs...I don't yet have a good idea for one though (^^;
I wonder if it would be worth having a set of examples that could also be tested against in jpm's repository...perhaps if there were an example there that demonstrated the :deps
thing in this issue, it could be linked to.
Sounds good, I'll close this issue and will open a new one, so it's a bit clearer that just the documentation is missing.
If a project.janet file contains both a
(declare-native)
module and janet-source-code that depends on it, the build fails, but only if an executable is to be build: Exampel project.janet extracted from jeff