Modules and subcommands

[casey]

This is apropos of @valscion's comment in #208, where he talks about some of the annoyances of using just to emulate subcommands. This is also somewhat related to #237 and #367, where sub-justfiles are discussed.

I'd like to come up with something that enables all of these things. Some ideas:

A statement mod foo parses foo.just or foo/mod.just as a submodule. Recipes in foo.just / foo/mod.just can now be called on the command line with just foo RECIPE.

If the submodule source file is foo.just, the current working directory for recipes in that submodule will be the current directory. If the it is in foo/mod.just, the current working directory will be foo/.

Questions:

• What do we do about directories with weird characters? I think punting is reasonable. But eventually we might want to support something like mod "path with spaces/bar". Any solution should be based on user feedback.

• How do we handle dependencies between recipes and variables in different submodules?

• Can you have a submodule in a submodule, and call those commands?

---

Notes for a first path for out-of-line modules only:

• make a module loading step after parsing
• look at parsed module and load all other modules, recursively
• module map has all modules in it
• pass module map to the analyzer
• analyzer produces justfile

---

• will need to modify error messages to include file path
• will need to modify argument parser to successfully call submodules
• will need to disallow recipes and modules to have same name

[valscion]

Thanks for opening this more focused issue :relaxed:

Currently for us, it would be enough to support one level deep submodules. But of course, it would be nice if the solution we could come up with would work for deeper submodules, too.

I'm not sure if just needs to automatically support directory traversal. To me it sounds reasonable to first keep on supporting a single Justfile, and that you could add special statements there to tell just about the existence of submodules. Like so:

# <root>/Justfile
mod foo

# <root>/foo/mod.just
test:
  echo "All OK"

And then you could call <root>/foo.just#test by running just foo test.

This way just wouldn't have to glob the filesystem to find .just files to figure out what submodules exist. That would also free just to use directory syntax for other purposes, e.g. sub-submodules. I could imagine one could run just foo/bar RECIPE which would try to something like this:

# <root>/Justfile
mod foo

# <root>/foo/mod.just
mod bar

# <root>/foo/bar/mod.just
RECIPE:
  echo "All OK"

---

I'm not excited about naming sub-modules with .just extension and having to either put them to <submodule>/mod.just or <submodule>.just file. If possible, I'd like to be able to control where those submodules are located. With something like this perhaps?

# <root>/Justfile
mod foo "tasks/foo/Justfile"

# <root>/tasks/foo/Justfile
RECIPE:
  echo "All OK"

which could be called like so:

just foo RECIPE

One could continue with the same way to enable sub-submodules:

# <root>/Justfile
mod foo "tasks/foo/Justfile"

# <root>/tasks/foo/Justfile
mod bar "bar/Justfile"

# <root>/tasks/foo/bar/Justfile
RECIPE:
  echo "All OK"

and then that could be called with something like this:

just foo/bar RECIPE

---

It could be that overloading the directory syntax is not wise, as just already has functionality for that:

# <root>/foo/Justfile
RECIPE:
  echo "All OK"

$ just foo/RECIPE

The syntax isn't the main point for sub-submodules. One possibility would be to denote sub-submodules with e.g. ::, making one of my previous examples callable like so:

just foo::bar RECIPE

I'd be fine with that kind of solution, too.

---

What do you think? What kind of discussion would you want to have here? I'd be happy to continue the conversation in any way you'd want to ☺️

[casey]

Thanks for opening this more focused issue ☺️

I'm just going respond like I didn't let your reply in my inbox for a month >_< I beg your forgiveness!

Currently for us, it would be enough to support one level deep submodules. But of course, it would be nice if the solution we could come up with would work for deeper submodules, too.

Agreed, it would be nice to do something that works with deeper submodules. I've heard tell that some companies use just, and it would be nice if I could infect their giant monorepos with justfiles.

I'm not sure if just needs to automatically support directory traversal. To me it sounds reasonable to first keep on supporting a single Justfile, and that you could add special statements there to tell just about the existence of submodules. Like so:

# <root>/Justfile
mod foo

# <root>/foo/mod.just
test:
  echo "All OK"

And then you could call <root>/foo.just#test by running just foo test.

This sounds good.

This way just wouldn't have to glob the filesystem to find .just files to figure out what submodules exist.

Yeah, I definitely want to avoid globbing.

That would also free just to use directory syntax for other purposes, e.g. sub-submodules. I could imagine one could run just foo/bar RECIPE which would try to something like this:

# <root>/Justfile
mod foo

# <root>/foo/mod.just
mod bar

# <root>/foo/bar/mod.just
RECIPE:
  echo "All OK"

We should come up with something that lets just foo bar RECIPE work in that case, since as you note / is already interpreted on the command line.

I'm not excited about naming sub-modules with .just extension and having to either put them to <submodule>/mod.just or <submodule>.just file. If possible, I'd like to be able to control where those submodules are located. With something like this perhaps?

# <root>/Justfile
mod foo "tasks/foo/Justfile"

# <root>/tasks/foo/Justfile
RECIPE:
  echo "All OK"

which could be called like so:

just foo RECIPE

I think using a string as in mod name "path" sound good, if you want just to look in a non-default location.

I was thinking that given mod NAME just should look in NAME.just or NAME/mod.just or NAME/Justfile, but I'm not sure if that's weird. What do you think?

What do you think? What kind of discussion would you want to have here? I'd be happy to continue the conversation in any way you'd want to ☺️

Here is great! I promise I'll respond faster next time 😅

[casey]

One thing that needs to get resolved:

Can a recipe in one module have dependencies on a recipe in another module? And if so, how do they refer to recipes in other modules?

Is this something that you need or would use? Also if there are others out there who would make use of modules, I'd definitely be interested in your thoughts.

[valscion]

I'm just going respond like I didn't let your reply in my inbox for a month >_< I beg your forgiveness!

I promise I'll respond faster next time 😅

Hey, it's completely OK to take your time to reply, or to not even reply at all :relaxed:. Nobody's getting paid on working on this, so any comment at all is only a value add, no matter how long it takes to respond. 💞

I was thinking that given mod NAME just should look in NAME.just or NAME/mod.just or NAME/Justfile, but I'm not sure if that's weird. What do you think?

Hmm yeah, we can make the location optional for sure — but I'm not sure if we know yet what would be a good choice from all of these, so I'm a bit weary on making the decision already in the first implementation.

I kinda like the name Justfile to describe files using this project. I'm a bit skeptical on using .just extension for modules — at least in the first iteration.

I'm worried about folder bloat if this default were any of the three you suggested. If, on the other hand, modules were to live inside one directory deeper first, then I don't have a strong feeling on any of the suggestions. What I mean is something like just_modules/NAME/Justfile, just_modules/NAME.just or just_modules/NAME/mod.just — i.e. that all submodules would be inside just_modules (the name is not important) directory first, so that on the root level, there would only be the root Justfile and another directory for the modules.

We should come up with something that lets just foo bar RECIPE work in that case, since as you note / is already interpreted on the command line.

This could work — it would also be possible to give a helpful error should someone have this kind of Justfile:

mod foo

foo:
  echo "hi"

and saying something like this:

Error: Names of modules and tasks must be unique. You have defined both a module named foo and a task named foo. Rename one of them to avoid this error.

> 1 | mod foo
    |     ^^^
  2 | 
> 3 | foo:
    | ^^^
  4 |   echo "Hello, world"

That way we wouldn't even have to figure out any special syntax 🙌

One thing that needs to get resolved:

Can a recipe in one module have dependencies on a recipe in another module? And if so, how do they refer to recipes in other modules?

Is this something that you need or would use? Also if there are others out there who would make use of modules, I'd definitely be interested in your thoughts.

Hoo boy, we're going to the dependency management domain 😅. If at all possible, I think we should avoid making any decisions on automatic behavior yet, in order to see how submodules would be used on their own.

If one would need to share recipes, one could refer to a module by path, e.g. by climbing up the tree:

# <root>/Justfile
mod foo

hello:
  foo bar greet World

# <root>/just_modules/foo/Justfile
mod bar "../bar/Justfile"

# <root>/just_modules/bar/Justfile
greet subject:
  echo "Hello {{subject}}"

Then just would need to handle cyclical dependencies somehow. I assume dependency trees will be quite small, so raising an error and refusing to work with cyclical dependencies could be an OK tradeoff.

So if one would have something like this:

# <root>/Justfile
mod foo

hello:
  foo bar greet World

# <root>/just_modules/foo/Justfile
mod bar "../bar/Justfile"

# <root>/just_modules/bar/Justfile
mod foo "../foo/Justfile"

greet subject:
  foo bar greet "Hello {{subject}}"

just could raise an error when loading the module trees before evaluating the module contents.

Error: Cyclical dependencies are not allowed. Module foo is being loaded from one of its submodules again, resulting in a cycle.

foo module was first loaded from <root>/Justfile:1:

> 1 | mod foo
    | ^^^^^^^
  2 | 
  3 | hello:

Cyclic dependency loading was discovered in <root>/just_modules/bar/Justfile:1:

> 1 | mod foo "../foo/Justfile"
    | ^^^     ^^^^^^^^^^^^^^^^^
  2 | 
  3 | greet subject:

[casey]

Hey, it's completely OK to take your time to reply, or to not even reply at all ☺️. Nobody's getting paid on working on this, so any comment at all is only a value add, no matter how long it takes to respond. 💞

Aww, thanks 😊

I was thinking that given mod NAME just should look in NAME.just or NAME/mod.just or NAME/Justfile, but I'm not sure if that's weird. What do you think?

Hmm yeah, we can make the location optional for sure — but I'm not sure if we know yet what would be a good choice from all of these, so I'm a bit weary on making the decision already in the first implementation.

Yeah, definitely. Hmm, so maybe the best thing to is to wait until we have more experience with the feature, before picking a default location or locations that just should look for submodules.

If we do the explicit-path form, as you suggested:

mod foo "foo/bar/whatever"

Then we can wait and see which, if any, default location make sense, based on user feedback

I'm worried about folder bloat if this default were any of the three you suggested. If, on the other hand, modules were to live inside one directory deeper first, then I don't have a strong feeling on any of the suggestions. What I mean is something like just_modules/NAME/Justfile, just_modules/NAME.just or just_modules/NAME/mod.just — i.e. that all submodules would be inside just_modules (the name is not important) directory first, so that on the root level, there would only be the root Justfile and another directory for the modules.

I think you have a good point about folder bloat, and we can use the explicit-path form at first to avoid committing to anything.

We should come up with something that lets just foo bar RECIPE work in that case, since as you note / is already interpreted on the command line.

This could work — it would also be possible to give a helpful error should someone have this kind of Justfile:

mod foo

foo:
  echo "hi"

and saying something like this:

Error: Names of modules and tasks must be unique. You have defined both a module named foo and a task named foo. Rename one of them to avoid this error.

> 1 | mod foo
    |     ^^^
  2 | 
> 3 | foo:
    | ^^^
  4 |   echo "Hello, world"

That way we wouldn't even have to figure out any special syntax 🙌

I think that's a good idea. One thing that I've learned from writing just is that it pays off to be as restrictive as possible initially.

If you allow a construct, it's annoying to deprecate it and can be dangerous to suddenly change its behavior in a way that users don't expect. However, if you didn't allow it in the first place, adding it later is very easy.

Hoo boy, we're going to the dependency management domain 😅. If at all possible, I think we should avoid making any decisions on automatic behavior yet, in order to see how submodules would be used on their own.

Yeah, good idea. This stuff is why the idea of adding submodules has kind of freaked me out, so it's probably best to implement a true MVP first, and then expand as necessary.

If one would need to share recipes, one could refer to a module by path, e.g. by climbing up the tree:

# <root>/Justfile
mod foo

hello:
  foo bar greet World

# <root>/just_modules/foo/Justfile
mod bar "../bar/Justfile"

# <root>/just_modules/bar/Justfile
greet subject:
  echo "Hello {{subject}}"

Then just would need to handle cyclical dependencies somehow. I assume dependency trees will be quite small, so raising an error and refusing to work with cyclical dependencies could be an OK tradeoff.

just actually already raises errors on cyclical dependencies. It can detect cyclical dependencies in recipes:

foo: bar
bar: foo

: just
error: Recipe `foo` has circular dependency `bar -> foo -> bar`
  |
1 | foo: bar
  |      ^^^

And cyclical dependencies in variable assignments:

foo = bar
bar = foo

: just
error: Variable `bar` depends on its own value: `bar -> foo -> bar`
  |
2 | bar = foo
  | ^^^

So if one would have something like this:

# <root>/Justfile
mod foo

hello:
  foo bar greet World

# <root>/just_modules/foo/Justfile
mod bar "../bar/Justfile"

# <root>/just_modules/bar/Justfile
mod foo "../foo/Justfile"

greet subject:
  foo bar greet "Hello {{subject}}"

just could raise an error when loading the module trees before evaluating the module contents.

That sounds good. Static analysis for the win 🧐

[valscion]

Great! I guess we're now in the "plan is ready, time to start coding" phase 😅. My rust skills are, err..., rusty — I have been meaning to get to know rust a bit but I haven't had the time. Is there any way I could help even though I would not be able to implement this?

Again, even if this would not be implemented, that would also be OK ☺️. Don't feel like you have any obligations to do this if you don't have the time or the motivation to do free work :smile:

[runeimp]

I like where this seems to be going for MVP. One addition I'd really appreciate is the ability to point to a module via ~ in the path. I share much of my setup between computers but often have different home directories on different systems. I also have some boilerplate I put in every single Justfile I've ever created. So if the following would resolve:

# <root>/Justfile
mod foo "~/.config/just_awesome/Justfile"

I would be extremely happy. 😃

[valscion]

@runeimp do you mean that ~ would point to $HOME or to the <root> or what? It isn't quite clear to me what you mean

[casey]

Fortunately, before needing to worry about implementing it, we can continue to noodle on the design!

I've been thinking that using strings to represent paths might be problematic. Paths differ on different platforms, e.g. unix and windows:

• ~ does not exist on Windows
• Drive prefixes (C:) do not exist on linux
• / vs \ delimiter

If module paths are strings, without some extra compatibility mechanism, justfiles will not be portable across platforms.

A couple ideas to resolve this:

1. Parse strings as unix-style on all platforms, and translate them into platform-native paths on non-unix platforms

2. Use a platform-neutral non-string representation, like foo::bar::baz, which translates to ./foo/bar/baz on linux and .\foo\bar\baz on windows

I like 2 a bit better, but it comes with problems of its own. If modules are required to be identifiers, are spaces and special characters allowed?

[valscion]

Fortunately, before needing to worry about implementing it, we can continue to noodle on the design!

Yay! :smile:

I've been thinking that using strings to represent paths might be problematic.

Might be, yes

1. Parse strings as unix-style on all platforms, and translate them into platform-native paths on non-unix platforms

I like this approach better than a custom syntax. This would map to how e.g. node does imports from other files — they just support / syntax for folder traversal. That's also how ruby works if I'm not mistaken.

If we were to take any other choice than using / syntax, I suppose it would be confusing to people. We could always show a helpful error when using / instead of :: or some similar name, though.

I am only speaking from my experiences, though. But from my point of my view, I would expect / syntax to "just work". Anything else would be confusing.

[runeimp]

@valscion I meant to point to $HOME so I can target common module(s) I will use on all systems I work on. So to be clear either ~ or some methodology that allows me to target my home directory whatever that name might be on any given system. I would settle for any decided system. But I'm partial to POSIX methodologies.

[runeimp]

@casey Actually Windows has supported POSIX/UNIX style paths since Windows XP. And in DOS prior to that if I remember correctly. It just prefers \ for historical reasons. So I highly recommend sticking with the POSIX/UNIX standard pathing.

[valscion]

Thanks for clarifying, @runeimp :relaxed:. And thanks for the additional tidbit about Windows supporting forward slash in paths. I agree that sticking with / pathing would be a good idea.

[casey]

I tried it out, and it looks like windows doesn't support ~. If we go with concrete / syntax, I'll parse it instead of using it directly, so that I can error on non-portable syntax.

[casey]

I wish I had time to dedicate to this, but it'll have to go on the back burner for now. It's a big project, with lots of implications codebase-wide, but if someone is interested in picking this up, I can provide some pointers.

[runeimp]

@casey Actually, ~ is supported by Windows PowerShell but not CMD. You'd have to do a swap for %UserProfile% in CMD/Command Prompt.

[casey]

I'm still thinking about this and definitely want to do it.

I landed a much improved lexer, and I'm rewriting the parser to better separate the actual parsing (turning a token stream into higher-level structures) and consistency checks (duplicate variable names, circular dependencies, etc). Both will make extensions like these easier.

I think the way that I can best move forward with this is to start with inline modules. From rust, for example:

// inline module
mod foo {
  fn bar() {}
}

// out of line module, loaded from `baz.rs` or `baz/mod.rs`
mod baz;

By starting with inline modules, we can nail down the semantics of submodules and subcommands, without having to think about loading multiple files from disk, which will be a separate complication.

We could lift rust's syntax:

mod foo {
  bar:
    echo baz
}

$ just foo bar
baz

Another option is to use whitespace:

mod foo
  bar:
    echo baz

Inter-module paths would be :: delimited:

# call foo::bar as a dependency
yik: foo::bar

I think that mod NAME { ... } is probably the most readable and the least alien. What do y'all think?

cc: @cledoux, since he was interested in including other justfiles, possibly conditionally, which this could be extended to. By adding some kind of conditional activation of modules,

@windows
mod build "build-windows.just"

@unix
mod build "build-unix.just"

[casey]

I'm leaning towards using something with {...}. Curly brace delimiters are widely used and will be easier for users to pick up and recognize.

Also thinking about "block recipes", which introduce an anonymous scope just for that recipe, for local variables:

# recipe `foo` with deps `a` and `b`
foo: a b {
   #  undecided mystery syntax in here
}

[ethankhall]

Hi @casey, I might have some free cycles coming up. Do you have a "current" design/expected syntax on what's wanted for this feature?

[casey]

Hi @ethankhall!

I'm still pretty uncertain about the syntax. I think the way to go is pick some syntax, implement modules with that syntax, and it behind a flag. Then we can bikeshed the syntax and details of the semantics after we have more experience with the feature.

Also, we should implement in-line modules first, and then later implement out-of-line modules (i.e. modules in another source file) since that's more complex.

Some syntax possibilities for a module foo containing a recipe bar:

curly braces

foo {
  bar:
    echo hello
}

double colon

foo::
  bar:
    echo hello

keyword + colon

mod foo:
  bar:
    echo hello

I'm leaning towards the double colon syntax:

• module paths are often delimited with ::, which will hopefully make it easier for users to learn and remember
• for better or worse, Just syntax is already terse and whitespace-sensitive, so those possible objections to the double-colon syntax are less salient
• the lexer already handles emitting whitespace-based INDENT and DEDENT tokens, so this should be the easiest to implement given what already exists
• GNU make has recipes that are written with a :: so existing syntax highlighting will work

Out-of-line modules can just omit the module suite:

foo:: # this loads `foo.just` or `foo/mod.just`

Or provide a path to the file to load:

foo:: path/to/some/file.just

But that can be done later.

How does that all sound? Please let me know if you think there are particularly strong reasons to prefer another syntax.

[casey]

This will likely be a pretty complex feature, and I'm very happy to provide as much support as needed to implement it!

[ethankhall]

Sounds good to me. I will look over the code base and see what I can find. I'll come up with an impl plan and post it here, or rough PR with my thoughts.

I'll reach out with questions!

[casey]

Sounds good! In general, parsing and consistency analysis is pretty gross, so feel free to shoot me any questions about it.

[casey]

To give an idea of where things stand with this, I actually started implementing modules, and you can see the initial implementation in this branch.

Adding modules wasn't so terrible, but I realized the feature would impact almost every part of Just. For example, error messages now need to be aware of modules to correctly report symbol paths, rules regarding variable scope need to be thought out and implemented, cross-module recipe dependencies need to be implemented, etcetera.

All in all, it's a massive undertaking, and I got kinda burnt out on it after I realized how much work it would take to bring it across the finish line, mostly because I'm working on another project that's taking a lot of bandwidth, so I don't have as much time as I'd like to devote to Just.

I think the way forward is probably to land a very conservative version of modules, namely one where cross-module variable and recipe references aren't allowed, so each module is more like a totally independent justfile, and the only way to call submodule recipes is on the command line:

foo::
  bar:
    echo 'bar!'

$ just foo bar
bar!

This would leave out a lot of obvious desiderata, like cross-module recipe dependencies and variable references, but would make landing the initial feature much more likely.

It might also be a good idea to only allow modules if a module-experiment setting is given, and exempting modules from stability guarantees. This would allow the design and syntax be a bit more sloppy, and make it easier to make progress without necessarily being 100% of all the details.

I'm hesitant to suggest that someone else pick up the torch and try to finish it and land it, just because I suspect that there are a lot of design gotchas, and so I'd have to do a lot of review and design as well. However, I still wanted to post and update jut to let peeps know where things stood.

[valscion]

Thanks for the status update 🙂. Seems like quite a massive undertaking indeed.

I'd like to make sure I didn't misunderstand: The conservative version you're describing would only work on the command line, and the only change to justfiles themselves would be the addition of optional module syntax? That is, the new possibility we'd get is the ability to group tasks in a single Justfile, right?

[casey]

@valscion Yup, that's right. Justfiles could contain submodules, and you could put exactly the same things in submodules that you can now, so you could group related functionality. A simple extension to this that could be landed afterwards would allow declaring the same modules "out of line", so that they would be loaded from another file, so recipes could be split between multiple files.

Initially it might look like this:

foo::
  bar:
   echo 'bar!'

Which might later be expressed with out of line modules like so:

# Justfile
foo:: "bar.just"

# bar.just
bar:
  echo 'bar!'

[valscion]

Interesting, thanks for expanding on the future idea ☺️

[rawkode]

@casey did this ever get implemented?

[casey]

@rawkode Nope, it did not. I started working on it, and published a WIP branch, but it's a very large change and I had other things on my plate, so I wasn't able to finish it.

I think if I (or someone else) were to pick it up a again, the best way to go would be to add a flag or setting that enabled modules, so that it could be incrementally implemented, and eventually the flag removed.

[rawkode]

@casey Would a naive / simple solution, to begin with, be appropriate?

I'm thinking scan the Justfile for mod xyz, copy all contents, and load a single Justfile?

[casey]

I think there are probably two different ways to go.

The first is to start implementing this behind an unstable flag. Because it's clearly delineated as unstable, we could introduce breaking changes as the feature is developed.

The second is to implement it in such a way that it can be developed in a backwards compatible fashion.

The latter is ideal, but since the modules are a complex feature, the former might be more pragmatic. For example, even basic questions such as the syntax of modules is unresolved, so marking the feature as unstable would allow for experimentation, instead of having to always land functionality in-tree and commit to it at the same time.

But, even in the latter case, I think it's okay to land something that is incomplete, i.e. is missing some functionality, but wouldn't want to land something that can't be incrementally developed into something that can be made stable. So, for example, I think an implementation based on literal text inclusion, a la C's #include "foo.h", wouldn't be a good idea, since it would have a lot of inherent problems.

I haven't thought a ton about how to proceed most incrementally, but an idea for a sequence of PRs that could be landed is:

1. Implement justfile annotations, as in #604, so that users could write #![feature(modules)] at the top of a justfile, to enable the unstable module feature.

2. Add module declaration syntax. This would error if you used it without the unstable#[feature(modules)], wouldn't actually do anything, and wouldn't have to be the final syntax. This could look like mod foo, mod "foo", or foo::. (The last one is less legible, but is the most make-y.)

3a. Load and parse, but don't analyze, a justfile when the module declaration syntax is encounter. So if the parser sees mod foo, it should load foo.just or foo/mod.just and parse it, returning an error if the file fails to load or parse. (analysis, which includes things like symbol resolution, can be skipped.) This will require adding a file-loader which includes an arena, so that child module files can be added to an arena, and thus the text can have the same lifetime as the parent file's text.

3b. Make sure that error messages when a module fails to parse are sensible. In particular, they should tell the user the module path where the error happened, like foo::bar::baz, as well as the filename foo/bar/baz.just.

4. Add functionality to allow users to call recipes in submodules. Each submodule would be still be totally separate, so no cross-module recipe, alias, or variable references. If you had a module foo with a recipe bar, you could call it from the command line with just foo bar.

5. Add cross-module recipe, variable, and alias references. (Can be done as multiple PRs.) So that a child module can call a parent module recipe, or vice versa.

6. Add import statements, so you can bring variables or recipes from other modules into scope.

7. Make it possible to declare a module in-line, without needing a separate file. This is more complex than it might seem, because of the indentation-based parsing.

(Aside: I found that a lot of the work for this feature had to do with error messages, since all of a sudden symbol references would have to mention the module they came from.)

I think the feature could be made stable once 4. is done, and 5-7 and beyond could be added in a backwards compatible way.

[ghost]

I think the "modules" approach adds too much complexity for what I want to do: just load tasks from other justfiles so that I can call them from the command line and use them on dependent new tasks.

I'm thinking in something like an "import" statement to define a new "domain": In a justfile: import [pathToAnotherJustfile] [domain]

Where [pathToAnotherJustfile] is a relative or absolute path to another justfile. Then I can call tasks defined on [pathToAnotherJustfile] with the just command: In command line: just domain.taskName

And I can also define tasks depending on tasks from another [domain]: In a justfile: otherTask: domain.taskName

There are a lot of details that I'm not addressing and would make this approach unfeasible, or something like this solution has already been considered and rejected... what do you think?

[casey]

So, I realized that you can sort of already do modules, although it's so absurd that it's verging on parody:

subcommand *args:
  #!/usr/bin/env just --justfile
  foo $arg:
    echo hello from foo: $arg

  bar:
    echo hello from bar

Just-fucking-ception.

It uses positional-arguments because nested interpolation doesn't work. I don't know if I can in good conscience recommend doing this, because lots of things probably break (like settings not being shared), but it's kind of hilarious and if you really need modules and subcommands, why not.

[casey]

(In retrospect it was inevitable.)

[MadBomber]

I have been playing with the concept of a pre-processor to implement module-like capability. I first tried m4 and its "include" macro. It worked; but, m4 has a problem with backticks. So I created a prototype in ruby which I then re-code for crystal because I'm not fully rust-ified.

https://github.com/MadBomber/experiments/tree/master/just_playing

This prototype looks for "main.just" in the current working directory and its hierarchy. When found it processes that file looking for inclusionary keywords that have a file name/path following. If found it inserts the content of the specified file (module) into the text at that point. When done it writes the processed text out to a file "justfile" in the current working directory.

There is no recursion supported..... by that I mean that an inclusionary file can not also include another file.

[MadBomber]

The justprep project has been a nice diversion. I'm using the "with" keyword in my main.just files more than any of the other possibilities. I think that is because is is shorter and easier to type. I've install the justprep executable from the crystal version in my ~/bin directory which is in my $PATH.

I have a "~/.bashrc__just" file that sets the following alias:

alias jj='justprep && just --no-dotenv'

In my home directory I have a file ".justfile" which only contains this:

# ~/.justfile

# List available recipes
@list:
  echo
  echo "Available Recipes at"
  echo `pwd`
  echo "are:"
  echo
  just -l --list-prefix 'just ' --list-heading ''
  echo

I use "with ~/.justfile" as my first actionable line in all of my other "main.just" files. This way the default is always the same except when I want a different default for a project.

By doing this inclusionary thing I avoid the problem of having to deal with duplicate recipes and such. When there is a duplicate just will tell me and I can rearrange the modules as necessary.

I think it worked out pretty nicely.

Dewayne o-*

[casey]

@MadBomber Awesome work! It makes me feel extra bad that I haven't just implemented modules / import statements yet 😅

[MadBomber]

Thank you. It is a stop gap measure until we get the time to implement some sort of recursive grammar that allows for the inclusionary/module feature. w/r/t error messages and such with the justprep pre-processor, those that come out of the just tool are fine. They point to the generated file and all of the line numbers are exclusive to that generated file. With the header/footer lines that wrap the included file its easy to manually find the actual source of a syntax error for example.

I think the next step is to add a command line option to just that would turn on/off duplicate errors. Something like --allow-duplicates OR maybe --warn-on-duplicates with the default behavior being the current functionality.

--allow-duplicates Would ignore any prior definitions of a recipe etc. and process the last definition as if it were the only one.

--warn-on-duplicates would give a WARNING message but still allow for recipes to be over-ridden.

With neither of these options present, the current behavior would be used to generate the ERROR message.

[casey]

It's a good thought, but one thing that would make --allow-duplicates or warn-on-duplicates problematic is that currently justfiles are completely order-independent. (Actually, that's with the exception of which recipe is defined first, which is used as the default recipe.) Recipes and variables can be declared in any order. --allow-duplicates would presumably use either the first or the last defined recipe, and ignore others, which would be a departure from the current behavior.

[MadBomber]

My preference would be for the last definition to be the one that is used when a duplicate condition exists. That way we can import generic recipes from other files and then over-ride a generic recipe in a specific project area.

with ../generic_recipes.just  # has a doit recipe
@doit:
  echo "my way"

[casey]

It's a bit of a weird feature, when considered apart from justprep, but that aside, I think I'd accept a PR that implemented --allow-duplicates.

[MadBomber]

I agree its weird out of context. Just pushed step one. I have the flag added to the config object and the help test modified to pass with the new option. Next step is to add the business logic to the analyzer and a few other files.

[casey]

@MadBomber Sounds good! Feel free to push a WIP draft PR if you want to get early feedback or have questions about anything.

[valscion]

Hmm I wonder if #1095 was supposed to entirely close this issue?

[casey]

Ah, sorry, no I definitely didn't meean to close this issue

[hhoeflin]

Cool project - this import sounds very interesting. I had 2 questions: a) @MadBomber - did you also try https://logological.org/gpp or just the m4 preprocessor? b) @casey For something like this, an option to read a justfile from stdin would be useful. How difficult do you think that would be?

Thanks - very great work. I love how many people create cool CLI tools in Rust these days!

[MadBomber]

@hhoeflin I've used M4 before to do similar things - docker compose junk. This time I wanted to see how close I could come with a Ruby/Crystal implementation that shared a common code base. Still not there. Like everyone else time is in short supply these days.

The justprep project is not general purpose. Its a specific stop-gap measure for use with the just utility until such time as the new modular architecture can be implemented. M4 and GPP could do the same thing; but, I would not have as much fun playing with Ruby/Crystal cross-breeding.

[casey]

@hhoeflin I don't think reading a justfile from stdin would be hard, exactly, but it might be a tricky PR, since it would require modifying a lot of error messages. Feel free to open an issue!

[hhoeflin]

Thank you, but looking at this https://serverfault.com/questions/398514/pass-a-pipe-to-a-command-that-expects-a-filename I learned it is not actually necessary and there is a good workaround, at least on linux in bash. Always something new to learn :)

[nickgarber]

As I learn about just I appreciate the balance being struck by the project lead and community.

So much so that it seems that just includes by default many of the workflow/ergonomic enhancements I've built on to GNU-make over time. This makes just feel like coming home.

The one thing I've looked for and not seen is GNU-make style includes, though it seems like there are a few related discussions.

---

I'd suggest that the ideas of modules and namespace could be made more independent from each other. Both are useful but could perhaps be implemented with some optional interdependence.

In fact, the ability to compose recipes from modular Makefiles based on path is one of most successful and comfortable patterns I've seen emerge.

For my use-cases I've taken to adding an -include .assets/makefile/*.mk to a minimal Makefile in the project root, with glob-included modular makefiles, each with prefixed target names according to their modules as a simple form of namespacing / collision avoidance.

It's very useful to be able to conceptually organize content while avoiding the disadvantages of recursive calls. It helps to keep the size of each individual file to a reasonable minimum, as well as making it easy to disable a module by renaming it from .mk to .mk.disabled.

I trust the course of this issue and just wanted to put forward the merits of this approach since it's something that's worked well for me that I can happily recommend.