alternative shell discussion

xynydev commented 7 months ago

I really like the fact that our modules are mostly shell scripts, as that is the most straightforward way of automating customizations on any Linux system. Even solutions that are not shell-script native, such as Vib with it's Go-based modules, end up frequently just generating RUN statements for Containerfiles.

On the other hand, bash is just fine. Most Linux users know a bit of bash, but not many know enough to fluently read and write all quirky syntaxes, bugs, and features that end up being needed in larger projects like this.

With that, I think it would be worth it to explore the prospects of using another (shell scripting) language for writing (at least some) of our modules.

There are many options for alternative shell scripting languages, but of them all, IMO the most promising is Nushell. It's typed, gives great errors, is very powerful and has very useful and readable helpers built in, and the syntax is clean and modern. Nushell supports reading many data types out of the box, including YAML and JSON, and manipulation of data is made elegant through pipelines.

I think it would be worth it to approach this topic through the following questions:

What would a module written in Nushell look like?
- Would it be more readable or easier to maintain?
- Would it be enough of an improvement over Bash?
- This is where we can get our hands dirty.
How would Nushell be included into our images / image builds?
- There isn't any official container as there is with mikefarah/yq, but it wouldn't be that hard for us to maintain one.
- Another option would be hardcoding the Github download link in generated Containerfiles.
- There is an official Fedora RPM package, but I would avoid non-OS-agnostic methods of installing a program required for the build process. For example, no packages exist yet for Debian/VanillaOS AFAIK.
Can justify or circumvent adding a large, mandatory binary to all images?
- The Nushell .tar.gz release is almost 40MB, while yq binaries are only around 3 MB gzipped.
- Adding Nushell to all images would make them way bigger, while not providing much better of an experience for end users.
- In my view this is the biggest decision to make here. We can't really make the inclusion of Nushell optional if some modules are written in it.
- One way to do this more cleanly could be to bind mount a container with the binary with each module's RUN statement, but that would still require the binary to be fetched each time the custom image is built.

Prior art:

All Atomic Studio build scripts are Nushell https://github.com/atomic-studio-org/Atomic-Studio/tree/main/config/scripts

tulilirockz commented 7 months ago

Honestly I really like nushell just because it lets me depend entirely on it, instead of using a bunch of programs and piping stuff through them, like, there is a http builtin that just straight up replaces curl for me, all the nushell primitives just replace jq, yq, or any-other-q, well, idk, its really consistent. It DOES have a bunch of breaking changes for now tho, as it is still in 0.92, but it will get better with time if anyone wants to adopt it in the future.

jonerrr commented 7 months ago

I think it would be a good idea to write a test module in nushell and see how it goes. Maybe the flatpak installer because it currently needs to be rewritten to support multiple repos. As for how it's packaged, I think its best to maintain a container image. Would it be possible to create a Github action that builds a new image when Nushell creates a Github release?

xynydev commented 7 months ago

Yes, it would be possible to make a GitHub Action for that. We should do some test refactorings first, though, and discuss the 40MB. I don't know how much time it would add in the GitHub builders, if it was just a bind-mounted program like the configs are currently. I guess the module repository container image could include a folder for binaries required to run the modules? Maybe we could split to core and extra modules, where the core modules wouldn't require Nushell, and would be simple essential modules, while extra modules would require it and be more complicated.

fiftydinar commented 7 months ago

Honestly I really like nushell just because it lets me depend entirely on it.

I also like how nushell centralizes a lot of currently used tools, to make things easier to maintain & troubleshoot in the future.

As you also said, it has a lot of breaking changes, so bringing it into BlueBuild should be done when the language is stabilized & when it's more thoroughly + successfully used in other projects.

I currently don't know how to write it, but I will definitely learn it in the future!

xynydev commented 7 months ago

Yeah, this is certainly not urgent, and we can wait until Nushell 1.0 to do more exploration.

xynydev commented 6 months ago

Just found out about https://elv.sh/, which also seems like a contender. Should compare that with nushell. Don't know if anything beats nushells data model, but elvish has a smaller binary.

Elvish also has built-in paraller processing and json loading, and even markdown printing which nu doesn't have.

It's also still pre-1.0, though.

https://www.reddit.com/r/Nushell/comments/10fnbt7/how_nushell_differs_from_elvish/

https://news.ycombinator.com/item?id=40316010

fiftydinar commented 6 months ago

I find Oils (formerly OilShell) interesting:

https://www.oilshell.org/

I like how it is perfectly compatible with bash & POSIX scripts when using OSH shell mode. I just changed #!/usr/bin/env bash to #!/usr/bin/env osh in default-flatpaks bash binaries & they work perfectly!

The difference in operation between OSH & bash is that OSH gives you clearer error & debug messages. There are probably more slight differences, as I don't know it in detail & I discovered it some days ago, but this feature by itself is amazing. So I believe that Oils in OSH shell mode can be a direct, minor update from Bash, without throwing Bash knowledge away.

YSH shell mode is a bit more radical Bash successor, without it being fully different, like some shell languages are. This one incorporates very nice improvements, like proper error log handling, which avoids occasional bad seterr behavior seen in bash: https://www.oilshell.org/release/latest/doc/error-handling.html

This is just 1 improvement, there are lots of it, which you can find in Oils docs.

Those are just initial things that I discovered about it, there is more to learn.

Developer's take on nushell vs oils (old, may still be true for some things in nushell): https://www.reddit.com/r/ProgrammingLanguages/comments/ddhmj9/comment/f2htup3/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button

If you want to try it, Oils is available in brew: https://formulae.brew.sh/formula/oil#default

gmpinder commented 6 months ago

We could totally support allowing nushell modules. All it would require is mounting the binary in the path. If they have an image we can pull that contains their bin like yq or cosign does, it would be super simple to support.

gmpinder commented 6 months ago

I would just like to avoid installing too many binaries that the user wouldn't necessarily want. Currently that list is yq, cosign, and bluebuild.

xynydev commented 6 months ago

We probably don't need yq, especially if using a language like Elvish or Nushell, which supports YAML ootb. I'd aim to replace yq with the alternative shell. Many current usages of yq are tasks that jq could probably do easily.

xynydev commented 6 months ago

Looking at oil & ysh now, it seems to be more than a slightly better bash, which is how I dismissed it earlier. Particularly looking at the "shell idioms" page gives a pretty good picture of the language.

In the reddit thread linked, this comment got me interested:

Another huge difference [between osh/ysh and nushell] is that Oil is a programming language and AFAIK nushell isn't. It doesn't have functions, loops, if statements, etc. It has an expression language for tables, but it's not a Turing-complete language.

Looking at the nu docs, this doesn't seem to be true anymore, though.

Ysh seems to be doing a lot of things right, but doesn't try to "rethink the shell". One thing that bums me out, though, is that the whole oil project is a jumble of python and c among others. Hopefully it is still faster than Xonsh, which I've heard is really slow.

Ysh, Elvish, and Nushell are still all viable candidates in my mind. Ysh is the most basic of the bunch, while Elvish and Nushell both have a lot of interesting features, like built-ins for different types of data, and speed optimizations with JIT and catching errors before running the script. Nushell also has a testing framework, which seems to be unique to it.

Something interesting Oil has is Hay, but there seems to be multiple other sublanguages and I'm not sure of their usability or integration with the shell language itself.

I like a lot of the philosophy behind Ysh, but the documentation for the language itself is the most lacking compared to the others. The website outlines a lot of the process and thoughts behind it, but actual tutorials and reference pages regarding the language are lesser.

I think a reasonable approach would be to just write the same module in each language, and trying to observe its benefits for the usecase. If there's not enough benefits to outweigh the "cost" of using an alternative shell, we can just keep using bash.

fiftydinar commented 6 months ago

I think a reasonable approach would be to just write the same module in each language, and trying to observe its benefits for the usecase. If there's not enough benefits to outweigh the "cost" of using an alternative shell, we can just keep using bash.

I agree with this idea.

Someday, I can try to rewrite default-flatpaks & unofficial wallpapers module in YSH.

However, I would like to do that when shell language becomes more stable, not now.

xynydev commented 6 months ago

Also, this was posted on the ublue discord: https://amber-lang.com/

Wouldn't need to ship an additional binary for the shell, but would still require using external programs for parsing data. I really like ECMAScript, but am unsure whether this language is mature enough for our needs.

gmpinder commented 6 months ago

Also, this was posted on the ublue discord: https://amber-lang.com/

Wouldn't need to ship an additional binary for the shell, but would still require using external programs for parsing data. I really like ECMAScript, but am unsure whether this language is mature enough for our needs.

This is probably the coolest one I've seen TBH

fiftydinar commented 6 months ago

Also, this was posted on the ublue discord: amber-lang.com

Wouldn't need to ship an additional binary for the shell, but would still require using external programs for parsing data. I really like ECMAScript, but am unsure whether this language is mature enough for our needs.

Looks interesting.

I remember that there are some tries to simplify bash by having a lot of built-in functions, but this goes further by changing syntax too.

I'll definitely follow its development.

xynydev commented 6 months ago

Ok, I tried Amber. Here are my findings:

For a programming language that advertises compiling to shell, it's not really a proper shell.
- Launching external programs requires surrounding the commands in $$
- There are no pipes
- All external commands require error catching blocks or being wrapped in an unsafe block
With all its glorious design features, there isn't really a way to parse the JSON command line argument that modules get, making this language pretty much useless for our needs

It feels like Amber was made for webdevs afraid of bash needing to write a simple "build.sh" or "install.sh".

Amber might be useful like NodeJS would be useful for writing scripts, with the additional feature of being able to run anywhere with Bash.

CarrotManMatt commented 6 months ago

I've also spent quite a bit of time using Nushell and would advocate for it's use as the way to write blue-build modules, over bash.

xynydev commented 6 months ago

Okay... I've been spending today on learning the basics of multiple alternative shells while writing a simple script in each one.

I chose to build an oversimplified version of default-flatpaks (specifically one that takes in this syntax, because the module needs a rewrite anyways, and I felt that this case works well for seeing some of the most important parts of each shell regarding our use case (parsing the JSON configuration passed to the script, launching external programs based on said configuration). Additionally, testing Flatpak commands locally is pretty straigthforward and non-destructive.

I tried to make the scripts as similar as possible to make syntax comparison easier, which worked well, as none of the shells I tried were radically different. (except Amber, which I dismissed in my earlier message, of course) The scripts wouldn't work as BlueBuild modules directly, since they call flatpak commands in the main script, which would have to be done in a separate script in the real world.

The examples are structured such that before the # ------- is code that could run in the build, which would output a configuration file to /usr/share/bluebuild/.e build, which would output a configuration file to /usr/share/bluebuild/. The code after that line is what could run after boot to set up the remotes / repos and Flatpaks. Both of those are of course very simplified for the sake of local testability and readability.

The notify: option does nothing in these examples. It could be used to conditionally do notify-sends, but I left that out for now.

Elvish

Overview:

Code mostly makes sense and is readable, though has some quirks
Syntax is lispy with some C-like elements
- The (+ 1 2) syntax might seem foreign to many contributors
Feature-set is between Bash and Nushell, basically everything we need exists
Binary size: 8.8M
Reasonable documentation website without search (googling with site:elv.sh was required)
- Have to hunt for some basic things before learning them, basically no discussions online
Good editor support in VSCode
JSON builtin

Specific notes:

Declaring "maps" with a syntax reminiscent of query-strings seems like a very odd choice
- Example from frontpage: [&name=a &cmd='apt update']
Quotes are basically never needed for strings, which is interesting but feels weird coming from bash
Separate equality operators for strings and numbers, and a third one for any two values of the same type; why?

`default-flatpaks.elv`

var default-config = '
{
    "notify": true,
    "scope": "system",
    "repo": {
        "url": "https://dl.flathub.org/repo/flathub.flatpakrepo",
        "name": "flathub",
        "title": "Flathub"
    },
    "install": []
}
'
var config = (echo $default-config $args[0] | jq -s add | from-json) # takes default config, overrides with user configsn)

echo (put $config | to-json) # debug

# -------

# add repo
echo Adding remote: $config[repo][title]
flatpak remote-add --if-not-exists $config[repo][name] $config[repo][url] --title $config[repo][title] --$config[scope]

# install all flatpaks
if (> (count $config[install]) 0) {
    echo Started install of $config[scope] Flatpaks from $config[repo][title]
    var install = $config[install]
    flatpak install --noninteractive --$config[scope] $config[repo][name] $@install # `$@` spreads the list
    echo Finished install of $config[scope] Flatpaks from $config[repo][title]
} else {
    echo "No Flatpaks to install..."
}

$ elvish elvish/default-flatpaks.elv '{"notify":true,"scope":"user","install":["org.gnome.Loupe","org.kde.kdenlive"]}'
{"repo":{"name":"flathub","title":"Flathub","url":"https://dl.flathub.org/repo/flathub.flatpakrepo"},"install":["org.gnome.Loupe","org.kde.kdenlive"],"notify":true,"scope":"user"}
Adding remote: Flathub
Started install of user Flatpaks from Flathub
Skipping: org.gnome.Loupe/x86_64/stable is already installed
Skipping: org.kde.kdenlive/x86_64/stable is already installed
Finished install of user Flatpaks from Flathub

$ elvish elvish/default-flatpaks.elv '{"notify":true,"scope":"user"}'
{"repo":{"name":"flathub","title":"Flathub","url":"https://dl.flathub.org/repo/flathub.flatpakrepo"},"install":[],"notify":true,"scope":"user"}
Adding remote: Flathub
No Flatpaks to install...

$ elvish elvish/default-flatpaks.elv invalid
jq: parse error: Invalid numeric literal at line 13, column 0
Exception: jq exited with 5
  /var/home/e/Zoding/bluebuild/shell-testing/elvish/default-flatpaks.elv:13:47-56: var config = (echo $default-config $args[0] | jq -s add | from-json) # takes default config, overrides with user configsn)

Nushell

Overview:

Code is very readable and makes sense, there is even a type system
Syntax is very C-like or even JS-like, with elements familiar to bash users
Feature-set is unbeatable
- Basically more features than we'll ever need
Binary size: 88M
- 10x from Elvish, mostly due to the huge amount of features
- This is my main concern, since including the shell in the final custom image would be really useful for some modules
Very good documentation website with a search, extensive content, and active community
Very good editor support in VSCode
YAML, JSON, etc. builtins

Specific notes:

Have to define a main function when taking arguments, which is pretty nice when there's many of them
Supports merging "records", which makes overriding a default config with the user configs a breeze
Very good error messages
There is talk of instability / frequent breaking changes, but with us using only a subset of Nushell features, maybe that wouldn't affect us that much(?)

`default-flatpaks.nu`

const defaultConfig = {
    notify: true
    scope: system
    repo: {
        url: "https://dl.flathub.org/repo/flathub.flatpakrepo"
        name: "flathub"
        title: "Flathub"
    }
    install: []
}

def main [configStr: string] {
    let config = $defaultConfig | merge ($configStr | from json) # takes default config, overrides with user configs

    print ($config | to json) # debug

    # -------

    # add repo
    print $"Adding remote: ($config | get repo.title)"
    flatpak remote-add --if-not-exists ($config | get repo.name) ($config | get repo.url) --title ($config | get repo.title) $"--($config | get scope)"

    # install all flatpaks
    if ($config | get install | length) > 0 {
        print $"Started install of ($config | get scope) Flatpaks from ($config | get repo.title)"
        flatpak install --noninteractive $"--($config | get scope)" ($config | get repo.name) ...($config | get install) # `...` spreads the list
        print $"Finished install of ($config | get scope) Flatpaks from ($config | get repo.title)"
    } else {
        print "No Flatpaks to install..."
    }
}

$ nu nushell/default-flatpaks.nu '{"notify":true,"scope":"user","install":["org.gnome.Loupe","org.kde.kdenlive"]}'
{
  "notify": true,
  "scope": "user",
  "repo":
  {
    "url": "https://dl.flathub.org/repo/flathub.flatpakrepo",
    "name": "flathub",
    "title": "Flathub"
  },
  "install":
  [
    "org.gnome.Loupe",
    "org.kde.kdenlive"
  ]
}
Adding remote: Flathub
Started install of user Flatpaks from Flathub
Skipping: org.gnome.Loupe/x86_64/stable is already installed
Skipping: org.kde.kdenlive/x86_64/stable is already installed
Finished install of user Flatpaks from Flathub

$ nu nushell/default-flatpaks.nu '{"notify":true,"scope":"user"}'
{
  "notify": true,
  "scope": "user",
  "repo":
  {
    "url": "https://dl.flathub.org/repo/flathub.flatpakrepo",
    "name": "flathub",
    "title": "Flathub"
  },
  "install": []
}
Adding remote: Flathub
No Flatpaks to install...

$ nu nushell/default-flatpaks.nu invalid
Error: nu::shell::pipeline_mismatch

  × Pipeline mismatch.
    ╭─[/var/home/e/Zoding/bluebuild/shell-testing/nushell/default-flatpaks.nu:13:18]
 12 │ def main [configStr: string] {
 13 │     let config = $defaultConfig | merge ($configStr | from json) # takes default config, overrides with user configs
    ·                  ───────┬──────   ──┬──
    ·                         │           ╰── expected: input, and argument, to be both record or both table
    ·                         ╰── value originates from here
 14 │     
    ╰────

Oilshell / Ysh

Overview:

Code feels very much like a Bash Improved with bash quirks cleared up and new features bolted on in the same framework
C-like "procs" and other code blocks
Feature-set similar to Elvish, though maybe a bit smaller
- Finding the json built-in, not to mention of the three "sublanguages" surprised me
Binary size: 2M
- This is pretty good taking into account that the same binary includes the bash-compatible OSH and the incrementally improving YSH
- Also LOC is apparently pretty small for the scope
Convoluted documentation website; hard to find the page you're looking for
- Centerpiece: A Tour of YSH
- But there's also a GitHub Wiki
- Googling with site:oilshell.org is definitely required for finding documentation of specific features, but that still might not lead to good results
Mediocre / bad editor support in VSCode; outdated syntax highlighting, no extra features
JSON & JSON8 builtins

Specific notes:

Something tells me this project is led by one passionate Unix Philosopher who likes to code and think about code
- There's many lighlty documented features and sublanguages, but no effective onboarding
There's actually current & mostly complete versions both in Python and as binaries from C++ (which is what I used here)
Has to be compiled from source, but that is not an obstacle
Adds a new syntax for working with "dicts": $[...], for which I cannot find much documentation for, which is also the only place for calling some builtins such as len() for getting the length of a list, and integrating it with other shell features remains unclear to me
JSON cannot be read into a "dict" in a normal variable declaration, instead json read is kind of passed a reference to a (non-existant) variable which it creates: json read (&config) -> creates config

`default-flatpaks.ysh`

const default_config = """
{
    "notify": true,
    "scope": "system",
    "repo": {
        "url": "https://dl.flathub.org/repo/flathub.flatpakrepo",
        "name": "flathub",
        "title": "Flathub"
    },
    "install": []
}
""" 

echo $default_config $1 | jq -s add | json read (&config) # takes default config, overrides with user configs

= config # debug, `=` used to pretty-print

# -------

# add repo
echo "Adding remote: $[config.repo.title]"
flatpak remote-add --if-not-exists $[config.repo.name] $[config.repo.url] --title $[config.repo.title] --$[config.scope]

# install all flatpaks
if ("$[len(config.install)]" !== "0") { # only hack I could make work for checking if the array is empty
    echo "Started install of $[config.scope] Flatpaks from $[config.repo.title]"
    flatpak install --noninteractive --$[config.scope] $[config.repo.name] @[config.install] # `@` spreads the list
    echo "Finished install of $[config.scope] Flatpaks from $[config.repo.title]"
} else {
    echo "No Flatpaks to install..."
}

$ ysh ./ysh/default-flatpaks.ysh '{"notify":true,"scope":"user","install":["org.gnome.Loupe","org.kde.kdenlive"]}'
(Dict 0x2719)   {"notify":true,"scope":"user","repo":{"url":"https://dl.flathub.org/repo/flathub.flatpakrepo","name":"flathub","title":"Flathub"},"install":["org.gnome.Loupe","org.kde.kdenlive"]}
Adding remote: Flathub
Started install of user Flatpaks from Flathub
Skipping: org.gnome.Loupe/x86_64/stable is already installed
Skipping: org.kde.kdenlive/x86_64/stable is already installed
Finished install of user Flatpaks from Flathub

$ ysh ./ysh/default-flatpaks.ysh '{"notify":true,"scope":"user"}'
(Dict 0x2721)   {"notify":true,"scope":"user","repo":{"url":"https://dl.flathub.org/repo/flathub.flatpakrepo","name":"flathub","title":"Flathub"},"install":[]}
Adding remote: Flathub
No Flatpaks to install...

$ ysh ./ysh/default-flatpaks.ysh invalid
jq: parse error: Invalid numeric literal at line 12, column 0
  echo $default_config $1 | jq -s add | json read (&config) # takes default config, overrides with user configs
                                             ^~~~
./ysh/default-flatpaks.ysh:14: json read: Unexpected EOF while parsing JSON (pos 0-0: '')
  echo $default_config $1 | jq -s add | json read (&config) # takes default config, overrides with user configs
                                        ^~~~
./ysh/default-flatpaks.ysh:14: errexit PID 58194: command.Simple failed with status 1
  echo $default_config $1 | jq -s add | json read (&config) # takes default config, overrides with user configs
                                        ^~~~
./ysh/default-flatpaks.ysh:14: errexit PID 58194: command.Pipeline failed with status 1

xynydev commented 6 months ago

pt. 2, not serious contenders, but I still wrote the same script with them

`default-flatpaks.bash`

writing this seriously made all the other shells look really good
shellcheck-approved

DEFAULT_CONFIG=$(
cat << EOF
{
    "notify": true,
    "scope": "system",
    "repo": {
        "url": "https://dl.flathub.org/repo/flathub.flatpakrepo",
        "name": "flathub",
        "title": "Flathub"
    },
    "install": []
}
EOF
)

CONFIG=$(echo "$DEFAULT_CONFIG" "$1" | jq -s add)

echo "$CONFIG" | jq -r . # pretty-print with `jq`

# -------

# add repo
echo "Adding remote: $(echo "$CONFIG" | jq -r .repo.title)"
flatpak remote-add --if-not-exists "$(echo "$CONFIG" | jq -r .repo.name)" "$(echo "$CONFIG" | jq -r .repo.url)" --title "$(echo "$CONFIG" | jq -r .repo.title)" "--$(echo "$CONFIG" | jq -r .scope)"

# install all flatpaks
if [[ "$(echo "$CONFIG" | jq -r '.install | length')" -gt 0 ]]; then
    echo "Started install of $(echo "$CONFIG" | jq -r .scope) Flatpaks from $(echo "$CONFIG" | jq -r .repo.title)"
    # shellcheck disable=SC2046
    flatpak install --noninteractive "--$(echo "$CONFIG" | jq -r .scope)" "$(echo "$CONFIG" | jq -r .repo.name)" $(echo "$CONFIG" | jq -r '.install | join(" ")') # intentional string splitting
    echo "Finished install of $(echo "$CONFIG" | jq -r .scope) Flatpaks from $(echo "$CONFIG" | jq -r .repo.title)"
else
    echo "No Flatpaks to install..."
fi

$ bash bash/default-flatpaks.bash '{"notify":true,"scope":"user","install":["org.gnome.Loupe","org.kde.kdenlive"]}'
{
  "notify": true,
  "scope": "user",
  "repo": {
    "url": "https://dl.flathub.org/repo/flathub.flatpakrepo",
    "name": "flathub",
    "title": "Flathub"
  },
  "install": [
    "org.gnome.Loupe",
    "org.kde.kdenlive"
  ]
}
Adding remote: Flathub
Started install of user Flatpaks from Flathub
Skipping: org.gnome.Loupe/x86_64/stable is already installed
Skipping: org.kde.kdenlive/x86_64/stable is already installed
Finished install of user Flatpaks from Flathub

$ bash bash/default-flatpaks.bash '{"notify":true,"scope":"user"}'
{
  "notify": true,
  "scope": "user",
  "repo": {
    "url": "https://dl.flathub.org/repo/flathub.flatpakrepo",
    "name": "flathub",
    "title": "Flathub"
  },
  "install": []
}
Adding remote: Flathub
No Flatpaks to install...

$ bash bash/default-flatpaks.bash invalid
jq: parse error: Invalid numeric literal at line 11, column 0
Adding remote: 
error: Flatpak system operation ConfigureRemote not allowed for user
No Flatpaks to install...
# comical fail, asks for my password to do whatever the fuck

`default-flatpaks.koi`

Köi is a minimalistic language, mainly aimed at shell scripting. https://koi-lang.dev/
writing this seriously made all the other shells look worse
sadly an infrequently updated single-maintainer project with sub-100 github stars, minimalist syntax and minimalist documentation, and incomprehensible errors

let defaultConfig = {
    notify: true
    scope: "system"
    repo: {
        url: "https://dl.flathub.org/repo/flathub.flatpakrepo"
        name: "flathub"
        title: "Flathub"
    }
    install: []
}

let config = defaultConfig + args[0].parseJson() # takes default config, overrides with user configs

print(config.toJson()) # debug

# -------

# add repo
print("Adding remote: {config.repo.title}")
flatpak remote-add --if-not-exists {config.repo.name} {config.repo.url} --title {config.repo.title} --{config.scope}

# install all flatpaks
if (config.install.len() > 0) {
    print("Started install of {config.scope} Flatpaks from {config.repo.title}")
    flatpak install --noninteractive --{config.scope} {config.repo.name} {config.install}
    print("Finished install of {config.scope} Flatpaks from {config.repo.title}")
} else {
    print("No Flatpaks to install...")
}

$ ./koi/koi ./koi/default-flatpaks.koi -- '{"notify":true,"scope":"user","install":["org.gnome.Loupe","org.kde.kdenlive"]}'
{"install":["org.gnome.Loupe","org.kde.kdenlive"],"notify":true,"repo":{"name":"flathub","title":"Flathub","url":"https://dl.flathub.org/repo/flathub.flatpakrepo"},"scope":"user"}
Adding remote: Flathub
Started install of user Flatpaks from Flathub
Skipping: org.gnome.Loupe/x86_64/stable is already installed
Skipping: org.kde.kdenlive/x86_64/stable is already installed
Finished install of user Flatpaks from Flathub

$ ./koi/koi ./koi/default-flatpaks.koi -- '{"notify":true,"scope":"user"}'
{"install":[],"notify":true,"repo":{"name":"flathub","title":"Flathub","url":"https://dl.flathub.org/repo/flathub.flatpakrepo"},"scope":"user"}
Adding remote: Flathub
No Flatpaks to install...

$ ./koi/koi ./koi/default-flatpaks.koi -- invalid
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Error("expected value", line: 1, column: 1)', src/interp/native.rs:108:47
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

`default-flatpaks.mjs`

javascript,,, for shell scripting? yes. https://github.com/google/zx
it's not bad
as web dev, i actually like it
the node binary is 111M, but i think that's installed by default either way, and zx on top of that is just ~900kb, that's basically for free
not actually a shell, uses bash under the hood
but actually a real programming language, access to fetch() apis etc
execution time is almost double that from real shells, probably due to slow startup'
- but can be improved by using bun, another 100M binary, lol not worth it

#!/usr/bin/npx zx

const defaultConfig = {
  notify: true,
  scope: "system",
  repo: {
      url: "https://dl.flathub.org/repo/flathub.flatpakrepo",
      name: "flathub",
      title: "Flathub"
  },
  install: []
}

const config = {...defaultConfig, ...JSON.parse(process.argv[3])} // takes default config, overrides with user configs (which are for some reason argv 3)

console.log(config) // debug

// -------

// add repo
console.log("Adding remote: " + config.repo.title)
await $`flatpak remote-add --user --if-not-exists ${config.repo.name} ${config.repo.url} --title ${config.repo.title} --${config.scope}`

// install all flatpaks
if (config.install.length > 0) {
    console.log(`Started install of ${config.scope} Flatpaks from ${config.repo.title}`)
    echo(await $`flatpak install --noninteractive --${config.scope} ${config.repo.name} ${config.install}`)
    console.log(`Finished install of ${config.scope} Flatpaks from ${config.repo.title}`)
} else {
    console.log("No Flatpaks to install...")
}

$ ./default-flatpaks.mjs '{"notify":true,"scope":"user","install":["org.gnome.Loupe","org.kde.kdenlive"]}'
{
  notify: true,
  scope: 'user',
  repo: {
    url: 'https://dl.flathub.org/repo/flathub.flatpakrepo',
    name: 'flathub',
    title: 'Flathub'
  },
  install: [ 'org.gnome.Loupe', 'org.kde.kdenlive' ]
}
Adding remote: Flathub
Started install of user Flatpaks from Flathub
Skipping: org.gnome.Loupe/x86_64/stable is already installed
Skipping: org.kde.kdenlive/x86_64/stable is already installed
Skipping: org.gnome.Loupe/x86_64/stable is already installed
Skipping: org.kde.kdenlive/x86_64/stable is already installed
Finished install of user Flatpaks from Flathub

$ ./default-flatpaks.mjs '{"notify":true,"scope":"user"}'
{
  notify: true,
  scope: 'user',
  repo: {
    url: 'https://dl.flathub.org/repo/flathub.flatpakrepo',
    name: 'flathub',
    title: 'Flathub'
  },
  install: []
}
Adding remote: Flathub
No Flatpaks to install...

$ ./default-flatpaks.mjs invalid
SyntaxError: Unexpected token 'i', "invalid" is not valid JSON
    at JSON.parse (<anonymous>)
    at file:///var/home/e/Zoding/bluebuild/shell-testing/node-zx/default-flatpaks.mjs:14:43
    at ModuleJob.run (node:internal/modules/esm/module_job:222:25)
    at async ModuleLoader.import (node:internal/modules/esm/loader:323:24)

xynydev commented 6 months ago

Thoughts:

I think YSH and Elvish are great, but
- They have some rough edges or warts
- The lacking documentation and smaller community makes them harder to use
Based on DX, I would actually only evaluate between the obvious choice, Nushell, and the inobvious choice, NodeJS + zx
- The feature-sets could be thought to be similar, though Nushell is definitely nicer as a shell
- JS is untyped (but can be extended with TS), while being more widely known
- JS is a real programming language, so more is possible, while specific things are harder
  - JS is also somewhat hated by some
- Nushell is smaller than NodeJS, but NodeJS would probably be included by default, making its bundle footprint smaller in the end
Overall, I would lean towards Nushell
- Best community support and prior usage in this space (just scroll up this thread)
- Best DX and programming that feels like programming compared to other shell languages tested
- Good UX too, with fancy errors (custom ones too), ansi builtin for colored output, etc.
- Each update always currently ships 5GB of changes anyways (due to upstream issues / unfinisheds), the Nushell binary would only be ~1-2% of that, and could likely be cached as it'd be added early in the build process
  - We'd have to include Nushell in the image if the intention is to use it in modules that have locally running parts
- Coming full circle, after all this, it feels good to know that Nushell is the best pick here and make an informed decision
- I don't have enough Nushell experience to know of what sorts of breaking changes there are; with a cursory glance in their changelogs the changes seem minor
  - @tulilirockz ?
  - We can always lock to a version and set a reminder to go through the nu changelogs every month or couple months before updating

Next steps; maybe making a nushell-container repo that builds a FROM scratch container with just a specific version of the Nushell binary. Ideally make the builds happen as seldom as possible to make best use of caching in custom image builds.

gmpinder commented 6 months ago

Next steps; maybe making a nushell-container repo that builds a FROM scratch container with just a specific version of the Nushell binary.

Do they not already supply an image with it? I was thinking of adding it like we do with yq or the bluebuild binary. Creating an image wouldn't be too bad, but I think we should use what's out there first

xynydev commented 6 months ago

Do they not already supply an image with it?

Their repo has a Dockerfile with an alpine/musl version, couldn't find the image published anywhere.

xynydev commented 2 months ago

Update: https://github.com/blue-build/nushell-image

blue-build / modules