Support built-in shell commands for an improved Windows support

[lunny]

For cross-platform usage, some built-in commands will helpful.

[andreynering]

Hi @lunny,

So, this is something that was discussed before, and I actually wanted to solve this from the beginning, but it requires a considerable amount of work.

You may want to take a look at https://github.com/mvdan/sh/issues/93 and https://github.com/mvdan/sh/issues/97.

In short, it requires someone to implement these built-ins on the https://github.com/mvdan/sh project to make them work on Windows.

It was also discussed before to just use https://github.com/ericlagergren/go-coreutils, but most of these implementations are done on a main package, and thus require work before being able to be imported on https://github.com/mvdan/sh.

If anyone reading this issue is willing to help, let me know. Just make sure to sync with the author on https://github.com/mvdan/sh/issues/93 before starting anything.

[andreynering]

Just found the issue I was looking for: https://github.com/ericlagergren/go-coreutils/issues/114

[ghost]

I wonder if this is also useful ? https://github.com/u-root/u-root

You can inject what commands you want via a template approach too: https://github.com/u-root/u-root/blob/master/templates.go

[ghost]

Or cant we use the ones from Mage ? as a base ?

[andreynering]

Hi @gedw99,

I wonder if this is also useful? https://github.com/u-root/u-root

Hmm... interesting project. It definitely have some core tools implemented. Would still require work, though, since these tools are main package (so, not importable).

Or cant we use the ones from Mage?

Mage fallsback to system tools just like Tusk.

[bigBron]

So, it's 2023 now, and there's no news?

[nathanblair]

I found, as a workaround for the absolute most trivial use case of reading a file, which I would usually reach to cat for, there is more that works in cmd on Windows and should be pretty cross-distro for UNIX-like systems.

e.g.

vars:
  VARIABLE:
    sh: more "{{.FILE_TO_READ_INTO_VARIABLE}}"

and that seems to populate the variable for Windows, macOS, and Linux (an Alpine instance).

Still would love a more stable API, but this is one case that's common that I wouldn't know another easy solution without hacking together a {{if}} {{else}} template.

[JohnHardy]

Hey @andreynering, since it's been a while it might be worth a note in the docs?

For example, in Platform specific tasks and commands we could say:

Writing fully cross platform tasks can still be difficult because not all shells work in the same way (e.g. `cp`, `mv`, `mkdir`, etc are different or have different names).

* [Use a specific shell](https://github.com/go-task/task/issues/892#issuecomment-1278217185) in the task script.
* Use [alternative implementations](https://gnuwin32.sourceforge.net/packages/coreutils.htm) of core utils or a specific shell on Windows like [git bash](https://gitforwindows.org/).
* Use [OS specific](https://taskfile.dev/usage/#os-specific-taskfiles) scripts.
* [Contribute](https://github.com/mvdan/sh/issues/93) to Task!

REF: https://github.com/mvdan/sh/issues/93

[JohnHardy]

Another approach could be to specify the required shell in the taskfile, so workarounds are more intentional and fail gracefully if the task can't execute. Similar to the shopt or platforms control that already exists.

We could also clean up the {{.SHELL}} workaround that makes the tasks hard to read.

vars:
  SHELL: '{{if eq .OS "Windows_NT"}}powershell{{end}}{{if eq .OS "darwin"}}pwsh{{end}}'

  clean:
    cmds:
      - '{{.SHELL}} rm -r ./builds/'
      - '{{.SHELL}} rm -r ./packs/'

Would, for instance, become:

shell:
    windows: powershell
    darwin: pwsh

  clean:
    cmds:
      - rm -r ./builds/
      - rm -r ./packs/

[leaanthony]

If there's support among us for built-in commands, I can help out. By built-in commands, I mean using something like this, instead of flaky shelling:

  clean:
    cmds:
      - rmdir: ./builds/
      - rm: ./packs/file.go
      - cp: ./packs-backup/file.go ./packs/file.go
      - mv: ./backup/manifest.xml ./site/

[JohnHardy]

I personally like it!

I dare say it would be easy enough to find the most commonly used parameters to avoid it becoming a shell replacement: https://discord.com/channels/974121106208354339/1025054680289660989/threads/1028716829981544559

[rondymesquita]

If there's support among us for built-in commands, I can help out. By built-in commands, I mean using something like this, instead of flaky shelling:

  clean:
    cmds:
      - rmdir: ./builds/
      - rm: ./packs/file.go
      - cp: ./packs-backup/file.go ./packs/file.go
      - mv: ./backup/manifest.xml ./site/

I'm not contributor of this project, but I was looking for that. I personaly, like this approach "shell-agnostic" of finding the common uses. Having a full shell seems too overpower and complicated.

With the above approach, we could expand the usage. Like, reading files, parsing JSON, replacing occurrence of strings, writing files and so on. We could create speciallized functions with different purposes: upload files, download. Similarly with github actions.

  read:
    cmds:
      - read: file.json           
        to: foo
      - print: {{.foo}}

[leaanthony]

@pd93 what's the blocker for shell command support like this right now? Is it just time?

[pd93]

Hey @leaanthony, it depends what kind of support we're talking about. If we're discussing mv, cp etc being available as regular shell commands on all systems e.g.

foo:
  cmds:
    - mv ./a ./b

Then this is still blocked by https://github.com/mvdan/sh/issues/93. This is quite a chunk of work though and afaik no-one has had the bandwidth to work on it. The outstanding work would probably be done in u-root which is a dependency of mvdan/sh.

The alternative to this is implementing support for some basic commands directly in Task using a syntax similar to what you suggested above.

foo:
  cmds:
    - mv: ./a ./b

While this is a good short-term solution, I would personally prefer proper shell support for the long-term for following reasons:

• Part of the simplicity of Task is its cross-platform shell and adding 1st party support takes away from this and adds another layer of complexity and something else we have to maintain and support. I can also already see the slippery slope of requests for new/missing commands as the full list of coreutils/builtins is huge.
• Users may also get confused about when they should write shell commands and when they need to use one of the special Task commands. We'd have to maintain a list of what you should use in different places.
• If we do add these commands, we are committing to support them until the next major version as we can't remove APIs without breaking backwards compatibility. If we do get full support for core utils in the shell, I would want to transition to this pretty quickly and everyone who uses the suggested syntax would have to migrate (albeit a simple migration).

Having said all of that, it might be some time until we get shell support, so we might need to consider something in the short term. We could consider making it experimental? Though I don't want users to start depending on an experimental feature - especially one we know we're going to remove.

In summary, the blocker here is resolving the above points and firmly deciding on a path forwards. After that, finding someone with some bandwidth to work on it :P

@andreynering, do you have any thoughts on the above points?

[leaanthony]

Thanks for the quick reply. The linked ticket hasn't been updated in 2 years so I don't think it's going to happen 😅

[andreynering]

I think that the work required to build something like cp:, mv: etc is similar to porting the actual builtins to mvdan/sh as requested on https://github.com/mvdan/sh/issues/93. So, I prefer to do that in the shell interpreter as we originally thought. This would be more future-proof, and more people would benefit from it (other users of the mvdan/sh package).

[W1M0R]

@leaanthony @andreynering The way I'm currently working around this issue, is to install https://github.com/uutils/coreutils on all platforms where I need the cross-platform commands. This works for me, but I realize other environments might be more constrained.