Can tasks/jobs be written in another language than bash

[blaggacao]

Similar to just how complicated would it be to extend make's stdenv to other languages...

GNU parallel might not always be the tool of choice & trust for parallelization... :sweat_smile:

[dsalaza4]

Currently makeDerivation only supports Bash. What we do is using Bash as an interface for running other languages (python script.py).

We could add an extra parameter for specifying the binary we want to use. Something to consider is that $out must be at least touched.

@kamadorueda What do you think about this?

[blaggacao]

Follow up question:

Can we explore standardizing the task interface around some sort of "message format" with the end goal of saddling makes jobs onto an orchestrator (such as airflow/zeebe/you-name-it)?

[kamadorueda]

@blaggacao You may want to read this first: https://github.com/fluidattacks/makes/issues/180#issuecomment-915563837

Similar to just how complicated would it be to extend make's stdenv to other languages...

We use bash in the stdenv because Nix, Nixpkgs, and pretty much every DevOps engineer knows bash. It's what the community uses

When you need to trigger other languages, you can simple do $ lang /path/to/my/code and that's it! The best of both worlds

GNU parallel might not always be the tool of choice & trust for parallelization... sweat_smile

100% of the parallelization happens within Nix itself. There is a small utility called makeScriptParallel in the framework (it uses GNU's parallel), but it's not even documented for public use, we don't use it at Fluid Attacks either, we can remove it

[blaggacao]

100% of the parallelization happens within Nix itself. 

I'm thinking of a use case where a baked task can shell out to a (slow) remote APIs and make use of the STATE directory to orchestrate some things from the CLI.

But a bash wrapper is just fine.

[kamadorueda]

I'm thinking of a use case where a baked task can shell out to a (slow) remote APIs and make use of the STATE directory to orchestrate some things from the CLI.

Can you elaborate a little bit on this please?

It really blow my mind to think on Makes triggerring AWS lambdas to execute the script instead of doing it on the same machine with bash, and then collecting results back

I would love to understand well your use case, do you have an example of the expected behaviour?

[blaggacao]

Concretely, I might be thinking of an integration with consul service backends, nomad job state or getting information about aws autosacling clients into my (temporary) local state.

Those are things that some decent parallelization seems worth it to keep CLI UX somewhat responsive. Also SDKs are available in different languages.

Re: lamda functions

This is a very interesting idea. I think in general adding an (optional) messaging layer to makes, would make all things that involve sending task inputs and outputs over the wire a fully native experience. (x-ref my orchestrator idea)

Maybe https://cloudevents.io and "AsyncAPI" are a good starting place for inspiration in this space.

[blaggacao]

I think we can close this. No further action is required from my point of view. I'll open separate issues for the items that pop up again in my journey.

[blaggacao]

I just stumbled yesterday on this RFC: https://github.com/NixOS/rfcs/pull/99.

It looks like switching to osh (as a "rational common subset of bash") is something that we should do right now.

oil, on the other hand, might be worth exploring as an "almost" bash, but proper programming language.