Idea: recast redirections as imperative

[andychu]

I think this is low priority, but in writing this section:

https://www.oilshell.org/blog/2021/12/backlog-language.html#redirects-to-memorize

I remember the case of piping stderr but not stdout:

https://stackoverflow.com/questions/2342826/how-can-i-pipe-stderr-and-not-stdout

command 3>&1 1>&2 2>&3

Note: this is also order dependent too:

command 2>&1 >/dev/null | grep 'something'

So without introducing too much syntax or language, we could use Oil blocks:

redirect {
  dup 3 1  # save
  dup 1 2  # swap
  dup 2 3 

  run ls /

  # at the end of this scope, the dup are restored
}

This probably isn't worth it, but I'm saving it for future conversations

This also obeys some abstraction?

proc swap-stdout-stderr {
  redirect {
    dup 3 1  # save
    dup 1 2  # swap
    dup 2 3 

    run @ARGV
  }
}

# Now use it like this:

swap-stdout-stderr ls /

[andychu]

Related to rejected ideas parse_amp and parse_redir #832 and #841

Those ideas aren't fully baked

And #673 {fd}<file.txt

[andychu]

Though there is an argument against using 3, because it conflicts with other programs. Should be

redirect {
  dup :tmp 1  # like read :line
  dup 1 2
  dup 2 $tmp

  run ls /
}

[orent]

Variables and file descriptors can be generalized into a common abstraction by noting that they are both mappings that a called command inherits from its parent. This mapping can be overridden when calling a command without altering the parent's mapping.

If we use a strawman prefix of '&' for turning these numeric fds into pseudo-variables then redirection would be done by assigning to &0, &1, &2 or overriding their value when calling a command, just like passing a variable value to a command. Their current values can saved by assigning them to a named variable like &SAVEFD or SAVEFD (depending of whether using a separate namespace from variables or sharing the same namespace). Behind the scenes it would obviously have some arbitrary fd number but it is named for the user.

The pseudo variable &1 can be considered to either contain the string value /dev/fd/1 or be unset. No other values are allowed. Assigning to or from it duplicates the fd. When such a value is stored in a named variable (assuming the shared namespace option) it is simply the string /dev/fd/NN. A reference count can be kept of variables matching this pattern and the corresponding fd is closed when it drops to zero (but only if really assigned from an fd in the first place).

[andychu]

Hm I see the first part -- environment variables are inherited, and so is the file descriptor table. (Although shell variables that are not exported aren't inherited.)

I'm not sure I see how the second works, or I suppose I'd be wary of introducing new semantics for the same syntax. What would it look like exactly? How to swap stdout and stderr?

I'd say the concrete reasons to do something with redirects are:

1. If it allows abstraction with Oil procs (shell functions), as mentioned. It does help to be be able to write a proc like swap-stdout-stderr mycommand, although I guess you can sort of do that now. I think you can parameterize it more with an imperative, but maybe that's not useful (?)
2. If it opens up some new patterns, like nonlinear pipelines. #843
   • Process sub gets you there somewhat, like tee >(sort) >(cat) or something, but I think it has limitations. (Those should be analyzed)

[andychu]

Still low priority, but here's an idea:

redirect {
  push '>' 1 out.txt
  push '&' 2 1        # because >& and <& are actually the same!

  ls /tmp
  echo hi
}

As a flexible alternative to:

{ ls /tmp; echo hi; } >out.txt 2>&1

You can also pipe it like

redirect {
  push '>' out.txt   # if 2 args, the default is 1
  push '&' 2 1

  ls /tmp
} | wc -l

The benefits are obviously larger for large examples.

fopen {L}>left.txt {R}> right.txt {
   echo 1 >& $L
   echo 2 >& $R
}

redirect {
  push '>' :L left.txt
  push '>' :R right.txt

  echo 1 >& $L
  echo 2 >& $L
}

---

More examples to port:

https://www.oilshell.org/blog/2021/12/backlog-language.html#redirects-to-memorize

[bar-g]

Just to document a thought: Maybe there could be an oil data structure like an array or dict that could represent redirection state and be printed and mutated with the nicer readable plain ysh syntax.

[bar-g]

The next thought from having a data structure (e.g. a dict) representation for some shell settings, here some command environment setting, might be to also have some configuration block support for it (idiomatic "readable" oil).

redirect {
  stderr = stdout
  stdout = /dev/null
}

Now, how would one apply such a (redirect) configuration to only a block of code? Appending another block with the code?

[bar-g]

This block-application problem let me to think about a possibly nice idea for a genreal feature for repetitive commands and blocks.

Think of something like apply <command>?. The command being applied to the next lines, and with an optionally following block only effective for the blocks environment?

Here an example with above redirect.

apply redirect:
  stderr = stdout
  stdout = /dev/null
{
   write "a lot of error message output from a block of code"
}

Ideally though maybe any command could be applied to mutliple lines, and get limited to an environment of a block. Maybe the same apply could even apply a block to a block

apply {
  cd $logdir
  apply redirect
    stderr = stdout
    stdout = /dev/null
} to {
   while read --line $logfile
   ... write...
}

Or maybe have different apply ... to ... and for repetitive lines something like several <command> or do <command>s (plural), or even just a tailing : sigil?

apply mkdir -p:
  /usr/local/lib/new
  /usr/share/doc/new
  ...
  /also/need/this/longer/path

# an empty line ends the "apply" (indentation is just visual syntax sugar)

[andychu]

Yeah there is a common issue where we need to push / pop like Python context managers. And sometimes you might want to do that in multiple places.

Examples of things that push and pop:

• redirects
• most builtins that take a block
  • shopt { }
  • shvar { }
  • cd { }

I think the real challenge here is how to implement something like this ...

But for right now I'd say it's still low priority. We need more people to use Oil, and if they run into stuff like this, we can consider some generalization.

[glyh]

Before this landed, is there a hacky way to suppress error messages from commands invoked by ysh? For now even if I have a try statement and redirection of error message, failing commands still yields something like:

      rustup toolchain list | grep $tag > /dev/null
                              ^~~~

[bar-g]

Hm, following an idea like the above seems to have lead to "Dataflow variables are spectacularly expressive in concurrent programming" Henri E. Bal , Jennifer G. Steiner , Andrew S. Tanenbaum

https://github.com/nextflow-io/nextflow nextfow.io

[andychu]

@glyh Please file this as a separate issue, and add more detail about what you're doing, and add the full error message. It isn't related to this issue

@bar-g Yes the code snippet on https://nextflow.io/ is very close to being expressible in YSH

This is what we're going for with https://www.oilshell.org/blog/2023/06/ysh-sketches.html

However, most likely it won't be done soon, unless there are more people working on the Oils code