[REQUEST] Teach `echo` to newline-terminate arguments

[Screwtapello]

Feature

Add a newline quoting style to the :echo command, which newline-terminates each argument. The Kakoune command:

echo -to-file /tmp/example.txt -quoting newline foo bar baz

...should produce the same result as the shell command:

printf "%s\n" foo bar baz > /tmp/example.txt

Usecase

Kakoune plugins often need to extract data from Kakoune so it can be processed in some external tool. The easiest way to do this is through command-line arguments ($@) and environment variables ($kak_selection, etc.). However, most Unix implementations reserve a fixed amount of space for arguments and environment variables, while Kakoune buffers can be arbitrarily large, so plugins occasionally crash in confusing hard-to-debug ways when that limit is exceeded.

To help plugins work around that limit, Kakoune added the $kak_command_fifo and $kak_response_fifo variables, along with the -to-file switch to the :echo command. In combination this lets plugins extract arbitrarily large amounts of data from Kakoune. Because Kakoune often works with lists of strings, the :echo command also supports the -quoting switch to change how multiple string arguments are combined into one data stream:

• raw separates each item with an ASCII space (0x20), which is ambiguous if any of the items also contain a space, but is straightforward and very easy to work with in other languages (str.split() in Python, read in POSIX shell, etc.)
• kakoune uses Kakoune's string-quoting style (delimiter doubling), which is not widely supported, but is easy to implement in most general-purpose languages
• shell uses POSIX shell's quoting syntax, since Kakoune plugins depend heavily on shell scripts, and POSIX shell does not make it easy to implement Kakoune's string-quoting style

However, most POSIX tools that deal with lists of strings do not use any of those quoting styles. Tools like sort, uniq, head, tail, comm, cut, xargs, grep, sed, awk, and more all expect a stream of newline-terminated items on standard input. Sure, newline-terminated items can be ambiguous if an item also contains a newline, but there's lots of things in Kakoune guaranteed to not contain newlines (%val{selections_desc} for example). Also, now that Kakoune has added support for echoing directly to stdin of a shell script I think newline-terminating items is going to be even more useful.

The specific usecase I have for this feature was writing a GNU TeXinfo reading plugin for Kakoune, like the :man and :doc plugins in the standard library. I wanted to implement hyperlinks the same way the :doc does, by saving the %val{selection_desc} and target of each hyperlink in a range-specs option, and then having a "follow link" command that compared the current cursor position to each item of the list to find the associated link target. Unfortunately, some TeXinfo files are very large, and the list of hyperlinks does not fit into the argument-and-environment-variable space on my machine.

Because some link targets contain spaces, I can't use raw quoting. I could use kakoune quoting if I wrote this one part of my plugin in (say) Python, but I'd be disappointed to add a Python dependency for this one feature when everything else is plain POSIX shell. I might be able to use shell quoting, but the only way to use it from a shell-script is eval "set -- $args" which I expect would bump into the same "not enough room for command line arguments" problem I had in the first place.

[Screwtapello]

An alternative I just thought of: If you put an expansion inside a double-quoted string, Kakoune will behave inconsistently: expanding selections_desc gives you one item with NUL-separated fields, but expanding an option will give you space-separated fields. If we wanted to give this feature a more useful specification, we could make it multiply the string.

Consider the following option:

declare-option str-list myopt foo bar baz

Expanding it directly, we get three items:

:echo -quoting kakoune %opt{myopt}
> 'foo' 'bar' 'baz'

With string multiplication, if we expand it inside a double-quoted string, we still get three items:

:echo -quoting kakoune "%opt{myopt}"
> 'foo' 'bar' 'baz'

However, we can put other things inside the double-quoted string, that get multiplied too:

:echo -quoting kakoune "Hello, %opt{myopt}!"
> 'Hello, foo!' 'Hello, bar!' 'Hello, baz!'

If there's more than one multiple-item expansion in the string, they get multiplied together:

:echo -quoting kakoune "%opt{myopt}-%opt{myopt}"
> 'foo-foo' 'foo-bar' 'foo-baz' 'bar-foo' 'bar-bar' 'bar-baz' 'baz-foo' 'baz-bar' 'baz-baz'

Then we could have newline-terminated arguments with:

declare-option str newline '
'
echo "%opt{myopt}%opt{newline}"

That would still wind up with spaces between each item, but it should be simple to add quoting style none that just concatenates all the items together.

(this idea comes from the "Concatenation" feature of the Plan9 rc shell)

[Delapouite]

Thanks for this detailed explanation.

As a complementary note, other shells also have similar cartesian product features, like in Bash:

$ echo {1..5}+{a,b,c}
1+a 1+b 1+c 2+a 2+b 2+c 3+a 3+b 3+c 4+a 4+b 4+c 5+a 5+b 5+c

or Fish : https://fishshell.com/docs/current/language.html#combining-lists-cartesian-product

[krobelus]

I might be able to use shell quoting, but the only way to use it from a shell-script is eval "set -- $args" which I expect would bump into the same "not enough room for command line arguments" problem I had in the first place.

bash can do that just fine - though eval set -- $(cat tmp) takes 7 seconds for a 100MB file of 10 million lines. dash OTOH bails out for input beyond 1MB, so it's correct that we need something different.

:echo -quoting kakoune "%opt{myopt}" 'foo' 'bar' 'baz'

yeah it's very surprising that this currently only outputs 'foo'. Printing multi-valued options like this is almost certainly an error.

I like cartesian product expansion. It's the behavior that makes the most sense.