search

scop / bash-completion

Programmable completion functions for bash
GNU General Public License v2.0
2.86k stars 379 forks source link

Excessive disk writes due to internal bash implementation of here-docs #199

Open dmerge opened 6 years ago

dmerge commented 6 years ago

Bash uses temporary files for here-documents (both << and <<<). See: https://lists.gnu.org/archive/html/bug-bash/2014-10/msg00057.html

If /tmp is not tmpfs, or TMPDIR is not set to tmpfs, this is verifiable by watching the IO_WBYTES column in htop for a given shell while completing directory contents. This value will increment in thousands each time tab is pressed. Over a typical shell session, with a habit to spam tab, this results in multiple GiBs written. Further verification is available using the fatrace command: # fatrace -C bash Using tab completion in another shell will show files of the form /tmp/sh-thd-* being created and deleted by bash.

Setting TMPDIR to tmpfs fixes this, but requires one to be aware of such issue, and doesn't prevent unnecessary temporary files. A solution is to pipe echo where possible, and to use process substitution in place of here-documents/here-strings. E.g. https://github.com/scop/bash-completion/blob/9039d771b63b562276151239c38f760418a9e5f4/bash_completion#L561 could be done < <(echo "$x"), or even remove the loop entirely and replace with toks+=( $x ), which should be safe given IFS is set to newline.

scop commented 6 years ago

The first, and immediate remedy I can offer is "just don't do that. then!" :). The second would be to ask to get this fixed in bash (e.g. using the buffering approach for "small" amounts of data).

As far as replacements go, I don't think I will be looking into anything that uses process substitution, because it doesn't work in posix mode, and toggling posix mode causes weird issues such as ones discussed in #190.

Other than that, I'm willing to look into tested patches, but I suggest starting small on them.

dmerge commented 6 years ago

just don't do that

As far as I can tell, it's unavoidable in some cases. Completing a directory with, say, 10,000 contents will create a temporary file of 10,000 lines with <<< "$x". I don't know if bash devs want this to get "fixed", given other shells appear to implement it as such also (so it's seekable for programs that expect it?).

I suggest starting small on them

I'm not sure about the completions, but for the main script, the only relevant code is the _filedir function. I will look into a PR.