FR: Support jq's tcl-style multiline comments

[emanuele6]

In jq you can write multiline comments like so:

# this is a comment \
this is also part of the comment

This is useful because sh does not support # \ as multiline comment, so you can use this kind of multiline comments to easily write shell commands that are ignored comments for jq; and use them to write complex shebangs; for example:

#!/bin/sh --
# \
exec jq --args -f "$0" -- "$@"

select(.foo | IN($ARGS.positional[]))

Currently, when I want to try running one of my jq scripts with gojq, instead of being able to just edit the exec command to say gojq instead of jq and run the script, or being able to run gojq "${opts[@]}" -f file.jq -- ... without modifying the file; I have to edit the file and comment out the exec line (otherwise it is a syntax error), and then run gojq with the same options that are listed in the comment manually:

#!/bin/sh --
# \
#exec jq --args -f "$0" -- "$@"

select(.foo | IN($ARGS.positional[]))

It would be nice if gojq also supported multiline comments, they would be useful to write shebangs for gojq script, and it would make it more compatible with standard jq.

[itchyny]

Does jq support multiline comments like shell does? How is this behavior useful actually?

 $ cat /tmp/test.jq
#! foo
# comment \
yes this is comment.
42,
# comment \
this is comment or no?
43

 $ jq -n -f /tmp/test.jq
jq: error: syntax error, unexpected IDENT, expecting end of file (Unix shell quoting issues?) at <top-level>, line 3:
this is comment or no?
jq: 1 compile error

[itchyny]

More simply,

 $ cat /tmp/test.jq
# comment \
this is comment or no?
42
 $ jq -n -f /tmp/test.jq
jq: error: syntax error, unexpected IDENT, expecting end of file (Unix shell quoting issues?) at <top-level>, line 2:
this is comment or no?
jq: 1 compile error

[emanuele6]

Does jq support multiline comments like shell does? How is this behavior useful actually?

Shell does not support # \ for multiline comments. Having a scripting language (e.g. Tcl) that supports # \ for multiline comments, while the shell does not support # \ for multiline comments is what is useful. If I want to write a jq script that should be executed with -n --args for example; I cannot use #!/bin/jq -n --args -f as shebang because shebangs only accept one argument after the interpreter path; that would run: /bin/jq '-n --args -f' ./foo.jq arg1 arg2 for ./foo.jq arg1 arg2 and fail. Also, even if I wanted to run the script with just -n, if I used #!/bin/jq -nf, I would not be able add -- to the shebang and users of the script may accedentaly pass an extra option to jq when trying to run the script with a file name that starts with -; ./foo.jq -s runs jq -nf ./foo.jq -s and -s is passed as an option to jq. With # \ you can fix that problem, by writing:

#!/bin/sh --
# \
exec jq -n --args -f "$0" -- "$@"

your jq code here

This script has a #!/bin/sh -- shebang so it will be executed by sh; exec jq -n --args -f "$0" -- "$@" will be part of a comment and ignored by jq, but it will not be part of a comment for sh; so sh will replace itself with that jq command where "$0" is the path to the script, and "$@" are the arguments of the script (and abort if jq is not found). Yet another benefit is that this will search jq in PATH instead of using /bin/jq as an hardcoded path.

---

 $ cat /tmp/test.jq
# comment \
this is comment or no?
42
 $ jq -n -f /tmp/test.jq
jq: error: syntax error, unexpected IDENT, expecting end of file (Unix shell quoting issues?) at <top-level>, line 2:
this is comment or no?
jq: 1 compile error

Mh, interesting. I have looked at the implemenation of this feature in jq; it seems it only supports comment continuation with \ on the first comment (after the #! shebang comment) in -f scripts, and only for one line. I assumed it fully supported line continuation for all comments like in Tcl:

# line 1 \
  line 2 \
  line 3
puts hello

But apparently the implementation of # \ is just weirdly hacked in. https://github.com/jqlang/jq/commit/8f6f28c8d3fd6fb85439add3e812edeb5a887999

[emanuele6]

With https://github.com/jqlang/jq/commit/cca1f7d18f2fa6721952645821ae429a0166d7e4, comments in jq work like in Tcl, so an odd amount of \ at the end of a line in a comment make the next line a comment too.

[itchyny]

Thanks for detailed explanation, gojq now supports this style of comment.

[emanuele6]

Thank you! @itchyny

---

By the way, I have noticed that gojq's comment implementation has a similar bug to jq's implementation before my patch: \r terminates comments. For example:

1 #<cr> + 2

In jq, before my patch, that caused a syntax error, because carriage return is not considered a whitespace character:

$ jq -n $'1 #\r + 2'
jq: error: syntax error, unexpected INVALID_CHARACTER, expecting end of file (Unix shell quoting issues?) at <top-level>, line 1:
 + 2   
jq: 1 compile error

After the patch, a rogue carriage return in the middle of the line is now considered part of the comment as you would expect:

$ ~/.source_code/jq/jq -n $'1 #\r + 2'
1

In gojq, since carriage return is considered a whitespace character, it will continue to read the rest of the line as code:

$ gojq -n $'1 #\r + 2'
3

You probably want to not use \r a comment terminator; also, if you want to support # \ in windows CRLF files (since gojq supports them, while jq does not), you should write code to skip the three character sequence \\, \r, \n.

[itchyny]

I think this is on intention, CR is a valid newline code while this is no longer used widely. But at least Vim supports this (the &fileformat is mac). Not commonly used anymore, but when used (consistently) it can be considered as a newline code.

[emanuele6]

Mh, I see. Still, you probably want to special case \\, \r to also skip the following \n if present or it won't work on CRLF files.

[itchyny]

Fixed now.