fq hangs when reading from STDIN

[pkoppstein]

What version are you using (fq -v)?

Both:
0.0.10 (darwin amd64)

and:
0.2.0 (darwin amd64)

How was fq installed?

brew install wader/tap/fq

Can you reproduce the problem using the latest release or master branch?

Yes (see above).

What did you do?

#!/bin/bash

# works with jq, gojq but not fq
JQ=fq
< /dev/random tr -cd '0-9' | fold -w 1 | $JQ -Mcnr '

### Generic Utility Functions
# Output: a PRN in range(0;$n) where $n is .
def prn:
  if . == 1 then 0
  else . as $n
  | (($n-1)|tostring|length) as $w
  | [limit($w; inputs)] | join("") | tonumber
  | if . < $n then . else ($n | prn) end
  end;

2 | prn
'

What result did you expect?

Same as for jq and gojq

What did you see instead?

The script hangs.

[wader]

Hey, thanks for the report. I think this boils down to that input behaves a bit differently forfq. Each input will read one full input file and not json fragments like jq, ex yes 1 | fq input also hangs. The reason it behaves like this currently is that some formats does not have any signaling except EOF to know that it ends. In addition fq also does not concatenate all inputs into one stream of bytes, ex for jq a json value might span over multiple files, that would be tricky with fq, ex would fq . a.mp3 b.mp3 be one big mp3 file in some cases? also by default fq probes format per input file.

But that said it might be possible with some refactoring to allow more per format behaviour to get something that is closer to how jq behaves. I haven't given it that much thought yet, so any help thinking about it is appreciated.

Also this should probably be documented somehow.

[wader]

Maybe i will revisit this at some point if decoder can be generators etc. Ok to close this for now?

[pkoppstein]

Ok to close this for now?

Thanks for your consideration. It's your project, so you should do what makes sense for you, though it does seem to me that leaving it open would help keep the issue alive and perhaps invite ideas or even contributions.

[wader]

Yeah make sense, i don't have a strong opinion really about closing issues that are not active, but try to cleanup some from time to time. Let keep it open for now.

btw thanks for all the jq related things your active on, learned a lot from your documentation, code and comments!

[wader]

Note to future me or someone else about is: Turning a decoder into iterators has some issues:

• How to handle sub-format that is a iterator? magically turn into array?
• Incompatible with fill gaps?