onelson / jq-rs

Rust crate to provide programmatic access to `jq`.
88 stars 9 forks source link

Multiple json input #27

Open MarcAntoine-Arnaud opened 4 years ago

MarcAntoine-Arnaud commented 4 years ago

JQ is able to merge multiple json in one.

Is it possible to define a method like: https://github.com/onelson/jq-rs/blob/master/src/jq.rs#L76

fn process_multiple(&mut self, source_values: Vec<JV>) -> Result<String> {
 ...
}

Is it simple a multiple call of jq_start with different inputs ?

onelson commented 4 years ago

Hi @MarcAntoine-Arnaud - interesting request. So, I suppose what you're looking for is essentially the same thing as jq_rs::compile() but without allocating a string per input.

I think essentially how I wrote the "compile and reuse" pattern is as you describe - calling jq_start for each input.

In jq proper, we're talking about something like:

jq . a.json b.json

just calling it with multiple input files, right?

I haven't done this before, so I wanted to check there's no magic to it.

mathiaHT commented 4 years ago

Hello,

I have the same need. I think the purpose of this issue was to bind the slurp functionality of JQ https://stedolan.github.io/jq/manual/v1.6/#Invokingjq:

Instead of running the filter for each JSON object in the input, read the entire input stream into a large array and run the filter just once.

onelson commented 4 years ago

Ah, maybe I misunderstood. I thought @MarcAntoine-Arnaud was asking about something akin to the --stream flag instead of --slurp. Slurp sounds as though it would have the opposite impact on memory consumption for large inputs.

Maybe there should be an issue for each flag.

mathiaHT commented 4 years ago

Slurp allows to merge all the inputs into one output according to a jq program

Yes, I think it would probably require an issue for each flag

SOF3 commented 5 months ago

Is https://github.com/onelson/jq-rs/blob/5a50b86010377178672266bcafd7bd925ac252c6/src/lib.rs#L177-L180 a consequence of this issue? An empty input yields an empty output because the program is never run for any input, which should not be a special case.