Compiled JQ in not execution safe for constant mutable structs

[afonsosribeiro]

If the jq code contains a constant map (ex {"key":1}). It will be compiled once and reused, but has it is a mutable struct it can be changd inside the Run, or returned as output and changed by the user.

Solution all mutable structs need to be instantiated at Runtime, never at compile time.

[itchyny]

I can't understand this issue. Can you provide some problematic code?

[afonsosribeiro]

Yes, maybe I also did not explain in the best way.

prog1.jq: {"count" : 1}

Outputs to the same memory location between executions: 0xc0002ccf30 map[count:1] 0xc0002ccf30 map[count:1]

prog2.jq {"count" : 1} | .count += 0

Outputs different memory locations between executions: 0xc0006c2540 map[count:1] 0xc0006c26f0 map[count:1]

I would assume, prog1 have the same behavior as prog2 in the output

[itchyny]

Okay, I'd like to know how you use the library. In what use case the current behavior causes problems?

[afonsosribeiro]

I'm implementing a parallel event processing system using a flow of multiple gojq programs. So the output of one jq program is used as the input of another.

Ex: I compile prog1.jq, execute prog1.jq in paralelel, both executions return a struct in the same memory position, which are then each used as variables in the next jq program of the flow in different go routines. It is safe to call this method in goroutines, to reuse a compiled [*Code]. But for arguments, do not give values sharing same data between goroutines.

That is how I found this situation, but I can see other scenarios where this can also be an issue.

[wader]

@afonsosribeiro same issue as in https://github.com/itchyny/gojq/issues/212 ? which is as i understand it that RunWithContext calls normalizeNumbers on the input which will modify maps and slices

[afonsosribeiro]

@wader I discovered because of it, but they are not the same issue. This issue concerns the return struct of a compiled jq code not the normalizeNumbers aspect of runnng jq code.

[wader]

Ex: I compile prog1.jq, execute prog1.jq in paralelel, both executions return a struct in the same memory position, which are then each used as variables in the next jq program of the flow in different go routines.

Ok but doesn't it end being the same issue with sharing input? or you mean gojq should guarantee to never return a "shared" variable even if it is a constant literal in your jq program etc?

[afonsosribeiro]

@wader No it is not the same issue