How can I slurp from code?

[amlwwalker]

I want to replicate

jq -s 'group_by(.["product"]) | map({cid: .[]["product"]......

and

jq -s '.' | jq -S 'sort_by(."publisher-id",....

from code, i.e using gojq as a library. How can I go about this? I can see that gojq knows what slurping is from here https://github.com/itchyny/gojq/blob/main/_gojq however I am not sure how I can call this option from code rather than using gojq as a client?

I don't think that gojq can do

--sort-keys / -S:
Output the fields of each object with the keys in sorted order.

(from jq docs)

right? How would I go about replicating the sort here in code? Thanks

[itchyny]

Where do you read the input values from? Collect them into a slice (an array) and use the value as the argument of gojq.Query.Run function. About the sorting key option, it depends on how you output maps. You can normally use gojq.Marshal, or if you want to implement yourself, collect the map keys and sort them.

[amlwwalker]

hey @itchyny thanks for responding. I am reading a file that each line is a (string) JSON input. I am passing each line one by one to gojq like so

func ParseLine(rule, content string) (string, error){
    query, err := gojq.Parse(rule)
    if err != nil {
        log.Fatalln("error passing rule ", err)
    }
    var input interface{}
    json.Unmarshal([]byte(content), &input)
    inputX, _ := json.Marshal(input)
    fmt.Println(string(inputX))
    iter := query.Run(input) // or query.RunWithContext
    v, ok := iter.Next()
    fmt.Println("v: ", v)
    if !ok {
        return "", errors.New("end of file")
    }
    if err, ok := v.(error); ok {
        log.Fatalln("v is an error ", err)
        return "", err
    }
    b, err := json.Marshal(v)
    return string(b), err
}

where rule is group_by(.["client-id"]) | map({cid: .[]["client-id"], sum: map(.["pred-req"]) | add}) | unique an example input line could be:

{"unix-ns":1666173821005104000,"cluster-id":"dev-client","deployment-group":"","client-id":"four","supplier-id":"supplier","publisher-id":"publisher","media-type":"media-type","device-type":"device-type","product":"MKT","hostname":"HOST","vm-id":"dev-vm","pred-req":1,"comp-bot":0,"comp-not-":1,"comp-revoke":0,"iab-":0,"iab-revoke":0,"prescan":0,"prescan":0,"prescan-revoke":0,"overtime":0,"latency-slo-violations":0,"version":"3.8.0-15-dirty","model":{"version":"test1","active":["default-model","one-model","two-model"]},"taxonomy":{}}

I get a failure with jq if not using slurp -s as well so I am pretty convinced that I need to be able to enable slurping in gojq here.

Sorting I can get to afterwards as I think I understand what you are saying about that, but can't progress until I can emulate the -s option in jq, in code

[itchyny]

Create a JSON decoder from content and decode all the JSONs, collect them to a slice.

package main

import (
    "encoding/json"
    "fmt"
    "io"
    "log"
    "strings"
)

func main() {
    // content contains the entire input, not a line
    content := `{"x":1}
{"x":2}
{"x":3}
`
    dec := json.NewDecoder(strings.NewReader(content))
    var inputs []interface{}
    for {
        var input interface{}
        if err := dec.Decode(&input); err != nil {
            if err == io.EOF {
                break
            }
            log.Fatalln(err)
        }
        inputs = append(inputs, input)
    }
    fmt.Println(inputs)
    // query.Run(inputs)
}

[amlwwalker]

thank you very much for this. I haven't quite understood how to order the output of the jq as you said with gojq.marshal but I ended up doing it with hash keys and maps. I think it works....