valyala / fastjson

Fast JSON parser and validator for Go. No custom structs, no code generation, no reflection
MIT License
2.3k stars 138 forks source link

How to iterate over `p.Parser` output and store the results? #72

Closed smjure closed 3 years ago

smjure commented 3 years ago

Im new to golang migrating from python. Am trying to convert a code from .py to .go. I have such a json file (measurements.log) with measurements:

[{"ev":"Turbine1","status":"good","sample":62879131982429,"quality":1,"inlet":[264.4,92.3,2589.4],"time":1605102482851,"z":1},{"ev":"Turbine2","status":"good","sample":62879131981000,"quality":1,"inlet":[262.2,98.3,2789.1],"time":1605102482851,"z":1},{"ev":"Turbine3","status":"good","sample":62879131982001,"quality":1,"inlet":[261.4,90.1,2289.4],"time":1605102482851,"z":1}]
[{"ev":"Pump2","status":"good","sample":62879131990000,"quality":2,"inlet":[42.9,99.23,3018.0],"time":1605102482852,"c":3,"outlet":[38.72,5.68,1031.23],"freq":4999}]
[{"ev":"Turbine3","status":"good","sample":62879131990002,"quality":1,"inlet":[254.14,93.33,2609.2],"time":1605102482853,"z":2},{"ev":"Turbine3","status":"good","sample":62879131990100,"quality":1,"inlet":[270.6,96.4,2603.22],"time":1605102482860,"z":1}]
.
.

There's millions of lines each containing up to ~50 json dicts over which I'd like to iterate and store them in results map/struct. I can parse a line, but then do not know how to iterate over each line and store results. I'd like to get sth like this:

results["Turbine1"] containig params: status, sample, quality, time, inlet,...
results["Turbine2"] containig params: status, sample, quality, time, inlet,...
results["Turbine3"] containig params: status, sample, quality, time, inlet,...
results["Pump2"] containig params: status, sample, quality, time, intle,...
.
.

I was playing a lot with encoding/json.Unmarshal and var results []map[string]interface{} but got completely stuck at this point without success. Any help on how to iterate over single line, parse json dicts and store them in appropriate map/struct is greatly appreciated. Here is a minimal working example I came up with so far:

package main

import (
    "bufio"
    // "encoding/json"
    "fmt"
    "os"

    "github.com/valyala/fastjson"
)

func main() {
    f, _ := os.Open("measurements.log")
    scanner := bufio.NewScanner(f)

    // var results []map[string]interface{}
    var p fastjson.Parser

    for scanner.Scan() {
        line := scanner.Text()
        v, _ := p.Parse(line) // fastjson.Parse(line)
        fmt.Println(v)
        fmt.Printf("v[0] = %q\n", v.GetStringBytes("0", "ev"))
        // json.Unmarshal(v, &results)
        fmt.Println()
    }
}