kindly / flatterer

Opinionated JSON to CSV/XLSX/SQLITE/PARQUET converter. Flattens JSON fast.
https://flatterer.opendata.coop
MIT License
182 stars 7 forks source link

pushdown to all levels? #63

Closed tooptoop4 closed 6 months ago

tooptoop4 commented 9 months ago

given this json

{
    "id": 1,
    "name":"don",
    "title": "A Game",
    "releaseDate": "2015-01-01",
    "things": [
      {"name":"Xbox","yoid":99, "zs": [{"cd":"a"},{"cd":"b"}]},
      {"name":"Playstation"}
    ],
    "rating": {
      "code": "E",
      "name": "Everyone"
    }
}

would like name of don to be in all 3 csvs (main.csv, things.csv, things_zs.csv) but its only appearing in main.csv would like name of Xbox to be in 2 csvs (things.csv, things_zs.csv) which it is

tried using: flatterer g2.json games_dir -d name

kindly commented 9 months ago

@tooptoop4

For individual JSON like this (not in a list or in json lines), if you want pushdown to work, you will need to add the --json-stream option. This will treat the whole JSON as the thing that is streamed, not just the "things" list above.

so flatterer g2.json games_dir -d name --json-stream should work

kindly commented 6 months ago

The above comment fixes this, at least for data that fits in memory. It is very difficult to do this when streaming objects in arrays, as what data is pushdown, can not be worked out until all the JSON is parsed.