Cinchoo / ChoETL

ETL framework for .NET (Parser / Writer for CSV, Flat, Xml, JSON, Key-Value, Parquet, Yaml, Avro formatted files)
MIT License
767 stars 133 forks source link

Flatten Json Array in Array #206

Open CreaterCalvin opened 2 years ago

CreaterCalvin commented 2 years ago

Hi @Cinchoo , I'm using your lib ChoETL.NETStandard ver 1.2.1.48-beta2 for flattening my json string:

{
    "id": "123456789",
    "TimeRanges": [
    [
      0,
      24
    ],
    [
      1,2
    ]
  ]
  }

My expected output after flattening is

[
  {
    "id": "123456789",
    "TimeRange": [0,1]
  },
  {
    "id": "123456789",
    "TimeRange": [1,2]
  }
]

But the actual output is

[
  {
    "id": "123456789",
    "TimeRanges.0": 0
  },
  {
    "id": "123456789",
    "TimeRanges.0": null
  },
  {
    "id": "123456789",
    "TimeRanges.0": 0
  },
  {
    "id": "123456789",
    "TimeRanges.0": null
  }
]

Config I use for flattening is:

var r = ChoJSONReader.LoadText(json)
               .Configure(c => c.DefaultArrayHandling = false)
               .Configure(c => c.FlattenNode = true)
               .Configure(c => c.UseNestedKeyFormat = true)
               .Configure(c => c.FlattenByNodeName = "TimeRanges")
               .Configure(c => c.NestedKeySeparator = '.')
               .Configure(c => c.NestedColumnSeparator = '.'

Here's the full code Thanks for your interest in the question Looking forward to hearing from you @Cinchoo

MykhailoNikolskyi-TinyCorp commented 8 months ago

Hi. @Cinchoo, for this issue update for ChoJSONRecordReader.FlattenNodeIfOn is required, to look like in parquet writer (without double enumeration). I'll attach file with suggested changes. Looking forward to see fix for this issue :)

ChoJSONRecordReader.zip