Cinchoo / ChoETL

ETL framework for .NET (Parser / Writer for CSV, Flat, Xml, JSON, Key-Value, Parquet, Yaml, Avro formatted files)
MIT License
801 stars 134 forks source link

csv to json including null #165

Open nklincoln opened 2 years ago

nklincoln commented 2 years ago

I'm attempting to perform the reverse of #143

In my case I would like to not include any fields where the resulting value is missing.

using System;
using System.Text;
using ChoETL;

namespace ChincooTester
{

    class Program
    {

        static void Main(string[] args)
        {
            string csv =
                @"Id,name,nestedobject/id,nestedobject/name,nestedarray/0/name, nestedarray/0/city, nestedarray/1/name, nestedarray/200/city
//1,name,2,objName,namelist10,citylist10,namelist11,citylist11
2,name1,3,obj3Nmae,namelist20,citylist20,,citylist21";

            StringBuilder json = new StringBuilder();
            using (var w = new ChoJSONWriter(json).Configure(c => c.DefaultArrayHandling = false))
            {
                using var r = ChoCSVReader.LoadText(csv).WithFirstLineHeader()
                    .Configure(c => c.NestedColumnSeparator = '/')
                    .WithMaxScanRows(1);

                w.Write(r);
            }

            Console.WriteLine(json.ToString());
        }
    }
}

Results in

  {
    "Id": "2",
    "name": "name1",
    "nestedobject": {
      "id": 3,
      "name": "obj3Name"
    },
    "nestedarray": [
      {
        "name": "namelist20",
        "city": "citylist20"
      },
      {
        "name": null
      }
    ]
  }

But experimenting with the possible "IgnoreValueField" options does not seem to prevent the appearance of the null item - am I missing something?

Cinchoo commented 2 years ago

Known issue, not handling the null values at the nested object level. Applied fix. Take ChoETL.JSON.NETStandard v1.2.1.29-beta8, try it.

Couple of ways you can achieve

Method 1:

            string csv =
                @"Id,name,nestedobject/id,nestedobject/name,nestedarray/0/name, nestedarray/0/city, nestedarray/1/name, nestedarray/200/city
1,,2,objName,namelist10,citylist10,namelist11,citylist11
2,name1,3,obj3Nmae,namelist20,citylist20,,citylist21";

            using (var w = new ChoJSONWriter(Console.Out)
                .Configure(c => c.DefaultArrayHandling = false)
                )
            {
                using (var r = ChoCSVReader.LoadText(csv)
                    .WithFirstLineHeader()
                    .Configure(c => c.NestedColumnSeparator = '/')
                    .WithMaxScanRows(1)
                    .IgnoreFieldValueMode(ChoIgnoreFieldValueMode.Any)
                    )
                    w.Write(r);
            }

Sample fiddle: https://dotnetfiddle.net/rQL7V0

Method 2:

            string csv =
                @"Id,name,nestedobject/id,nestedobject/name,nestedarray/0/name, nestedarray/0/city, nestedarray/1/name, nestedarray/200/city
1,,2,objName,namelist10,citylist10,namelist11,citylist11
2,name1,3,obj3Nmae,namelist20,citylist20,,citylist21";

            StringBuilder json = new StringBuilder();
            using (var w = new ChoJSONWriter(json)
                .Configure(c => c.DefaultArrayHandling = false)
                .JsonSerializationSettings(s => s.NullValueHandling = NullValueHandling.Ignore)
                .Configure(c => c.TurnOnAutoDiscoverJsonConverters = true)
                )
            {
                using (var r = ChoCSVReader.LoadText(csv).WithFirstLineHeader()
                    .Configure(c => c.NestedColumnSeparator = '/')
                    .WithMaxScanRows(1)
                    )
                    w.Write(r);
            }

            Console.WriteLine(json.ToString());

Sample fiddle: https://dotnetfiddle.net/EeUiQy

Cinchoo commented 2 years ago

With the latest release, conversion is simplified as below (on the JSON writer end)

            string csv =
                @"Id,name,nestedobject/id,nestedobject/name,nestedarray/0/name, nestedarray/0/city, nestedarray/1/name, nestedarray/200/city
1,,2,objName,namelist10,citylist10,namelist11,citylist11
2,name1,3,obj3Nmae,namelist20,citylist20,,citylist21";

            using (var w = new ChoJSONWriter(Console.Out)
                   .Configure(c => c.DefaultArrayHandling = false)
                   .IgnoreFieldValueMode(ChoIgnoreFieldValueMode.Null)
                )
            {
                using (var r = ChoCSVReader.LoadText(csv).WithFirstLineHeader()
                    .Configure(c => c.NestedColumnSeparator = '/')
                    .WithMaxScanRows(1)
                    )
                    w.Write(r);
            }

Sample fiddle: https://dotnetfiddle.net/FmQUtM