Cinchoo / ChoETL

ETL framework for .NET (Parser / Writer for CSV, Flat, Xml, JSON, Key-Value, Parquet, Yaml, Avro formatted files)
MIT License
758 stars 133 forks source link

Ignore Field Not Working? #245

Open Binary-Samurai opened 1 year ago

Binary-Samurai commented 1 year ago

I have tried to follow the steps in #28 to Ignore empty headers and ignore empty fields/whitepsace in my output. But I am still getting back a JSON that is out of sequence with my headers after ignoring the empty headers.

Code

StringBuilder json = new StringBuilder();
            using (ChoCSVReader<dynamic> r = ChoCSVReader.LoadText(csv)
                .WithDelimiter(",")
                .WithFirstLineHeader()
                .Configure(c => c.FileHeaderConfiguration.IgnoreColumnsWithEmptyHeader = true)
                .Configure(c => c.QuoteAllFields = true)
                .Configure(c => c.NullValue = " "))
            {
                using (ChoJSONWriter<dynamic> w = new ChoJSONWriter(json)
                    .Configure(c => c.NullValueHandling = ChoNullValueHandling.Ignore))
                {
                    w.Write(r);
                }
            }

            return json.ToString();

CSV

Freight Terms,,Shipper Location ID,Shipper First Name,,,,Deliver to Line 1,Deliver to Line 2,Deliver to City,Deliver to State,Deliver to Zip,,,,,Handling Unit Count,Piece Count,,,,,,Product ID,BOL (DN en SAP),Carrier,PO Number,SO Number,PO Date,Batch,Customer Part Number,Shipping Notes,T_Tags,Send Freight bill to,Deletion flag
PREPAID US, ,US24,WOLONG US LRD, , , ,JEMA MOTORS AND AUTOMATION,10827 ELGAR LANE,TOMBALL,TX,77375, , , , ,PC,2, , , , , ,5KS143CWL205,80048901, ,Test AVRT SFTP 1,1000020196,0,31012, ,shipping marks header shipping marks line 1,t tags header t tags line 1,"WOLONG ELECTRIC INDUSTRIAL MOTORS c/o Interlog Services, North America 25 Research Drive Ann Arbor, MI 48103 LOC. 31 ACCT BCJM401754", 
PREPAID US, ,US24,WOLONG US LRD, , , ,JEMA MOTORS AND AUTOMATION,10827 ELGAR LANE,TOMBALL,TX,77375, , , , ,PC,1, , , , , ,5KS143STE109,80048901, ,Test AVRT SFTP 1,1000020196,0,PRT591098, ,shipping marks header shipping marks line 2,t tags header t tags line 2,"WOLONG ELECTRIC INDUSTRIAL MOTORS c/o Interlog Services, North America 25 Research Drive Ann Arbor, MI 48103 LOC. 31 ACCT BCJM401754", 

Output

[
  {
    "Freight Terms": "PREPAID US",
    "Shipper Location ID": "",
    "Shipper First Name": "US24",
    "Deliver to Line 1": "WOLONG US LRD",
    "Deliver to Line 2": "",
    "Deliver to City": "",
    "Deliver to State": "",
    "Deliver to Zip": "JEMA MOTORS AND AUTOMATION",
    "Handling Unit Count": "10827 ELGAR LANE",
    "Piece Count": "TOMBALL",
    "Product ID": "TX",
    "BOL (DN en SAP)": "77375",
    "Carrier": "",
    "PO Number": "",
    "SO Number": "",
    "PO Date": "",
    "Batch": "PC",
    "Customer Part Number": "2",
    "Shipping Notes": "",
    "T_Tags": "",
    "Send Freight bill to": "",
    "Deletion flag": ""
  },
  {
    "Freight Terms": "PREPAID US",
    "Shipper Location ID": "",
    "Shipper First Name": "US24",
    "Deliver to Line 1": "WOLONG US LRD",
    "Deliver to Line 2": "",
    "Deliver to City": "",
    "Deliver to State": "",
    "Deliver to Zip": "JEMA MOTORS AND AUTOMATION",
    "Handling Unit Count": "10827 ELGAR LANE",
    "Piece Count": "TOMBALL",
    "Product ID": "TX",
    "BOL (DN en SAP)": "77375",
    "Carrier": "",
    "PO Number": "",
    "SO Number": "",
    "PO Date": "",
    "Batch": "PC",
    "Customer Part Number": "1",
    "Shipping Notes": "",
    "T_Tags": "",
    "Send Freight bill to": "",
    "Deletion flag": ""
  }
]
Cinchoo commented 1 year ago

It is bug, applied fix. take the v1.2.1.51-beta2 and try it. Let me know.