Cinchoo / ChoETL

ETL framework for .NET (Parser / Writer for CSV, Flat, Xml, JSON, Key-Value, Parquet, Yaml, Avro formatted files)
MIT License
801 stars 134 forks source link

Large Xml to csv is tooo slow(Json to csv) #66

Closed Kunjahamed-P closed 4 years ago

Kunjahamed-P commented 4 years ago

when try with large xml to csv or json to csv it take more time. code is


  StringBuilder csvData = new StringBuilder();
            using (var r = new ChoJSONReader(Json))
            {
                using (var w = new ChoCSVWriter(csvData).WithFirstLineHeader())
                {
                    w.Write(r);
                }
            }

or

            StringBuilder sb = new StringBuilder();
            using (var p = ChoXmlReader.LoadText(doc.InnerXml))
            {
                using (var w = new ChoCSVWriter(sb).WithFirstLineHeader())
                    w.Write(p);
            }
Cinchoo commented 4 years ago

Would be nice if you can post working example with sample input. It help to investigate it. Thanks.

Kunjahamed-P commented 4 years ago

the code is

JObject caqhJson = JObject.Parse(File.ReadAllText(@"path/JSON2CSV/rows.json"));

            StringBuilder csvData = new StringBuilder();
            using (var r = new ChoJSONReader(caqhJson))
            {
                using (var w = new ChoCSVWriter(csvData).WithFirstLineHeader())
                {
                    w.Write(r);
                }
            }

the actual data is cannot be share, u can use these json file https://data.ct.gov/api/views/rybz-nyjw/rows.json?accessType=DOWNLOAD the result is same

Cinchoo commented 4 years ago

found the issue, fixed and pushed new package. Please take the latest and try it.

Thanks for reporting it.

Kunjahamed-P commented 4 years ago

i try json to csv with latest ChoETL.JSON.NETStandard but still issue exist. image

Cinchoo commented 4 years ago

Please try installing prerelease packages. try.

Install-Package ChoETL.NETStandard -Version 1.1.0.5-alpha8
Install-Package ChoETL.JSON.NETStandard -Version 1.1.0.5-alpha8
Kunjahamed-P commented 4 years ago

its working fine, thank you so much. hopefully new stable version will update soon...

neuli1980 commented 4 years ago

found the issue, fixed and pushed new package. Please take the latest and try it.

Thanks for reporting it.

Where is the source for this fix? There is no new commit for this issue.