search

zemirco / json2csv

Convert json to csv with column titles
http://zemirco.github.io/json2csv
MIT License
2.72k stars 363 forks source link

Invalid String Length Error #87

Closed ankurkaushal closed 8 years ago

ankurkaushal commented 8 years ago

I have been using json2csv for big data records & apparently it seems it can't handle large datasets.

Here's the error I get:

error: uncaughtException: Invalid string length date=Tue Nov 24 2015 21:58:18 GMT+0000 (UTC), pid=818, uid=0, gid=0, cwd=/scripts/latest_activity_engagement_report, execPath=/root/.nvm/versions/node/v4.2.2/bin/node, version=v4.2.2, argv=[/root/.nvm/versions/node/v4.2.2/bin/node, /scripts/latest_activity_engagement_report/index.js], rss=1325998080, heapTotal=1287887104, heapUsed=1265062400, loadavg=[1.00341796875, 0.7333984375, 0.43896484375], uptime=20235596, trace=[column=11, file=/scripts/latest_activity_engagement_report/node_modules/json2csv/lib/json2csv.js, function=null, line=189, method=null, native=false, column=null, file=null, function=Array.forEach, line=null, method=forEach, native=true, column=15, file=/scripts/latest_activity_engagement_report/node_modules/json2csv/lib/json2csv.js, function=createColumnContent, line=144, method=null, native=false, column=15, file=/scripts/latest_activity_engagement_report/node_modules/json2csv/lib/json2csv.js, function=null, line=28, method=null, native=false, column=3, file=/scripts/latest_activity_engagement_report/node_modules/json2csv/lib/json2csv.js, function=checkParams, line=86, method=null, native=false, column=3, file=/scripts/latest_activity_engagement_report/node_modules/json2csv/lib/json2csv.js, function=module.exports, line=22, method=exports, native=false, column=3, file=/scripts/latest_activity_engagement_report/index.js, function=generateCSV, line=307, method=null, native=false, column=13, file=/scripts/latest_activity_engagement_report/index.js, function=null, line=172, method=null, native=false, column=13, file=/scripts/latest_activity_engagement_report/dbHelper.js, function=null, line=51, method=null, native=false, column=12, file=/scripts/latest_activity_engagement_report/node_modules/mongodb/lib/utils.js, function=handleCallback, line=96, method=null, native=false], stack=[RangeError: Invalid string length,     at /scripts/latest_activity_engagement_report/node_modules/json2csv/lib/json2csv.js:189:11,     at Array.forEach (native),     at createColumnContent (/scripts/latest_activity_engagement_report/node_modules/json2csv/lib/json2csv.js:144:15),     at /scripts/latest_activity_engagement_report/node_modules/json2csv/lib/json2csv.js:28:15,     at checkParams (/scripts/latest_activity_engagement_report/node_modules/json2csv/lib/json2csv.js:86:3),     at module.exports (/scripts/latest_activity_engagement_report/node_modules/json2csv/lib/json2csv.js:22:3),     at generateCSV (/scripts/latest_activity_engagement_report/index.js:307:3),     at /scripts/latest_activity_engagement_report/index.js:172:13,     at /scripts/latest_activity_engagement_report/dbHelper.js:51:13,     at handleCallback (/scripts/latest_activity_engagement_report/node_modules/mongodb/lib/utils.js:96:12)]

I am using the latest version i.e. 3.0.1. Any idea what might be causing this?

knownasilya commented 8 years ago

How big is your data set? Any way you could provide a sample?

ankurkaushal commented 8 years ago

Unfortunately I can't provide a sample but the datasets are quite large; some having more than 600,000 records. I actually ended up using inbuilt streams in Node which works perfectly for my company's use case.

knownasilya commented 8 years ago

Have you tried https://github.com/zemirco/json2csv-stream?

ankurkaushal commented 8 years ago

To be honest, I didn't since we were little short on time. Will surely try it in the next project.

knownasilya commented 8 years ago

I'm going to close this, since the streams project is a better use-case for huge data and there is no way for me to reproduce the issue. Maybe one day, we'll merge the two.

narainsagar commented 8 years ago

@ankurkaushal Hi, I was also facing same problem from past last two days.. I tried with many libs and at last I found a great library which handles large dataset problem nicely....

https://www.npmjs.com/package/csvwriter

You must try this.. I have exported upto 5 lacs json objects..

Hope this helps others as well, when they come over this issue.

Cheers, Thanks.

knownasilya commented 8 years ago

Thanks for the link.

fungiboletus commented 6 years ago

Same issue here with a 13GB jsonl file. I'm looking for a command line tool supporting jsonl.

[EDIT] json2csv in go looks like a good alternative.