papa parser streamming large files(500mega+), memory error occure.

[niceliberty]

sorry. My English is poor.

data.csv file size maybe 500mega or more.

but memory error occure during load file(IE / Chrome), firefox is not error.

How can I solved...?

so I set "RemoteChunkSize=50242880" but same error occure.

url : http://www.phyzen.com:8080/demo/GBSMarker_gamja/index.html?name=gamja

source like this.

Papa.parse("../data.csv",{ download: true, delimiter: ",", // auto-detect newline: "\n", // auto-detect header: false, dynamicTyping: false, preview: 0, encoding: "", worker: true, comments: false, step: function(result, handle) { if (result.data[0][0]) { data.push(result.data[0]); } }, complete: function() { parseGene(Date.now(), scaffold, data, 0, "", [], [], 0); }, error: function(e) { console.log(e); }, skipEmptyLines: false, chunk: function(results) { rows.concat(rows, results.data); }, fastMode: false });

[mholt]

It's probably because you're saving all the data into an array (data.push) even though the point of streaming is to not do that.

This isn't an issue with Papa Parse, so I'm closing it. Good luck!

[niceliberty]

remove data into array in source, but same error

like this.

Papa.parse("../data.csv",{
                download: true,
                delimiter: "",  // auto-detect
                newline: "",    // auto-detect
                header: false,
                dynamicTyping: false,
                preview: 0,
                encoding: "",
                worker: true,
                comments: false,
                skipEmptyLines: false,
                chunk: undefined,
                fastMode: false
            });

[mholt]

It's because now you're not streaming, so it's the same thing as before, all the data is loaded into memory. You need to not do that.

[niceliberty]

sorry... How can I solve this problem...

[mholt]

Use streaming. But don't store all the results in memory like you were doing.

[niceliberty]

console message like this.

IE : http://file.okky.kr/images/1461198785397.jpg chrome : http://file.okky.kr/images/1461198882978.jpg firefox : http://file.okky.kr/images/1461198887326.jpg

[niceliberty]

this program is display all data in array so all data must into array,

http://file.okky.kr/images/1461217794699.jpg