StringifyObject write "deep" into target file

[mm-gmbd]

I have a large JSON object structured like the following:

v1: {
  key1: {
    ...data...
  },
  key2: {
    ...data...
  },
  ...
}

I cannot JSON.stringify the entire object as the resulting string is too long and I get the String RangeError. I'd like to do something like the following with JSONStream.stringifyObject:

var out = fs.createWriteStream(filename, { flags : 'w' });
var JSONStream = require("JSONStream");
var es = require('event-stream');

es.readable(function (count, next) {

    this.emit('data', ["v1", {}]); //emit base object to begin

    for (var key in data.v1) {
      this.emit('data', ["v1."+key, data.v1[key])
    }

    this.emit('end');
    next();
}).pipe(JSONStream.stringifyObject()).pipe(out);

Essentially, I'd like to write the top level v1 object as an empty object, but then fill in the individual key# properties one by one, because I know that, individually, those will not exceed the Javascript string length limits. However, the resulting file looks like the following:

{
  v1: {}
  ,
  v1.key1: { ...data... }
  ,
  v1.key2: { ...data... }
  ,
  ...
}

Is there a way to specify some sort of option/delimiter that would make JSONStream know to insert the provided key (with something like .'s as delimiters), rather than creating entirely new keys?

[dominictarr]

I think the answer is to use the open,sep,close arguments to stringify https://github.com/dominictarr/JSONStream#jsonstreamstringifyopen-sep-close

but make those be the partially valid json objects, effectively '{"rows": [' +data + ']}' something like that...

[mm-gmbd]

Hmm -- can you provide any more color to this? Using the example in the ticket, I'm still not exactly sure how to use stringify... The docs say:

If you only write one item this will be valid JSON.

If you write many items, you can use a RegExp to split it into valid chunks.

Can you provide an example for using a RegExp for this? Maybe using the example structure I provided in the original question? Thanks for the tool -- I think it has the capacity to be really useful in my application, but the documentation is just a little light.

[cbuteau]

I also would love the answer to this question. Given a gigantic object that JSON.stringify bombs on ....How do you write that object using JSONStream.

I tried the example mm-gmbd provided and it does not even write the right property names.

Given the code below what would I replace it with?

var output = JSON.stringify(final);
fs.writeFileSync(argv.file, output, { options: 'w' });

[hmontazeri]

Could you please update the documentation with an example for using open-sep-close? I don't really get it...

[cbuteau]

I just wrote my own manual stringifier...

with javascripts easy ability to enumerate properties the code was not that complex...it was incredibly slow so I enhanced it by using a class that managed multiple buffers and then pressed it all to one string in the end.

Good luck everyone I am unsubscribing from this thread.

[hmontazeri]

@cbuteau yeah.. did so too yesterday... working fine...