Unable to create _metadata

[mupadi]

Summary

Unable to create _metadata Can you combine those 1,2,3,... small json files into one _metadata. or run the program just to create _metadata. Because I made 200,000 more and it looks like the JSON.stringify serialization function seems to have a limit, so could you add a feature to combine those 1,2,3,... files into a file like _metadata.

Until now I haven't been able to solve the problem even using the @bolshoytoster fork. And can you add a program for the _metadata file to be converted into an excel file like .csv or .xls.

Thank you @bolshoytoster

Basic example

RangeError: Invalid string length at JSON.stringify () at startCreating (D:\hashlips_art_engine-main\src\main.js:552:22) at D:\hashlips_art_engine-main\index.js:8:3 at Object. (D:\hashlips_art_engine-main\index.js:9:3) at Module._compile (node:internal/modules/cjs/loader:1155:14) at Object.Module._extensions..js (node:internal/modules/cjs/loader:1209:10) at Module.load (node:internal/modules/cjs/loader:1033:32) at Function.Module._load (node:internal/modules/cjs/loader:868:12) at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:81:12) at node:internal/main/run_main_module:22:47

Motivation

In order to be able to create more artwork and overcome the limitations that JSON.stringify has in creating _metadata.

And for excel to make it easier to collect data and edit _metadata and be able to convert it back into a .json file.

[bolshoytoster]

@mupadi I've just made a fix for this now. You can either redownload src/main.js or just change the line ~552: https://github.com/bolshoytoster/hashlips_art_engine/blob/0f8ef6dc3cb54368d51048ebda1264f56eb04816/src/main.js#L551-L553

to

  writeMetaData(`[${metadataList.map(e => JSON.stringify(e, null, 2)).join(',')}]`);

---

This uses code from this stackoverflow answer, and the output won't be properly formatted (bad indentation) but it should still work.

[mupadi]

still can not and still displays it

RangeError: Invalid string length at Array.join () at startCreating (D:\hashlips_art_engine-main\src\main.js:552:71) at D:\hashlips_art_engine-main\index.js:8:3 at Object. (D:\hashlips_art_engine-main\index.js:9:3) at Module._compile (node:internal/modules/cjs/loader:1155:14) at Object.Module._extensions..js (node:internal/modules/cjs/loader:1209:10) at Module.load (node:internal/modules/cjs/loader:1033:32) at Function.Module._load (node:internal/modules/cjs/loader:868:12) at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:81:12) at node:internal/main/run_main_module:22:47

I see from the stackoverflow answer link that @bolshoytoster sent there that it says you can use JSONStream stringify. How to use the JSONStream stringify? Can it be added in the JSONStream function or added for utils in hashlips_art_engine? I was hoping @bolshoytoster could make the function work around JSONStream limits on creating large _metadata on processing large numbers of objects.

Thank you @bolshoytoster for creating the function JSONStream

[bolshoytoster]

@mupadi

RangeError: Invalid string length

I think it's just a limitation with javascript then.

How to use the JSONStream stringify?

Run npm install JSONStream, then replace the line ~552 (the one you changed earlier) with:

  var JSONStream = require( "JSONStream" );

  var transformStream = JSONStream.stringify();
  var outputStream = fs.createWriteStream(`${buildDir}/json/_metadata.json`);
  transformStream.pipe(outputStream);
  metadataList.forEach(transformStream.write);
  transformStream.end();

I've also added this to the repo, so you could just redownload instead.