shuttle-hq / synth

The Declarative Data Generator
https://www.getsynth.com/
Apache License 2.0
1.39k stars 109 forks source link

Missing warning or error when generation would exhaust time / memory #193

Open llogiq opened 3 years ago

llogiq commented 3 years ago

Describe the bug One can ask synth to create an array of a length that will no longer fit in memory. As we write everything into memory before printing, this will simply fill all available memory until either the system crashes (Windows) or the OOM killer kills it (and possibly other programs, Linux & MacOS).

To Reproduce Steps to reproduce the behavior:

  1. Schema (if applicable)
    {
    "type": "array",
    "length": {
        "type": "number",
        "subtype": "u32"
    },
    "content": {
        "type": "object"
    }
    }
  2. There is no observable error, the program will run for a good while until all memory is consumed.

Expected behavior Given that we can calculate the heap size of all our objects , we should be able to use this calculation to return an error if we go over a certain amount of memory. That limit may be soft or configurable.

Environment (please complete the following information):

catkins commented 1 year ago

On top of ☝️, it would be nifty if it was possible to do the generation in batches (when possible) such that it didn't need to hold the entire set in memory.