search

bentonam / fakeit

Generates JSON documents based on models defined in YAML and adds them to a Couchbase Bucket
MIT License
86 stars 21 forks source link

Heap Issues #147

Open hanej opened 7 years ago

hanej commented 7 years ago

This is an amazing application by the way and it's greatly helping my project.

I'm trying to generate over a million records for Couchbase but keep running out of memory even if I increase the heap in node. Is there a way to flush a number of records while they are being generated instead of waiting until completion? This way it can be reclaimed in the heap.

bentonam commented 7 years ago

Thanks @hanej we are glad to hear that you are finding the project useful, any feedback you have is greatly appreciated.

Would you be able to provide or create a gist of your models, any input you are using as well as the command?

We are always trying to improve the project and increase performance, FakeIt was never really meant to generate millions of documents due to the exact issue you're describing. We do have a few ideas on how to get to millions of documents, but those models would have to have no dependants.

A solution I have heard people use to get to millions of documents is spinning up several nodes/containers and using offsets within their models when issuing the fakeit command. This spreads the generation load across multiple nodes while still allowing you to get to a million documents. For example, 5 nodes running FakeIt each generating ~200k documents.

hanej commented 7 years ago

My models have nested dependencies. I've stripped out everything but the essentials just to get the last bits generated. I'm going to try the offset approach. I think I can do that pretty easily.