search

pennelise / GOOPy

2 stars 1 forks source link

Make sure you don't run out of memory with tons of observations. #8

Open pennelise opened 1 year ago

hannahnesser commented 11 months ago

I think this is as easy as capping the number of observations ingested at one time--not sure where the best place to do this is? Maybe in the demo script? It does seem like an easy thing to instruct the user to watch out for. But not sure--maybe this is a thing to do once we're a bit farther along in development.

hannahnesser commented 6 months ago

Fairly certain I have done this by allowing a "file_length_threshold" argument as one of the two key inputs to apply_operator (though perhaps this should be specified in config, instead?). Open to suggestions on better ways of handling this!