search

nextml / NEXT

NEXT is a machine learning system that runs in the cloud and makes it easy to develop, evaluate, and apply active learning in the real-world. Ask better questions. Get better results. Faster. Automated.
http://nextml.org
Apache License 2.0
160 stars 53 forks source link

Butler.memory key not unique to experiments #173

Closed dconathan closed 7 years ago

dconathan commented 7 years ago

Thoughts on Butler.memory... the way it's implemented now, there's no sense of creating "buckets" for each experiment/algorithm, etc. So in my example my app stores a feature matrix under the key "features". If I run two instances of this experiment, the keys clash.

Butler should probably automatically (optionally?) prepend its memory keys with exp_uid and/or alg_label, right? I mean, in theory we could leave it up to the user (in case they want to have multiple experiments share the same data... but this seems like a bad thing for the default behavior...)

dconathan commented 7 years ago

Another thought is to make memory a property of Collection instead of Butler. So you'd do butler.experiment.memory.set() instead of butler.memory.set() ... this way we could mirror the same "unique to experiment/unique to algorithm" data scheme.

lalitkumarj commented 7 years ago

I think memory should be a collection - for exactly the reasons you stated.

dconathan commented 7 years ago

Cool. Just to clarify... do you mean "be a collection" or "be a property of collection", e.g. butler.experiment.memory. I have a use case for this (i.e. some things need to exist in memory and be unique to experiment, some things unique to algorithm), so I'm going to work on this implementation.

I will leave butler.memory in place to not break existing things, and also there are possible use cases for having a "NEXT global memory" accessible by all experiments.