Closed magicicada closed 4 years ago
Heads up @magicicada @bobturneruk @aflag @WPettersson @alex-konovalov @may1066 @mrow84 - the "Simple Network Sim" label was applied to this issue.
I may have confused the issue in our standup. If you are doing what is effectively a "single run", from the perspective of the data pipeline, that generates multiple realisations of the model, then that should just work as normal as long as you can work around the memory issues that you referred to.
My confusion was caused by thinking that you were going to track each realisation separately, and then try to relate the runs subsequently - it is that that is not well-modelled in the data registry.
I may have confused the issue in our standup. If you are doing what is effectively a "single run", from the perspective of the data pipeline, that generates multiple realisations of the model, then that should just work as normal as long as you can work around the memory issues that you referred to.
My confusion was caused by thinking that you were going to track each realisation separately, and then try to relate the runs subsequently - it is that that is not well-modelled in the data registry.
Yes definitely the first case. Even with stochastic mode and multiprocessed runs, we should always be able to remain fully reproducible on the random numbers generated, and this means indeed it will be a single run from the perspective of the data pipeline. As @magicicada mentions we can key the runs by child seed id or run id. But bundling all results together results in rather chunky dataframes (could go to a few gb for large networks)
What would be really nice would be to have all "runs" in the same output file but with different components.
What would be really nice would be to have all "runs" in the same output file but with different components.
That is perfectly fine afaik, the only issue is you managing your interaction with the API so that you do all the writes in the same session. We could potentially make it so a session could be longer than a process, but it would take some work, so I'd rather not unless strictly necessary.
Output from inference runs where multiple series are generated. Output of all runs should be saved with a view to disk, with a view to being able to put these in the data pipeline. Changes to the data registry (i.e. a field that defines a "group" of runs) may be required. Another option would be to concatenate outputs with a run number or seed column.