carbon flow models very memory-intensive

[fvilla]

@lambdatronic

[fvilla]

I mean, seriously. Carbon flow on san_pedro512 will eat 7/8 gigs of RAM and break (many minutes later) with a GC error. I'm not even sure people will look at the flow results that much (not until we get better emission layers at least) so that's a pretty big deal-breaker for carbon. Anything that could be done would be greatly appreciated (btw those models don't even seem to use the CPU that much).

[lambdatronic]

Damn. I believe this might have to do with the version of Clojure we're running. I need to go back and look at this code in the context of chunked seqs to see how I can speed it up. I think improper laziness is probably killing the model here.

[lambdatronic]

Okay, I took a look into this, and it looks like the issue here is not really specific to carbon. The problem is that the way I have been storing the SPAN simulation information ultimately creates a data structure of size O(S_K_U), where S, K, U are the number of above-threshold source, sink, and use points respectively. The reason we are running into a problem with carbon, is that carbon often actually invokes the worst case scenario of O(N^3), wherein all grid cells on the map are source, sink, and use points simultaneously. For all of our other models, we haven't gotten anywhere near this upper bound yet, so our model runs have been fine. Unfortunately, in order to resolve this problem, I'll have to redesign the SPAN infrastructure and all of the flow models - not a simple task. I spotted this problem almost a year ago, I think, but there were too many fires in the interim that kept me from getting around to this job. For now, I recommend just running carbon at a lower res (or with a really large downscaling factor) until I finish an optimized rewrite of the system. I'll let everyone know when this is done.

[fvilla]

OK, I'm inclined to use only the source models in the storylines instead of the flow models until this is resolved, rather than downscaling. People expect carbon to be "easy" and have good resolution, and the results of the source models are mostly what they're interested in. Let me know if there's any disagreement.

@bvoigt @kbagstad

[kbagstad]

I agree that we should drop the flow models from the carbon storyline for the time being I agree with your points below Ferd - but also that we should still include the sink and use data/models as well in the storyline at this point. There's useful information in there about carbon storage in vegetation and soils, fire threat, etc. that is useful to keep around.

Ken

On Fri, Oct 21, 2011 at 3:51 AM, Ferdinando Villa < reply@reply.github.com>wrote:

OK, I'm inclined to use only the source models in the storylines instead of the flow models until this is resolved, rather than downscaling. People expect carbon to be "easy" and have good resolution, and the results of the source models are mostly what they're interested in. Let me know if there's any disagreement.

@bvoigt @kbagstad

Reply to this email directly or view it on GitHub: https://github.com/ariesteam/aries/issues/28#issuecomment-2479507