actuarialopensource / benchmarks

Some performance tests for actuarial applications
MIT License
14 stars 4 forks source link

Add memory complexity analyses #43

Closed serenity4 closed 1 year ago

serenity4 commented 1 year ago

Implements memory complexity analyses as discussed in #34, with plots showing the linearity with respect to the number of model points and apparent independence in terms of number of timesteps.

MatthewCaseres commented 1 year ago

We have comparisons between the two models in terms of time. We have demonstrated the allocations scale linearly with modelpointsets. But our comparisons in terms of memory complexity don't have the side-by-side lines as we do for runtime. I imagine this is due to difficulties with GC and stuff as you have mentioned, but some sort of procedure to compare the two implementations is pretty valuable. Ideally the procedure will be simple enough to follow so that when someone claims an O(P) Python model we can demonstrate that they have done it right as well.

serenity4 commented 1 year ago

Alright, I can give it a shot with some memory profiling as mentioned in https://github.com/actuarialopensource/benchmarks/issues/34#issuecomment-1704299551

MatthewCaseres commented 1 year ago

I'm thinking about it more and I'm pretty convinced that the complexity is O(P) and not O(PT) based on the work that is done. Issue #34 is not about demonstrating that Python is O(PT) but it is about Julia being O(P) and I think the work done here closes out that issue. The Python implementation is "obviously" O(P*T) because it literally allocates a matrix of that size.