CPCLAB-UNIPI / SIPPY

Systems Identification Package for PYthon
GNU Lesser General Public License v3.0
269 stars 92 forks source link

Rewrote parts of algorithm_1 which were memory inefficient #19

Closed NitroxKnight closed 4 years ago

NitroxKnight commented 4 years ago

Dear CPCLAB-UNIPI,

I have been using parts of your code for my system identification framework and noticed that part of the state-space algorithm: algorithm_1 was memory inefficient due to the memory scaling quadratically with the number of samples.

This can be demonstrated by setting tfin in the ss.py i.e. https://github.com/CPCLAB-UNIPI/SIPPY/blob/master/Examples/SS.py#L37 to 50,000 at which it creates a matrix of approximate size 50,000 by 50,000 (12+GB of memory) and thus creates a memory error. This large matrix creation happens in three places

I have altered the order of operations such that in N4SID and MOESP these matrixes do not get created and added an argument to the full_matrices=False to np.linalg.svd(O_i) which will not create the large V_n matrix which is not required.

With my changes I get have tested up to tfin= 500,000 samples which take ~1 GB of memory with indistinguishable results from the original script at any tfin

If you have any further questions I will do my best to answer them.

Best Regards, Gerben

p.s. this error still persists in SVD_weighted_K which should be an issue.

CPCLAB-UNIPI commented 4 years ago

Dear NitroxKnight, we are analyzing your commits and considering to implement your modifications as soon as we have tested all the relevant cases.

Best Regards,

CPCLAB-UNIPI