I have been using parts of your code for my system identification framework and noticed that part of the state-space algorithm: algorithm_1 was memory inefficient due to the memory scaling quadratically with the number of samples.
This can be demonstrated by setting tfin in the ss.py i.e. https://github.com/CPCLAB-UNIPI/SIPPY/blob/master/Examples/SS.py#L37 to 50,000 at which it creates a matrix of approximate size 50,000 by 50,000 (12+GB of memory) and thus creates a memory error. This large matrix creation happens in three places
I have altered the order of operations such that in N4SID and MOESP these matrixes do not get created and added an argument to the full_matrices=False to np.linalg.svd(O_i) which will not create the large V_n matrix which is not required.
With my changes I get have tested up to tfin= 500,000 samples which take ~1 GB of memory with indistinguishable results from the original script at any tfin
If you have any further questions I will do my best to answer them.
Best Regards,
Gerben
p.s. this error still persists in SVD_weighted_K which should be an issue.
Dear CPCLAB-UNIPI,
I have been using parts of your code for my system identification framework and noticed that part of the state-space algorithm:
algorithm_1
was memory inefficient due to the memory scaling quadratically with the number of samples.This can be demonstrated by setting
tfin
in the ss.py i.e. https://github.com/CPCLAB-UNIPI/SIPPY/blob/master/Examples/SS.py#L37 to 50,000 at which it creates a matrix of approximate size 50,000 by 50,000 (12+GB of memory) and thus creates a memory error. This large matrix creation happens in three placesW1
https://github.com/CPCLAB-UNIPI/SIPPY/blob/master/sippy/OLSims_methods.py#L31V_n
matrix created in https://github.com/CPCLAB-UNIPI/SIPPY/blob/master/sippy/OLSims_methods.py#L33I have altered the order of operations such that in
N4SID
andMOESP
these matrixes do not get created and added an argument to thefull_matrices=False
tonp.linalg.svd(O_i)
which will not create the largeV_n
matrix which is not required.With my changes I get have tested up to
tfin=
500,000 samples which take ~1 GB of memory with indistinguishable results from the original script at anytfin
If you have any further questions I will do my best to answer them.
Best Regards, Gerben
p.s. this error still persists in
SVD_weighted_K
which should be an issue.