search

brando90 / ultimate-anatome

Ἀνατομή is a PyTorch library to analyze representation of neural networks
MIT License
11 stars 1 forks source link

Running mean and std/variance #10

Open brando90 opened 3 years ago

brando90 commented 3 years ago

std: https://stackoverflow.com/questions/1174984/how-to-efficiently-calculate-a-running-standard-deviation

mean should be easy: https://stackoverflow.com/questions/14313510/how-to-calculate-rolling-moving-average-using-python-numpy-scipy

alexpreynolds commented 3 years ago

Have you looked at the welford package: https://pypi.org/project/welford/ ?

From the description:

Python (Numpy) implementation of Welford’s algorithm, which is online or parallel algorithm for calculating variances.

The algorithm is described in the followings,

Wikipedia:Welford Online Algorithm Wikipedia:Welford Parallel Algorithm

Welford’s method is more numerically stable than the standard method. The theoritical background of Welford’s method is mentioned in detail on the following blog articles. Please refer them if you are interested in.

http://www.johndcook.com/blog/standard_deviation https://jonisalonen.com/2013/deriving-welfords-method-for-computing-variance/

This library is inspired by the jvf’s implementation, which is implemented without using numpy library.

brando90 commented 3 years ago

hmmm, I will check it out.

I was thinking to just copy paste the numpy version and write torch code instead though so idk if this will allow me.

Perhaps I can just compute the std values and means in torch and then give that to that code to compute the running stuff.

On Nov 26, 2021, at 7:50 PM, Alex Reynolds @.***> wrote:

Have you looked at the welford package: https://pypi.org/project/welford/ https://pypi.org/project/welford/ ?

From the description:

Python (Numpy) implementation of Welford’s algorithm, which is online or parallel algorithm for calculating variances.

The algorithm is described in the followings,

Wikipedia:Welford Online Algorithm https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Online_algorithm Wikipedia:Welford Parallel Algorithm https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Parallel_algorithm Welford’s method is more numerically stable than the standard method. The theoritical background of Welford’s method is mentioned in detail on the following blog articles. Please refer them if you are interested in.

http://www.johndcook.com/blog/standard_deviation http://www.johndcook.com/blog/standard_deviation https://jonisalonen.com/2013/deriving-welfords-method-for-computing-variance/ https://jonisalonen.com/2013/deriving-welfords-method-for-computing-variance/ This library is inspired by the jvf’s implementation, which is implemented without using numpy library.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/brando90/ultimate-anatome/issues/10#issuecomment-980484698, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAOE6LVVZKGW76NQFWPUYSLUOA2QHANCNFSM5I3TNQ5Q. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.