search

CalculatedContent / WeightWatcher

The WeightWatcher tool for predicting the accuracy of Deep Neural Networks
Apache License 2.0
1.44k stars 124 forks source link

A generic measure for generalization #2

Closed kenluck2001 closed 5 years ago

kenluck2001 commented 5 years ago

This is a great library.

Is it possible for you to provide a summary metric that can show that one network has a higher capacity than other networks on a given data?

Is it possible for you to provide a summary metric that can show that one network has a higher capacity than other networks based only on the weight distribution?

Can you try out generic information theoretic measures to provide a measure of the capacity of the neural network based on observing the weight distribution alone?

Unfortunately, I am in Python 2 land at the moment and plan to migrate to Python 3 in the near future. I cannot run the code in the default implementation without tweaks.

The get_summary method provide statistics like min, avg, max. This is different from a specific metric that can tell the capacity of a network. For example, being able to tell that GPT 3 is better than GPT 2 based on the weight distribution alone and providing this information in a single value. Rather than providing a chart that can provide subjective interpretation based on the technical abilities of the analyst.

Kenneth Odoh

charlesmartin14 commented 5 years ago

Oh python2. Thats difficult to support

On the other questions, sure we can make a support tool . What framework and version ? Keras 2 ? Pytorch ? TF ?

kenluck2001 commented 5 years ago

Hi Charles,

I have looked at the project in more details. The capacity is already what I was expecting. I am closing the ticket. Keep up the great work. Thanks for your reply.

Kenneth