Consider adding the spectral dimension from Deepka Dhar's work

If I understand his concept of spectral dimension properly, I believe this is can be applied to deep learning to examine the fractal dimension of the weight matrices as they evolve in time. We use related techniques in the weightwatcher package. Indeed, I suspect that deep learning models behave optimally when the weight matrices display signatures of a strongly correlated system., His work definitely looks useful here.

ChatGPT suggests the following code

def spectral_dimension(eigenvalues):
    eigenvalues = np.sort(np.abs(eigenvalues))
    eigenvalues = eigenvalues[eigenvalues > 0]
    return -2 * np.sum(np.log(eigenvalues) / np.log(eigenvalues[-1])) / eigenvalues.shape[0]

I'm not sure yet if this is correct but it looks very similar to what we are doing with the detX constraint

Deepak Dhar is a theoretical physicist and professor at the University of Michigan. He is known for his work in the field of condensed matter physics, specifically in the area of phase transitions and critical phenomena.

The spectral dimension is a concept in the study of fractal geometry and refers to the dimensionality of a space as measured by the diffusion of a random walker on the space. Dhar has proposed a theory of the spectral dimension in the context of condensed matter systems and has applied this concept to study phase transitions and critical phenomena in various systems.

Yes, the spectral dimension can be used to measure the effective dimension of a real matrix. By analyzing the scaling behavior of the eigenvalues of a matrix, one can determine the spectral dimension and use it as a measure of the effective dimensionality of the matrix. This has applications in fields such as data analysis, network science, and complex systems.

CalculatedContent / WeightWatcher

Consider adding the spectral dimension from Deepka Dhar's work #177