Multiscale Fourier Features

[zoemcc]

Implement the Multiscale Fourier Features (MFF) neural network architecture proposed in this paper "On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networks" paper by Sifan Wang, Hanwen Wang, Paris Perdikaris. The paper comes with a github repo which should guide implementation.

The architecture is proposed in equations (3.29 - 3.34) in the paper and figure (8.b) represents it pictorially. This architecture should be implemented as a function that takes in the hyperparameters of the network and outputs a DiffEqFlux FastChain that encapsulates the network. This might mean that it should be implemented in DiffEqFlux instead of here, but the issue makes more sense here. The different hyperparameters that seem relevant to me to include as arguments to the function are:

• Input and output dimensions of the network
• Number of hidden layers following the Fourier basis
• Number of Fourier basis blocks and their scale parameters
• Whether an input dimension is a time dimension (and thus that dimension should be split off and use its own Fourier basis and then merged using equation 3.33 as in section 4.3 of the paper)

(i.e.)

function MultiScaleFourierFeatureNetwork(hyperparameters...) 

return FastChain(...)
end

The exact form of the hyperparameters seems slightly difficult to define in a way that has good ergonomics though, since you would have to define scale parameters for each of the time-based Fourier bases and for each of the space-based Fourier bases. Perhaps there should just be two different functions, one that generates MFF features for either space or time inputs (or treats space-time together) and the space-time MFF function splits the input along space and time dimensions, calls the interior MFF function, and then splices them together using the elementwise multiplication mechanism in the paper.

This is not too difficult of an implementation project, most of the difficulty is in figuring out a good interface and processing that into the correct DiffEqFlux building blocks. I think that architectures like this are likely to lead to big improvements in the algorithm, since the standard feedforward nets are not well suited to this problem due to spectral bias (as discussed in the paper). There is a lot of interesting and useful theory in the paper (mostly using the NTK theory), but understanding the theory in the first half of the paper isn't necessary to implement the method.

[YichengDWu]

Randomly generating some untrainable Fourier basis functions is far from being a good model. Of course this paper is "cheating" because it sets hyperparameters based on the spectrum of the real solution. Siren performs much better and is more robust than it: https://www.vincentsitzmann.com/siren/

[YichengDWu]

Available in https://milkshakeforreal.github.io/Sophon.jl/dev/. However, I don't have much time to test this model in experiments. If you want to start some experiments you can have a try.

[harishwar017]

hey @zoemcc I would like to take a crack at this. Please do let me know if it is still relevant and not implemented here yet. Thanks!

[ChrisRackauckas]

It has not been implemented here yet. It's still relevant and the Sophon implementation might be a good reference to use.

[YichengDWu]

Sophon needs a new maintainer. I'd be happy to transfer it to the SciML organization since I'm no longer maintaining it.