Architecture explorations

[balthazarneveu]

Baseline

• [x] Implement and train flat convolutions

• [ ] Implement and train a Conv+RNN

• 1 1.5e-5 FlatConv

U-Net

Home-made ResUNet

• [x] code Res-UNet architecture from scratch

• [x] improve and study architecture impact

• 2 : 4.3e-6 depth=4 , hidden=16 , k_size= 5 extension factor=1.5 ( batch=16)

• 3 : 5.8e-6 depth=4 , hidden=16 , k_size= 5 extension factor=1.5 (batch=32)

• 4 4.2e-6 depth=4 , hidden=16 , k_size= 7 extension factor=1.5 (batch=16) -> larger receptive field

• 5 depth=4 , hidden=32, k_size= 5 extension factor=1.5 (batch=8) -> thicker

• extension factor?

• deeper?

Parametric ResUNet - depth (should maybe allow using weight decay)

Wave U-net

• [x] Check key designs
• [x] compare it to the ResUNet
• [x] Implement
• [x] 7 layers or 6 layers > 1M parameters = :x: Overfitting

Transformer

• [ ] Reduce the number of tokens + reconstruct full resolution after

HF class on audio