Add support for HuBERT batch norm instead of weight norm in pos_conv_emb

gallilmaimon commented 1 week ago

Feature request

In the Hubert model convolutional positional encoding - add a support for batch norm instead of weight norm as added in Fairseq at - https://github.com/facebookresearch/fairseq/commit/4db264940f281a6f47558d17387b1455d4abd8d9
This in turn will allow supporting mhubert-base-25hz which was introduced in - https://arxiv.org/abs/2305.13009

Motivation

This will give the opportunity to support other textlesslib/fairseq models and specifically the newer, better HuBERT model - mhubert-base-25hz which uses this and is currently unsupported.
This model is frequently used for training speech language models, and would benefit the community as well as myself in a project I am working on.

Your contribution

I can create a PR to implement this, but would love some guidance @ylacombe

avishaiElmakies commented 6 days ago

Hi, would love this feature as well @ylacombe

ylacombe commented 3 days ago

Hey @avishaiElmakies and @gallilmaimon , this would indeed be a great addition.

Would you like to open a PR to correct this ? You'd have to add the possibility to use batch norm in the configuration_hubert.py, propagate the change to the modeling file and the converting file, and finally add an integration test. How does it sound?

Thanks

gallilmaimon commented 2 days ago

@ylacombe Sounds good. I will work on something and let you know when the PR is ready

huggingface / transformers