[Question] Are numerical and categorical features used for fine-tuning the BERT(or LLMs)?

georgian-io / Multimodal-Toolkit

Multimodal model for text and tabular data with HuggingFace transformers as building block for text data

https://multimodal-toolkit.readthedocs.io

Apache License 2.0

588 stars 84 forks source link

[Question] Are numerical and categorical features used for fine-tuning the BERT(or LLMs)? #74

Closed anirbandey303 closed 6 months ago

anirbandey303 commented 6 months ago

Hi Developers,

I have a naive question, could you please help me understand:

During the model training, are the weights and biases of the BERT model(or any other supported model) changing? (IF yes,) is it taking the numerical and categorical features into account or is it being used as a feature (along with the vector-embeddings from the LLM) by the MLP that does the classification?

akashsaravanan-georgian commented 6 months ago

Hi @anirbandey303,

Yes, the pre-trained language model (such as BERT) weights do change.

The numerical/categorical features are not used by the language (BERT) model. They are sent as features to the MLP alongside the output of the language model. The architecture diagram here might help!

anirbandey303 commented 6 months ago

Perfect, that clears out my confusion. Thanks a lot for the prompt response. 👍

akashsaravanan-georgian commented 6 months ago

Happy to help :)