search

adapter-hub / adapters

A Unified Library for Parameter-Efficient and Modular Transfer Learning
https://docs.adapterhub.ml
Apache License 2.0
2.48k stars 331 forks source link

Supporting BigScience’s BLOOM model #429

Open yongzx opened 1 year ago

yongzx commented 1 year ago

🌟 New Model: BLOOM

Model description

The BLOOM model has been proposed with its various versions through the BigScience Workshop. BigScience is inspired by other open science initiatives where researchers have pooled their time and resources to collectively achieve a higher impact. The architecture of BLOOM is essentially similar to GPT3 (auto-regressive model for next token prediction), but has been trained on 46 different languages and 13 programming languages. Several smaller versions of the models have been trained on the same dataset. BLOOM is available in the following versions: bloom-560m, bloom-1b1, bloom-1b7, bloom-3b, bloom-7b1, bloom (176B parameters)

One important note is that BLOOM is developed with the most recent transformer library version so we need to check for backward compatibility.

Open source status

xszheng2020 commented 1 year ago

Hi, @yongzx may I ask whether you have supported BLOOM models yet? Thanks!

yongzx commented 1 year ago

Hi @xszheng2020, apologies for my late reply. Right now my forked repo (https://github.com/yongzx/adapter-transformers) is able to support BLOOM for some of the methods such as Pfeiffer, MAD-X, and IA3. I haven't pushed the forked commit yet as I haven't written the unit test cases.

MaximilianKummeth commented 1 year ago

Hey @yongzx did you already manage to create the unit tests to push your commits?

karim1104 commented 1 year ago

Hi @xszheng2020, apologies for my late reply. Right now my forked repo (https://github.com/yongzx/adapter-transformers) is able to support BLOOM for some of the methods such as Pfeiffer, MAD-X, and IA3. I haven't pushed the forked commit yet as I haven't written the unit test cases.

Hi Yong, does that mean if we install your fork through pip, we'll get the support for BLOOM?

yongzx commented 1 year ago

Hi yes. I think you can check out the usage of it in our bigscience multilingual modeling group: https://github.com/bigscience-workshop/multilingual-modeling .

Thanks for bringing this up. I haven't gotten the chance to work on this.