Quantized versions

[alexandreteles]

Hello, I've been playing around with the model and made quantized versions available at:

• alexandreteles/bonito-v1-awq (can be used directly with vllm)
• alexandreteles/bonito-v1-gguf (for llama.cpp inference)

As soon as you have decided on a license, I will update the repositories accordingly. On a different topic, are there prospects to add more conversational tasks to the dataset in the future?

Thank you for this, it is an amazing project remarkably close to something I wanted to do myself. Good luck with publishing!

[nihalnayak]

Thank you so much for creating the quantized versions of Bonito. We really appreciate it. We will update the license shortly.

Regarding adding more conversational tasks, we are still deciding on the next steps in the project. In the meantime, you can check out the humpback paper. They are generating long-form datasets, which might be of interest to you.

[nihalnayak]

We've updated the license to Apache-2.

Thank you again!

[alexandreteles]

We've updated the license to Apache-2.

I've updated the license on HuggingFace.

check out the humpback paper. They are generating long-form datasets, which might be of interest to you.

Will do, thank you very much for the heads up!

[krittaprot]

Hello,

I was wondering if there might be plans to develop and share a tutorial Colab notebook specifically for using the quantized model with the Hugging Face Transformers API.

Creating such a tutorial could greatly enhance its accessibility and would be incredibly beneficial for a wide range of users, myself included. I've run into issues trying to load the full model into the Colab T4 GPU environment due to its size, which has prevented the Bonito library from working as expected, both locally and on platforms like Colab/Kaggle notebooks.

Originally, I had plans to use the Gemini Free Edition API for Q&A task dataset preparation for an LLM SFT project. However, the release of Bonito seems to offer a promising alternative for dataset preparation.

Really appreciate the effort you've put into developing this library and thank you for considering my request.

[asmith26]

I've have also been trying to run bonito on colab but am hitting problems, thus as per @krittaprot , I would also greatly appreciate a tutorial Colab notebook.

Thanks for this amazing work!

[avitrost]

@krittaprot @asmith26 Thank you for your patience! We have created a tutorial for the quantized model that can run on a T4 instance here.