fast-patching library to be compatible with colab and environments where latest transformers, tensorflow is installed.

NVIDIA / kvpress

LLM KV cache compression made easy

Apache License 2.0

232 stars 9 forks source link

fast-patching library to be compatible with colab and environments where latest transformers, tensorflow is installed. #4

Closed sleepingcat4 closed 1 week ago

sleepingcat4 commented 1 week ago

I tried installing the library and running some experiments on Google colab. Unfortunately, it was breaking due to recent tensorflow changes and numpy changes. HF did not account them before pushing latest transformers version and older is not compatible with this library since meta-llama class failed error happens with them.

A fast-patch for this will be downgrading tensorflow and numpy versions slightly. It's not a KVPress specific error as it is happening due to transformers library. Particularly for this line:

from transformers.pipelines import PIPELINE_REGISTRY

I have added a requirements.txt file in the main-dir so that others don't face this error. I will also open an issue on HF as well.

Collab I experimented with: https://colab.research.google.com/drive/1482Nc7IfE-s5sl7dBjEbWfOjz59QFA1l?usp=sharing

sleepingcat4 commented 1 week ago

https://github.com/huggingface/transformers/issues/34839

sleepingcat4 commented 1 week ago

I added a note on readme as well

maxjeblick commented 1 week ago

Thanks a lot for the PR! We've also just now noticed some errors with colab, see #3. To my understanding,

default tensorflow version used in colab requires numpy < 2.x. This raises an issue when importing transformers. Uninstalling tensorflow fixes the issue.
importing numpy after installing kv-press shows numpy version 1.26 which causes a subsequent import error as well. Restarting the kernel fixes this issue.

Regarding your PR, this repository uses poetry for dependency management. We also don't want to add tensorflow as a dependency, as we are not using it. kvpress==0.0.1 package installs numpy==2.1.3, I don't think downgrading to numpy==2.1.0 would fix the colab issues observed above.

I'm currently working on a colab-specific notebook PR that fixes those issues, it should be ready this week. So stay tuned!

sleepingcat4 commented 1 week ago

@maxjeblick umm, I didn't get the error from KVPress tho. the error was from transformers library itself. the line quoted earlier was depending on tensorflow's August release and the October version is breaking it.

KVPress does not need to change numpy's np.bool to bool but that's different from the error I received.

also without the latest version of transformers, KVPress won't even work since Metallama classes aren't available in them.

so, I'm not sure how it can be a Colab specific issue rather it's a transformers issue.

SimJeg commented 1 week ago

@sleepingcat4 we just merged a #6 which adds colab notebook examples. It uninstalls tensorflow, install kvpress and ask the user to restart the notebook if needed. Hopefully the issue will be fixed soon by colab team. Thanks for your contribution !