search

dusty-nv / jetson-containers

Machine Learning Containers for NVIDIA Jetson and JetPack-L4T
MIT License
2.18k stars 446 forks source link

Upgrade `flash-attention` to `2.5.7`, fix `text-generation-webui` container #494

Closed ms1design closed 5 months ago

ms1design commented 5 months ago

Hi,

dusty-nv commented 5 months ago

@ms1design can you comment on why to remove the auto_awq patch from the oobabooga's requirements.txt?

auto_awq container gets added to text-gen-webui container on JP6: https://github.com/dusty-nv/jetson-containers/blob/97fd4fcc3a896a9074dbeef6e78cf01a910d944b/packages/llm/text-generation-webui/config.py#L32

So I remove that from requirements.txt so it doesn't go out installing some other version, but rather keeps the version from container. It looks like this is now autoawq==0.2.3; platform_system == "Linux" or platform_system == "Windows" in their requirements.txt

I'm guessing you hit some issue with this

ms1design commented 5 months ago

Missed that you migrated it to config.py, restored.

dusty-nv commented 5 months ago

Thanks @ms1design - I added a config.py to flash-attention container to support proper versioning in 98952b4

dusty-nv commented 5 months ago

flash_attn-2.5.7-cp310-cp310-linux_aarch64.whl