search

yangkevin2 / doc-story-generation

MIT License
145 stars 19 forks source link

flair==0.10 requires torch and will not accept pre-installed ROCM pytorch #6

Open bennmann opened 10 months ago

bennmann commented 10 months ago

Still coming back to project from time to time in hopes of getting it to work with llama 70B (1t/sec 4bit quantized locally).

While trying to get a local model to work with this repo, I found that requirement "flair==0.10" would not accept AMD ROCM pytorch even when invoking the appropriate --index-url flag and ROCM nightly URL in pip.

If you ever release a follow-up to your work I encourage you to implement a local model version, including dependencies which are not NVIDIA only.

yangkevin2 commented 10 months ago

Hey, yeah, the state of the art is indeed moving pretty fast these days. We're actually intending to open-source a version of this project that has the same high-level structure but supports newer LMs (e.g., LLama2), and is also rewritten to be easier to work with and modify. Flair will no longer be required in that version (and in the current version, you can probably get away with turning off the part that does use flair, with only a small loss). We're hoping to have the new version out in the next couple of weeks.

bennmann commented 9 months ago

Thank you Kevin!!

If you would like an AMD ROCM tester or any help getting AMD consumer cards to work with the new launch, I am happy to help even just with random errors as well. I have a couple hundred hours struggling through various AMD dependency purgatories at this point with multiple LLMs and have a local machine for inference with 16GB VRAM.

yangkevin2 commented 9 months ago

Sure, will let you know when it's released!

yangkevin2 commented 9 months ago

(Sorry, this is taking a bit longer than expected to get approved, but I'm hoping the new version should be out soon.)

yangkevin2 commented 9 months ago

The new code is up at https://github.com/facebookresearch/doc-storygen-v2 !