raznem / parsera

Lightweight library for scraping web-sites with LLMs
https://docs.parsera.org
GNU General Public License v2.0
897 stars 58 forks source link

Support HuggingFace Transformers #12

Closed SamTheCoder777 closed 3 months ago

SamTheCoder777 commented 3 months ago

This PR introduces support for HuggingFace Transformers in Parsera by adding a new HuggingFaceModel. Users only need to install Transformers when they choose to use it, as the module is not loaded until Parsera is initialized with HuggingFaceModel as the model.

This PR introduces support for HuggingFace Transformers by adding a new class, ParseraHuggingFace. The class is designed for lazy loading, ensuring that users only install the transformers library if they choose to use it.

List of changes:

- Added: ParseraHuggingFace class for HuggingFace integration.

portoaj commented 3 months ago

I think this a great feature but I think making a whole new ParseraHuggingFace is unnecessary unless there's something that I'm missing. I think it would be better if you could either make the ParseraHuggingFace class inherit _run, run, and arun from Parsera since those seem to be mostly untouched, or even better just add the HuggingFace support to the existing Parsera class potentially with some helper methods.

SamTheCoder777 commented 3 months ago

I’ve removed the unnecessary ParseraHuggingFace class to avoid code duplication. Now, HuggingFaceModel integrates directly with Parsera as it's already a BaseChatModel instance.