search

SeldonIO / MLServer

An inference server for your machine learning models, including support for multiple frameworks, multi-model serving and more
https://mlserver.readthedocs.io/en/latest/
Apache License 2.0
708 stars 180 forks source link

Documentation for parameter usage in HuggingFace runtime missing #1016

Open JochenGuckSnk opened 1 year ago

JochenGuckSnk commented 1 year ago

I am interested in using MLServer with HuggingFace models. To get the full potential of these models, it is essential to modify the model parameters (https://huggingface.co/docs/transformers/main_classes/text_generation). I have tried to change these parameters, but it is not straightforward to me. Therefore, it might be a good idea to extend the documentation with a section on how to modify parameters like the temperature or the num_return_sequences.

coctel99 commented 1 year ago

Hi! Are there any news or plans on adding opportunity to pass such transformers config parameters as 'max_length' using Seldon HuggingFace server?

adriangonz commented 1 year ago

There're plans but we haven't got around it yet. We'll provide an update on this issue once it's done.

Alternatively, contributions are always welcome :wink:

coctel99 commented 8 months ago

@adriangonz Do you have any updates? Or approximate dates?

adriangonz commented 8 months ago

Hey @coctel99, I'm no longer working for Seldon and no longer contributing to MLServer so I don't have any extra context on this one.

@sakoush has taken the reins of the project, so it'd be best to check with him.

sakoush commented 8 months ago

@ramonpzg for reference

ramonpzg commented 7 months ago

Hi @coctel99 -- I am taking care of this one for the next release :)

If you are in the community, I'll send you a message once it is merged.