livepeer / ai-worker

https://livepeer.ai
MIT License
17 stars 27 forks source link

enable specifying schedulers for text-to-image and image-to-image #132

Open ad-astra-video opened 2 months ago

ad-astra-video commented 2 months ago

This PR adds ability to set schedulers using a json string in the request for t2i and i2i pipeline. name of the scheduler can be a preset or a class name for the scheduler from diffusers.

There are presets to make setting the main schedulers easier from https://huggingface.co/docs/diffusers/en/api/schedulers/overview.

Scheduler is created by mapping the preset name to the diffusers scheduler class using importlib so the schedulers are not imported until needed. If a scheduler fails to be added the pipeline raises an error. If no scheduler is specified the default scheduler is applied.

performance testing loading no scheduler (reassigns default scheduler) 2024-07-28 01:12:00,677 - app.pipelines.text_to_image - INFO - setting pipeline scheduler to: 2024-07-28 01:12:00,677 - app.pipelines.text_to_image - INFO - loading scheduler took 0.0002524852752685547s

loading DPM++ 2M scheduler from preset on RealVisXL_V4.0 model 2024-07-28 01:13:44,851 - app.pipelines.text_to_image - INFO - setting pipeline scheduler to: {"name":"DPM++ 2M","args":{}} 2024-07-28 01:13:44,858 - app.pipelines.text_to_image - INFO - loading scheduler took 0.00660252571105957s

loading DPM2 scheduler from preset on RealVisXL_V4.0 model 2024-07-28 01:10:39,576 - app.pipelines.text_to_image - INFO - setting pipeline scheduler to: {"name":"DPM2","args":{}} 2024-07-28 01:10:39,817 - app.pipelines.text_to_image - INFO - loading scheduler took 0.24109697341918945s

loading Euler flow match for SD3, setting shift arg. 2024-07-28 01:42:36,853 - app.pipelines.text_to_image - INFO - setting pipeline scheduler to: {"name":"Euler flow match","args":{"shift":"3"}} 2024-07-28 01:42:36,856 - app.pipelines.text_to_image - INFO - loading scheduler took 0.002814054489135742s

format of json string in http request: { "name": "Euler", "args": {} }

ad-astra-video commented 3 weeks ago

Updated this to 0.2.0 release