search

golemfactory / ya-runtime-ai

1 stars 1 forks source link

gsb http proxy converts `500` into `200`. Incompatible model passes model loading step. #100

Closed pwalski closed 1 month ago

pwalski commented 2 months ago

Issues:

I tried to use example Golem.Tools/App with model https://huggingface.co/stabilityai/stable-diffusion-2-1/resolve/main/v2-1_768-ema-pruned.safetensors?download=true?download=true Runtime started correctly, model got loaded (loading incompatible model sometimes fails), but requests are failing, and logs print recommendations about adding some config options to Automatic commandline which user is unable to do:

[2024-04-17T12:04:02.511+0200 DEBUG ya_gsb_http_proxy::gsb_to_http] Calling http://localhost:7861/sdapi/v1/txt2img
[2024-04-17T12:04:02.511+0200 DEBUG reqwest::connect] starting new connection: http://localhost:7861/
[2024-04-17T12:04:02.511+0200 DEBUG hyper::client::connect::dns] resolving host="localhost"
[2024-04-17T12:04:02.511+0200 DEBUG hyper::client::connect::http] connecting to [::1]:7861
[2024-04-17T12:04:02.751+0200 DEBUG ya_runtime_ai] Successfully sent activity usage message
[2024-04-17T12:04:02.781+0200 DEBUG hyper::client::connect::http] connecting to 127.0.0.1:7861
[2024-04-17T12:04:02.781+0200 DEBUG hyper::client::connect::http] connected to 127.0.0.1:7861
[2024-04-17T12:04:02.781+0200 DEBUG hyper::proto::h1::io] flushed 53 bytes
[2024-04-17T12:04:02.783+0200 DEBUG hyper::proto::h1::io] parsed 5 headers
[2024-04-17T12:04:02.783+0200 DEBUG hyper::proto::h1::conn] incoming body is content-length (22 bytes)
[2024-04-17T12:04:02.827+0200 DEBUG hyper::client::connect::http] connecting to 127.0.0.1:7861
[2024-04-17T12:04:02.843+0200 DEBUG hyper::client::connect::http] connected to 127.0.0.1:7861
[2024-04-17T12:04:02.850+0200 DEBUG hyper::proto::h1::io] flushed 466 bytes
[2024-04-17T12:04:03.757+0200 DEBUG ya_runtime_ai] Successfully sent activity usage message
[2024-04-17T12:04:03.787+0200 DEBUG reqwest::connect] starting new connection: http://localhost:7861/
[2024-04-17T12:04:03.787+0200 DEBUG hyper::client::connect::dns] resolving host="localhost"
[2024-04-17T12:04:03.787+0200 DEBUG hyper::client::connect::http] connecting to [::1]:7861
[2024-04-17T12:04:04.095+0200 DEBUG hyper::client::connect::http] connecting to 127.0.0.1:7861
[2024-04-17T12:04:04.096+0200 DEBUG hyper::client::connect::http] connected to 127.0.0.1:7861
[2024-04-17T12:04:04.096+0200 DEBUG hyper::proto::h1::io] flushed 53 bytes
[2024-04-17T12:04:04.098+0200 DEBUG hyper::proto::h1::io] parsed 5 headers
[2024-04-17T12:04:04.098+0200 DEBUG hyper::proto::h1::conn] incoming body is content-length (22 bytes)
[2024-04-17T12:04:04.388+0200 DEBUG ya_runtime_ai::process::automatic::monitor] > 
  0%|          | 0/250 [00:00<?, ?it/s]
  0%|          | 0/250 [00:00<?, ?it/s]
[2024-04-17T12:04:04.550+0200 DEBUG ya_runtime_ai::process::automatic::monitor] > INFO:     127.0.0.1:64285 - "POST /sdapi/v1/txt2img HTTP/1.1" 500 Internal Server Error
[2024-04-17T12:04:04.550+0200 DEBUG ya_runtime_ai::process::automatic::monitor] > *** API error: POST: http://localhost:7861/sdapi/v1/txt2img {'error': 'NansException', 'detail': '', 'body': '', 'errors': 'A tensor with all NaNs was produced in Unet. This could be either because there\'s not enough precision to represent the picture, or because your video card does not support half type. Try setting the "Upcast cross attention layer to float32" option in Settings > Stable Diffusion or using the --no-half commandline argument to fix this. Use --disable-nan-check commandline argument to disable this check.'}
[2024-04-17T12:04:04.550+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >     Traceback (most recent call last):
[2024-04-17T12:04:04.550+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >       File "D:\Code\gamerhash-facade\modules\plugins\ya-automatic-ai\automatic\system\python\lib\site-packages\anyio\streams\memory.py", line 98, in receive
[2024-04-17T12:04:04.550+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >         return self.receive_nowait()
[2024-04-17T12:04:04.550+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >       File "D:\Code\gamerhash-facade\modules\plugins\ya-automatic-ai\automatic\system\python\lib\site-packages\anyio\streams\memory.py", line 93, in receive_nowait
[2024-04-17T12:04:04.550+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >         raise WouldBlock
[2024-04-17T12:04:04.550+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >     anyio.WouldBlock
[2024-04-17T12:04:04.550+0200 DEBUG ya_runtime_ai::process::automatic::monitor] > 
[2024-04-17T12:04:04.550+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >     During handling of the above exception, another exception occurred:
[2024-04-17T12:04:04.550+0200 DEBUG ya_runtime_ai::process::automatic::monitor] > 
[2024-04-17T12:04:04.550+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >     Traceback (most recent call last):
[2024-04-17T12:04:04.550+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >       File "D:\Code\gamerhash-facade\modules\plugins\ya-automatic-ai\automatic\system\python\lib\site-packages\starlette\middleware\base.py", line 78, in call_next
[2024-04-17T12:04:04.550+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >         message = await recv_stream.receive()
[2024-04-17T12:04:04.550+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >       File "D:\Code\gamerhash-facade\modules\plugins\ya-automatic-ai\automatic\system\python\lib\site-packages\anyio\streams\memory.py", line 118, in receive
[2024-04-17T12:04:04.550+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >         raise EndOfStream
[2024-04-17T12:04:04.550+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >     anyio.EndOfStream
[2024-04-17T12:04:04.550+0200 DEBUG ya_runtime_ai::process::automatic::monitor] > 
[2024-04-17T12:04:04.550+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >     During handling of the above exception, another exception occurred:
[2024-04-17T12:04:04.550+0200 DEBUG ya_runtime_ai::process::automatic::monitor] > 
[2024-04-17T12:04:04.550+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >     Traceback (most recent call last):
[2024-04-17T12:04:04.550+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >       File "D:\Code\gamerhash-facade\modules\plugins\ya-automatic-ai\automatic\webui\modules\api\api.py", line 187, in exception_handling
[2024-04-17T12:04:04.550+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >         return await call_next(request)
[2024-04-17T12:04:04.550+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >       File "D:\Code\gamerhash-facade\modules\plugins\ya-automatic-ai\automatic\system\python\lib\site-packages\starlette\middleware\base.py", line 84, in call_next
[2024-04-17T12:04:04.550+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >         raise app_exc
[2024-04-17T12:04:04.550+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >       File "D:\Code\gamerhash-facade\modules\plugins\ya-automatic-ai\automatic\system\python\lib\site-packages\starlette\middleware\base.py", line 70, in coro
[2024-04-17T12:04:04.550+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >         await self.app(scope, receive_or_disconnect, send_no_error)
[2024-04-17T12:04:04.550+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >       File "D:\Code\gamerhash-facade\modules\plugins\ya-automatic-ai\automatic\system\python\lib\site-packages\starlette\middleware\base.py", line 108, in __call__
[2024-04-17T12:04:04.550+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >         response = await self.dispatch_func(request, call_next)
[2024-04-17T12:04:04.550+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >       File "D:\Code\gamerhash-facade\modules\plugins\ya-automatic-ai\automatic\webui\modules\api\api.py", line 151, in log_and_time
[2024-04-17T12:04:04.550+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >         res: Response = await call_next(req)
[2024-04-17T12:04:04.550+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >       File "D:\Code\gamerhash-facade\modules\plugins\ya-automatic-ai\automatic\system\python\lib\site-packages\starlette\middleware\base.py", line 84, in call_next
[2024-04-17T12:04:04.550+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >         raise app_exc
[2024-04-17T12:04:04.550+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >       File "D:\Code\gamerhash-facade\modules\plugins\ya-automatic-ai\automatic\system\python\lib\site-packages\starlette\middleware\base.py", line 70, in coro
[2024-04-17T12:04:04.550+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >         await self.app(scope, receive_or_disconnect, send_no_error)
[2024-04-17T12:04:04.550+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >       File "D:\Code\gamerhash-facade\modules\plugins\ya-automatic-ai\automatic\system\python\lib\site-packages\starlette\middleware\cors.py", line 84, in __call__
[2024-04-17T12:04:04.550+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >         await self.app(scope, receive, send)
[2024-04-17T12:04:04.550+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >       File "D:\Code\gamerhash-facade\modules\plugins\ya-automatic-ai\automatic\system\python\lib\site-packages\starlette\middleware\gzip.py", line 26, in __call__
[2024-04-17T12:04:04.550+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >         await self.app(scope, receive, send)
[2024-04-17T12:04:04.550+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >       File "D:\Code\gamerhash-facade\modules\plugins\ya-automatic-ai\automatic\system\python\lib\site-packages\starlette\middleware\exceptions.py", line 79, in __call__
[2024-04-17T12:04:04.550+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >         raise exc
[2024-04-17T12:04:04.550+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >       File "D:\Code\gamerhash-facade\modules\plugins\ya-automatic-ai\automatic\system\python\lib\site-packages\starlette\middleware\exceptions.py", line 68, in __call__
[2024-04-17T12:04:04.550+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >         await self.app(scope, receive, sender)
[2024-04-17T12:04:04.550+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >       File "D:\Code\gamerhash-facade\modules\plugins\ya-automatic-ai\automatic\system\python\lib\site-packages\fastapi\middleware\asyncexitstack.py", line 21, in __call__
[2024-04-17T12:04:04.550+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >         raise e
[2024-04-17T12:04:04.550+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >       File "D:\Code\gamerhash-facade\modules\plugins\ya-automatic-ai\automatic\system\python\lib\site-packages\fastapi\middleware\asyncexitstack.py", line 18, in __call__
[2024-04-17T12:04:04.550+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >         await self.app(scope, receive, send)
[2024-04-17T12:04:04.550+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >       File "D:\Code\gamerhash-facade\modules\plugins\ya-automatic-ai\automatic\system\python\lib\site-packages\starlette\routing.py", line 718, in __call__
[2024-04-17T12:04:04.550+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >         await route.handle(scope, receive, send)
[2024-04-17T12:04:04.550+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >       File "D:\Code\gamerhash-facade\modules\plugins\ya-automatic-ai\automatic\system\python\lib\site-packages\starlette\routing.py", line 276, in handle
[2024-04-17T12:04:04.551+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >         await self.app(scope, receive, send)
[2024-04-17T12:04:04.551+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >       File "D:\Code\gamerhash-facade\modules\plugins\ya-automatic-ai\automatic\system\python\lib\site-packages\starlette\routing.py", line 66, in app
[2024-04-17T12:04:04.551+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >         response = await func(request)
[2024-04-17T12:04:04.551+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >       File "D:\Code\gamerhash-facade\modules\plugins\ya-automatic-ai\automatic\system\python\lib\site-packages\fastapi\routing.py", line 237, in app
[2024-04-17T12:04:04.551+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >         raw_response = await run_endpoint_function(
[2024-04-17T12:04:04.551+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >       File "D:\Code\gamerhash-facade\modules\plugins\ya-automatic-ai\automatic\system\python\lib\site-packages\fastapi\routing.py", line 165, in run_endpoint_function
[2024-04-17T12:04:04.551+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >         return await run_in_threadpool(dependant.call, **values)
[2024-04-17T12:04:04.551+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >       File "D:\Code\gamerhash-facade\modules\plugins\ya-automatic-ai\automatic\system\python\lib\site-packages\starlette\concurrency.py", line 41, in run_in_threadpool
[2024-04-17T12:04:04.551+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >         return await anyio.to_thread.run_sync(func, *args)
[2024-04-17T12:04:04.551+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >       File "D:\Code\gamerhash-facade\modules\plugins\ya-automatic-ai\automatic\system\python\lib\site-packages\anyio\to_thread.py", line 33, in run_sync
[2024-04-17T12:04:04.551+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >         return await get_asynclib().run_sync_in_worker_thread(
[2024-04-17T12:04:04.551+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >       File "D:\Code\gamerhash-facade\modules\plugins\ya-automatic-ai\automatic\system\python\lib\site-packages\anyio\_backends\_asyncio.py", line 877, in run_sync_in_worker_thread
[2024-04-17T12:04:04.551+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >         return await future
[2024-04-17T12:04:04.551+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >       File "D:\Code\gamerhash-facade\modules\plugins\ya-automatic-ai\automatic\system\python\lib\site-packages\anyio\_backends\_asyncio.py", line 807, in run
[2024-04-17T12:04:04.551+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >         result = context.run(func, *args)
[2024-04-17T12:04:04.551+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >       File "D:\Code\gamerhash-facade\modules\plugins\ya-automatic-ai\automatic\webui\modules\api\api.py", line 380, in text2imgapi
[2024-04-17T12:04:04.551+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >         processed = process_images(p)
[2024-04-17T12:04:04.551+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >       File "D:\Code\gamerhash-facade\modules\plugins\ya-automatic-ai\automatic\webui\modules\processing.py", line 732, in process_images
[2024-04-17T12:04:04.551+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >         res = process_images_inner(p)
[2024-04-17T12:04:04.551+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >       File "D:\Code\gamerhash-facade\modules\plugins\ya-automatic-ai\automatic\webui\modules\processing.py", line 867, in process_images_inner
[2024-04-17T12:04:04.551+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >         samples_ddim = p.sample(conditioning=p.c, unconditional_conditioning=p.uc, seeds=p.seeds, subseeds=p.subseeds, subseed_strength=p.subseed_strength, prompts=p.prompts)
[2024-04-17T12:04:04.551+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >       File "D:\Code\gamerhash-facade\modules\plugins\ya-automatic-ai\automatic\webui\modules\processing.py", line 1140, in sample
[2024-04-17T12:04:04.551+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >         samples = self.sampler.sample(self, x, conditioning, unconditional_conditioning, image_conditioning=self.txt2img_image_conditioning(x))
[2024-04-17T12:04:04.551+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >       File "D:\Code\gamerhash-facade\modules\plugins\ya-automatic-ai\automatic\webui\modules\sd_samplers_kdiffusion.py", line 235, in sample
[2024-04-17T12:04:04.551+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >         samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args=self.sampler_extra_args, disable=False, callback=self.callback_state, **extra_params_kwargs))
[2024-04-17T12:04:04.551+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >       File "D:\Code\gamerhash-facade\modules\plugins\ya-automatic-ai\automatic\webui\modules\sd_samplers_common.py", line 261, in launch_sampling
[2024-04-17T12:04:04.551+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >         return func()
[2024-04-17T12:04:04.551+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >       File "D:\Code\gamerhash-facade\modules\plugins\ya-automatic-ai\automatic\webui\modules\sd_samplers_kdiffusion.py", line 235, in <lambda>
[2024-04-17T12:04:04.551+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >         samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args=self.sampler_extra_args, disable=False, callback=self.callback_state, **extra_params_kwargs))
[2024-04-17T12:04:04.551+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >       File "D:\Code\gamerhash-facade\modules\plugins\ya-automatic-ai\automatic\system\python\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
[2024-04-17T12:04:04.551+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >         return func(*args, **kwargs)
[2024-04-17T12:04:04.551+0200 DEBUG hyper::proto::h1::io] parsed 4 headers
[2024-04-17T12:04:04.551+0200 DEBUG hyper::proto::h1::conn] incoming body is content-length (465 bytes)
[2024-04-17T12:04:04.551+0200 DEBUG hyper::proto::h1::conn] incoming body completed
[2024-04-17T12:04:04.551+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >       File "D:\Code\gamerhash-facade\modules\plugins\ya-automatic-ai\automatic\webui\repositories\k-diffusion\k_diffusion\sampling.py", line 128, in sample_euler
[2024-04-17T12:04:04.551+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >         denoised = model(x, sigma_hat * s_in, **extra_args)
[2024-04-17T12:04:04.551+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >       File "D:\Code\gamerhash-facade\modules\plugins\ya-automatic-ai\automatic\system\python\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
[2024-04-17T12:04:04.551+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >         return forward_call(*args, **kwargs)
[2024-04-17T12:04:04.551+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >       File "D:\Code\gamerhash-facade\modules\plugins\ya-automatic-ai\automatic\webui\modules\sd_samplers_cfg_denoiser.py", line 201, in forward
[2024-04-17T12:04:04.551+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >         devices.test_for_nans(x_out, "unet")
[2024-04-17T12:04:04.551+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >       File "D:\Code\gamerhash-facade\modules\plugins\ya-automatic-ai\automatic\webui\modules\devices.py", line 136, in test_for_nans
[2024-04-17T12:04:04.551+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >         raise NansException(message)
[2024-04-17T12:04:04.551+0200 DEBUG ya_runtime_ai::process::automatic::monitor] >     modules.devices.NansException: A tensor with all NaNs was produced in Unet. This could be either because there's not enough precision to represent the picture, or because your video card does not support half type. Try setting the "Upcast cross attention layer to float32" option in Settings > Stable Diffusion or using the --no-half commandline argument to fix this. Use --disable-nan-check commandline argument to disable this check.
[2024-04-17T12:04:04.551+0200 DEBUG ya_runtime_ai::process::automatic::monitor] > 
[2024-04-17T12:04:04.551+0200 DEBUG ya_runtime_ai::process::automatic::monitor] > ---
[2024-04-17T12:04:04.551+0200 DEBUG hyper::client::pool] pooling idle connection for ("http", localhost:7861)
[2024-04-17T12:04:04.552+0200 INFO  ya_gsb_http_proxy::gsb_to_http] sending GsbEvent nr 0
[2024-04-17T12:04:04.766+0200 DEBUG ya_runtime_ai] Successfully sent activity usage message

User received following response 200 OK:

{"error":"NansException","detail":"","body":"","errors":"A tensor with all NaNs was produced in Unet. This could be either because there's not enough precision to represent the picture, or because your video card does not support half type. Try setting the \"Upcast cross attention layer to float32\" option in Settings > Stable Diffusion or using the --no-half commandline argument to fix this. Use --disable-nan-check commandline argument to disable this check."}

It seems that Automatic API returned 500, but gsb http proxy returns 200.

Versions:

PS D:\Code\gamerhash-facade> .\modules\golem\yagna.exe --version
yagna 0.16.0-ai-rc13 (9ab5b417 2024-04-05 build #422)
[2024-04-17 12:25:20.241469 +02:00] ERROR [src\main.rs:166] ya-runtime-ai 0.1.1

(SHA256 B6F47B478BE36118D3DB732173C65C194BFF64342E28BFEC776F182A37DA69A4) (ya-runtime-ai still prints 0.1.1)

Originally posted by @pwalski in https://github.com/golemfactory/ya-runtime-ai/issues/79#issuecomment-2060920089

pwalski commented 2 months ago

EDIT: OUTDATED

pwalski commented 2 months ago

Error response returned by Automatic API recommends using --disable-nan-check and --no-half parameters. It happens because there were some problems with model I used. ya-runtime-ai provides a way to configure --ckpt cmdline param using golem.srv.comp.ai.model property. There is no other way to add additional cmdline parameters, and adding some of them might be necessary to work with some models.

Maybe we should have a property where user could add additional cmdline parameters? This list would be validated using a whitelist of allowed parameters (like --no-half, --no-half-vae, --medvram, --disable-nan-check, etc. More of them here https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Command-Line-Arguments-and-Settings)

It seems like a property specific to Automatic runtime, but vllm might also need an additional --chat-template cmdline arg, so adding some general extra runtime cmdline args property could be useful later, and only a validation of this property (checking against whitelist) would be a runtime specific implementation.