[bug]: Invoke AI not work with Runpod (always disconnected)

e471wo commented 1 year ago

Is there an existing issue for this?

[X] I have searched the existing issues

OS

Linux

GPU

cuda

VRAM

No response

What version did you experience this issue on?

3.0.1post3

What happened?

Hi All,

For the first time, I tried to use Invoke. I use the runpod service as there is a docker updated with the last version of invoke. After the installation (who take 10 minutes), I have access to the interface... BUT impossible to generate something as the button invoke is not active.

But more than that, Impossible to be connected. (Always disconnected). I checkd the log, but sadly, there is nothing. I don't know if it's only runpod or all cloud service... And most important, no idea how to correct. What to change...

any idea ? thanks.

Screenshots

Manage Help Pods Pod Name/ID Filter Pod Name/ID Filter GPU Type GPU Type Status Savings InvokeAI - A Stable Diffusion Toolkit ID: 7niiskrji9w3mb 1 x A40 4 vCPU 50 GB RAM ghcr.io/invoke-ai/invokeai:v3.0-latest-cuda On-Demand - Secure Cloud Running 10 GB Disk 20 GB Pod Volume Volume Path: /workspace HR 734 Mbps 725 Mbps 1870 MBps Pod Uptime: 3h 12 seconds ago Pod Utilization CPU 0% Mem 3% Disk Utilization Container 84% Volume 37% GPU Utilization 0 0% GPU Memory Used 0 0% $0.79/hr Note: All pod prices are updated every Sunday at midnight to match standard prices on deploy page. © RunPod 2023

My Pods 0 matches 2023-08-04T11:30:36.405560739Z Setting up openssh-server (1:8.9p1-3ubuntu0.3) ... 2023-08-04T11:30:36.489294783Z debconf: unable to initialize frontend: Dialog 2023-08-04T11:30:36.489326203Z debconf: (TERM is not set, so the dialog frontend is not usable.) 2023-08-04T11:30:36.489331423Z debconf: falling back to frontend: Readline 2023-08-04T11:30:36.599784210Z 2023-08-04T11:30:36.599816663Z Creating config file /etc/ssh/sshd_config with new version 2023-08-04T11:30:37.691670912Z Creating SSH2 RSA key; this may take some time ... 2023-08-04T11:30:37.700330621Z 3072 SHA256:CB/iEbkzq3qrDgsRNqbYvemq8e/auZ/Ow4BPifYT9iI root@bb63deaa983d (RSA) 2023-08-04T11:30:37.718009254Z Creating SSH2 ECDSA key; this may take some time ... 2023-08-04T11:30:37.726119404Z 256 SHA256:F9IcgkXTw26+9HH0aKcvfMFY1v1RhS0EQcG1kMXLiHY root@bb63deaa983d (ECDSA) 2023-08-04T11:30:37.745244931Z Creating SSH2 ED25519 key; this may take some time ... 2023-08-04T11:30:37.752977715Z 256 SHA256:h024PmQYsNxkY97Dq7EDAW7Rw9hPoTdOf7KXrYaF1/s root@bb63deaa983d (ED25519) 2023-08-04T11:30:37.858858183Z invoke-rc.d: could not determine current runlevel 2023-08-04T11:30:37.863221686Z invoke-rc.d: policy-rc.d denied execution of start. 2023-08-04T11:30:38.020403552Z Created symlink /etc/systemd/system/sshd.service → /lib/systemd/system/ssh.service. 2023-08-04T11:30:38.020445063Z Created symlink /etc/systemd/system/multi-user.target.wants/ssh.service → /lib/systemd/system/ssh.service. 2023-08-04T11:30:38.269643265Z Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-08-04T11:30:38.525280511Z ~ /opt/invokeai 2023-08-04T11:30:38.529265060Z /opt/invokeai 2023-08-04T11:30:38.555350908Z * Starting OpenBSD Secure Shell server sshd 2023-08-04T11:30:38.571423945Z ...done. 2023-08-04T11:30:44.104486059Z [2023-08-04 11:30:44,104]::[InvokeAI]::INFO --> Patchmatch initialized 2023-08-04T11:30:44.203457343Z /opt/venv/invokeai/lib/python3.10/site-packages/torchvision/transforms/functional_tensor.py:5: UserWarning: The torchvision.transforms.functional_tensor module is deprecated in 0.15 and will be removed in 0.17. Please don't rely on it. You probably just need to use APIs in torchvision.transforms.functional or in torchvision.transforms.v2.functional. 2023-08-04T11:30:44.203486930Z warnings.warn( 2023-08-04T11:30:44.836930496Z [2023-08-04 11:30:44,836]::[uvicorn.error]::INFO --> Started server process [7] 2023-08-04T11:30:44.836960095Z [2023-08-04 11:30:44,836]::[uvicorn.error]::INFO --> Waiting for application startup. 2023-08-04T11:30:44.837108611Z [2023-08-04 11:30:44,837]::[InvokeAI]::INFO --> InvokeAI version 3.0.1post3 2023-08-04T11:30:44.837150150Z [2023-08-04 11:30:44,837]::[InvokeAI]::INFO --> Root directory = /workspace/invokeai 2023-08-04T11:30:44.862112156Z [2023-08-04 11:30:44,861]::[InvokeAI]::INFO --> GPU device = cuda NVIDIA A40 2023-08-04T11:30:44.869922479Z [2023-08-04 11:30:44,869]::[InvokeAI]::INFO --> Scanning /workspace/invokeai/models for new models 2023-08-04T11:30:45.282325223Z [2023-08-04 11:30:45,282]::[InvokeAI]::INFO --> Scanned 5 files and directories, imported 0 models 2023-08-04T11:30:45.282353839Z [2023-08-04 11:30:45,282]::[InvokeAI]::INFO --> Model manager service initialized 2023-08-04T11:30:45.293679178Z [2023-08-04 11:30:45,293]::[uvicorn.error]::INFO --> Application startup complete. 2023-08-04T11:30:45.294425485Z [2023-08-04 11:30:45,294]::[uvicorn.error]::INFO --> Uvicorn running on http://0.0.0.0:9090 (Press CTRL+C to quit) 2023-08-04T11:35:50.496256417Z [2023-08-04 11:35:50,495]::[uvicorn.access]::INFO --> 100.64.0.21:56838 - "GET / HTTP/1.1" 200 2023-08-04T11:35:50.681527616Z [2023-08-04 11:35:50,681]::[uvicorn.access]::INFO --> 100.64.0.21:56838 - "GET /assets/index-e8055717.js HTTP/1.1" 200 2023-08-04T11:35:51.233496849Z [2023-08-04 11:35:51,233]::[uvicorn.access]::INFO --> 100.64.0.21:56838 - "GET /locales/en.json HTTP/1.1" 200 2023-08-04T11:35:51.268821374Z [2023-08-04 11:35:51,268]::[uvicorn.access]::INFO --> 100.64.0.21:56838 - "GET /assets/ThemeLocaleProvider-8a2a915e.js HTTP/1.1" 200 2023-08-04T11:35:51.298561091Z [2023-08-04 11:35:51,298]::[uvicorn.access]::INFO --> 100.64.0.21:56854 - "GET /assets/ThemeLocaleProvider-5b992bc7.css HTTP/1.1" 200 2023-08-04T11:35:51.307374514Z [2023-08-04 11:35:51,307]::[uvicorn.access]::INFO --> 100.64.0.21:56866 - "GET /assets/logo-13003d72.png HTTP/1.1" 200 2023-08-04T11:35:51.308846020Z [2023-08-04 11:35:51,308]::[uvicorn.access]::INFO --> 100.64.0.21:56838 - "GET /assets/MantineProvider-d5b0fbfc.js HTTP/1.1" 200 2023-08-04T11:35:51.474236863Z [2023-08-04 11:35:51,473]::[uvicorn.access]::INFO --> 100.64.0.21:56876 - "GET /assets/App-6125620a.css HTTP/1.1" 200 2023-08-04T11:35:51.482508959Z [2023-08-04 11:35:51,482]::[uvicorn.access]::INFO --> 100.64.0.21:56874 - "GET /assets/App-784c01fa.js HTTP/1.1" 200 2023-08-04T11:35:51.724648664Z [2023-08-04 11:35:51,724]::[uvicorn.access]::INFO --> 100.64.0.21:56874 - "GET /assets/favicon-0d253ced.ico HTTP/1.1" 200 2023-08-04T11:35:52.138580710Z [2023-08-04 11:35:52,138]::[uvicorn.access]::INFO --> 100.64.0.25:56796 - "GET / HTTP/1.1" 200 2023-08-04T11:35:52.229590292Z [2023-08-04 11:35:52,229]::[uvicorn.access]::INFO --> 100.64.0.21:56874 - "GET /assets/inter-latin-wght-normal-450f3ba4.woff2 HTTP/1.1" 200 2023-08-04T11:35:56.860676924Z [2023-08-04 11:35:56,860]::[InvokeAI]::INFO --> NSFW checker initialized 2023-08-04T11:35:56.862440678Z [2023-08-04 11:35:56,862]::[uvicorn.access]::INFO --> 100.64.0.21:56874 - "GET /api/v1/app/config HTTP/1.1" 200 2023-08-04T11:35:56.865514152Z [2023-08-04 11:35:56,865]::[uvicorn.access]::INFO --> 100.64.0.21:56876 - "GET /api/v1/app/version HTTP/1.1" 200 2023-08-04T11:35:56.869922829Z [2023-08-04 11:35:56,869]::[uvicorn.access]::INFO --> 100.64.0.21:56890 - "GET /api/v1/models/?model_type=embedding HTTP/1.1" 200 2023-08-04T11:35:56.871236298Z [2023-08-04 11:35:56,871]::[uvicorn.access]::INFO --> 100.64.0.21:56904 - "GET /api/v1/models/?base_models=sd-1&base_models=sd-2&base_models=sdxl&model_type=main HTTP/1.1" 200 2023-08-04T11:35:56.872891980Z [2023-08-04 11:35:56,872]::[uvicorn.access]::INFO --> 100.64.0.21:56920 - "GET /api/v1/boards/?all=true HTTP/1.1" 200 2023-08-04T11:35:56.873352604Z [2023-08-04 11:35:56,873]::[uvicorn.access]::INFO --> 100.64.0.21:56934 - "GET /api/v1/models/?model_type=lora HTTP/1.1" 200 2023-08-04T11:35:56.874101521Z [2023-08-04 11:35:56,874]::[uvicorn.access]::INFO --> 100.64.0.21:56948 - "GET /api/v1/models/?model_type=vae HTTP/1.1" 200 2023-08-04T11:35:56.876011152Z [2023-08-04 11:35:56,875]::[uvicorn.access]::INFO --> 100.64.0.21:56950 - "GET /api/v1/images/?board_id=none&categories=general&is_intermediate=false&limit=0&offset=0 HTTP/1.1" 200 2023-08-04T11:35:56.876785222Z [2023-08-04 11:35:56,876]::[uvicorn.access]::INFO --> 100.64.0.21:56956 - "GET /api/v1/images/?board_id=none&categories=control&categories=mask&categories=user&categories=other&is_intermediate=false&limit=0&offset=0 HTTP/1.1" 200 2023-08-04T11:35:56.877363670Z [2023-08-04 11:35:56,877]::[uvicorn.access]::INFO --> 100.64.0.21:56962 - "GET /api/v1/images/?board_id=none&categories=general&is_intermediate=false&limit=100&offset=0 HTTP/1.1" 200 2023-08-04T11:35:56.877737118Z [2023-08-04 11:35:56,877]::[uvicorn.access]::INFO --> 100.64.0.21:56966 - "GET /api/v1/models/?model_type=controlnet HTTP/1.1" 200 2023-08-04T11:37:50.649605652Z [2023-08-04 11:37:50,648]::[uvicorn.access]::INFO --> 100.64.0.21:39492 - "GET /api/v1/images/?is_intermediate=true HTTP/1.1" 200 2023-08-04T11:38:57.165036937Z [2023-08-04 11:38:57,164]::[uvicorn.access]::INFO --> 100.64.0.21:55654 - "GET /api/v1/images/?board_id=none&categories=control&categories=mask&categories=user&categories=other&is_intermediate=false&limit=100&offset=0 HTTP/1.1" 200 2023-08-04T11:39:39.542210163Z [2023-08-04 11:39:39,541]::[uvicorn.access]::INFO --> 100.64.0.21:56520 - "GET /api/v1/models/?base_models=sd-1&base_models=sd-2&base_models=sdxl&base_models=sdxl-refiner&model_type=main HTTP/1.1" 200 2023-08-04T11:40:24.074523720Z [2023-08-04 11:40:24,074]::[uvicorn.access]::INFO --> 100.64.0.21:33826 - "GET /api/v1/boards/?all=true HTTP/1.1" 200 2023-08-04T11:40:24.077110634Z [2023-08-04 11:40:24,076]::[uvicorn.access]::INFO --> 100.64.0.21:33824 - "GET /api/v1/models/?model_type=lora HTTP/1.1" 200 2023-08-04T11:40:24.079082889Z [2023-08-04 11:40:24,078]::[uvicorn.access]::INFO --> 100.64.0.21:33842 - "GET /api/v1/images/?board_id=none&categories=control&categories=mask&categories=user&categories=other&is_intermediate=false&limit=0&offset=0 HTTP/1.1" 200 2023-08-04T11:40:24.081034678Z [2023-08-04 11:40:24,080]::[uvicorn.access]::INFO --> 100.64.0.21:33844 - "GET /api/v1/models/?model_type=vae HTTP/1.1" 200 2023-08-04T11:40:24.081911775Z [2023-08-04 11:40:24,081]::[uvicorn.access]::INFO --> 100.64.0.21:33846 - "GET /api/v1/models/?base_models=sd-1&base_models=sd-2&base_models=sdxl&model_type=main HTTP/1.1" 200 2023-08-04T11:40:24.082875024Z [2023-08-04 11:40:24,082]::[uvicorn.access]::INFO --> 100.64.0.21:33860 - "GET /api/v1/app/config HTTP/1.1" 200 2023-08-04T11:40:24.083689274Z [2023-08-04 11:40:24,083]::[uvicorn.access]::INFO --> 100.64.0.21:33864 - "GET /api/v1/images/?board_id=none&categories=general&is_intermediate=false&limit=100&offset=0 HTTP/1.1" 200 2023-08-04T11:40:24.084113326Z [2023-08-04 11:40:24,083]::[uvicorn.access]::INFO --> 100.64.0.21:33870 - "GET /api/v1/models/?model_type=controlnet HTTP/1.1" 200 2023-08-04T11:40:24.084638288Z [2023-08-04 11:40:24,084]::[uvicorn.access]::INFO --> 100.64.0.21:33872 - "GET /api/v1/app/version HTTP/1.1" 200 2023-08-04T11:40:24.085323753Z [2023-08-04 11:40:24,085]::[uvicorn.access]::INFO --> 100.64.0.21:33896 - "GET /api/v1/images/?board_id=none&categories=general&is_intermediate=false&limit=0&offset=0 HTTP/1.1" 200 2023-08-04T11:40:24.085782437Z [2023-08-04 11:40:24,085]::[uvicorn.access]::INFO --> 100.64.0.21:33882 - "GET /api/v1/models/?model_type=embedding HTTP/1.1" 200 2023-08-04T11:42:46.862984381Z [2023-08-04 11:42:46,862]::[uvicorn.access]::INFO --> 100.64.0.21:36622 - "GET /api/v1/images/?is_intermediate=true HTTP/1.1" 200 2023-08-04T11:42:55.607753855Z [2023-08-04 11:42:55,607]::[uvicorn.access]::INFO --> 100.64.0.21:50370 - "GET / HTTP/1.1" 304 2023-08-04T11:42:56.593832079Z [2023-08-04 11:42:56,593]::[uvicorn.access]::INFO --> 100.64.0.21:50370 - "GET /api/v1/app/config HTTP/1.1" 200 2023-08-04T11:42:56.625093097Z [2023-08-04 11:42:56,624]::[uvicorn.access]::INFO --> 100.64.0.21:50374 - "GET /api/v1/app/version HTTP/1.1" 200 2023-08-04T11:42:56.626870020Z [2023-08-04 11:42:56,626]::[uvicorn.access]::INFO --> 100.64.0.21:50370 - "GET /api/v1/models/?model_type=lora HTTP/1.1" 200 2023-08-04T11:42:56.628206431Z [2023-08-04 11:42:56,628]::[uvicorn.access]::INFO --> 100.64.0.21:50384 - "GET /api/v1/models/?model_type=embedding HTTP/1.1" 200 2023-08-04T11:42:56.630272788Z [2023-08-04 11:42:56,630]::[uvicorn.access]::INFO --> 100.64.0.21:50398 - "GET /api/v1/models/?model_type=vae HTTP/1.1" 200 2023-08-04T11:42:56.631281171Z [2023-08-04 11:42:56,631]::[uvicorn.access]::INFO --> 100.64.0.21:50406 - "GET /api/v1/boards/?all=true HTTP/1.1" 200 2023-08-04T11:42:56.632690265Z [2023-08-04 11:42:56,632]::[uvicorn.access]::INFO --> 100.64.0.21:50418 - "GET /api/v1/models/?base_models=sd-1&base_models=sd-2&base_models=sdxl&model_type=main HTTP/1.1" 200 2023-08-04T11:42:56.635182965Z [2023-08-04 11:42:56,635]::[uvicorn.access]::INFO --> 100.64.0.21:50426 - "GET /api/v1/images/?board_id=none&categories=general&is_intermediate=false&limit=0&offset=0 HTTP/1.1" 200 2023-08-04T11:42:56.635987730Z [2023-08-04 11:42:56,635]::[uvicorn.access]::INFO --> 100.64.0.21:50442 - "GET /api/v1/models/?model_type=controlnet HTTP/1.1" 200 2023-08-04T11:42:56.637105672Z [2023-08-04 11:42:56,636]::[uvicorn.access]::INFO --> 100.64.0.21:50444 - "GET /api/v1/images/?board_id=none&categories=control&categories=mask&categories=user&categories=other&is_intermediate=false&limit=0&offset=0 HTTP/1.1" 200 2023-08-04T11:42:56.637827688Z [2023-08-04 11:42:56,637]::[uvicorn.access]::INFO --> 100.64.0.21:50456 - "GET /api/v1/images/?board_id=none&categories=general&is_intermediate=false&limit=100&offset=0 HTTP/1.1" 200 2023-08-04T11:47:00.522198264Z [2023-08-04 11:47:00,521]::[uvicorn.access]::INFO --> 100.64.0.21:46462 - "GET /api/v1/models/?base_models=sd-1&base_models=sd-2&base_models=sdxl&base_models=sdxl-refiner&model_type=main HTTP/1.1" 200 2023-08-04T13:10:47.307073285Z [2023-08-04 13:10:47,306]::[uvicorn.access]::INFO --> 100.64.0.21:41352 - "GET /api/v1/images/?is_intermediate=true HTTP/1.1" 200 2023-08-04T13:10:51.529214614Z [2023-08-04 13:10:51,528]::[uvicorn.access]::INFO --> 100.64.0.21:49892 - "GET / HTTP/1.1" 304 2023-08-04T13:10:52.218038962Z [2023-08-04 13:10:52,217]::[uvicorn.access]::INFO --> 100.64.0.21:49892 - "GET /locales/en.json HTTP/1.1" 200 2023-08-04T13:10:53.174140417Z [2023-08-04 13:10:53,173]::[uvicorn.access]::INFO --> 100.64.0.21:49920 - "GET /api/v1/app/config HTTP/1.1" 200 2023-08-04T13:10:53.175556249Z [2023-08-04 13:10:53,175]::[uvicorn.access]::INFO --> 100.64.0.21:49906 - "GET /api/v1/app/version HTTP/1.1" 200 2023-08-04T13:10:53.180344333Z [2023-08-04 13:10:53,180]::[uvicorn.access]::INFO --> 100.64.0.21:49924 - "GET /api/v1/models/?model_type=vae HTTP/1.1" 200 2023-08-04T13:10:53.188493103Z [2023-08-04 13:10:53,188]::[uvicorn.access]::INFO --> 100.64.0.21:49950 - "GET /api/v1/models/?model_type=lora HTTP/1.1" 200 2023-08-04T13:10:53.189253243Z [2023-08-04 13:10:53,189]::[uvicorn.access]::INFO --> 100.64.0.21:49934 - "GET /api/v1/boards/?all=true HTTP/1.1" 200 2023-08-04T13:10:53.189874783Z [2023-08-04 13:10:53,189]::[uvicorn.access]::INFO --> 100.64.0.21:49928 - "GET /api/v1/models/?model_type=controlnet HTTP/1.1" 200 2023-08-04T13:10:53.190962348Z [2023-08-04 13:10:53,190]::[uvicorn.access]::INFO --> 100.64.0.21:49952 - "GET /api/v1/models/?model_type=embedding HTTP/1.1" 200 2023-08-04T13:10:53.191986939Z [2023-08-04 13:10:53,191]::[uvicorn.access]::INFO --> 100.64.0.21:49956 - "GET /api/v1/models/?base_models=sd-1&base_models=sd-2&base_models=sdxl&model_type=main HTTP/1.1" 200 2023-08-04T13:10:53.209389418Z [2023-08-04 13:10:53,209]::[uvicorn.access]::INFO --> 100.64.0.21:49920 - "GET /api/v1/images/?board_id=none&categories=control&categories=mask&categories=user&categories=other&is_intermediate=false&limit=0&offset=0 HTTP/1.1" 200 2023-08-04T13:10:53.211949741Z [2023-08-04 13:10:53,211]::[uvicorn.access]::INFO --> 100.64.0.21:49906 - "GET /api/v1/images/?board_id=none&categories=general&is_intermediate=false&limit=0&offset=0 HTTP/1.1" 200 2023-08-04T13:10:53.214634717Z [2023-08-04 13:10:53,214]::[uvicorn.access]::INFO --> 100.64.0.21:49962 - "GET /api/v1/images/?board_id=none&categories=general&is_intermediate=false&limit=100&offset=0 HTTP/1.1" 200 2023-08-04T13:15:02.772173565Z [2023-08-04 13:15:02,771]::[InvokeAI]::INFO --> Scanning /workspace/invokeai/models for new models 2023-08-04T13:15:02.840575448Z [2023-08-04 13:15:02,840]::[InvokeAI]::INFO --> Scanned 5 files and directories, imported 0 models 2023-08-04T13:15:02.840794808Z [2023-08-04 13:15:02,840]::[uvicorn.access]::INFO --> 100.64.0.21:59886 - "POST /api/v1/models/sync HTTP/1.1" 201 2023-08-04T13:15:02.928522896Z [2023-08-04 13:15:02,928]::[uvicorn.access]::INFO --> 100.64.0.21:59886 - "GET /api/v1/models/?base_models=sd-1&base_models=sd-2&base_models=sdxl&model_type=main HTTP/1.1" 200 2023-08-04T13:40:36.747324225Z [2023-08-04 13:40:36,746]::[uvicorn.access]::INFO --> 100.64.0.21:52614 - "GET /api/v1/images/?is_intermediate=true HTTP/1.1" 200 2023-08-04T13:49:05.441566122Z [2023-08-04 13:49:05,441]::[uvicorn.access]::INFO --> 100.64.0.21:57968 - "GET /assets/ThemeLocaleProvider-8a2a915e.js HTTP/1.1" 304 2023-08-04T13:49:06.188671469Z [2023-08-04 13:49:06,188]::[uvicorn.access]::INFO --> 100.64.0.21:57968 - "GET /api/v1/app/config HTTP/1.1" 200 2023-08-04T13:49:06.222357468Z [2023-08-04 13:49:06,221]::[uvicorn.access]::INFO --> 100.64.0.21:57968 - "GET /api/v1/app/version HTTP/1.1" 200 2023-08-04T13:49:06.245479446Z [2023-08-04 13:49:06,245]::[uvicorn.access]::INFO --> 100.64.0.21:57980 - "GET /api/v1/models/?model_type=embedding HTTP/1.1" 200 2023-08-04T13:49:06.255468625Z [2023-08-04 13:49:06,255]::[uvicorn.access]::INFO --> 100.64.0.21:57968 - "GET /api/v1/models/?model_type=lora HTTP/1.1" 200 2023-08-04T13:49:06.271847592Z [2023-08-04 13:49:06,271]::[uvicorn.access]::INFO --> 100.64.0.21:57988 - "GET /api/v1/models/?base_models=sd-1&base_models=sd-2&base_models=sdxl&model_type=main HTTP/1.1" 200 2023-08-04T13:49:06.279110761Z [2023-08-04 13:49:06,278]::[uvicorn.access]::INFO --> 100.64.0.21:57980 - "GET /api/v1/images/?board_id=none&categories=general&is_intermediate=false&limit=100&offset=0 HTTP/1.1" 200 2023-08-04T13:49:06.281725939Z [2023-08-04 13:49:06,281]::[uvicorn.access]::INFO --> 100.64.0.21:58002 - "GET /api/v1/images/?board_id=none&categories=general&is_intermediate=false&limit=0&offset=0 HTTP/1.1" 200 2023-08-04T13:49:06.282619428Z [2023-08-04 13:49:06,282]::[uvicorn.access]::INFO --> 100.64.0.21:58014 - "GET /api/v1/models/?model_type=controlnet HTTP/1.1" 200 2023-08-04T13:49:06.284918597Z [2023-08-04 13:49:06,284]::[uvicorn.access]::INFO --> 100.64.0.21:58026 - "GET /api/v1/models/?model_type=vae HTTP/1.1" 200 2023-08-04T13:49:06.285654752Z [2023-08-04 13:49:06,285]::[uvicorn.access]::INFO --> 100.64.0.21:58042 - "GET /api/v1/boards/?all=true HTTP/1.1" 200 2023-08-04T13:49:06.288466019Z [2023-08-04 13:49:06,288]::[uvicorn.access]::INFO --> 100.64.0.21:57968 - "GET /api/v1/images/?board_id=none&categories=control&categories=mask&categories=user&categories=other&is_intermediate=false&limit=0&offset=0 HTTP/1.1" 200 2023-08-04T13:50:08.489225422Z [2023-08-04 13:50:08,488]::[uvicorn.access]::INFO --> 100.64.0.21:39472 - "GET /api/v1/images/?is_intermediate=true HTTP/1.1" 200 2023-08-04T13:51:27.986555968Z [2023-08-04 13:51:27,986]::[uvicorn.access]::INFO --> 100.64.0.21:53148 - "GET /api/v1/images/?is_intermediate=true HTTP/1.1" 200

Additional context

No response

Contact Details

No response

KarimJedda commented 1 year ago

Yeah, I think something is wrong with the proxy/LB runpod is using to serve the service.

If you check the Developer tools for your browser you'll see errors about mixed content on "http://" endpoint (vs https://).

I solved this using SSH tunneling to the service.

Steps:

Using the invokeai template, replace the docker container command with bash -c 'sleep infinity', this will make sure your container doesn't stop and bypasses all the stuff defined in the invokeai container, also, expose port 22 over TCP
Connect through SSH (over TCP) using ssh -L 9090:localhost:9090 root@....
cd /opt
source venv/invokeai/bin/activate
export INVOKEAI_ROOT=/workspace
invokeai-configure
invokeai --web

Screenshot from 2023-08-07 19-33-11

Let me know if you're stuck.

e471wo commented 1 year ago

I m sorry, but I m little stuck.

Here what I do.

first I use this image (docker):

InvokeAI - A Stable Diffusion Toolkit ghcr.io/invoke-ai/invokeai :v3.0-latest-cuda

there is a official image but impossible for me to build it.

after the server I change the docker command

I launch. In the log I have that (after the download of a lot of thing...):

2023-08-08T02:00:35.449932851-07:00 Setting up python3-distro (1.7.0-1) ...
2023-08-08T02:00:35.579824772-07:00 Setting up openssh-client (1:8.9p1-3ubuntu0.3) ...
2023-08-08T02:00:35.628852231-07:00 update-alternatives: using /usr/bin/ssh to provide /usr/bin/rsh (rsh) in auto mode
2023-08-08T02:00:35.628953382-07:00 update-alternatives: warning: skip creation of /usr/share/man/man1/rsh.1.gz because associated file /usr/share/man/man1/ssh.1.gz (of link group rsh) doesn't exist
2023-08-08T02:00:35.632136370-07:00 update-alternatives: using /usr/bin/slogin to provide /usr/bin/rlogin (rlogin) in auto mode
2023-08-08T02:00:35.632143450-07:00 update-alternatives: warning: skip creation of /usr/share/man/man1/rlogin.1.gz because associated file /usr/share/man/man1/slogin.1.gz (of link group rlogin) doesn't exist
2023-08-08T02:00:35.634984805-07:00 update-alternatives: using /usr/bin/scp to provide /usr/bin/rcp (rcp) in auto mode
2023-08-08T02:00:35.634991885-07:00 update-alternatives: warning: skip creation of /usr/share/man/man1/rcp.1.gz because associated file /usr/share/man/man1/scp.1.gz (of link group rcp) doesn't exist
2023-08-08T02:00:35.641987209-07:00 Setting up ssh-import-id (5.11-0ubuntu1) ...
2023-08-08T02:00:35.768591560-07:00 Setting up xauth (1:1.1-1build2) ...
2023-08-08T02:00:35.771675117-07:00 Setting up openssh-sftp-server (1:8.9p1-3ubuntu0.3) ...
2023-08-08T02:00:35.772960843-07:00 Setting up openssh-server (1:8.9p1-3ubuntu0.3) ...
2023-08-08T02:00:35.840695267-07:00 debconf: unable to initialize frontend: Dialog
2023-08-08T02:00:35.840714927-07:00 debconf: (TERM is not set, so the dialog frontend is not usable.)
2023-08-08T02:00:35.840718537-07:00 debconf: falling back to frontend: Readline
2023-08-08T02:00:35.932364378-07:00 
2023-08-08T02:00:35.932385169-07:00 Creating config file /etc/ssh/sshd_config with new version
2023-08-08T02:00:36.135353948-07:00 Creating SSH2 RSA key; this may take some time ...
2023-08-08T02:00:36.140344148-07:00 3072 SHA256:YkGdtSXye9BSgg5/875rzjNg0j5tswsJ6P95c3Aw/i0 root@4e5801c3e4a6 (RSA)
2023-08-08T02:00:36.150582410-07:00 Creating SSH2 ECDSA key; this may take some time ...
2023-08-08T02:00:36.155663972-07:00 256 SHA256:EMM9D9C3KYi3YPMijaM0V1BaNEN1KPUeg/TKWcedQVE root@4e5801c3e4a6 (ECDSA)
2023-08-08T02:00:36.168145892-07:00 Creating SSH2 ED25519 key; this may take some time ...
2023-08-08T02:00:36.172706277-07:00 256 SHA256:NXHviuGO+YnDhsRh7Q96LyHcmLAf5Ta5/MSybwIzTYE root@4e5801c3e4a6 (ED25519)
2023-08-08T02:00:36.245667043-07:00 invoke-rc.d: could not determine current runlevel
2023-08-08T02:00:36.248584468-07:00 invoke-rc.d: policy-rc.d denied execution of start.
2023-08-08T02:00:36.354949076-07:00 Created symlink /etc/systemd/system/sshd.service → /lib/systemd/system/ssh.service.
2023-08-08T02:00:36.354960366-07:00 Created symlink /etc/systemd/system/multi-user.target.wants/ssh.service → /lib/systemd/system/ssh.service.
2023-08-08T02:00:36.427372247-07:00 Processing triggers for libc-bin (2.35-0ubuntu3.1) ...
2023-08-08T02:00:36.462749382-07:00 ~ /opt/invokeai
2023-08-08T02:00:36.465253572-07:00 /opt/invokeai
2023-08-08T02:00:36.481111843-07:00  * Starting OpenBSD Secure Shell server sshd
2023-08-08T02:00:36.488175828-07:00    ...done.
2023-08-08T02:00:36.491797181-07:00 going to sleep

so it's good as you said. BUT after that...(I use the terminal of runpod eastead of SSH).

In the configuration, I have to delete the key of huffing (--------). If not ->x issue.
I start invoke with invoke --web --host 0.0.0.0 (or simply invoke.py --web, I tried both)
I go to runpod click to run http 9090...
invokeAI is launched

but I still disconnected.

the funny thing is that. If I shutdown the web server

I can still connect to Invoke (but it's still disconnect). so it's seems that I still on the old image. seems the sleep didn't work ? I sill to digg but if you have an idea. thanks.

ricardodolnl commented 1 year ago

There is also an official runpod template. It's probably better to use that one. Haven't tried it yet so I don't know if this one does work instead of the one you're using. Schermafbeelding 2023-08-08 om 12 04 43

e471wo commented 1 year ago

I tried too. I have exactly the same issue. No connection. I keep digging on both image.

also, I have no models...

KarimJedda commented 1 year ago

Yes @e471wo, if you try to connect to the invokeAi Web interface through the link that Runpod provides, it will not work with the current version because of a mix of things. The connection with the invokeAI backend is somehow made over "http" instead of "https" which the browser won't serve. A SSH tunnel bypasses this.

Try doing the same again, but don't use the web console, use the SSH tunnel over TCP:

Here you need to add 22:

And start your pod. After that, on the terminal in your computer, run: ssh -L 9090:localhost:9090 root@.... -p ... -i ~/.ssh/id_... with the correct parameters. Runpod will show you 2 ways to SSH, it's the bottom one.

Important: I recommend generating a separate public key following Runpod's guide and adding that in your profile settings on Runpod.

Then, in that terminal window, run the same commands and once that's done, open "http://localhost:9090" on your local computer browser. InvokeAi should launch and it will work.

I hope this helps. Good luck!

e471wo commented 1 year ago

Hi, thanks for your help. your solution work at home. Just wanted a solution from my job where I have no SSH acces :-D.

I took a month on Invoke.AI, I will see ... (sadly, I thing that 15 euro/month for 300 minuts is ...expensive).

thanks for you help.

KarimJedda commented 1 year ago

Glad I could be of assistance. Those cases where the employer blocks SSH are quite tough indeed. 👀

ricardodolnl commented 1 year ago

I've contacted RunPod and they just let me know they're working on a fix.

KarimJedda commented 1 year ago

From the latest release notes, this should have fixed it: https://github.com/invoke-ai/InvokeAI/pull/4116 Trying it out later tonight 🎉

e471wo commented 1 year ago

Yeah, they told me the same. Thanks for you help.

invoke-ai / InvokeAI