Finally, it's killed。

[Aceyng]

Has this issue been opened before?

• [x] It is not in the FAQ, I checked.
• [x] It is not in the issues, I searched.

Describe the bug

Last log: Finally, it's killed。 2023-05-08 16:03:05 Mounted .cache 2023-05-08 16:03:05 Mounted openpose 2023-05-08 16:03:05 Mounted LyCORIS 2023-05-08 16:03:05 Mounted ModelScope 2023-05-08 16:03:05 Mounted LDSR 2023-05-08 16:03:05 Mounted BLIP 2023-05-08 16:03:05 Mounted Hypernetworks 2023-05-08 16:03:05 Mounted VAE 2023-05-08 16:03:05 Mounted GFPGAN 2023-05-08 16:03:06 Mounted RealESRGAN 2023-05-08 16:03:06 Mounted Deepdanbooru 2023-05-08 16:03:06 Mounted ScuNET 2023-05-08 16:03:06 Mounted ControlNet 2023-05-08 16:03:06 Mounted .cache 2023-05-08 16:03:06 Mounted StableDiffusion 2023-05-08 16:03:06 Mounted embeddings 2023-05-08 16:03:06 Mounted ESRGAN 2023-05-08 16:03:06 Mounted config.json 2023-05-08 16:03:06 Mounted SwinIR 2023-05-08 16:03:06 Mounted Lora 2023-05-08 16:03:06 Mounted MiDaS 2023-05-08 16:03:06 Mounted styles.csv 2023-05-08 16:03:06 Mounted ui-config.json 2023-05-08 16:03:06 Mounted BSRGAN 2023-05-08 16:03:06 Mounted Codeformer 2023-05-08 16:03:06 Mounted extensions 2023-05-08 16:03:13 Warning: caught exception 'Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx', memory monitor disabled 2023-05-08 16:03:13 No module 'xformers'. Proceeding without it. 2023-05-08 16:03:14 Removing empty folder: /stable-diffusion-webui/models/BSRGAN 2023-05-08 16:03:14 Loading weights [c6bbc15e32] from /stable-diffusion-webui/models/Stable-diffusion/sd-v1-5-inpainting.ckpt 2023-05-08 16:03:18 Creating model from config: /stable-diffusion-webui/configs/v1-inpainting-inference.yaml 2023-05-08 16:03:18 LatentInpaintDiffusion: Running in eps-prediction mode 2023-05-08 16:03:19 DiffusionWrapper has 859.54 M params. 2023-05-08 16:05:05 Killed

auto-cpu

Hardware / Software

• OS: Ubuntu 20.04
• OS version:
• WSL version (if applicable): WSL2
• Docker Version: 23.0.5
• Docker compose version:
• Repo version: master
• RAM: 16GB
• GPU/VRAM:

Steps to Reproduce

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Additional context

And，After running the container for the first time, it threw: webui-docker-auto-cpu-1 | Killed webui-docker-auto-cpu-1 exited with code 137

What is the reason for the above? Thank you.

[DevilaN]

137 is mostly Out Of Memory Kill. docker inspect CONTAINER_ID_HERE should give hint about what's happened.

[Aceyng]

137 is mostly Out Of Memory Kill. docker inspect CONTAINER_ID_HERE should give hint about what's happened. Can you see any key information about this? Thank you. docker inspect c35e838c9646ec13d3120dc96b930068b8afd4f20b9481bbdbddf01cffb4c1ca [ { "Id": "c35e838c9646ec13d3120dc96b930068b8afd4f20b9481bbdbddf01cffb4c1ca", "Created": "2023-05-08T07:32:59.13549732Z", "Path": "/docker/entrypoint.sh", "Args": [ "/bin/sh", "-c", "python -u webui.py --listen --port 7860 ${CLI_ARGS}" ], "State": { "Status": "exited", "Running": false, "Paused": false, "Restarting": false, "OOMKilled": false, "Dead": false, "Pid": 0, "ExitCode": 137, "Error": "", "StartedAt": "2023-05-08T08:03:05.553904873Z", "FinishedAt": "2023-05-08T08:05:05.514785974Z" }, "Image": "sha256:5f11d9aad2138498dc34135e99907ee1eb6499f35d57db82e6f4b64acfa88feb", "ResolvConfPath": "/var/lib/docker/containers/c35e838c9646ec13d3120dc96b930068b8afd4f20b9481bbdbddf01cffb4c1ca/resolv.conf", "HostnamePath": "/var/lib/docker/containers/c35e838c9646ec13d3120dc96b930068b8afd4f20b9481bbdbddf01cffb4c1ca/hostname", "HostsPath": "/var/lib/docker/containers/c35e838c9646ec13d3120dc96b930068b8afd4f20b9481bbdbddf01cffb4c1ca/hosts", "LogPath": "/var/lib/docker/containers/c35e838c9646ec13d3120dc96b930068b8afd4f20b9481bbdbddf01cffb4c1ca/c35e838c9646ec13d3120dc96b930068b8afd4f20b9481bbdbddf01cffb4c1ca-json.log", "Name": "/webui-docker-auto-cpu-1", "RestartCount": 0, "Driver": "overlay2", "Platform": "linux", "MountLabel": "", "ProcessLabel": "", "AppArmorProfile": "", "ExecIDs": null, "HostConfig": { "Binds": [ "/var/yangroom/3sd/data:/data:rw", "/var/yangroom/3sd/output:/output:rw" ], "ContainerIDFile": "", "LogConfig": { "Type": "json-file", "Config": {} }, "NetworkMode": "webui-docker_default", "PortBindings": { "7860/tcp": [ { "HostIp": "", "HostPort": "7860" } ] }, "RestartPolicy": { "Name": "", "MaximumRetryCount": 0 }, "AutoRemove": false, "VolumeDriver": "", "VolumesFrom": null, "CapAdd": null, "CapDrop": null, "CgroupnsMode": "host", "Dns": null, "DnsOptions": null, "DnsSearch": null, "ExtraHosts": [], "GroupAdd": null, "IpcMode": "private", "Cgroup": "", "Links": null, "OomScoreAdj": 0, "PidMode": "", "Privileged": false, "PublishAllPorts": false, "ReadonlyRootfs": false, "SecurityOpt": null, "UTSMode": "", "UsernsMode": "", "ShmSize": 67108864, "Runtime": "runc", "ConsoleSize": [ 0, 0 ], "Isolation": "", "CpuShares": 0, "Memory": 0, "NanoCpus": 0, "CgroupParent": "", "BlkioWeight": 0, "BlkioWeightDevice": null, "BlkioDeviceReadBps": null, "BlkioDeviceWriteBps": null, "BlkioDeviceReadIOps": null, "BlkioDeviceWriteIOps": null, "CpuPeriod": 0, "CpuQuota": 0, "CpuRealtimePeriod": 0, "CpuRealtimeRuntime": 0, "CpusetCpus": "", "CpusetMems": "", "Devices": null, "DeviceCgroupRules": null, "DeviceRequests": null, "KernelMemory": 0, "KernelMemoryTCP": 0, "MemoryReservation": 0, "MemorySwap": 0, "MemorySwappiness": null, "OomKillDisable": false, "PidsLimit": null, "Ulimits": null, "CpuCount": 0, "CpuPercent": 0, "IOMaximumIOps": 0, "IOMaximumBandwidth": 0, "MaskedPaths": [ "/proc/asound", "/proc/acpi", "/proc/kcore", "/proc/keys", "/proc/latency_stats", "/proc/timer_list", "/proc/timer_stats", "/proc/sched_debug", "/proc/scsi", "/sys/firmware" ], "ReadonlyPaths": [ "/proc/bus", "/proc/fs", "/proc/irq", "/proc/sys", "/proc/sysrq-trigger" ] }, "GraphDriver": { "Data": { "LowerDir": "/var/lib/docker/overlay2/361caa8fe0c780cbc0ee06718313a59252249cc6d8e1ed02fc043fb37f667e38-init/diff:/var/lib/docker/overlay2/3be0zpkiii4qri8x0qqottk38/diff:/var/lib/docker/overlay2/nh2kq8g47ex3z09dl3yu2bchf/diff:/var/lib/docker/overlay2/s9pjl1rmvsbvo8gucnu9zplgw/diff:/var/lib/docker/overlay2/yptjw4zjh67t8smncjrsgsnll/diff:/var/lib/docker/overlay2/2l4ck7whza7frf94jrh1n6030/diff:/var/lib/docker/overlay2/rw7n958hgol3min916mt1t2xe/diff:/var/lib/docker/overlay2/qe4cccsee64f7u7xr2fftevpz/diff:/var/lib/docker/overlay2/cwtu9z819fe8949whetmi6uwp/diff:/var/lib/docker/overlay2/nhjmdffn6nm943x399bpkma1r/diff:/var/lib/docker/overlay2/mqb1uiwf558fsgkwqhswco827/diff:/var/lib/docker/overlay2/e50f50w32s3xug8o3q7pjksyb/diff:/var/lib/docker/overlay2/eyww3nn8hybim7wq3mtm3dr9k/diff:/var/lib/docker/overlay2/vaq0gelaza8zav6ezjhqmunhf/diff:/var/lib/docker/overlay2/c5xiew9v4hae03g4ha1qphjtf/diff:/var/lib/docker/overlay2/639d3763afd9d0dc9c360b186a04bc4d89d033ce6a84462adffab0e5756438e5/diff:/var/lib/docker/overlay2/fde995e8737870b44dc7814f4a11232b35bb2613c6056dcfc16c5bdaa52884b7/diff:/var/lib/docker/overlay2/8a245269512950569bcfc484e02af3db948ffd8e4e06c30aa43dc3e01f145092/diff:/var/lib/docker/overlay2/70e5c23eb231b37e4930db2f247afdcd3a5c7dc0d1cf049d94454c236731f960/diff:/var/lib/docker/overlay2/68c89f41fd8d23ca27fea7f303e55eb63d84a54f707b65180ffcf490c7d21402/diff", "MergedDir": "/var/lib/docker/overlay2/361caa8fe0c780cbc0ee06718313a59252249cc6d8e1ed02fc043fb37f667e38/merged", "UpperDir": "/var/lib/docker/overlay2/361caa8fe0c780cbc0ee06718313a59252249cc6d8e1ed02fc043fb37f667e38/diff", "WorkDir": "/var/lib/docker/overlay2/361caa8fe0c780cbc0ee06718313a59252249cc6d8e1ed02fc043fb37f667e38/work" }, "Name": "overlay2" }, "Mounts": [ { "Type": "bind", "Source": "/var/yangroom/3sd/output", "Destination": "/output", "Mode": "rw", "RW": true, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/var/yangroom/3sd/data", "Destination": "/data", "Mode": "rw", "RW": true, "Propagation": "rprivate" } ], "Config": { "Hostname": "c35e838c9646", "Domainname": "", "User": "", "AttachStdin": false, "AttachStdout": true, "AttachStderr": true, "ExposedPorts": { "7860/tcp": {} }, "Tty": false, "OpenStdin": false, "StdinOnce": false, "Env": [ "CLI_ARGS=--no-half --precision full --allow-code --enable-insecure-extension-access --api", "PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "LANG=C.UTF-8", "GPG_KEY=A035C8C19219BA821ECEA86B64E628F8D684696D", "PYTHON_VERSION=3.10.9", "PYTHON_PIP_VERSION=22.3.1", "PYTHON_SETUPTOOLS_VERSION=65.5.1", "PYTHON_GET_PIP_URL=https://github.com/pypa/get-pip/raw/1a96dc5acd0303c4700e02655aefd3bc68c78958/public/get-pip.py", "PYTHON_GET_PIP_SHA256=d1d09b0f9e745610657a528689ba3ea44a73bd19c60f4c954271b790c71c2653", "DEBIAN_FRONTEND=noninteractive", "PIP_PREFER_BINARY=1", "ROOT=/stable-diffusion-webui", "LD_PRELOAD=libtcmalloc.so", "NVIDIA_DRIVER_CAPABILITIES=compute,utility", "NVIDIA_VISIBLE_DEVICES=all" ], "Cmd": [ "/bin/sh", "-c", "python -u webui.py --listen --port 7860 ${CLI_ARGS}" ], "Image": "sd-auto:55", "Volumes": null, "WorkingDir": "/stable-diffusion-webui", "Entrypoint": [ "/docker/entrypoint.sh" ], "OnBuild": null, "Labels": { "com.docker.compose.config-hash": "9f5126e8d94e2918c6371acf077346250a8674619f31ba08abce49dd5de15408", "com.docker.compose.container-number": "1", "com.docker.compose.depends_on": "", "com.docker.compose.image": "sha256:5f11d9aad2138498dc34135e99907ee1eb6499f35d57db82e6f4b64acfa88feb", "com.docker.compose.oneoff": "False", "com.docker.compose.project": "webui-docker", "com.docker.compose.project.config_files": "/var/yangroom/3sd/docker-compose.yml", "com.docker.compose.project.working_dir": "/var/yangroom/3sd", "com.docker.compose.service": "auto-cpu", "com.docker.compose.version": "2.17.2", "desktop.docker.io/wsl-distro": "Ubuntu" }, "StopSignal": "SIGINT" }, "NetworkSettings": { "Bridge": "", "SandboxID": "5c5e41fcce0b5e5f0f53da0494d403e2d52d336aaa764c1cea147cbca8f9b8f6", "HairpinMode": false, "LinkLocalIPv6Address": "", "LinkLocalIPv6PrefixLen": 0, "Ports": {}, "SandboxKey": "/var/run/docker/netns/5c5e41fcce0b", "SecondaryIPAddresses": null, "SecondaryIPv6Addresses": null, "EndpointID": "", "Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "IPAddress": "", "IPPrefixLen": 0, "IPv6Gateway": "", "MacAddress": "", "Networks": { "webui-docker_default": { "IPAMConfig": null, "Links": null, "Aliases": [ "webui-docker-auto-cpu-1", "auto-cpu", "c35e838c9646" ], "NetworkID": "40a32a1fa2358875bd2526216531987d3660cad97f074c06e8e1a9bf177daae2", "EndpointID": "", "Gateway": "", "IPAddress": "", "IPPrefixLen": 0, "IPv6Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "MacAddress": "", "DriverOpts": null } } } } ]

[Aceyng]

I am using "auto-cpu" mode.

[AbdBarho]

@Aceyng the bug template exists for a reason, please fill it out to make our life easier.

In any case, just like @DevilaN said, it is likely OOM, how much ram do you have?

[Aceyng]

@Aceyng the bug template exists for a reason, please fill it out to make our life easier.

In any case, just like @DevilaN said, it is likely OOM, how much ram do you have?

My laptop is 16GB, I wonder if 16GB is too low，When I started this container, the memory skyrocketed to 92%，If that's the case, are there any other tags that can accommodate smaller memory? Are there any tags below 5.0.3 that occupy smaller memory. Thank you!

[DevilaN]

@Aceyng, your container reports

 "OOMKilled": false,

Can you monitor your ram usage during startup? This is weird, as Killed info comes up almost 2 minutes after diffusion wrapper ended it's thing. Is this vanilla install or you have something in startup scripts that might install / change things in container?

I have 16GB RAM too and when starting profile auto-cpu I have only 7GB total RAM usage in my system (including FF browser and other apps running) just after auto-cpu started.

[Aceyng]

@Aceyng, your container reports

 "OOMKilled": false,

Can you monitor your ram usage during startup? This is weird, as Killed info comes up almost 2 minutes after diffusion wrapper ended it's thing. Is this vanilla install or you have something in startup scripts that might install / change things in container?

I have 16GB RAM too and when starting profile auto-cpu I have only 7GB total RAM usage in my system (including FF browser and other apps running) just after auto-cpu started.

The error was reported when starting the container. 'killed' occurred approximately two minutes after starting the container, and then there was no operation (nor access http://localhost:7860 ）Suddenly killed.

[DevilaN]

Just try docker stats in terminal and take a look what happens with memory during startup of auto-cpu service.

[Aceyng]

Just try docker stats in terminal and take a look what happens with memory during startup of auto-cpu service.

This is the running status and screenshot of the container before exiting, and then it throws' killed 'to exit.

[DevilaN]

And there it is. MEM USAGE / LIMIT shows that max memory allowed here is 7.625 GiB.

This is windows power shell. And windows :/ Gad Demyt! I missed info about WSL2. I thought you are using Ubuntu as a native system.

There is a limit of resources for docker virtual machine and default one is 8GB. You can change docker for windows settings so WSL2 will take more RAM, but I don't think we will get away with using more than 12GB here. Try to bump it up until it is working or your system refuses to work with so much memory reserved by other app.

[AbdBarho]

@DevilaN thank you for debugging this!

@Aceyng you can find many resources online on changing the WSL memory limits, it can be done by creating a .wslconfig file, either in the host or inside the WSL VM.

[Aceyng]

@DevilaN @AbdBarho I will modify the memory configuration before trying again. Thank you very much for your guidance~

[Aceyng]

I configured the wsl2 memory to run normally after 9GB, which is perfect. Thanks again @DevilaN @AbdBarho