Image upload always fails with "InternalFailure: Worker closed"

[Kurokha]

I've set up Picsur v0.3.1 with the standard docker configuration behind an nginx proxy. The container is up and running, the app can be accessed via web, login and changing settings work like a charm. But I'm unable to upload images. It's "Processing" them for about a second and then shows a red error "Worker closed". A similar message appears in the logs:

[Nest] 36 - 08/30/2022, 9:04:12 AM ERROR [MainExceptionFilter] (172.21.0.1 -> POST /api/image/upload) InternalFailure: Worker closed

The database container is not showing any errors at all. I've tried Firefox and Edge as well as upload per file selector or drag and drop, but it's always the same result.

Docker Inspect

{ "AppArmorProfile": "docker-default", "Args": [ "/bin/sh", "-c", "/bin/bash -c \"yarn workspace picsur-backend start:prod\"" ], "Config": { "AttachStderr": true, "AttachStdin": false, "AttachStdout": true, "Cmd": [ "/bin/sh", "-c", "/bin/bash -c \"yarn workspace picsur-backend start:prod\"" ], "Domainname": "", "Entrypoint": [ "docker-entrypoint.sh" ], "Env": [ "PICSUR_DB_HOST=picsur_postgres", "PICSUR_PORT=5007", "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "NODE_VERSION=18.8.0", "YARN_VERSION=1.22.19", "PICSUR_PRODUCTION=true" ], "ExposedPorts": { "5007/tcp": {} }, "Hostname": "9531e92981de", "Image": "ghcr.io/rubikscraft/picsur:latest", "Labels": { "com.docker.compose.config-hash": "e35a85a18e6bab7941072f9b70240c37a6ecd909a4044fa691710a01a08b754d", "com.docker.compose.container-number": "1", "com.docker.compose.depends_on": "", "com.docker.compose.image": "sha256:5d57948b0f8181bc4bc3e17cd148581faaf541a93246fc708dba3c7671fbea58", "com.docker.compose.oneoff": "False", "com.docker.compose.project": "picsur", "com.docker.compose.project.config_files": "/data/compose/15/docker-compose.yml", "com.docker.compose.project.working_dir": "/data/compose/15", "com.docker.compose.service": "picsur", "com.docker.compose.version": "2.5.1" }, "OnBuild": null, "OpenStdin": false, "StdinOnce": false, "Tty": false, "User": "", "Volumes": null, "WorkingDir": "/picsur" }, "Created": "2022-08-30T08:40:10.100727336Z", "Driver": "overlay2", "ExecIDs": null, "GraphDriver": { "Data": { "LowerDir": "/var/lib/docker/overlay2/af894f23d8b4433b75dfaa6b16dff8986611c57b0453dd61f4a305ee1e1f2a4b-init/diff:/var/lib/docker/overlay2/0c281565965ea9e58c308cb0c09a27f6efc3df50ae46e97e089eb97b36ebfd48/diff:/var/lib/docker/overlay2/ed8ea733826ec00794d5d1f0e4a78aee85d23545712749a66d80b3e2174b4b1d/diff:/var/lib/docker/overlay2/391c7d17d7c59ce4775ddb23eea006900d51b1ece4175024cb2858b162fa105e/diff:/var/lib/docker/overlay2/aab5da82745621ba20b67449bdd536a09bb543ca7fc0bf2ddd34dc0bdf3474f9/diff:/var/lib/docker/overlay2/4260416ba9094b0c4e264a59e917eedcbf2d91e71f421cbf89309e52ee0c6d07/diff:/var/lib/docker/overlay2/344b49b09803093aa26870adb3582e58559b0886a09f70a416553eb011b43f6d/diff:/var/lib/docker/overlay2/35b0bdc67499777a58e0db1a31081b0224cfff150cf766b937706e205f2260a2/diff:/var/lib/docker/overlay2/8b65ac6c6223432afdb452290536e6044cdec6f8bc05bfdf577ef452d22767aa/diff:/var/lib/docker/overlay2/47dd543c72ce0e72ab665da7ee3e010eb1d751bb0b8ad0fc5cdc01b2d0bcd32a/diff:/var/lib/docker/overlay2/61e6d08a26dc7323df978985ede581fd07b488507d4028a80c290f0ca7041980/diff:/var/lib/docker/overlay2/e9af43e1c17855988f730f977a90264d710e64afe624afbc7ed14c3fcb636f6f/diff:/var/lib/docker/overlay2/5732c53dc0cab00f93ee3017bf608a21aaca4e28a36130e8f156438aaf975a80/diff:/var/lib/docker/overlay2/f769223b4ea87e0bd5b5ec3f08557b2f6947272379d57582400a4023fadeba29/diff:/var/lib/docker/overlay2/9752d19b41eff8f74f81f452e6c013ddf88a6207a2b093e92cf4ac9da68d7df4/diff", "MergedDir": "/var/lib/docker/overlay2/af894f23d8b4433b75dfaa6b16dff8986611c57b0453dd61f4a305ee1e1f2a4b/merged", "UpperDir": "/var/lib/docker/overlay2/af894f23d8b4433b75dfaa6b16dff8986611c57b0453dd61f4a305ee1e1f2a4b/diff", "WorkDir": "/var/lib/docker/overlay2/af894f23d8b4433b75dfaa6b16dff8986611c57b0453dd61f4a305ee1e1f2a4b/work" }, "Name": "overlay2" }, "HostConfig": { "AutoRemove": false, "Binds": [], "BlkioDeviceReadBps": null, "BlkioDeviceReadIOps": null, "BlkioDeviceWriteBps": null, "BlkioDeviceWriteIOps": null, "BlkioWeight": 0, "BlkioWeightDevice": null, "CapAdd": null, "CapDrop": null, "Cgroup": "", "CgroupParent": "", "CgroupnsMode": "private", "ConsoleSize": [ 0, 0 ], "ContainerIDFile": "", "CpuCount": 0, "CpuPercent": 0, "CpuPeriod": 0, "CpuQuota": 0, "CpuRealtimePeriod": 0, "CpuRealtimeRuntime": 0, "CpuShares": 0, "CpusetCpus": "", "CpusetMems": "", "DeviceCgroupRules": null, "DeviceRequests": null, "Devices": null, "Dns": null, "DnsOptions": null, "DnsSearch": null, "ExtraHosts": [], "GroupAdd": null, "IOMaximumBandwidth": 0, "IOMaximumIOps": 0, "IpcMode": "private", "Isolation": "", "KernelMemory": 0, "KernelMemoryTCP": 0, "Links": null, "LogConfig": { "Config": {}, "Type": "json-file" }, "MaskedPaths": [ "/proc/asound", "/proc/acpi", "/proc/kcore", "/proc/keys", "/proc/latency_stats", "/proc/timer_list", "/proc/timer_stats", "/proc/sched_debug", "/proc/scsi", "/sys/firmware" ], "Memory": 0, "MemoryReservation": 0, "MemorySwap": 0, "MemorySwappiness": null, "NanoCpus": 0, "NetworkMode": "picsur_default", "OomKillDisable": null, "OomScoreAdj": 0, "PidMode": "", "PidsLimit": null, "PortBindings": { "5007/tcp": [ { "HostIp": "", "HostPort": "5007" } ] }, "Privileged": false, "PublishAllPorts": false, "ReadonlyPaths": [ "/proc/bus", "/proc/fs", "/proc/irq", "/proc/sys", "/proc/sysrq-trigger" ], "ReadonlyRootfs": false, "RestartPolicy": { "MaximumRetryCount": 0, "Name": "unless-stopped" }, "Runtime": "runc", "SecurityOpt": null, "ShmSize": 67108864, "UTSMode": "", "Ulimits": null, "UsernsMode": "", "VolumeDriver": "", "VolumesFrom": null }, "HostnamePath": "/var/lib/docker/containers/9531e92981de994e5ccb420996a962670e4b9cac2cc4858b7ba36654bd275a97/hostname", "HostsPath": "/var/lib/docker/containers/9531e92981de994e5ccb420996a962670e4b9cac2cc4858b7ba36654bd275a97/hosts", "Id": "9531e92981de994e5ccb420996a962670e4b9cac2cc4858b7ba36654bd275a97", "Image": "sha256:5d57948b0f8181bc4bc3e17cd148581faaf541a93246fc708dba3c7671fbea58", "LogPath": "/var/lib/docker/containers/9531e92981de994e5ccb420996a962670e4b9cac2cc4858b7ba36654bd275a97/9531e92981de994e5ccb420996a962670e4b9cac2cc4858b7ba36654bd275a97-json.log", "MountLabel": "", "Mounts": [], "Name": "/picsur", "NetworkSettings": { "Bridge": "", "EndpointID": "", "Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "HairpinMode": false, "IPAddress": "", "IPPrefixLen": 0, "IPv6Gateway": "", "LinkLocalIPv6Address": "", "LinkLocalIPv6PrefixLen": 0, "MacAddress": "", "Networks": { "picsur_default": { "Aliases": [ "picsur", "picsur", "9531e92981de" ], "DriverOpts": null, "EndpointID": "5f9bdbf4368a62ef36734678e5f8220b5314c3d5fa2e37bad53847f33bed3053", "Gateway": "172.21.0.1", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "IPAMConfig": null, "IPAddress": "172.21.0.2", "IPPrefixLen": 16, "IPv6Gateway": "", "Links": null, "MacAddress": "02:42:ac:15:00:02", "NetworkID": "bfe69a75414bf9c5f804919914f1b552bb80b2e9c4531f3ba6a15e02a24fdd3c" } }, "Ports": { "5007/tcp": [ { "HostIp": "0.0.0.0", "HostPort": "5007" }, { "HostIp": "::", "HostPort": "5007" } ] }, "SandboxID": "4b92c5e64cc5b3f2360e6bac6379545fbf540c6a8b9b5f9d49a308087c74d557", "SandboxKey": "/var/run/docker/netns/4b92c5e64cc5", "SecondaryIPAddresses": null, "SecondaryIPv6Addresses": null }, "Path": "docker-entrypoint.sh", "Platform": "linux", "Portainer": { "ResourceControl": { "Id": 7, "ResourceId": "2_picsur", "SubResourceIds": [], "Type": 6, "UserAccesses": [], "TeamAccesses": [], "Public": false, "AdministratorsOnly": true, "System": false } }, "ProcessLabel": "", "ResolvConfPath": "/var/lib/docker/containers/9531e92981de994e5ccb420996a962670e4b9cac2cc4858b7ba36654bd275a97/resolv.conf", "RestartCount": 0, "State": { "Dead": false, "Error": "", "ExitCode": 0, "FinishedAt": "0001-01-01T00:00:00Z", "OOMKilled": false, "Paused": false, "Pid": 439855, "Restarting": false, "Running": true, "StartedAt": "2022-08-30T08:40:10.459133454Z", "Status": "running" } }

Unfortunately I can't find any further information to help identify the issue. Do you have an idea what else I could check? Any help is greatly appreciated!

[CaramelFur]

Hmm, weird, have you tried a couple different image files? And if its just one image file, could you send me that one? Could you also give me the specifications of the device this is running on? It might just be running out of ram or cpu.

Edit: You can also start the docker container with env "PICSUR_PRODUCTION=false", this way it will print more verbose logs. That should give more insight. It will also dump the admin password in the logs, so do mind that.

[Kurokha]

I've tried different files and formats, yes. They all cause the same error. Here are my docker stats:

CONTAINER ID	NAME	CPU %	MEM USAGE / LIMIT	MEM %	NET I/O	BLOCK I/O	PIDS
b0b5f9270f05	picsur	0.00%	177.8MiB / 7.762GiB	2.24%	1.5MB / 40.8kB	0B / 90.1kB	30

Setting PICSUR_PRODUCTION: false gives me these log lines when trying to upload an image file.

[Nest] 37 - 08/30/2022, 9:56:39 AM VERBOSE [SharpWrapper49374] Worker 49374 initialized with 60000ms timeout and 1024MB memory limit [Nest] 37 - 08/30/2022, 9:56:40 AM VERBOSE [SharpWrapper49374] Worker 49374 exited with code null and signal SIGILL [Nest] 37 - 08/30/2022, 9:56:40 AM ERROR [MainExceptionFilter] (172.21.0.1 -> POST /api/image/upload) InternalFailure: Worker closed [Nest] 37 - 08/30/2022, 9:56:40 AM DEBUG [MainExceptionFilter] Error: Worker closed at Fail (file:///picsur/shared/dist/types/failable.js:120:46) at SharpWrapper.finish (file:///picsur/backend/dist/workers/sharp.wrapper.js:109:20) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async ImageConverterService.convertStill (file:///picsur/backend/dist/managers/image/image-converter.service.js:93:24) at async ImageProcessorService.process (file:///picsur/backend/dist/managers/image/image-processor.service.js:21:20) at async ImageManagerService.upload (file:///picsur/backend/dist/managers/image/image.service.js:56:31) at async ImageManageController.uploadImage (file:///picsur/backend/dist/routes/image/image-manage.controller.js:29:37)

[CaramelFur]

It encountered a SIGILL, meaning the CPU encountered an instruction it can't execute. What CPU and what architecture are you running?

[Kurokha]

It's a KVM running on an AMD EPYC 7702, x86_64

[CaramelFur]

Ah nice, could you send the output of the cpuinfo info command? That way I can compare supported instructions and hopefully figure out which one is missing.

(I'm guessing its avx, but I wanna know for sure)

[Kurokha]

There you go:

processor : 0 vendor_id : AuthenticAMD cpu family : 6 model : 13 model name : QEMU Virtual CPU version 2.5+ stepping : 3 microcode : 0x1000065 cpu MHz : 1996.186 cache size : 512 KB physical id : 0 siblings : 8 core id : 0 cpu cores : 8 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx lm nopl cpuid tsc_known_freq pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt aes xsave rdrand hypervisor lahf_lm cmp_legacy abm 3dnowprefetch ssbd ibpb vmmcall arch_capabilities bugs : fxsave_leak sysret_ss_attrs null_seg spectre_v1 spectre_v2 spec_store_bypass bogomips : 3992.37 TLB size : 1024 4K pages clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management:

[CaramelFur]

yep, it indeed looks like avx is missing, but its not the epycs fault, its qemu that doesnt support it for some reason. But I will try and see if I can compile this project without avx.

[Kurokha]

Ahh, fair enough. Thanks for looking into this!

[CaramelFur]

I have made a new build that should solve the problem, can you give this container a try: ghcr.io/rubikscraft/picsur:alpha-0.3.2

[Kurokha]

Awesome, that fixed the issue. No errors at all and image upload works perfectly now.

[CaramelFur]

Ah that's excellent