strahd-zarovich
commented
3 months ago I am having this issue too. Woke up to a lot of Spaghetti today. I do not know how to pull all the logs, but here is what I am working with.
moonraker-obico: v2.0.1-0-g1a527d58 Standalone Obico server one on Linux machine and one on UnRAID server both have the same issue.
I had to hard code my webcam it was not picked up automatically like it was in the past.
No video after the print was manually cancelled, so no time-lapse to look through.
Weirdly my phone app say there is no video when the print was happening, but the server saw the video.
1012302887
kennethjiang
shom
on web ui not Failure Detection.
docker logs obico-server-web-1
0 static files copied to '/app/static_build', 327 unmodified, 1030 post-processed. daphne.cli INFO Starting server at tcp:port=3334:interface=0.0.0.0 daphne.server INFO HTTP/2 support enabled daphne.server INFO Configuring endpoint tcp:port=3334:interface=0.0.0.0 daphne.server INFO Listening on TCP address 0.0.0.0:3334 api.consumers WARNING WebConsumer: Closed websocket with code: None api.consumers WARNING WebConsumer: Closed websocket with code: None api.consumers WARNING WebConsumer: Closed websocket with code: None api.octoprint_messages WARNING current_print_ts not present. Received status: {} api.octoprint_messages WARNING current_print_ts not present. Received status: {} api.octoprint_messages WARNING current_print_ts not present. Received status: {} api.octoprint_messages WARNING current_print_ts not present. Received status: {} api.octoprint_messages WARNING current_print_ts not present. Received status: {} api.consumers WARNING OctoPrintConsumer: Closed websocket with code: 1006 api.octoprint_messages WARNING current_print_ts not present. Received status: {} api.octoprint_messages WARNING current_print_ts not present. Received status: {'settings': {'webcams': None, 'temperature': {'profiles': []}, 'agent': {'name': 'moonraker_obico', 'version': '2.0.1'}, 'platform_uname': ['Linux', 'klipper', '6.6.31+rpt-rpi-2712', '#1 SMP PREEMPT Debian 1:6.6.31-1+rpt1 (2024-05-29)', 'aarch64', '', 'Raspberry Pi 5 Model B Rev 1.0\x00'], 'installed_plugins': []}} django.request WARNING Not Found: /ent/api/printers/3/ext/ app.models WARNING current_print_ts=-1 received when current print is still active. Force-closing print. print_id: 5 - printer_id: 3 api.consumers WARNING WebConsumer: Closed websocket with code: None 182.114.247.211:38662 - - [09/Jun/2024:02:20:49] "GET /" 302 - 182.114.247.211:38662 - - [09/Jun/2024:02:20:49] "GET /printers/" 200 19765 182.114.247.211:38662 - - [09/Jun/2024:02:20:49] "GET /api/v1/printers/?with_archived=true" 200 399 182.114.247.211:38662 - - [09/Jun/2024:02:20:49] "GET /printers/3/control/" 200 19671 182.114.247.211:38668 - - [09/Jun/2024:02:20:49] "WSCONNECTING /ws/web/3/" - - 182.114.247.211:38662 - - [09/Jun/2024:02:20:49] "GET /api/v1/prints/?start=0&limit=1&filter_by_printer_ids[]=3&sorting=date_desc" 200 2 182.114.247.211:38668 - - [09/Jun/2024:02:20:49] "WSCONNECT /ws/web/3/" - - 182.114.247.211:38670 - - [09/Jun/2024:02:20:49] "WSCONNECTING /ws/janus/3/" - - 182.114.247.211:38670 - - [09/Jun/2024:02:20:50] "WSCONNECT /ws/janus/3/" - - 182.114.247.211:38672 - - [09/Jun/2024:02:20:50] "WSCONNECTING /ws/janus/3/" - - 182.114.247.211:38672 - - [09/Jun/2024:02:20:50] "WSCONNECT /ws/janus/3/" - - 182.114.247.211:38668 - - [09/Jun/2024:02:20:52] "WSDISCONNECT /ws/web/3/" - - 182.114.247.211:38774 - - [09/Jun/2024:02:21:54] "WSCONNECTING /ws/dev/" - - 182.114.247.211:38774 - - [09/Jun/2024:02:21:54] "WSCONNECT /ws/dev/" - - 182.114.247.211:38842 - - [09/Jun/2024:02:22:33] "WSCONNECTING /ws/web/3/" - - 182.114.247.211:38842 - - [09/Jun/2024:02:22:33] "WSCONNECT /ws/web/3/" - - 182.114.247.211:38844 - - [09/Jun/2024:02:22:33] "GET /api/v1/prints/?start=0&limit=1&filter_by_printer_ids[]=3&sorting=date_desc" 200 2 182.114.247.211:38846 - - [09/Jun/2024:02:22:33] "POST /api/v1/octo/pic/" 200 15 182.114.247.211:38844 - - [09/Jun/2024:02:22:33] "GET /media/tsd-pics/snapshots/3/latest_unrotated.jpg?digest=YidceDhlXHhlNlx4OTBceGY4XHhkZWZceGNhXHhlN3FceDA4XHhiNj9ceDFmXHhlMFx4ODRceDEyXHg4M3ckXHg4Nlx4ZDFceGJmXHg5NG5ceGFmXHg4Y1x4ZTZceDFmXHg5NFVceDlmdyc=" 200 250997 182.114.247.211:38848 - - [09/Jun/2024:02:22:33] "POST /api/v1/octo/pic/" 200 15 182.114.247.211:38852 - - [09/Jun/2024:02:22:34] "POST /api/v1/octo/pic/" 200 15 182.114.247.211:38844 - - [09/Jun/2024:02:22:34] "GET /api/v1/prints/?start=0&limit=1&filter_by_printer_ids[]=3&sorting=date_desc" 200 2 182.114.247.211:38844 - - [09/Jun/2024:02:22:36] "GET /api/v1/prints/?start=0&limit=1&filter_by_printer_ids[]=3&sorting=date_desc" 200 2 182.114.247.211:38844 - - [09/Jun/2024:02:22:39] "GET /api/v1/prints/?start=0&limit=1&filter_by_printer_ids[]=3&sorting=date_desc" 200 2 182.114.247.211:38844 - - [09/Jun/2024:02:22:43] "GET /api/v1/prints/?start=0&limit=1&filter_by_printer_ids[]=3&sorting=date_desc" 200 2 182.114.247.211:38842 - - [09/Jun/2024:02:22:44] "WSDISCONNECT /ws/web/3/" - - 182.114.247.211:38844 - - [09/Jun/2024:02:22:48] "GET /api/v1/prints/?start=0&limit=1&filter_by_printer_ids[]=3&sorting=date_desc" 200 2 182.114.247.211:40016 - - [09/Jun/2024:02:27:43] "WSCONNECTING /ws/web/3/" - - 182.114.247.211:40016 - - [09/Jun/2024:02:27:43] "WSCONNECT /ws/web/3/" - - 182.114.247.211:40020 - - [09/Jun/2024:02:27:43] "POST /api/v1/octo/pic/" 200 15 182.114.247.211:40026 - - [09/Jun/2024:02:27:43] "POST /api/v1/octo/pic/" 200 15 182.114.247.211:40042 - - [09/Jun/2024:02:27:44] "POST /api/v1/octo/pic/" 200 15 182.114.247.211:40146 - - [09/Jun/2024:02:27:46] "WSCONNECTING /ws/web/3/" - - 182.114.247.211:40146 - - [09/Jun/2024:02:27:46] "WSCONNECT /ws/web/3/" - - 182.114.247.211:40154 - - [09/Jun/2024:02:27:46] "POST /api/v1/octo/pic/" 200 15 182.114.247.211:40162 - - [09/Jun/2024:02:27:47] "POST /api/v1/octo/pic/" 200 15 182.114.247.211:40170 - - [09/Jun/2024:02:27:47] "POST /api/v1/octo/pic/" 200 15 182.114.247.211:40146 - - [09/Jun/2024:02:27:48] "WSDISCONNECT /ws/web/3/" - - 182.114.247.211:40212 - - [09/Jun/2024:02:27:48] "POST /api/v1/octo/pic/" 200 15 182.114.247.211:40224 - - [09/Jun/2024:02:27:48] "POST /api/v1/octo/pic/" 200 15 182.114.247.211:40240 - - [09/Jun/2024:02:27:49] "POST /api/v1/octo/pic/" 200 15 182.114.247.211:40472 - - [09/Jun/2024:02:27:58] "POST /api/v1/octo/g_code_files/" 201 293 182.114.247.211:40474 - - [09/Jun/2024:02:27:58] "GET /api/v1/prints/?start=0&limit=1&filter_by_printer_ids[]=3&sorting=date_desc" 200 681 182.114.247.211:40476 - - [09/Jun/2024:02:27:58] "POST /api/v1/octo/pic/" 200 15 182.114.247.211:40478 - - [09/Jun/2024:02:27:59] "POST /api/v1/octo/pic/" 200 15 182.114.247.211:40480 - - [09/Jun/2024:02:27:59] "POST /api/v1/octo/pic/" 200 15 182.114.247.211:40484 - - [09/Jun/2024:02:28:00] "POST /api/v1/octo/pic/" 200 15 182.114.247.211:40474 - - [09/Jun/2024:02:28:03] "GET /api/v1/prints/?start=0&limit=1&filter_by_printer_ids[]=3&sorting=date_desc" 200 698 182.114.247.211:40474 - - [09/Jun/2024:02:28:04] "GET /api/v1/prints/?start=0&limit=1&filter_by_printer_ids[]=3&sorting=date_desc" 200 698 182.114.247.211:40474 - - [09/Jun/2024:02:28:07] "GET /api/v1/prints/?start=0&limit=1&filter_by_printer_ids[]=3&sorting=date_desc" 200 698 182.114.247.211:40474 - - [09/Jun/2024:02:28:10] "GET /api/v1/prints/?start=0&limit=1&filter_by_printer_ids[]=3&sorting=date_desc" 200 698 182.114.247.211:40474 - - [09/Jun/2024:02:28:14] "GET /api/v1/prints/?start=0&limit=1&filter_by_printer_ids[]=3&sorting=date_desc" 200 698 182.114.247.211:40474 - - [09/Jun/2024:02:28:19] "GET /api/v1/prints/?start=0&limit=1&filter_by_printer_ids[]=3&sorting=date_desc" 200 698 182.114.247.211:38774 - - [09/Jun/2024:02:28:30] "WSDISCONNECT /ws/dev/" - - 115.49.104.158:40244 - - [09/Jun/2024:02:29:06] "GET /api/v1/octo/printer/" 200 100 115.49.104.158:40256 - - [09/Jun/2024:02:29:06] "WSCONNECTING /ws/dev/" - - 115.49.104.158:40256 - - [09/Jun/2024:02:29:06] "WSCONNECT /ws/dev/" - - 115.49.104.158:40266 - - [09/Jun/2024:02:29:07] "GET /ent/api/printers/3/ext/" 404 179 182.114.247.211:40474 - - [09/Jun/2024:02:29:08] "GET /api/v1/prints/?start=0&limit=1&filter_by_printer_ids[]=3&sorting=date_desc" 200 891 115.49.104.158:40268 - - [09/Jun/2024:02:29:08] "POST /api/v1/octo/pic/" 200 15 115.49.104.158:40274 - - [09/Jun/2024:02:29:08] "POST /api/v1/octo/pic/" 200 15 115.49.104.158:40290 - - [09/Jun/2024:02:29:08] "POST /api/v1/octo/pic/" 200 15 115.49.104.158:40300 - - [09/Jun/2024:02:29:08] "POST /api/v1/octo/pic/" 200 15 115.49.104.158:40308 - - [09/Jun/2024:02:29:08] "POST /api/v1/octo/pic/" 200 15 115.49.104.158:40324 - - [09/Jun/2024:02:29:08] "POST /api/v1/octo/pic/" 200 15 115.49.104.158:37456 - - [09/Jun/2024:02:29:34] "POST /api/v1/octo/g_code_files/" 200 876 182.114.247.211:40474 - - [09/Jun/2024:02:29:34] "GET /api/v1/prints/?start=0&limit=1&filter_by_printer_ids[]=3&sorting=date_desc" 200 685 115.49.104.158:37458 - - [09/Jun/2024:02:29:34] "POST /api/v1/octo/pic/" 200 15 115.49.104.158:37474 - - [09/Jun/2024:02:29:35] "POST /api/v1/octo/pic/" 200 15 115.49.104.158:37490 - - [09/Jun/2024:02:29:35] "POST /api/v1/octo/pic/" 200 15 115.49.104.158:49880 - - [09/Jun/2024:02:29:36] "POST /api/v1/octo/pic/" 200 15 115.49.104.158:55246 - - [09/Jun/2024:02:29:46] "POST /api/v1/octo/pic/" 200 15 115.49.104.158:60446 - - [09/Jun/2024:02:29:57] "POST /api/v1/octo/pic/" 200 15 115.49.104.158:54866 - - [09/Jun/2024:02:30:07] "POST /api/v1/octo/pic/" 200 15 115.49.104.158:52410 - - [09/Jun/2024:02:30:17] "POST /api/v1/octo/pic/" 200 15 115.49.104.158:47002 - - [09/Jun/2024:02:30:27] "POST /api/v1/octo/pic/" 200 15 115.49.104.158:49486 - - [09/Jun/2024:02:30:37] "POST /api/v1/octo/pic/" 200 15 115.49.104.158:50770 - - [09/Jun/2024:02:30:48] "POST /api/v1/octo/pic/" 200 15 182.114.247.211:40882 - - [09/Jun/2024:02:30:50] "GET /printers/3/control/" 200 19670 182.114.247.211:40016 - - [09/Jun/2024:02:30:50] "WSDISCONNECT /ws/web/3/" - - 182.114.247.211:40894 - - [09/Jun/2024:02:30:51] "WSCONNECTING /ws/web/3/" - -
docker logs -f obico-server-tasks-1
/usr/local/lib/python3.10/site-packages/celery/platforms.py:829: SecurityWarning: You're running the worker with superuser privileges: this is absolutely not recommended!
Please specify a different user using the --uid option.
User information: uid=0 euid=0 gid=0 egid=0
warnings.warn(SecurityWarning(ROOT_DISCOURAGED.format(
-------------- celery@tasks v5.3.4 (emerald-rush) --- * ----- -- *** ---- Linux-5.10.0-28-amd64-x86_64-with 2024-06-09 02:20:05
[tasks] . app.tasks.compile_timelapse . app.tasks.detect_timelapse . app.tasks.preprocess_timelapse . app.tasks.process_print_events . app.tasks.prune_channel_presence . app.tasks.prune_channel_rooms . channels_presence.tasks.prune_presence . channels_presence.tasks.prune_rooms . notifications.tasks.send_failure_alerts . notifications.tasks.send_printer_notifications
[2024-06-09 02:20:05,954: INFO/MainProcess] Connected to redis://redis:6379// [2024-06-09 02:20:05,957: INFO/MainProcess] mingle: searching for neighbors [2024-06-09 02:20:06,966: INFO/MainProcess] mingle: all alone [2024-06-09 02:20:06,981: INFO/MainProcess] celery@tasks ready. [2024-06-09 02:20:07,584: INFO/Beat] beat: Starting... [2024-06-09 02:24:25,088: INFO/Beat] Scheduler: Sending due task app.tasks.prune_channel_presence (app.tasks.prune_channel_presence) [2024-06-09 02:24:25,096: INFO/Beat] Scheduler: Sending due task app.tasks.prune_channel_rooms (app.tasks.prune_channel_rooms) [2024-06-09 02:24:25,098: INFO/MainProcess] Task app.tasks.prune_channel_presence[862b2f92-4f23-42af-8af5-7efbc1d4d066] received [2024-06-09 02:24:25,101: INFO/MainProcess] Task app.tasks.prune_channel_rooms[71692adc-37fe-4cdb-8160-397a9935db17] received [2024-06-09 02:24:25,129: INFO/ForkPoolWorker-3] Task app.tasks.prune_channel_presence[862b2f92-4f23-42af-8af5-7efbc1d4d066] succeeded in 0.026097160996869206s: None [2024-06-09 02:24:25,128: INFO/ForkPoolWorker-2] Task app.tasks.prune_channel_rooms[71692adc-37fe-4cdb-8160-397a9935db17] succeeded in 0.025799182010814548s: None [2024-06-09 02:27:58,545: INFO/MainProcess] Task app.tasks.process_print_events[47f10758-c05f-4d36-b6c9-c1966828eff6] received [2024-06-09 02:27:58,613: INFO/ForkPoolWorker-3] Task app.tasks.process_print_events[47f10758-c05f-4d36-b6c9-c1966828eff6] succeeded in 0.06760313600534573s: None [2024-06-09 02:29:07,944: INFO/MainProcess] Task app.tasks.process_print_events[81193750-b677-409b-936f-e1a3d49bba23] received [2024-06-09 02:29:07,961: ERROR/ForkPoolWorker-3] Task app.tasks.process_print_events[81193750-b677-409b-936f-e1a3d49bba23] raised unexpected: FileNotFoundError(2, 'No such file or directory') Traceback (most recent call last): File "/usr/local/lib/python3.10/site-packages/celery/app/trace.py", line 477, in trace_task R = retval = fun(*args, *kwargs) File "/usr/local/lib/python3.10/site-packages/celery/app/trace.py", line 760, in __protected_call__ return self.run(args, **kwargs) File "/app/app/tasks.py", line 45, in process_print_events process_print_end_event(print_event) File "/app/app/tasks.py", line 56, in process_print_end_event if will_record_timelapse(_print): File "/app/app/tasks.py", line 271, in will_record_timelapse last_pic = last_pic_of_print(_print, 'raw') File "/app/lib/utils.py", line 74, in last_pic_of_print print_pics = list_dir(f'{path_prefix}/{_print.printer.id}/{_print.id}/', settings.PICS_CONTAINER, long_term_storage=False) File "/app/lib/file_storage.py", line 23, in list_dir return file_storage.list_dir(dir_path, container) File "/app/lib/fs_file_storage.py", line 25, in list_dir return [ path.join(path.normpath(dir_path), f) for f in os.listdir(fqp) ] FileNotFoundError: [Errno 2] No such file or directory: '/app/static_build/media/tsd-pics/raw/3/5/' [2024-06-09 02:29:08,031: INFO/MainProcess] Task notifications.tasks.send_printer_notifications[0aaf1474-cb39-429e-8fce-592f1f75622c] received [2024-06-09 02:29:08,042: INFO/ForkPoolWorker-3] Task notifications.tasks.send_printer_notifications[0aaf1474-cb39-429e-8fce-592f1f75622c] succeeded in 0.00925327000732068s: None [2024-06-09 02:29:34,712: INFO/MainProcess] Task app.tasks.process_print_events[af678704-f596-492e-9b10-5a02802df257] received [2024-06-09 02:29:34,740: INFO/ForkPoolWorker-3] Task app.tasks.process_print_events[af678704-f596-492e-9b10-5a02802df257] succeeded in 0.02662443400186021s: None [2024-06-09 02:30:34,191: INFO/MainProcess] Task notifications.tasks.send_printer_notifications[6bfed8e1-0adf-4f26-b4bb-d0e6b577bd68] received [2024-06-09 02:30:34,207: INFO/ForkPoolWorker-3] Task notifications.tasks.send_printer_notifications[6bfed8e1-0adf-4f26-b4bb-d0e6b577bd68] succeeded in 0.014067399999476038s: None
docker logs -f obico-server-ml_api-1
CUDA Version 11.4.3
Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
This container image and its contents are governed by the NVIDIA Deep Learning Container License. By pulling and using the container, you accept the terms and conditions of this license: https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license
A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.
[2024-06-09 02:20:03 +0000] [1] [INFO] Starting gunicorn 21.2.0 [2024-06-09 02:20:03 +0000] [1] [INFO] Listening at: http://0.0.0.0:3333 (1) [2024-06-09 02:20:03 +0000] [1] [INFO] Using worker: sync [2024-06-09 02:20:03 +0000] [28] [INFO] Booting worker with pid: 28
Let's try darknet lib built with GPU support - /darknet/libdarknet_gpu.so Done! Hooray! Now we have darknet with GPU support.
----- Trying to load weights: /app/lib/../model/model-weights.darknet - use_gpu = True ----- Try to load cfg: /app/model/model.cfg, weights: /app/lib/../model/model-weights.darknet, clear = 0 0 : compute_capability = 860, cudnn_half = 1, GPU: NVIDIA GeForce RTX 3050 net.optimized_memory = 0 mini_batch = 1, batch = 8, time_steps = 1, train = 0 layer filters size/strd(dil) input output 0 Create CUDA-stream - 0 Create cudnn-handle 0 conv 32 3 x 3/ 1 416 x 416 x 3 -> 416 x 416 x 32 0.299 BF 1 max 2x 2/ 2 416 x 416 x 32 -> 208 x 208 x 32 0.006 BF 2 conv 64 3 x 3/ 1 208 x 208 x 32 -> 208 x 208 x 64 1.595 BF 3 max 2x 2/ 2 208 x 208 x 64 -> 104 x 104 x 64 0.003 BF 4 conv 128 3 x 3/ 1 104 x 104 x 64 -> 104 x 104 x 128 1.595 BF 5 conv 64 1 x 1/ 1 104 x 104 x 128 -> 104 x 104 x 64 0.177 BF 6 conv 128 3 x 3/ 1 104 x 104 x 64 -> 104 x 104 x 128 1.595 BF 7 max 2x 2/ 2 104 x 104 x 128 -> 52 x 52 x 128 0.001 BF 8 conv 256 3 x 3/ 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 9 conv 128 1 x 1/ 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 10 conv 256 3 x 3/ 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 11 max 2x 2/ 2 52 x 52 x 256 -> 26 x 26 x 256 0.001 BF 12 conv 512 3 x 3/ 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 13 conv 256 1 x 1/ 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 14 conv 512 3 x 3/ 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 15 conv 256 1 x 1/ 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 16 conv 512 3 x 3/ 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 17 max 2x 2/ 2 26 x 26 x 512 -> 13 x 13 x 512 0.000 BF 18 conv 1024 3 x 3/ 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 19 conv 512 1 x 1/ 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 20 conv 1024 3 x 3/ 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 21 conv 512 1 x 1/ 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 22 conv 1024 3 x 3/ 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 23 conv 1024 3 x 3/ 1 13 x 13 x1024 -> 13 x 13 x1024 3.190 BF 24 conv 1024 3 x 3/ 1 13 x 13 x1024 -> 13 x 13 x1024 3.190 BF 25 route 16 -> 26 x 26 x 512 26 conv 64 1 x 1/ 1 26 x 26 x 512 -> 26 x 26 x 64 0.044 BF 27 reorg / 2 26 x 26 x 64 -> 13 x 13 x 256 28 route 27 24 -> 13 x 13 x1280 29 conv 1024 3 x 3/ 1 13 x 13 x1280 -> 13 x 13 x1024 3.987 BF 30 conv 30 1 x 1/ 1 13 x 13 x1024 -> 13 x 13 x 30 0.010 BF 31 detection mask_scale: Using default '1.000000' Total BFLOPS 29.338 avg_outputs = 607364 Allocate additional workspace_size = 48.74 MB Try to load weights: /app/lib/../model/model-weights.darknet Loading weights from /app/lib/../model/model-weights.darknet... seen 64, trained: 1427 K-images (22 Kilo-batches_64) Done! Loaded 32 layers from weights-file Loaded - names_list: /app/model/names, classes = 1 Succeeded!
Screenshots
Hosting environment (please complete the following information):