Closed YBonline closed 1 year ago
What type of mount is /db/?
Filesystem is ext4, SSD, separate drive from /media/frigate
I started seeing this too, let me know what I can do to help debug.
0.13.0-dacf45c
2023-07-18 09:50:05.972838129 Exception in thread timeline_processor:
2023-07-18 09:50:05.972852518 Traceback (most recent call last):
2023-07-18 09:50:05.972855499 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql
2023-07-18 09:50:05.975147184 cursor.execute(sql, params or ())
2023-07-18 09:50:05.975158221 sqlite3.OperationalError: database is locked
2023-07-18 09:50:05.975159517
2023-07-18 09:50:05.975162406 During handling of the above exception, another exception occurred:
2023-07-18 09:50:05.975163536
2023-07-18 09:50:05.975165202 Traceback (most recent call last):
2023-07-18 09:50:05.975169462 File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner
2023-07-18 09:50:05.985262396 self.run()
2023-07-18 09:50:05.985274137 File "/opt/frigate/frigate/timeline.py", line 46, in run
2023-07-18 09:50:05.985275859 self.handle_object_detection(
2023-07-18 09:50:05.985278155 File "/opt/frigate/frigate/timeline.py", line 92, in handle_object_detection
2023-07-18 09:50:05.985279914 Timeline.insert(timeline_entry).execute()
2023-07-18 09:50:05.985282137 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 1966, in inner
2023-07-18 09:50:05.985283970 return method(self, database, *args, **kwargs)
2023-07-18 09:50:05.985292933 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2037, in execute
2023-07-18 09:50:05.985379413 return self._execute(database)
2023-07-18 09:50:05.985382413 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2842, in _execute
2023-07-18 09:50:05.985385062 return super(Insert, self)._execute(database)
2023-07-18 09:50:05.985387339 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2556, in _execute
2023-07-18 09:50:05.985500801 return self.handle_result(database, cursor)
2023-07-18 09:50:05.985504061 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2851, in handle_result
2023-07-18 09:50:05.985572801 return database.last_insert_id(cursor, self._query_type)
2023-07-18 09:50:05.985576153 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3749, in last_insert_id
2023-07-18 09:50:05.985577967 return cursor.lastrowid
2023-07-18 09:50:05.985580449 File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 87, in lastrowid
2023-07-18 09:50:05.985581819 self._wait()
2023-07-18 09:50:05.985584264 File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 62, in _wait
2023-07-18 09:50:05.985585690 raise self._exc
2023-07-18 09:50:05.985588097 File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 177, in execute
2023-07-18 09:50:05.985645059 cursor = self.database._execute(obj.sql, obj.params)
2023-07-18 09:50:05.985648226 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql
2023-07-18 09:50:05.986163018 cursor.execute(sql, params or ())
2023-07-18 09:50:05.986285387 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3014, in __exit__
2023-07-18 09:50:05.987740153 reraise(new_type, new_type(exc_value, *exc_args), traceback)
2023-07-18 09:50:05.987876578 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 192, in reraise
2023-07-18 09:50:05.988446832 raise value.with_traceback(tb)
2023-07-18 09:50:05.988571202 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql
2023-07-18 09:50:05.990128541 cursor.execute(sql, params or ())
2023-07-18 09:50:05.990474519 peewee.OperationalError: database is locked
2023-07-18 12:55:50.776418134 Exception in thread event_cleanup:
2023-07-18 12:55:50.776429856 Traceback (most recent call last):
2023-07-18 12:55:50.776481448 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql
2023-07-18 12:55:50.777896732 cursor.execute(sql, params or ())
2023-07-18 12:55:50.777944121 sqlite3.OperationalError: database is locked
2023-07-18 12:55:50.777946121
2023-07-18 12:55:50.777948491 During handling of the above exception, another exception occurred:
2023-07-18 12:55:50.777949750
2023-07-18 12:55:50.778240618 Traceback (most recent call last):
2023-07-18 12:55:50.778247618 File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner
2023-07-18 12:55:50.778670929 self.run()
2023-07-18 12:55:50.778706855 File "/opt/frigate/frigate/events/cleanup.py", line 164, in run
2023-07-18 12:55:50.779064556 self.expire("clips")
2023-07-18 12:55:50.779100407 File "/opt/frigate/frigate/events/cleanup.py", line 125, in expire
2023-07-18 12:55:50.779403479 update_query.execute()
2023-07-18 12:55:50.779438015 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 1966, in inner
2023-07-18 12:55:50.780254842 return method(self, database, *args, **kwargs)
2023-07-18 12:55:50.780291212 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2037, in execute
2023-07-18 12:55:50.781140019 return self._execute(database)
2023-07-18 12:55:50.781146408 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2556, in _execute
2023-07-18 12:55:50.782277361 return self.handle_result(database, cursor)
2023-07-18 12:55:50.782289621 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2567, in handle_result
2023-07-18 12:55:50.783319408 return database.rows_affected(cursor)
2023-07-18 12:55:50.783329704 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3759, in rows_affected
2023-07-18 12:55:50.784712729 return cursor.rowcount
2023-07-18 12:55:50.784725248 File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 93, in rowcount
2023-07-18 12:55:50.785003412 self._wait()
2023-07-18 12:55:50.785041560 File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 62, in _wait
2023-07-18 12:55:50.785281336 raise self._exc
2023-07-18 12:55:50.785288761 File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 177, in execute
2023-07-18 12:55:50.785544111 cursor = self.database._execute(obj.sql, obj.params)
2023-07-18 12:55:50.785551944 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql
2023-07-18 12:55:50.786711971 cursor.execute(sql, params or ())
2023-07-18 12:55:50.786721953 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3014, in __exit__
2023-07-18 12:55:50.787886128 reraise(new_type, new_type(exc_value, *exc_args), traceback)
2023-07-18 12:55:50.787897350 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 192, in reraise
2023-07-18 12:55:50.788201199 raise value.with_traceback(tb)
2023-07-18 12:55:50.788209088 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql
2023-07-18 12:55:50.789472114 cursor.execute(sql, params or ())
2023-07-18 12:55:50.789526855 peewee.OperationalError: database is locked
This didn't fix it for me:
2023-07-19 08:36:32.553079912 [2023-07-19 08:36:32] frigate.record.maintainer WARNING : Failed to probe corrupt segment /tmp/cache/warehouse_workbench-20230719083338.mp4
2023-07-19 08:36:32.553236125 [2023-07-19 08:36:32] frigate.record.maintainer WARNING : Discarding a corrupt recording segment: /tmp/cache/warehouse_workbench-20230719083338.mp4
2023-07-19 09:37:30.323723133 Exception in thread event_processor:
2023-07-19 09:37:30.323897768 Traceback (most recent call last):
2023-07-19 09:37:30.323918369 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql
2023-07-19 09:37:30.331318561 cursor.execute(sql, params or ())
2023-07-19 09:37:30.331342251 sqlite3.OperationalError: database is locked
2023-07-19 09:37:30.331344599
2023-07-19 09:37:30.331347339 During handling of the above exception, another exception occurred:
2023-07-19 09:37:30.331349284
2023-07-19 09:37:30.331351318 Traceback (most recent call last):
2023-07-19 09:37:30.331359311 File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner
2023-07-19 09:37:30.331632840 self.run()
2023-07-19 09:37:30.331635196 File "/opt/frigate/frigate/events/maintainer.py", line 93, in run
2023-07-19 09:37:30.331723972 self.handle_object_detection(event_type, camera, event_data)
2023-07-19 09:37:30.331726662 File "/opt/frigate/frigate/events/maintainer.py", line 205, in handle_object_detection
2023-07-19 09:37:30.331789908 Event.insert(event)
2023-07-19 09:37:30.331792344 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 1966, in inner
2023-07-19 09:37:30.332129113 return method(self, database, *args, **kwargs)
2023-07-19 09:37:30.332139754 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2037, in execute
2023-07-19 09:37:30.332569313 return self._execute(database)
2023-07-19 09:37:30.332579136 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2842, in _execute
2023-07-19 09:37:30.333172138 return super(Insert, self)._execute(database)
2023-07-19 09:37:30.333179884 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2556, in _execute
2023-07-19 09:37:30.333545795 return self.handle_result(database, cursor)
2023-07-19 09:37:30.333552348 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2851, in handle_result
2023-07-19 09:37:30.333964995 return database.last_insert_id(cursor, self._query_type)
2023-07-19 09:37:30.333971215 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3749, in last_insert_id
2023-07-19 09:37:30.334540225 return cursor.lastrowid
2023-07-19 09:37:30.334547590 File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 87, in lastrowid
2023-07-19 09:37:30.334622120 self._wait()
2023-07-19 09:37:30.334628949 File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 62, in _wait
2023-07-19 09:37:30.334667776 raise self._exc
2023-07-19 09:37:30.334675984 File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 177, in execute
2023-07-19 09:37:30.334724209 cursor = self.database._execute(obj.sql, obj.params)
2023-07-19 09:37:30.334732501 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql
2023-07-19 09:37:30.335214326 cursor.execute(sql, params or ())
2023-07-19 09:37:30.335221475 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3014, in __exit__
2023-07-19 09:37:30.335653469 reraise(new_type, new_type(exc_value, *exc_args), traceback)
2023-07-19 09:37:30.335659650 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 192, in reraise
2023-07-19 09:37:30.335711924 raise value.with_traceback(tb)
2023-07-19 09:37:30.335717649 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql
2023-07-19 09:37:30.336193640 cursor.execute(sql, params or ())
2023-07-19 09:37:30.336199661 peewee.OperationalError: database is locked
2023-07-19 10:02:14.205352389 Exception in thread event_cleanup:
2023-07-19 10:02:14.205530698 Traceback (most recent call last):
2023-07-19 10:02:14.205540822 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql
2023-07-19 10:02:14.214614187 cursor.execute(sql, params or ())
2023-07-19 10:02:14.214620420 sqlite3.OperationalError: database is locked
2023-07-19 10:02:14.214622804
2023-07-19 10:02:14.214625694 During handling of the above exception, another exception occurred:
2023-07-19 10:02:14.214627684
2023-07-19 10:02:14.214629899 Traceback (most recent call last):
2023-07-19 10:02:14.214632570 File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner
2023-07-19 10:02:14.215264752 self.run()
2023-07-19 10:02:14.215289119 File "/opt/frigate/frigate/events/cleanup.py", line 164, in run
2023-07-19 10:02:14.216935461 self.expire("clips")
2023-07-19 10:02:14.216959001 File "/opt/frigate/frigate/events/cleanup.py", line 125, in expire
2023-07-19 10:02:14.218279606 update_query.execute()
2023-07-19 10:02:14.219404135 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 1966, in inner
2023-07-19 10:02:14.223553696 return method(self, database, *args, **kwargs)
2023-07-19 10:02:14.223558200 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2037, in execute
2023-07-19 10:02:14.225136113 return self._execute(database)
2023-07-19 10:02:14.225146470 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2556, in _execute
2023-07-19 10:02:14.226925532 return self.handle_result(database, cursor)
2023-07-19 10:02:14.226937722 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2567, in handle_result
2023-07-19 10:02:14.228378208 return database.rows_affected(cursor)
2023-07-19 10:02:14.228579252 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3759, in rows_affected
2023-07-19 10:02:14.232971992 return cursor.rowcount
2023-07-19 10:02:14.232989583 File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 93, in rowcount
2023-07-19 10:02:14.240234468 self._wait()
2023-07-19 10:02:14.240249777 File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 62, in _wait
2023-07-19 10:02:14.242112263 raise self._exc
2023-07-19 10:02:14.249262345 File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 177, in execute
2023-07-19 10:02:14.251695665 cursor = self.database._execute(obj.sql, obj.params)
2023-07-19 10:02:14.251710549 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql
2023-07-19 10:02:14.265251398 cursor.execute(sql, params or ())
2023-07-19 10:02:14.265266152 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3014, in __exit__
2023-07-19 10:02:14.271666464 reraise(new_type, new_type(exc_value, *exc_args), traceback)
2023-07-19 10:02:14.273429818 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 192, in reraise
2023-07-19 10:02:14.285790808 raise value.with_traceback(tb)
2023-07-19 10:02:14.285811216 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql
2023-07-19 10:02:14.303426253 cursor.execute(sql, params or ())
2023-07-19 10:02:14.303631589 peewee.OperationalError: database is locked
This doesn't seem right, if I'm correct you have 25 cameras which means the db has a timeout of 250 seconds. If the db is unable to write that change in 250 seconds that seems like an issue with the host / storage
Your camera count is indeed correct. So yesterday I was playing with a few things to try to troubleshoot this issue, and tried moving the DB file from my main host drive to the RAID array where I store my backup files, which is ZFS, still local. Using dd to do a speed test I was getting 6x-10x faster read/write speeds, but no effect at all on this issue.
During further troubleshooting, I've discovered that the auto-expiration process where the recordings storage drive fills up takes a TON of resources, in fact, I achieved a 60% reduction in CPU usage by adjusting my auto expiration settings to make sure that enough space is freed for the entire days recordings a day in advance. I was going to make a separate issue for that once I test one more time to make sure that was in fact true. But I still get my DB lock errors, even with drastically less load and many less hits on the DB.
I guess the next thing I could do is delete the database file and start over to make sure there isn't something in there that is causing it to move really slow if you want me to try that
I guess the next thing I could do is delete the database file and start over to make sure there isn't something in there that is causing it to move really slow if you want me to try that
Could be worth a try, can you also show your docker compose for frigate?
Ok, my suspicion is an empty DB will not give me the issue, and I believe it will take like 2 weeks for it to fill back up, which is why I didn't jump to that as of yet, but here's my docker-compose:
version: "3.9"
services:
frigate:
container_name: frigate
privileged: true # this may not be necessary for all setups
restart: unless-stopped
image: ghcr.io/blakeblackshear/frigate:dev-f2ff55d
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 1
capabilities: [gpu]
shm_size: "1521mb" # update for your cameras based on calculation above
devices:
- /dev/bus/usb:/dev/bus/usb # passes the USB Coral, needs to be modified for other versions
- /dev/dri/card0 # for intel hwaccel, needs to be updated for your hardware
volumes:
- /etc/localtime:/etc/localtime:ro
- /etc/frigate/config.yml:/config/config.yml
- /media/frigate/storage:/media/frigate
- /var/lib/frigate:/db
- type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
target: /tmp/cache
tmpfs:
size: 2500000000
ports:
- "5000:5000"
- "1935:1935" # RTMP feeds
- "8554:8554" # RTSP feeds
- "8555:8555/tcp" # WebRTC over tcp
- "8555:8555/udp" # WebRTC over udp
environment:
FRIGATE_RTSP_PASSWORD: removed
PLUS_API_KEY: removed
How large is the db currently?
3.2G Jul 19 11:57 frigate.db 64K Jul 19 12:10 frigate.db-shm 21M Jul 19 12:10 frigate.db-wal
Oh wow, that is massive. Okay so I see a couple things:
VACUUM
and show the new sizes after running that?So to run the manual vacuum I had to shutdown Frigate as DB was always locked, but after that: 3.2G Jul 19 12:20 frigate.db 6.3M Jul 19 12:21 frigate.db-shm 3.2G Jul 19 12:20 frigate.db-wal
Everything got bigger? I changed my config directory to expose the entire directory, will report back if that makes any difference
Once the write ahead log is merged to the db it should get smaller. It makes sense that it got bigger because the write ahead log is keeping the changes separately until they are merged.
So for reference, the SHM and WAL files went back down to size (160K and 79M)... and since the change enabling full config directory access, had it happen a few more times, here's the full log from end of startup till now (1.5 hours). Have not yet tried deleting the DB and running it yet, wanted to see if you had other ideas before I tried that. Also my order of the dual edge Google Coral is finally getting delivered today from my order last year (currently running USB), and wanted to try to make sure I'm not changing too much at once, not that I really expect much difference from that.
2023-07-19 14:05:04.935783849 [2023-07-19 14:05:04] frigate.app INFO : Capture process started for gp_bldg_03_side: 1514
2023-07-19 14:05:06.993078352 [2023-07-19 14:05:06] frigate.detectors.plugins.edgetpu_tfl INFO : TPU found
2023-07-19 14:05:09.284169961 [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f7624032480] moov atom not found
2023-07-19 14:05:09.370823522 [2023-07-19 14:05:09] frigate.record.maintainer WARNING : Failed to probe corrupt segment /tmp/cache/gp636_doorbell-20230719140434.mp4
2023-07-19 14:05:09.370925365 [2023-07-19 14:05:09] frigate.record.maintainer WARNING : Discarding a corrupt recording segment: /tmp/cache/gp636_doorbell-20230719140434.mp4
2023-07-19 14:05:09.478313324 [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f7624440b00] moov atom not found
2023-07-19 14:05:09.483466714 [ERROR:0@7.197] global cap.cpp:164 open VIDEOIO(CV_IMAGES): raised OpenCV exception:
2023-07-19 14:05:09.483470895
2023-07-19 14:05:09.483473733 OpenCV(4.7.0) /io/opencv/modules/videoio/src/cap_images.cpp:267: error: (-215:Assertion failed) number < max_number in function 'icvExtractPattern'
2023-07-19 14:05:09.483480065
2023-07-19 14:05:09.483481632
2023-07-19 14:05:09.511689656 [2023-07-19 14:05:09] frigate.record.maintainer WARNING : Failed to probe corrupt segment /tmp/cache/gp_clubhouse_firepit-20230719140433.mp4
2023-07-19 14:05:09.511978141 [2023-07-19 14:05:09] frigate.record.maintainer WARNING : Discarding a corrupt recording segment: /tmp/cache/gp_clubhouse_firepit-20230719140433.mp4
2023-07-19 14:05:09.525444910 [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f7624440b00] moov atom not found
2023-07-19 14:05:09.556675963 [2023-07-19 14:05:09] frigate.record.maintainer WARNING : Failed to probe corrupt segment /tmp/cache/gp_clubhouse_gym_1-20230719140441.mp4
2023-07-19 14:05:09.556830086 [2023-07-19 14:05:09] frigate.record.maintainer WARNING : Discarding a corrupt recording segment: /tmp/cache/gp_clubhouse_gym_1-20230719140441.mp4
2023-07-19 14:05:09.559521070 [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f7624440b00] moov atom not found
2023-07-19 14:05:09.597711532 [2023-07-19 14:05:09] frigate.record.maintainer WARNING : Failed to probe corrupt segment /tmp/cache/gp_clubhouse_gym_2-20230719140441.mp4
2023-07-19 14:05:09.597829647 [2023-07-19 14:05:09] frigate.record.maintainer WARNING : Discarding a corrupt recording segment: /tmp/cache/gp_clubhouse_gym_2-20230719140441.mp4
2023-07-19 14:05:09.627912344 [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f7624440b00] moov atom not found
2023-07-19 14:05:09.628009285 [ERROR:0@7.342] global cap.cpp:164 open VIDEOIO(CV_IMAGES): raised OpenCV exception:
2023-07-19 14:05:09.628012755
2023-07-19 14:05:09.628017077 OpenCV(4.7.0) /io/opencv/modules/videoio/src/cap_images.cpp:267: error: (-215:Assertion failed) number < max_number in function 'icvExtractPattern'
2023-07-19 14:05:09.628019815
2023-07-19 14:05:09.628022305
2023-07-19 14:05:09.665363215 [2023-07-19 14:05:09] frigate.record.maintainer WARNING : Failed to probe corrupt segment /tmp/cache/gp_clubhouse_pool_porch-20230719140442.mp4
2023-07-19 14:05:09.665956867 [2023-07-19 14:05:09] frigate.record.maintainer WARNING : Discarding a corrupt recording segment: /tmp/cache/gp_clubhouse_pool_porch-20230719140442.mp4
2023-07-19 14:05:09.667042493 [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f7624440b00] moov atom not found
2023-07-19 14:05:09.699408018 [2023-07-19 14:05:09] frigate.record.maintainer WARNING : Failed to probe corrupt segment /tmp/cache/gp_entrance_75-20230719140433.mp4
2023-07-19 14:05:09.699874644 [2023-07-19 14:05:09] frigate.record.maintainer WARNING : Discarding a corrupt recording segment: /tmp/cache/gp_entrance_75-20230719140433.mp4
2023-07-19 14:05:09.699979262 [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f7624440b00] moov atom not found
2023-07-19 14:05:09.700097107 [ERROR:0@7.414] global cap.cpp:164 open VIDEOIO(CV_IMAGES): raised OpenCV exception:
2023-07-19 14:05:09.700100732
2023-07-19 14:05:09.700105043 OpenCV(4.7.0) /io/opencv/modules/videoio/src/cap_images.cpp:267: error: (-215:Assertion failed) number < max_number in function 'icvExtractPattern'
2023-07-19 14:05:09.700107732
2023-07-19 14:05:09.700110122
2023-07-19 14:05:09.737633344 [2023-07-19 14:05:09] frigate.record.maintainer WARNING : Failed to probe corrupt segment /tmp/cache/gp_entrance_archer-20230719140441.mp4
2023-07-19 14:05:09.738021571 [2023-07-19 14:05:09] frigate.record.maintainer WARNING : Discarding a corrupt recording segment: /tmp/cache/gp_entrance_archer-20230719140441.mp4
2023-07-19 14:05:09.743464666 [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f7624440b00] moov atom not found
2023-07-19 14:05:09.744108921 [ERROR:0@7.458] global cap.cpp:164 open VIDEOIO(CV_IMAGES): raised OpenCV exception:
2023-07-19 14:05:09.744121871
2023-07-19 14:05:09.744126825 OpenCV(4.7.0) /io/opencv/modules/videoio/src/cap_images.cpp:267: error: (-215:Assertion failed) number < max_number in function 'icvExtractPattern'
2023-07-19 14:05:09.744130023
2023-07-19 14:05:09.744132524
2023-07-19 14:05:09.777590583 [2023-07-19 14:05:09] frigate.record.maintainer WARNING : Failed to probe corrupt segment /tmp/cache/warehouse_exterior_nw-20230719140434.mp4
2023-07-19 14:05:09.778164379 [2023-07-19 14:05:09] frigate.record.maintainer WARNING : Discarding a corrupt recording segment: /tmp/cache/warehouse_exterior_nw-20230719140434.mp4
2023-07-19 14:05:10.062995434 [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f7624440b00] moov atom not found
2023-07-19 14:05:10.063151669 [ERROR:0@7.777] global cap.cpp:164 open VIDEOIO(CV_IMAGES): raised OpenCV exception:
2023-07-19 14:05:10.063154037
2023-07-19 14:05:10.063156859 OpenCV(4.7.0) /io/opencv/modules/videoio/src/cap_images.cpp:267: error: (-215:Assertion failed) number < max_number in function 'icvExtractPattern'
2023-07-19 14:05:10.063158864
2023-07-19 14:05:10.063160374
2023-07-19 14:05:10.096662415 [2023-07-19 14:05:10] frigate.record.maintainer WARNING : Failed to probe corrupt segment /tmp/cache/warehouse_interior_overview-20230719140443.mp4
2023-07-19 14:05:10.097464861 [2023-07-19 14:05:10] frigate.record.maintainer WARNING : Discarding a corrupt recording segment: /tmp/cache/warehouse_interior_overview-20230719140443.mp4
2023-07-19 14:28:03.633009785 [2023-07-19 14:28:03] frigate.object_processing WARNING : Unable to create jpg because frame 1689791280.763218 is not in the cache
2023-07-19 14:28:41.539640682 [2023-07-19 14:28:41] frigate.object_processing WARNING : Unable to create jpg because frame 1689791280.763218 is not in the cache
2023-07-19 14:28:41.539740623 [2023-07-19 14:28:41] frigate.object_processing WARNING : Unable to save snapshot for 1689791259.540438-63sls2.
2023-07-19 14:28:41.539803005 [2023-07-19 14:28:41] frigate.object_processing WARNING : Unable to create clean png because frame 1689791280.763218 is not in the cache
2023-07-19 14:28:41.539854370 [2023-07-19 14:28:41] frigate.object_processing WARNING : Unable to save clean snapshot for 1689791259.540438-63sls2.
2023-07-19 14:28:41.540489346 [2023-07-19 14:28:41] frigate.object_processing WARNING : Unable to create jpg because frame 1689791280.763218 is not in the cache
2023-07-19 14:29:51.403473181 [2023-07-19 14:29:51] frigate.object_processing WARNING : Unable to create jpg because frame 1689791390.547153 is not in the cache
2023-07-19 14:30:03.539377830 [2023-07-19 14:30:03] frigate.object_processing WARNING : Unable to create jpg because frame 1689791390.547153 is not in the cache
2023-07-19 14:30:03.539512287 [2023-07-19 14:30:03] frigate.object_processing WARNING : Unable to save snapshot for 1689791317.765963-afqor4.
2023-07-19 14:30:03.539595885 [2023-07-19 14:30:03] frigate.object_processing WARNING : Unable to create clean png because frame 1689791390.547153 is not in the cache
2023-07-19 14:30:03.539671668 [2023-07-19 14:30:03] frigate.object_processing WARNING : Unable to save clean snapshot for 1689791317.765963-afqor4.
2023-07-19 14:30:03.541160656 [2023-07-19 14:30:03] frigate.object_processing WARNING : Unable to create jpg because frame 1689791390.547153 is not in the cache
2023-07-19 14:49:41.306030604 Exception in thread timeline_processor:
2023-07-19 14:49:41.306416853 Traceback (most recent call last):
2023-07-19 14:49:41.306426516 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql
2023-07-19 14:49:41.308120146 cursor.execute(sql, params or ())
2023-07-19 14:49:41.308151042 sqlite3.OperationalError: database is locked
2023-07-19 14:49:41.308157901
2023-07-19 14:49:41.308159676 During handling of the above exception, another exception occurred:
2023-07-19 14:49:41.308160993
2023-07-19 14:49:41.308162528 Traceback (most recent call last):
2023-07-19 14:49:41.308170216 File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner
2023-07-19 14:49:41.309166539 self.run()
2023-07-19 14:49:41.309185550 File "/opt/frigate/frigate/timeline.py", line 46, in run
2023-07-19 14:49:41.309591198 self.handle_object_detection(
2023-07-19 14:49:41.309695258 File "/opt/frigate/frigate/timeline.py", line 92, in handle_object_detection
2023-07-19 14:49:41.309785574 Timeline.insert(timeline_entry).execute()
2023-07-19 14:49:41.309792028 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 1966, in inner
2023-07-19 14:49:41.310130337 return method(self, database, *args, **kwargs)
2023-07-19 14:49:41.310138295 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2037, in execute
2023-07-19 14:49:41.310457614 return self._execute(database)
2023-07-19 14:49:41.310467678 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2842, in _execute
2023-07-19 14:49:41.310898007 return super(Insert, self)._execute(database)
2023-07-19 14:49:41.310908975 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2556, in _execute
2023-07-19 14:49:41.311272285 return self.handle_result(database, cursor)
2023-07-19 14:49:41.311440908 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2851, in handle_result
2023-07-19 14:49:41.316831388 return database.last_insert_id(cursor, self._query_type)
2023-07-19 14:49:41.316835305 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3749, in last_insert_id
2023-07-19 14:49:41.317430543 return cursor.lastrowid
2023-07-19 14:49:41.317441077 File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 87, in lastrowid
2023-07-19 14:49:41.317857224 self._wait()
2023-07-19 14:49:41.317889316 File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 62, in _wait
2023-07-19 14:49:41.318009519 raise self._exc
2023-07-19 14:49:41.318035054 File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 177, in execute
2023-07-19 14:49:41.318157035 cursor = self.database._execute(obj.sql, obj.params)
2023-07-19 14:49:41.318186073 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql
2023-07-19 14:49:41.318997723 cursor.execute(sql, params or ())
2023-07-19 14:49:41.319008706 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3014, in __exit__
2023-07-19 14:49:41.319752205 reraise(new_type, new_type(exc_value, *exc_args), traceback)
2023-07-19 14:49:41.319765290 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 192, in reraise
2023-07-19 14:49:41.319841621 raise value.with_traceback(tb)
2023-07-19 14:49:41.319849799 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql
2023-07-19 14:49:41.320656212 cursor.execute(sql, params or ())
2023-07-19 14:49:41.320669587 peewee.OperationalError: database is locked
2023-07-19 14:53:14.763705979 [2023-07-19 14:53:14] frigate.video ERROR : warehouse_interior_front: Unable to read frames from ffmpeg process.
2023-07-19 14:53:14.764016476 [2023-07-19 14:53:14] frigate.video ERROR : warehouse_interior_front: ffmpeg process is not running. exiting capture thread...
2023-07-19 14:53:15.095173898 [2023-07-19 14:53:15] watchdog.warehouse_interior_front ERROR : Ffmpeg process crashed unexpectedly for warehouse_interior_front.
2023-07-19 14:53:15.095381689 [2023-07-19 14:53:15] watchdog.warehouse_interior_front ERROR : The following ffmpeg logs include the last 100 lines prior to exit.
2023-07-19 14:53:15.095584574 [2023-07-19 14:53:15] ffmpeg.warehouse_interior_front.detect ERROR : rtsp://127.0.0.1:8554/warehouse_interior_front_sub: Connection timed out
2023-07-19 14:53:15.095745892 [2023-07-19 14:53:15] ffmpeg.warehouse_interior_front.detect ERROR : Last message repeated 1 times
2023-07-19 14:53:20.241772808 [2023-07-19 14:53:20] frigate.video ERROR : warehouse_interior_front: Unable to read frames from ffmpeg process.
2023-07-19 14:53:20.242285765 [2023-07-19 14:53:20] frigate.video ERROR : warehouse_interior_front: ffmpeg process is not running. exiting capture thread...
2023-07-19 14:53:25.125974072 [2023-07-19 14:53:25] watchdog.warehouse_interior_front ERROR : Ffmpeg process crashed unexpectedly for warehouse_interior_front.
2023-07-19 14:53:25.126114786 [2023-07-19 14:53:25] watchdog.warehouse_interior_front ERROR : The following ffmpeg logs include the last 100 lines prior to exit.
2023-07-19 14:53:25.126223301 [2023-07-19 14:53:25] ffmpeg.warehouse_interior_front.detect ERROR : rtsp://127.0.0.1:8554/warehouse_interior_front_sub: Invalid data found when processing input
2023-07-19 14:53:25.156582141 [2023-07-19 14:53:25] ffmpeg.warehouse_interior_front.record ERROR : [segment @ 0x556e1fda4580] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly
2023-07-19 14:53:25.172398503 [2023-07-19 14:53:25] ffmpeg.warehouse_interior_front.record ERROR : [aac @ 0x556e1fda3740] channel element 0.0 is not allocated
2023-07-19 14:53:25.172559015 [2023-07-19 14:53:25] ffmpeg.warehouse_interior_front.record ERROR : Error while decoding stream #0:1: Invalid data found when processing input
2023-07-19 14:53:25.172644575 [2023-07-19 14:53:25] ffmpeg.warehouse_interior_front.record ERROR : [aac @ 0x556e1fda3740] channel element 0.0 is not allocated
2023-07-19 14:53:25.172735064 [2023-07-19 14:53:25] ffmpeg.warehouse_interior_front.record ERROR : Error while decoding stream #0:1: Invalid data found when processing input
2023-07-19 14:53:25.172807134 [2023-07-19 14:53:25] ffmpeg.warehouse_interior_front.record ERROR : [aac @ 0x556e1fda3740] Input buffer exhausted before END element found
2023-07-19 14:53:25.172877940 [2023-07-19 14:53:25] ffmpeg.warehouse_interior_front.record ERROR : Error while decoding stream #0:1: Invalid data found when processing input
2023-07-19 14:53:25.172950672 [2023-07-19 14:53:25] ffmpeg.warehouse_interior_front.record ERROR : rtsp://127.0.0.1:8554/warehouse_interior_front: Connection timed out
2023-07-19 14:53:25.173033584 [2023-07-19 14:53:25] watchdog.warehouse_interior_front INFO : Terminating the existing ffmpeg process...
2023-07-19 14:53:25.173110939 [2023-07-19 14:53:25] watchdog.warehouse_interior_front INFO : Waiting for ffmpeg to exit gracefully...
2023-07-19 14:53:27.744338974 [2023-07-19 14:53:27] frigate.video ERROR : warehouse_interior_front: Unable to read frames from ffmpeg process.
2023-07-19 14:53:27.744830786 [2023-07-19 14:53:27] frigate.video ERROR : warehouse_interior_front: ffmpeg process is not running. exiting capture thread...
2023-07-19 14:53:35.191596475 [2023-07-19 14:53:35] watchdog.warehouse_interior_front ERROR : Ffmpeg process crashed unexpectedly for warehouse_interior_front.
2023-07-19 14:53:35.191735098 [2023-07-19 14:53:35] watchdog.warehouse_interior_front ERROR : The following ffmpeg logs include the last 100 lines prior to exit.
2023-07-19 14:53:35.191823828 [2023-07-19 14:53:35] ffmpeg.warehouse_interior_front.detect ERROR : [rtsp @ 0x5564b1ac8f00] method DESCRIBE failed: 404 Not Found
2023-07-19 14:53:35.191906075 [2023-07-19 14:53:35] ffmpeg.warehouse_interior_front.detect ERROR : rtsp://127.0.0.1:8554/warehouse_interior_front_sub: Server returned 404 Not Found
2023-07-19 14:53:35.226665625 [2023-07-19 14:53:35] ffmpeg.warehouse_interior_front.record ERROR : [rtsp @ 0x55dfcb431000] method DESCRIBE failed: 404 Not Found
2023-07-19 14:53:35.226853984 [2023-07-19 14:53:35] ffmpeg.warehouse_interior_front.record ERROR : rtsp://127.0.0.1:8554/warehouse_interior_front: Server returned 404 Not Found
2023-07-19 14:53:35.227014950 [2023-07-19 14:53:35] watchdog.warehouse_interior_front INFO : Terminating the existing ffmpeg process...
2023-07-19 14:53:35.243611619 [2023-07-19 14:53:35] watchdog.warehouse_interior_front INFO : Waiting for ffmpeg to exit gracefully...
2023-07-19 14:53:35.245432689 [2023-07-19 14:53:35] frigate.video ERROR : warehouse_interior_front: Unable to read frames from ffmpeg process.
2023-07-19 14:53:35.246636847 [2023-07-19 14:53:35] frigate.video ERROR : warehouse_interior_front: ffmpeg process is not running. exiting capture thread...
2023-07-19 14:53:45.261952582 [2023-07-19 14:53:45] watchdog.warehouse_interior_front ERROR : Ffmpeg process crashed unexpectedly for warehouse_interior_front.
2023-07-19 14:53:45.262069500 [2023-07-19 14:53:45] watchdog.warehouse_interior_front ERROR : The following ffmpeg logs include the last 100 lines prior to exit.
2023-07-19 14:53:45.262156610 [2023-07-19 14:53:45] ffmpeg.warehouse_interior_front.detect ERROR : [rtsp @ 0x559925dd0f00] method DESCRIBE failed: 404 Not Found
2023-07-19 14:53:45.262241758 [2023-07-19 14:53:45] ffmpeg.warehouse_interior_front.detect ERROR : rtsp://127.0.0.1:8554/warehouse_interior_front_sub: Server returned 404 Not Found
2023-07-19 14:53:45.298182131 [2023-07-19 14:53:45] ffmpeg.warehouse_interior_front.record ERROR : [rtsp @ 0x55a330ac6000] method DESCRIBE failed: 404 Not Found
2023-07-19 14:53:45.298318891 [2023-07-19 14:53:45] ffmpeg.warehouse_interior_front.record ERROR : rtsp://127.0.0.1:8554/warehouse_interior_front: Server returned 404 Not Found
2023-07-19 14:53:45.298413437 [2023-07-19 14:53:45] watchdog.warehouse_interior_front INFO : Terminating the existing ffmpeg process...
2023-07-19 14:53:45.298490568 [2023-07-19 14:53:45] watchdog.warehouse_interior_front INFO : Waiting for ffmpeg to exit gracefully...
2023-07-19 14:53:45.315449319 [2023-07-19 14:53:45] frigate.video ERROR : warehouse_interior_front: Unable to read frames from ffmpeg process.
2023-07-19 14:53:45.315818754 [2023-07-19 14:53:45] frigate.video ERROR : warehouse_interior_front: ffmpeg process is not running. exiting capture thread...
2023-07-19 14:53:55.330039150 [2023-07-19 14:53:55] watchdog.warehouse_interior_front ERROR : Ffmpeg process crashed unexpectedly for warehouse_interior_front.
2023-07-19 14:53:55.330118729 [2023-07-19 14:53:55] watchdog.warehouse_interior_front ERROR : The following ffmpeg logs include the last 100 lines prior to exit.
2023-07-19 14:53:55.330186618 [2023-07-19 14:53:55] ffmpeg.warehouse_interior_front.detect ERROR : [rtsp @ 0x563eee392f00] method DESCRIBE failed: 404 Not Found
2023-07-19 14:53:55.330246707 [2023-07-19 14:53:55] ffmpeg.warehouse_interior_front.detect ERROR : rtsp://127.0.0.1:8554/warehouse_interior_front_sub: Server returned 404 Not Found
2023-07-19 14:53:55.364380958 [2023-07-19 14:53:55] ffmpeg.warehouse_interior_front.record ERROR : [rtsp @ 0x556e0d041000] method DESCRIBE failed: 404 Not Found
2023-07-19 14:53:55.364716917 [2023-07-19 14:53:55] ffmpeg.warehouse_interior_front.record ERROR : rtsp://127.0.0.1:8554/warehouse_interior_front: Server returned 404 Not Found
2023-07-19 14:53:55.364776006 [2023-07-19 14:53:55] watchdog.warehouse_interior_front INFO : Terminating the existing ffmpeg process...
2023-07-19 14:53:55.364850965 [2023-07-19 14:53:55] watchdog.warehouse_interior_front INFO : Waiting for ffmpeg to exit gracefully...
2023-07-19 14:53:55.382431141 [2023-07-19 14:53:55] frigate.video ERROR : warehouse_interior_front: Unable to read frames from ffmpeg process.
2023-07-19 14:53:55.383017393 [2023-07-19 14:53:55] frigate.video ERROR : warehouse_interior_front: ffmpeg process is not running. exiting capture thread...
2023-07-19 14:54:05.398514691 [2023-07-19 14:54:05] watchdog.warehouse_interior_front ERROR : Ffmpeg process crashed unexpectedly for warehouse_interior_front.
2023-07-19 14:54:05.398680563 [2023-07-19 14:54:05] watchdog.warehouse_interior_front ERROR : The following ffmpeg logs include the last 100 lines prior to exit.
2023-07-19 14:54:05.398837422 [2023-07-19 14:54:05] ffmpeg.warehouse_interior_front.detect ERROR : [rtsp @ 0x5630e5f43f00] method DESCRIBE failed: 404 Not Found
2023-07-19 14:54:05.398981089 [2023-07-19 14:54:05] ffmpeg.warehouse_interior_front.detect ERROR : rtsp://127.0.0.1:8554/warehouse_interior_front_sub: Server returned 404 Not Found
2023-07-19 14:54:05.438075497 [2023-07-19 14:54:05] ffmpeg.warehouse_interior_front.record ERROR : [rtsp @ 0x5648aaedf000] method DESCRIBE failed: 404 Not Found
2023-07-19 14:54:05.438562790 [2023-07-19 14:54:05] ffmpeg.warehouse_interior_front.record ERROR : rtsp://127.0.0.1:8554/warehouse_interior_front: Server returned 404 Not Found
2023-07-19 14:54:05.438575256 [2023-07-19 14:54:05] watchdog.warehouse_interior_front INFO : Terminating the existing ffmpeg process...
2023-07-19 14:54:05.438750594 [2023-07-19 14:54:05] watchdog.warehouse_interior_front INFO : Waiting for ffmpeg to exit gracefully...
2023-07-19 14:54:05.458499396 [2023-07-19 14:54:05] frigate.video ERROR : warehouse_interior_front: Unable to read frames from ffmpeg process.
2023-07-19 14:54:05.458648523 [2023-07-19 14:54:05] frigate.video ERROR : warehouse_interior_front: ffmpeg process is not running. exiting capture thread...
2023-07-19 14:54:15.472376672 [2023-07-19 14:54:15] watchdog.warehouse_interior_front ERROR : Ffmpeg process crashed unexpectedly for warehouse_interior_front.
2023-07-19 14:54:15.472479926 [2023-07-19 14:54:15] watchdog.warehouse_interior_front ERROR : The following ffmpeg logs include the last 100 lines prior to exit.
2023-07-19 14:54:15.472560808 [2023-07-19 14:54:15] ffmpeg.warehouse_interior_front.detect ERROR : [rtsp @ 0x55b4f3f8ef00] method DESCRIBE failed: 404 Not Found
2023-07-19 14:54:15.472654510 [2023-07-19 14:54:15] ffmpeg.warehouse_interior_front.detect ERROR : rtsp://127.0.0.1:8554/warehouse_interior_front_sub: Server returned 404 Not Found
2023-07-19 14:54:15.506323201 [2023-07-19 14:54:15] ffmpeg.warehouse_interior_front.record ERROR : [rtsp @ 0x55a1067f7000] method DESCRIBE failed: 404 Not Found
2023-07-19 14:54:15.506531602 [2023-07-19 14:54:15] ffmpeg.warehouse_interior_front.record ERROR : rtsp://127.0.0.1:8554/warehouse_interior_front: Server returned 404 Not Found
2023-07-19 14:54:15.506675749 [2023-07-19 14:54:15] watchdog.warehouse_interior_front INFO : Terminating the existing ffmpeg process...
2023-07-19 14:54:15.506788169 [2023-07-19 14:54:15] watchdog.warehouse_interior_front INFO : Waiting for ffmpeg to exit gracefully...
2023-07-19 14:54:15.519536333 [2023-07-19 14:54:15] frigate.video ERROR : warehouse_interior_front: Unable to read frames from ffmpeg process.
2023-07-19 14:54:15.519856166 [2023-07-19 14:54:15] frigate.video ERROR : warehouse_interior_front: ffmpeg process is not running. exiting capture thread...
2023-07-19 14:54:25.536218035 [2023-07-19 14:54:25] watchdog.warehouse_interior_front ERROR : Ffmpeg process crashed unexpectedly for warehouse_interior_front.
2023-07-19 14:54:25.536510239 [2023-07-19 14:54:25] watchdog.warehouse_interior_front ERROR : The following ffmpeg logs include the last 100 lines prior to exit.
2023-07-19 14:54:25.536820054 [2023-07-19 14:54:25] ffmpeg.warehouse_interior_front.detect ERROR : [rtsp @ 0x55d05d6b0f00] method DESCRIBE failed: 404 Not Found
2023-07-19 14:54:25.537005601 [2023-07-19 14:54:25] ffmpeg.warehouse_interior_front.detect ERROR : rtsp://127.0.0.1:8554/warehouse_interior_front_sub: Server returned 404 Not Found
2023-07-19 14:54:25.571962159 [2023-07-19 14:54:25] ffmpeg.warehouse_interior_front.record ERROR : [rtsp @ 0x55a6c3884000] method DESCRIBE failed: 404 Not Found
2023-07-19 14:54:25.572266912 [2023-07-19 14:54:25] ffmpeg.warehouse_interior_front.record ERROR : rtsp://127.0.0.1:8554/warehouse_interior_front: Server returned 404 Not Found
2023-07-19 14:54:25.572913306 [2023-07-19 14:54:25] watchdog.warehouse_interior_front INFO : Terminating the existing ffmpeg process...
2023-07-19 14:54:25.573179693 [2023-07-19 14:54:25] watchdog.warehouse_interior_front INFO : Waiting for ffmpeg to exit gracefully...
2023-07-19 14:54:25.589177460 [2023-07-19 14:54:25] frigate.video ERROR : warehouse_interior_front: Unable to read frames from ffmpeg process.
2023-07-19 14:54:25.589423031 [2023-07-19 14:54:25] frigate.video ERROR : warehouse_interior_front: ffmpeg process is not running. exiting capture thread...
2023-07-19 14:54:35.603474275 [2023-07-19 14:54:35] watchdog.warehouse_interior_front ERROR : Ffmpeg process crashed unexpectedly for warehouse_interior_front.
2023-07-19 14:54:35.603686937 [2023-07-19 14:54:35] watchdog.warehouse_interior_front ERROR : The following ffmpeg logs include the last 100 lines prior to exit.
2023-07-19 14:54:35.603920018 [2023-07-19 14:54:35] ffmpeg.warehouse_interior_front.detect ERROR : [rtsp @ 0x55af9e860f00] method DESCRIBE failed: 404 Not Found
2023-07-19 14:54:35.604109203 [2023-07-19 14:54:35] ffmpeg.warehouse_interior_front.detect ERROR : rtsp://127.0.0.1:8554/warehouse_interior_front_sub: Server returned 404 Not Found
2023-07-19 14:54:35.649154771 [2023-07-19 14:54:35] ffmpeg.warehouse_interior_front.record ERROR : [rtsp @ 0x5599dad84000] method DESCRIBE failed: 404 Not Found
2023-07-19 14:54:35.649247565 [2023-07-19 14:54:35] ffmpeg.warehouse_interior_front.record ERROR : rtsp://127.0.0.1:8554/warehouse_interior_front: Server returned 404 Not Found
2023-07-19 14:54:35.649305490 [2023-07-19 14:54:35] watchdog.warehouse_interior_front INFO : Terminating the existing ffmpeg process...
2023-07-19 14:54:35.649354888 [2023-07-19 14:54:35] watchdog.warehouse_interior_front INFO : Waiting for ffmpeg to exit gracefully...
2023-07-19 14:54:40.764049917 [2023-07-19 14:54:40] frigate.video ERROR : warehouse_interior_front: Unable to read frames from ffmpeg process.
2023-07-19 14:54:40.764573431 [2023-07-19 14:54:40] frigate.video ERROR : warehouse_interior_front: ffmpeg process is not running. exiting capture thread...
2023-07-19 14:54:45.686344141 [2023-07-19 14:54:45] watchdog.warehouse_interior_front ERROR : Ffmpeg process crashed unexpectedly for warehouse_interior_front.
2023-07-19 14:54:45.686514163 [2023-07-19 14:54:45] watchdog.warehouse_interior_front ERROR : The following ffmpeg logs include the last 100 lines prior to exit.
2023-07-19 14:54:45.686627434 [2023-07-19 14:54:45] ffmpeg.warehouse_interior_front.detect ERROR : rtsp://127.0.0.1:8554/warehouse_interior_front_sub: Invalid data found when processing input
2023-07-19 14:54:45.717520347 [2023-07-19 14:54:45] watchdog.warehouse_interior_front ERROR : No new recording segments were created for warehouse_interior_front in the last 120s. restarting the ffmpeg record process...
2023-07-19 14:54:45.717743917 [2023-07-19 14:54:45] watchdog.warehouse_interior_front INFO : Terminating the existing ffmpeg process...
2023-07-19 14:54:45.717866264 [2023-07-19 14:54:45] watchdog.warehouse_interior_front INFO : Waiting for ffmpeg to exit gracefully...
2023-07-19 14:59:24.645395062 Exception in thread event_processor:
2023-07-19 14:59:24.645402112 Traceback (most recent call last):
2023-07-19 14:59:24.645568576 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql
2023-07-19 14:59:24.678170706 cursor.execute(sql, params or ())
2023-07-19 14:59:24.678371111 sqlite3.OperationalError: database is locked
2023-07-19 14:59:24.678373407
2023-07-19 14:59:24.678375380 During handling of the above exception, another exception occurred:
2023-07-19 14:59:24.678376756
2023-07-19 14:59:24.678430985 Traceback (most recent call last):
2023-07-19 14:59:24.678439777 File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner
2023-07-19 14:59:24.693580562 self.run()
2023-07-19 14:59:24.693774376 File "/opt/frigate/frigate/events/maintainer.py", line 93, in run
2023-07-19 14:59:24.699297875 self.handle_object_detection(event_type, camera, event_data)
2023-07-19 14:59:24.699324883 File "/opt/frigate/frigate/events/maintainer.py", line 205, in handle_object_detection
2023-07-19 14:59:24.717302611 Event.insert(event)
2023-07-19 14:59:24.717326310 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 1966, in inner
2023-07-19 14:59:24.725860641 return method(self, database, *args, **kwargs)
2023-07-19 14:59:24.725894414 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2037, in execute
2023-07-19 14:59:24.736404946 return self._execute(database)
2023-07-19 14:59:24.736411984 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2842, in _execute
2023-07-19 14:59:24.742417231 return super(Insert, self)._execute(database)
2023-07-19 14:59:24.742454397 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2556, in _execute
2023-07-19 14:59:24.752735931 return self.handle_result(database, cursor)
2023-07-19 14:59:24.757469824 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2851, in handle_result
2023-07-19 14:59:24.800752316 return database.last_insert_id(cursor, self._query_type)
2023-07-19 14:59:24.801096127 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3749, in last_insert_id
2023-07-19 14:59:24.828738636 return cursor.lastrowid
2023-07-19 14:59:24.828765844 File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 87, in lastrowid
2023-07-19 14:59:24.845037833 self._wait()
2023-07-19 14:59:24.845068045 File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 62, in _wait
2023-07-19 14:59:24.848293298 raise self._exc
2023-07-19 14:59:24.848564637 File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 177, in execute
2023-07-19 14:59:24.877475296 cursor = self.database._execute(obj.sql, obj.params)
2023-07-19 14:59:24.877522099 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql
2023-07-19 14:59:24.913647512 cursor.execute(sql, params or ())
2023-07-19 14:59:24.913672069 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3014, in __exit__
2023-07-19 14:59:24.942764555 reraise(new_type, new_type(exc_value, *exc_args), traceback)
2023-07-19 14:59:24.942788804 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 192, in reraise
2023-07-19 14:59:24.944917497 raise value.with_traceback(tb)
2023-07-19 14:59:24.944923921 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql
2023-07-19 14:59:24.967764664 cursor.execute(sql, params or ())
2023-07-19 14:59:24.967797397 peewee.OperationalError: database is locked
2023-07-19 15:00:37.476707097 [2023-07-19 15:00:37] frigate.object_processing WARNING : Unable to create jpg because frame 1689793233.618259 is not in the cache
2023-07-19 15:00:37.477011756 [2023-07-19 15:00:37] frigate.object_processing WARNING : Unable to save snapshot for 1689793232.872475-fj58un.
2023-07-19 15:00:37.477031841 [2023-07-19 15:00:37] frigate.object_processing WARNING : Unable to create clean png because frame 1689793233.618259 is not in the cache
2023-07-19 15:00:37.477899857 [2023-07-19 15:00:37] frigate.object_processing WARNING : Unable to save clean snapshot for 1689793232.872475-fj58un.
2023-07-19 15:00:37.478222216 [2023-07-19 15:00:37] frigate.object_processing WARNING : Unable to create jpg because frame 1689793233.618259 is not in the cache
2023-07-19 15:04:51.087972174 Exception in thread event_cleanup:
2023-07-19 15:04:51.087992548 Traceback (most recent call last):
2023-07-19 15:04:51.087994659 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql
2023-07-19 15:04:51.137087370 cursor.execute(sql, params or ())
2023-07-19 15:04:51.137681407 sqlite3.OperationalError: database is locked
2023-07-19 15:04:51.137691704
2023-07-19 15:04:51.137694009 During handling of the above exception, another exception occurred:
2023-07-19 15:04:51.137695615
2023-07-19 15:04:51.137697497 Traceback (most recent call last):
2023-07-19 15:04:51.137705651 File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner
2023-07-19 15:04:51.149135325 self.run()
2023-07-19 15:04:51.149160687 File "/opt/frigate/frigate/events/cleanup.py", line 164, in run
2023-07-19 15:04:51.150719048 self.expire("clips")
2023-07-19 15:04:51.150726503 File "/opt/frigate/frigate/events/cleanup.py", line 125, in expire
2023-07-19 15:04:51.150888196 update_query.execute()
2023-07-19 15:04:51.150905226 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 1966, in inner
2023-07-19 15:04:51.158851509 return method(self, database, *args, **kwargs)
2023-07-19 15:04:51.158935061 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2037, in execute
2023-07-19 15:04:51.190647638 return self._execute(database)
2023-07-19 15:04:51.190694665 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2556, in _execute
2023-07-19 15:04:51.208109332 return self.handle_result(database, cursor)
2023-07-19 15:04:51.208134021 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2567, in handle_result
2023-07-19 15:04:51.209236994 return database.rows_affected(cursor)
2023-07-19 15:04:51.209321316 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3759, in rows_affected
2023-07-19 15:04:51.231111916 return cursor.rowcount
2023-07-19 15:04:51.231134510 File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 93, in rowcount
2023-07-19 15:04:51.245487015 self._wait()
2023-07-19 15:04:51.245766792 File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 62, in _wait
2023-07-19 15:04:51.253091979 raise self._exc
2023-07-19 15:04:51.253124326 File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 177, in execute
2023-07-19 15:04:51.253385616 cursor = self.database._execute(obj.sql, obj.params)
2023-07-19 15:04:51.253397615 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql
2023-07-19 15:04:51.267243391 cursor.execute(sql, params or ())
2023-07-19 15:04:51.267345892 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3014, in __exit__
2023-07-19 15:04:51.270761863 reraise(new_type, new_type(exc_value, *exc_args), traceback)
2023-07-19 15:04:51.270775679 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 192, in reraise
2023-07-19 15:04:51.271138336 raise value.with_traceback(tb)
2023-07-19 15:04:51.271149300 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql
2023-07-19 15:04:51.309389243 cursor.execute(sql, params or ())
2023-07-19 15:04:51.310975993 peewee.OperationalError: database is locked
I have a hard time understanding how the DB could be so far behind that it takes 250 seconds and is still locked, but maybe something else is going on. I have made a few improvements in #7227 that should help somewhat
Note I experienced it with 1 camera, so with the recent commit of lowering the timeout from 60 -> 10 I didn't bother testing it yet. I will try the latest changes once they get merged and report back. It has been so rare for me that I won't know for awhile either though (it hasn't happened in 24 hours)
That just seems weird, 60 seconds with the activity of one camera again seems like an issue with the host / storage that holds the DB from what I can tell. My 6 cameras (so 60 seconds timeout before and after the change) have not had this issue ever.
I don't believe I mentioned it previously and don't know if it will help, but the last build I ran for a considerable amount of time was the dev build immediately preceding your birdseye view revamp (the last dev build with the grid). On that build, I never once had a DB locked error, and the object recognition was consistently under 1 second. That build was fully rock solid with a large amount of cameras from my experience. Unfortunately I changed a couple cameras around the same time I switched newer dev builds (after you applied the last round of birdseye view fixes) so I wasn't sure if something else I changed caused some funkiness, but thought I would mention that in case it would help.
2023-07-19 17:47:51.442605605 [2023-07-19 17:47:51] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for gp636_doorbell. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-07-19 17:47:51.442720836 [2023-07-19 17:47:51] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for gp_bldg_03_side. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-07-19 17:47:51.442991148 [2023-07-19 17:47:51] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for gp_bldg_08_front. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-07-19 17:47:51.443188533 [2023-07-19 17:47:51] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for gp_clubhouse_back_door. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-07-19 17:47:51.443389411 [2023-07-19 17:47:51] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for gp_clubhouse_banquet. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-07-19 17:47:51.443588370 [2023-07-19 17:47:51] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for gp_clubhouse_firepit. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-07-19 17:47:51.443797216 [2023-07-19 17:47:51] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for gp_clubhouse_front_door. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-07-19 17:47:51.444014054 [2023-07-19 17:47:51] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for gp_clubhouse_gym_1. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-07-19 17:47:51.444222003 [2023-07-19 17:47:51] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for gp_clubhouse_gym_2. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-07-19 17:47:51.444418403 [2023-07-19 17:47:51] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for gp_clubhouse_parking. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-07-19 17:47:51.444635019 [2023-07-19 17:47:51] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for gp_clubhouse_pool. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-07-19 17:47:51.444823123 [2023-07-19 17:47:51] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for gp_clubhouse_pool_porch. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-07-19 17:47:51.445018796 [2023-07-19 17:47:51] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for gp_entrance_archer. Keeping the 5 most recent segments out of 7 and discarding the rest...
2023-07-19 17:47:51.445347171 [2023-07-19 17:47:51] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for warehouse_exterior_nw. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-07-19 17:47:51.445533760 [2023-07-19 17:47:51] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for warehouse_exterior_se. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-07-19 17:47:51.445722186 [2023-07-19 17:47:51] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for warehouse_exterior_sw. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-07-19 17:47:51.445919373 [2023-07-19 17:47:51] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for warehouse_interior_front. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-07-19 17:47:51.446102757 [2023-07-19 17:47:51] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for warehouse_interior_overview. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-07-19 17:47:51.446332000 [2023-07-19 17:47:51] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for warehouse_workbench. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-07-19 17:49:32.299686551 [2023-07-19 17:49:32] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for gp636_doorbell. Keeping the 5 most recent segments out of 10 and discarding the rest...
2023-07-19 17:49:32.300532919 [2023-07-19 17:49:32] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for gp_bldg_03_side. Keeping the 5 most recent segments out of 10 and discarding the rest...
2023-07-19 17:49:32.301066652 [2023-07-19 17:49:32] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for gp_bldg_08_front. Keeping the 5 most recent segments out of 10 and discarding the rest...
2023-07-19 17:49:32.301604455 [2023-07-19 17:49:32] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for gp_bldg_08_rear. Keeping the 5 most recent segments out of 11 and discarding the rest...
2023-07-19 17:49:32.302353840 [2023-07-19 17:49:32] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for gp_clubhouse_back_door. Keeping the 5 most recent segments out of 10 and discarding the rest...
2023-07-19 17:49:32.303078195 [2023-07-19 17:49:32] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for gp_clubhouse_banquet. Keeping the 5 most recent segments out of 10 and discarding the rest...
2023-07-19 17:49:32.304277275 [2023-07-19 17:49:32] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for gp_clubhouse_firepit. Keeping the 5 most recent segments out of 10 and discarding the rest...
2023-07-19 17:49:32.304986473 [2023-07-19 17:49:32] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for gp_clubhouse_front_door. Keeping the 5 most recent segments out of 10 and discarding the rest...
2023-07-19 17:49:32.305632511 [2023-07-19 17:49:32] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for gp_clubhouse_gym_1. Keeping the 5 most recent segments out of 10 and discarding the rest...
2023-07-19 17:49:32.306422428 [2023-07-19 17:49:32] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for gp_clubhouse_gym_2. Keeping the 5 most recent segments out of 10 and discarding the rest...
2023-07-19 17:49:32.307211452 [2023-07-19 17:49:32] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for gp_clubhouse_parking. Keeping the 5 most recent segments out of 10 and discarding the rest...
2023-07-19 17:49:32.307832624 [2023-07-19 17:49:32] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for gp_clubhouse_pool. Keeping the 5 most recent segments out of 10 and discarding the rest...
2023-07-19 17:49:32.308520954 [2023-07-19 17:49:32] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for gp_clubhouse_pool_porch. Keeping the 5 most recent segments out of 10 and discarding the rest...
2023-07-19 17:49:32.309208895 [2023-07-19 17:49:32] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for gp_entrance_archer. Keeping the 5 most recent segments out of 10 and discarding the rest...
2023-07-19 17:49:32.309906754 [2023-07-19 17:49:32] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for warehouse_exterior_nw. Keeping the 5 most recent segments out of 10 and discarding the rest...
2023-07-19 17:49:32.310921008 [2023-07-19 17:49:32] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for warehouse_exterior_se. Keeping the 5 most recent segments out of 10 and discarding the rest...
2023-07-19 17:49:32.311427649 [2023-07-19 17:49:32] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for warehouse_exterior_sw. Keeping the 5 most recent segments out of 10 and discarding the rest...
2023-07-19 17:49:32.312631965 [2023-07-19 17:49:32] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for warehouse_interior_front. Keeping the 5 most recent segments out of 10 and discarding the rest...
2023-07-19 17:49:32.313533449 [2023-07-19 17:49:32] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for warehouse_interior_overview. Keeping the 5 most recent segments out of 10 and discarding the rest...
2023-07-19 17:49:32.314128081 [2023-07-19 17:49:32] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for warehouse_workbench. Keeping the 5 most recent segments out of 10 and discarding the rest...
I don't see any changes that would increase DB activity and cause this issue, seems like it would likely be whatever changes were made to cameras.
however I started paying a bit more attention to other delays, and I've noticed pretty extreme delays on the object recognition count getting into Home Assistant, so I think something else is in play here
I don't believe db delays will affect object count via MQTT at all so it seems unrelated
The DB locking is definitely causing an effect on the object count that Home Assistant is receiving. Every time the DB locks, while Frigate seems to recover from it eventually and I just have some time of lost recordings, Home Assistant seems to start adding to the object count like crazy, and I end up with Home Assistant reading like 23 people in a view with 0. Every time the DB locks, I have to restart the Home Assistant plugin, it never resets to 0. Even restarting Frigate doesn't reset Home Assistant's object counters back to 0 after the DB locks.
Even after restarting Frigate, have to restart the Home Assistant plugin after the database error for my detections/automations to work correctly)
1.5 hours and no DB errors with #7227 and actually no more of the can't keep up errors since the group of them at the 30 minute point. I have gone 10 hours without the error in the past though
I meant if the DB is slow to respond but not actually having the lock error I don't believe it will affect the mqtt object count.
If frigate actually has a crash from the db locked error then it will absolutely affect the mqtt object count because frigate is crashed and not working correctly.
In any case though, it is good to hear that that change has so far fixed the issue.
Unfortunately, it did eventually come in multiple times, but it did take significantly longer to appear. I also seemed to have a memory leak that sucked all my 70GB of RAM until Frigate stopped working entirely :-/
2023-07-19 20:57:54.938386870 Exception in thread event_processor:
2023-07-19 20:57:54.938421216 Traceback (most recent call last):
2023-07-19 20:57:54.938424705 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql
2023-07-19 20:57:54.942297772 cursor.execute(sql, params or ())
2023-07-19 20:57:54.942314561 sqlite3.OperationalError: database is locked
2023-07-19 20:57:54.942317619
2023-07-19 20:57:54.942320863 During handling of the above exception, another exception occurred:
2023-07-19 20:57:54.942322978
2023-07-19 20:57:54.942325600 Traceback (most recent call last):
2023-07-19 20:57:54.942328281 File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner
2023-07-19 20:57:54.942852870 self.run()
2023-07-19 20:57:54.942869188 File "/opt/frigate/frigate/events/maintainer.py", line 93, in run
2023-07-19 20:57:54.943567915 self.handle_object_detection(event_type, camera, event_data)
2023-07-19 20:57:54.943748845 File "/opt/frigate/frigate/events/maintainer.py", line 205, in handle_object_detection
2023-07-19 20:57:54.944025244 Event.insert(event)
2023-07-19 20:57:54.944038745 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 1966, in inner
2023-07-19 20:57:54.944824895 return method(self, database, *args, **kwargs)
2023-07-19 20:57:54.944831462 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2037, in execute
2023-07-19 20:57:54.945590176 return self._execute(database)
2023-07-19 20:57:54.945597157 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2842, in _execute
2023-07-19 20:57:54.946829045 return super(Insert, self)._execute(database)
2023-07-19 20:57:54.946853511 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2556, in _execute
2023-07-19 20:57:54.948082069 return self.handle_result(database, cursor)
2023-07-19 20:57:54.948103109 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2851, in handle_result
2023-07-19 20:57:54.948926720 return database.last_insert_id(cursor, self._query_type)
2023-07-19 20:57:54.948931465 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3749, in last_insert_id
2023-07-19 20:57:54.950009943 return cursor.lastrowid
2023-07-19 20:57:54.950020689 File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 87, in lastrowid
2023-07-19 20:57:54.950124173 self._wait()
2023-07-19 20:57:54.950132095 File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 62, in _wait
2023-07-19 20:57:54.950193910 raise self._exc
2023-07-19 20:57:54.950200070 File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 177, in execute
2023-07-19 20:57:54.950278410 cursor = self.database._execute(obj.sql, obj.params)
2023-07-19 20:57:54.950285888 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql
2023-07-19 20:57:54.951382985 cursor.execute(sql, params or ())
2023-07-19 20:57:54.952043540 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3014, in __exit__
2023-07-19 20:57:54.952806685 reraise(new_type, new_type(exc_value, *exc_args), traceback)
2023-07-19 20:57:54.952813306 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 192, in reraise
2023-07-19 20:57:54.952916305 raise value.with_traceback(tb)
2023-07-19 20:57:54.952922694 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql
2023-07-19 20:57:54.953647736 cursor.execute(sql, params or ())
2023-07-19 20:57:54.953670615 peewee.OperationalError: database is locked
I also noticed /tmp/cache seems to both be way fuller then usual, along with the max size seemingly being the full size of my database (????) drive? I am not super familiar with docker, so I don't know if I am onto something or not?
/tmp/cache | 52.13 GiB | 654.96 GiB
My docker-compose has /tmp/cache configured to 2.5GB of memory I believe:
I'm not sure if its actually mounting any tmpfs, I don't see it under mount either inside of the docker container:
root@ff56cc3a3c0d:/opt/frigate# mount
overlay on / type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/LVMYR4Z4VTZYUOA4X6KFXVJIPC:/var/lib/docker/overlay2/l/THJDIXRBQIVGDHXOFYBVE5UBT5:/var/lib/docker/overlay2/l/H7T2WJMK4NUFWQOOGA5VDRKOR4:/var/lib/docker/overlay2/l/HAXVSU6FNUVYYNKAWQYO6WHNGY:/var/lib/docker/overlay2/l/TZFLCTHSVGMD5VUKQPDZSKJV4C:/var/lib/docker/overlay2/l/ZWXTBJE535RGOXUUZ4P6GAKUMN:/var/lib/docker/overlay2/l/ADIRBFCCWJL3RVLUQGLGNPGUM5:/var/lib/docker/overlay2/l/N2RXCEC77GFR2G7XH43LR64SXA,upperdir=/var/lib/docker/overlay2/98304f27ab12a452e7edc0c231c9f69e0996d7524707be953d1e84e9f10b964f/diff,workdir=/var/lib/docker/overlay2/98304f27ab12a452e7edc0c231c9f69e0996d7524707be953d1e84e9f10b964f/work)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev type tmpfs (rw,nosuid,size=65536k,mode=755,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=1557504k,inode64)
/dev/mapper/ubuntu--vg-ubuntu--lv on /config type ext4 (rw,relatime)
ubntvar on /db type zfs (rw,xattr,noacl)
/dev/mapper/ubuntu--vg-ubuntu--lv on /usr/share/zoneinfo/Etc/UTC type ext4 (ro,relatime)
/opt/frigate/frigate on /media/frigate type zfs (rw,xattr,noacl)
ubntvar on /etc/resolv.conf type zfs (rw,xattr,noacl)
ubntvar on /etc/hostname type zfs (rw,xattr,noacl)
ubntvar on /etc/hosts type zfs (rw,xattr,noacl)
tmpfs on /proc/driver/nvidia type tmpfs (rw,nosuid,nodev,noexec,relatime,mode=555,inode64)
/dev/mapper/ubuntu--vg-ubuntu--lv on /usr/bin/nvidia-smi type ext4 (ro,nosuid,nodev,relatime)
/dev/mapper/ubuntu--vg-ubuntu--lv on /usr/bin/nvidia-debugdump type ext4 (ro,nosuid,nodev,relatime)
/dev/mapper/ubuntu--vg-ubuntu--lv on /usr/bin/nvidia-persistenced type ext4 (ro,nosuid,nodev,relatime)
/dev/mapper/ubuntu--vg-ubuntu--lv on /usr/bin/nvidia-cuda-mps-control type ext4 (ro,nosuid,nodev,relatime)
/dev/mapper/ubuntu--vg-ubuntu--lv on /usr/bin/nvidia-cuda-mps-server type ext4 (ro,nosuid,nodev,relatime)
/dev/mapper/ubuntu--vg-ubuntu--lv on /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.525.125.06 type ext4 (ro,nosuid,nodev,relatime)
/dev/mapper/ubuntu--vg-ubuntu--lv on /usr/lib/x86_64-linux-gnu/libnvidia-cfg.so.525.125.06 type ext4 (ro,nosuid,nodev,relatime)
/dev/mapper/ubuntu--vg-ubuntu--lv on /usr/lib/x86_64-linux-gnu/libcuda.so.525.125.06 type ext4 (ro,nosuid,nodev,relatime)
/dev/mapper/ubuntu--vg-ubuntu--lv on /usr/lib/x86_64-linux-gnu/libcudadebugger.so.525.125.06 type ext4 (ro,nosuid,nodev,relatime)
/dev/mapper/ubuntu--vg-ubuntu--lv on /usr/lib/x86_64-linux-gnu/libnvidia-opencl.so.525.125.06 type ext4 (ro,nosuid,nodev,relatime)
/dev/mapper/ubuntu--vg-ubuntu--lv on /usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.525.125.06 type ext4 (ro,nosuid,nodev,relatime)
/dev/mapper/ubuntu--vg-ubuntu--lv on /usr/lib/x86_64-linux-gnu/libnvidia-allocator.so.525.125.06 type ext4 (ro,nosuid,nodev,relatime)
/dev/mapper/ubuntu--vg-ubuntu--lv on /usr/lib/x86_64-linux-gnu/libnvidia-compiler.so.525.125.06 type ext4 (ro,nosuid,nodev,relatime)
/dev/mapper/ubuntu--vg-ubuntu--lv on /usr/lib/x86_64-linux-gnu/libnvidia-nvvm.so.525.125.06 type ext4 (ro,nosuid,nodev,relatime)
/dev/mapper/ubuntu--vg-ubuntu--lv on /usr/lib/x86_64-linux-gnu/libnvidia-encode.so.525.125.06 type ext4 (ro,nosuid,nodev,relatime)
/dev/mapper/ubuntu--vg-ubuntu--lv on /usr/lib/x86_64-linux-gnu/libnvidia-opticalflow.so.525.125.06 type ext4 (ro,nosuid,nodev,relatime)
/dev/mapper/ubuntu--vg-ubuntu--lv on /usr/lib/x86_64-linux-gnu/libnvcuvid.so.525.125.06 type ext4 (ro,nosuid,nodev,relatime)
/dev/mapper/ubuntu--vg-ubuntu--lv on /lib/firmware/nvidia/525.125.06/gsp_ad10x.bin type ext4 (ro,nosuid,nodev,relatime)
/dev/mapper/ubuntu--vg-ubuntu--lv on /lib/firmware/nvidia/525.125.06/gsp_tu10x.bin type ext4 (ro,nosuid,nodev,relatime)
tmpfs on /run/nvidia-persistenced/socket type tmpfs (rw,nosuid,nodev,noexec,relatime,size=6537212k,mode=755,inode64)
udev on /dev/nvidiactl type devtmpfs (ro,nosuid,noexec,relatime,size=32569268k,nr_inodes=8142317,mode=755,inode64)
udev on /dev/nvidia-uvm type devtmpfs (ro,nosuid,noexec,relatime,size=32569268k,nr_inodes=8142317,mode=755,inode64)
udev on /dev/nvidia-uvm-tools type devtmpfs (ro,nosuid,noexec,relatime,size=32569268k,nr_inodes=8142317,mode=755,inode64)
udev on /dev/nvidia0 type devtmpfs (ro,nosuid,noexec,relatime,size=32569268k,nr_inodes=8142317,mode=755,inode64)
proc on /proc/driver/nvidia/gpus/0000:3b:00.0 type proc (ro,nosuid,nodev,noexec,relatime)
root@ff56cc3a3c0d:/opt/frigate# ^C
root@ff56cc3a3c0d:/opt/frigate#
No sign of it on mount in the host either.
Well, what is actually inside of /tmp/cache?
Seems like there's definitely other issues going on with the setup then
the only thing in there is short clip files, max size 15MB, its just getting backed up. I got the DB is locked error after 1.5 hours this morning
the only thing in there is short clip files, max size 15MB, its just getting backed up
but how many? if frigate's /tmp/cache is 50+ GB then there must be a lot otherwise it seems that is unrelated to frigate
Actually it seems that the Frigate's storage UI is wrong on the size of /tmp/cache. Its still reading the same 52.13 GB but the actual size of clips is 85MB. It appears to be using the size of the /var/lib/docker merged directory for the container, which seems to be where /tmp/cache is stored. From what I can tell, it doesn't appear to be stored in memory, although my docker-compose indicates it should, correct?
No non-clip files in there, and It does get dumped on each restart. I also got multiple more DB is locked errors overnight
root@ff56cc3a3c0d:/tmp/cache# dd if=/dev/zero of=test1.img bs=2G count=1 oflag=dsync 0+1 records in 0+1 records out 2147479552 bytes (2.1 GB, 2.0 GiB) copied, 14.0936 s, 152 MB/s
I've not seen that before, it definitely should be stored in memory and it is for me. I don't see anything wrong with the docker compose definition
I changed the size to 1GB and now it appears to be a tmpfs stored in memory... back to 2.5GB and stored on disk... so not sure how to explain that. I'm leaving it at 1GB to see what happens now... tons of motion activity at the moment and it got to 300MB and dropped a round of frames.
2023-07-20 09:40:41.657642645 [2023-07-20 09:40:41] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for gp636_doorbell. Keeping the 5 most recent segments out of 7 and discarding the rest... 2023-07-20 09:40:41.657752769 [2023-07-20 09:40:41] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for gp_bldg_03_side. Keeping the 5 most recent segments out of 7 and discarding the rest... 2023-07-20 09:40:41.659271033 [2023-07-20 09:40:41] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for gp_bldg_08_front. Keeping the 5 most recent segments out of 7 and discarding the rest... 2023-07-20 09:40:41.660399177 [2023-07-20 09:40:41] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for gp_bldg_08_rear. Keeping the 5 most recent segments out of 7 and discarding the rest... 2023-07-20 09:40:41.661322977 [2023-07-20 09:40:41] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for gp_clubhouse_back_door. Keeping the 5 most recent segments out of 7 and discarding the rest... 2023-07-20 09:40:41.662558977 [2023-07-20 09:40:41] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for gp_clubhouse_banquet. Keeping the 5 most recent segments out of 7 and discarding the rest... 2023-07-20 09:40:41.664081407 [2023-07-20 09:40:41] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for gp_clubhouse_firepit. Keeping the 5 most recent segments out of 7 and discarding the rest... 2023-07-20 09:40:41.666221622 [2023-07-20 09:40:41] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for gp_clubhouse_front_door. Keeping the 5 most recent segments out of 7 and discarding the rest... 2023-07-20 09:40:41.667167385 [2023-07-20 09:40:41] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for gp_clubhouse_gym_1. Keeping the 5 most recent segments out of 7 and discarding the rest... 2023-07-20 09:40:41.669037147 [2023-07-20 09:40:41] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for gp_clubhouse_gym_2. Keeping the 5 most recent segments out of 7 and discarding the rest... 2023-07-20 09:40:41.670609249 [2023-07-20 09:40:41] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for gp_clubhouse_parking. Keeping the 5 most recent segments out of 7 and discarding the rest... 2023-07-20 09:40:41.671477552 [2023-07-20 09:40:41] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for gp_clubhouse_pool. Keeping the 5 most recent segments out of 7 and discarding the rest... 2023-07-20 09:40:41.672420197 [2023-07-20 09:40:41] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for gp_clubhouse_pool_porch. Keeping the 5 most recent segments out of 6 and discarding the rest... 2023-07-20 09:40:41.672907205 [2023-07-20 09:40:41] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for gp_entrance_archer. Keeping the 5 most recent segments out of 6 and discarding the rest... 2023-07-20 09:40:41.674062259 [2023-07-20 09:40:41] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for warehouse_exterior_nw. Keeping the 5 most recent segments out of 6 and discarding the rest... 2023-07-20 09:40:41.674429415 [2023-07-20 09:40:41] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for warehouse_exterior_se. Keeping the 5 most recent segments out of 7 and discarding the rest... 2023-07-20 09:40:41.675965486 [2023-07-20 09:40:41] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for warehouse_exterior_sw. Keeping the 5 most recent segments out of 6 and discarding the rest... 2023-07-20 09:40:41.676171432 [2023-07-20 09:40:41] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for warehouse_interior_front. Keeping the 5 most recent segments out of 6 and discarding the rest... 2023-07-20 09:40:41.677865370 [2023-07-20 09:40:41] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for warehouse_interior_overview. Keeping the 5 most recent segments out of 6 and discarding the rest... 2023-07-20 09:40:41.678571948 [2023-07-20 09:40:41] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for warehouse_workbench. Keeping the 5 most recent segments out of 7 and discarding the rest...
Then started acting normally for the last 10 minutes... will report back on DB errors with /tmp/cache working and at a smaller size
2023-07-20 10:09:48.028974729 Exception in thread timeline_processor:
2023-07-20 10:09:48.029013745 Traceback (most recent call last):
2023-07-20 10:09:48.029017379 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql
2023-07-20 10:09:48.030638148 cursor.execute(sql, params or ())
2023-07-20 10:09:48.030677871 sqlite3.OperationalError: database is locked
2023-07-20 10:09:48.030684321
2023-07-20 10:09:48.030691964 During handling of the above exception, another exception occurred:
2023-07-20 10:09:48.030697874
2023-07-20 10:09:48.030704264 Traceback (most recent call last):
2023-07-20 10:09:48.030740670 File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner
2023-07-20 10:09:48.031225014 self.run()
2023-07-20 10:09:48.031230337 File "/opt/frigate/frigate/timeline.py", line 46, in run
2023-07-20 10:09:48.031365907 self.handle_object_detection(
2023-07-20 10:09:48.031371721 File "/opt/frigate/frigate/timeline.py", line 92, in handle_object_detection
2023-07-20 10:09:48.031474146 Timeline.insert(timeline_entry).execute()
2023-07-20 10:09:48.031497626 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 1966, in inner
2023-07-20 10:09:48.032126401 return method(self, database, *args, *kwargs)
2023-07-20 10:09:48.032132125 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2037, in execute
2023-07-20 10:09:48.034483710 return self._execute(database)
2023-07-20 10:09:48.034494078 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2842, in _execute
2023-07-20 10:09:48.035138616 return super(Insert, self)._execute(database)
2023-07-20 10:09:48.035145561 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2556, in _execute
2023-07-20 10:09:48.035717367 return self.handle_result(database, cursor)
2023-07-20 10:09:48.035723638 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2851, in handle_result
2023-07-20 10:09:48.036345082 return database.last_insert_id(cursor, self._query_type)
2023-07-20 10:09:48.036351020 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3749, in last_insert_id
2023-07-20 10:09:48.037243452 return cursor.lastrowid
2023-07-20 10:09:48.037249780 File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 87, in lastrowid
2023-07-20 10:09:48.037535868 self._wait()
2023-07-20 10:09:48.037542145 File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 62, in _wait
2023-07-20 10:09:48.037614664 raise self._exc
2023-07-20 10:09:48.037620536 File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 177, in execute
2023-07-20 10:09:48.037702816 cursor = self.database._execute(obj.sql, obj.params)
2023-07-20 10:09:48.037708534 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql
2023-07-20 10:09:48.038485549 cursor.execute(sql, params or ())
2023-07-20 10:09:48.038491485 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3014, in exit
2023-07-20 10:09:48.039161180 reraise(new_type, new_type(exc_value, exc_args), traceback)
2023-07-20 10:09:48.039166886 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 192, in reraise
2023-07-20 10:09:48.039252382 raise value.with_traceback(tb)
2023-07-20 10:09:48.039258314 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql
2023-07-20 10:09:48.039965084 cursor.execute(sql, params or ())
2023-07-20 10:09:48.039989002 peewee.OperationalError: database is locked
:-(
@YBonline you can try the latest commit on that PR, I have made additional improvements to the recordings management that should reduce DB activity
No luck with the second last commit, trying now with the one with .execute()
2023-07-20 13:36:22.165349158 [2023-07-20 13:36:20] frigate.app INFO : Capture process started for gp_bldg_03_side: 1516
2023-07-20 13:51:18.799544924 [2023-07-20 13:51:18] frigate.object_processing WARNING : Unable to create jpg because frame 1689875475.620079 is not in the cache
2023-07-20 13:52:21.452289462 [2023-07-20 13:52:21] frigate.object_processing WARNING : Unable to create jpg because frame 1689875540.741912 is not in the cache
2023-07-20 13:52:21.453911583 [2023-07-20 13:52:21] frigate.object_processing WARNING : Unable to create jpg because frame 1689875540.358542 is not in the cache
2023-07-20 13:52:25.947611268 [2023-07-20 13:52:25] frigate.object_processing WARNING : Unable to create jpg because frame 1689875540.358542 is not in the cache
2023-07-20 13:52:25.947702198 [2023-07-20 13:52:25] frigate.object_processing WARNING : Unable to save snapshot for 1689875486.544059-wv22p2.
2023-07-20 13:52:25.947759549 [2023-07-20 13:52:25] frigate.object_processing WARNING : Unable to create clean png because frame 1689875540.358542 is not in the cache
2023-07-20 13:52:25.947811091 [2023-07-20 13:52:25] frigate.object_processing WARNING : Unable to save clean snapshot for 1689875486.544059-wv22p2.
2023-07-20 13:52:25.950705363 [2023-07-20 13:52:25] frigate.object_processing WARNING : Unable to create jpg because frame 1689875540.358542 is not in the cache
2023-07-20 13:53:15.905574432 Exception in thread timeline_processor:
2023-07-20 13:53:15.905985036 Traceback (most recent call last):
2023-07-20 13:53:15.906006217 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql
2023-07-20 13:53:15.908779298 cursor.execute(sql, params or ())
2023-07-20 13:53:15.908826641 sqlite3.OperationalError: database is locked
2023-07-20 13:53:15.908833678
2023-07-20 13:53:15.908842433 During handling of the above exception, another exception occurred:
2023-07-20 13:53:15.908854992
2023-07-20 13:53:15.908860004 Traceback (most recent call last):
2023-07-20 13:53:15.908881896 File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner
2023-07-20 13:53:15.911555984 self.run()
2023-07-20 13:53:15.911591193 File "/opt/frigate/frigate/timeline.py", line 46, in run
2023-07-20 13:53:15.912064219 self.handle_object_detection(
2023-07-20 13:53:15.912086784 File "/opt/frigate/frigate/timeline.py", line 81, in handle_object_detection
2023-07-20 13:53:15.912238196 Timeline.insert(timeline_entry).execute()
2023-07-20 13:53:15.912256738 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 1966, in inner
2023-07-20 13:53:15.913653870 return method(self, database, *args, *kwargs)
2023-07-20 13:53:15.913690581 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2037, in execute
2023-07-20 13:53:15.917654278 return self._execute(database)
2023-07-20 13:53:15.917661539 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2842, in _execute
2023-07-20 13:53:15.918568999 return super(Insert, self)._execute(database)
2023-07-20 13:53:15.918575514 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2556, in _execute
2023-07-20 13:53:15.919376106 return self.handle_result(database, cursor)
2023-07-20 13:53:15.919398521 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2851, in handle_result
2023-07-20 13:53:15.920233103 return database.last_insert_id(cursor, self._query_type)
2023-07-20 13:53:15.920240267 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3749, in last_insert_id
2023-07-20 13:53:15.921319357 return cursor.lastrowid
2023-07-20 13:53:15.921329068 File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 87, in lastrowid
2023-07-20 13:53:15.921754032 self._wait()
2023-07-20 13:53:15.921760632 File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 62, in _wait
2023-07-20 13:53:15.921858956 raise self._exc
2023-07-20 13:53:15.921865309 File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 177, in execute
2023-07-20 13:53:15.922238758 cursor = self.database._execute(obj.sql, obj.params)
2023-07-20 13:53:15.922259582 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql
2023-07-20 13:53:15.925387773 cursor.execute(sql, params or ())
2023-07-20 13:53:15.925411232 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3014, in exit
2023-07-20 13:53:15.926899574 reraise(new_type, new_type(exc_value, exc_args), traceback)
2023-07-20 13:53:15.926918710 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 192, in reraise
2023-07-20 13:53:15.927116317 raise value.with_traceback(tb)
2023-07-20 13:53:15.927134392 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql
2023-07-20 13:53:15.928703866 cursor.execute(sql, params or ())
2023-07-20 13:53:15.928727669 peewee.OperationalError: database is locked
The next commit won't help because without execute()
it wasn't even entering the item into the DB. I think at this point we need debug logs to understand what is going on because it makes absolutely no sense how the DB write is taking longer than 250 seconds
logger:
logs:
peewee: debug
should enable debug logs so we can get better info
nevermind, doesn't seem like that shows the timing of the queries
actually running that would still potentially be helpful because we can see how long there is between queries
There seems to be delete operations in progress when it crashes which seems to be in line with the other issue I've had of when Frigate is automatically deleting old events, it uses a ton of CPU time.
That is a lot of single timeline deletes, I think I found a better way to clean these up with bulk deletes that should hopefully help, just updated the PR
Alright, I'm running that but I'm getting so much stuff in the log file that I can't seem to go through that log now... is it logged to separate files somewhere with the frigate log separated from the debug info for peewee?
The peewee debug info is part of the frigate logs, there is no difference.
Alright, well I will continue to monitor, but so far 2.5 hours and no database is locked, and I'm more hopeful this time as Frigate seems more performant as well, Birdseye is updating quicker, etc. Probably won't know for sure until the morning if the error is fully gone but this definitely helped a lot so far.
I think we are good!!!! 13 hours, no database locks, no signs of memory usage spikes, or any other issue for me now. I did shorten up my storage times for motion events since if it was auto expiring based off of free space, the problem and resource usage was far worse, so I'm going to go ahead and re-enable my original setting to see if that causes any issue, but it is looking like #7227 does fix the core issue for me for sure, thank you again! My load average is also now staying around 10 (with little motion) and 15 (with tons of motion) now, this is the best I've ever had running Frigate, by a large margin (nearly 50% reduction)
If the original settings cause issues please create a new issue
Alright, just wanted to follow up and report that this does seem to be fully fixed. In fact, the could not keep up with the cache errors, and most of my other camera errors seemed to have been related to this as well. Thanks again for all of the help!
I haven't seen it since I started running the fixes either
Describe the problem you are having
I've been having a "Database is locked" error randomly on the latest dev build. I haven't been able to figure out a cause yet, but it happens much more frequently when my storage drive for the videos is full and Frigate is deleting them continuously to free up space for the next recording. It does still happen without it.
After the database is locked error occurs, the Home Assistant's object count becomes permanently too high. Even restarting Frigate does not cause it to return to normal (although restarting the plugin in HA without restarting Frigate does fix it) Frigate does seem to start working normally again after just dropping some recordings while the database is locked.
Version
0.13.0-8E584CF
Frigate config file
Relevant log output
FFprobe output from your camera
Frigate stats
Operating system
HassOS
Install method
HassOS Addon
Coral version
USB
Network connection
Wired
Camera make and model
NA
Any other information that may be helpful
No response