Open jin-eld opened 1 week ago
I think I might know what I did wrong, so it seems in the dev version the configuration has changed and the folder
option is obsolete, one has to use a storage
section.
In docker compose I now changed my tmpfs
to:
tmpfs:
- /ramdisk:rw,size=2g,mode=1777
And in Viseron I added:
storage:
recorder:
tiers:
- path: /ramdisk
events:
max_age:
days: 1
max_size:
gb: 1.5
move_on_shutdown: true
- path: /recordings
events:
max_age:
days: 7
max_size:
gb: 80
snapshots:
tiers:
- path: /ramdisk
move_on_shutdown: true
max_age:
days: 1
max_size:
gb: 0.5
Will need to make a few tests where something gets detected and recorded to see if the recordings finally end up where they should (for example if they will get moved after a shutdown).
Retested with the new storage settings, something is still off, the log shows:
viseron | [2024-11-18 19:49:54] [INFO ] [viseron.components.nvr.nvr.camera_1] - Stopping recording in: 1
viseron | [2024-11-18 19:49:55] [DEBUG ] [viseron.components.edgetpu.object_detector.camera_1] - Objects: []
viseron | [2024-11-18 19:49:55] [DEBUG ] [viseron.domains.camera.fragmenter.camera_1] - Checking for new segments to fragment in /tmp/viseron/ramdisk/segments/camera_1
viseron | [2024-11-18 19:49:55] [DEBUG ] [viseron.components.edgetpu.object_d[2024-11-18 19:49:56] [DEBUG ] [viseron.components.edgetpu.object_detector.camera_1] - Objects: [], message repeated 2 times
viseron | [2024-11-18 19:49:56] [DEBUG ] [viseron.domains.camera.fragmenter.camera_1] - Checking for new segments to fragment in /tmp/viseron/ramdisk/segments/camera_1
viseron | [2024-11-18 19:49:56] [DEBUG ] [viseron.domains.camera.fragmenter.camera_1] - Processing 1731955790.m4s
viseron | [2024-11-18 19:49:56] [DEBUG ] [viseron.components.storage.tier_handler.camera_1.tier_0] - File created: /ramdisk/segments/camera_1/1731955790.m4s
viseron | [2024-11-18 19:49:57] [DEBUG ] [viseron.components.edgetpu.object_detector.camera_1] - Objects: []
viseron | [2024-11-18 19:49:57] [DEBUG ] [viseron.domains.camera.fragmenter.camera_1] - Checking for new segments to fragment in /tmp/viseron/ramdisk/segments/camera_1
viseron | [2024-11-18 19:49:57] [DEBUG ] [viseron.components.storage.tier_handler.camera_1.tier_0] - File modified (delayed event): /ramdisk/segments/camera_1/1731955790.m4s
viseron | [2024-11-18 19:49:57] [DEBUG ] [viseron.components.edgetpu.object_detector.camera_1] - Objects: []
viseron | [2024-11-18 19:49:57] [INFO ] [viseron.components.nvr.nvr.camera_1] - Stopping recording in: 0
viseron | [2024-11-18 19:49:57] [INFO ] [viseron.components.ffmpeg.recorder.camera_1] - Stopping recorder
viseron | [2024-11-18 19:49:57] [DEBUG ] [viseron.states] - Setting state of binary_sensor.camera_1_recorder to state: off, attributes {'name': 'east Recorder', 'domain': 'binary_sensor', 'id': 96, 'start_time': datetime.datetime(2024, 11, 18, 18, 49, 28, 374093, tzinfo=datetime.timezone.utc), 'start_timestamp': 1731955768.374093, 'end_time': datetime.datetime(2024, 11, 18, 18, 49, 57, 627797, tzinfo=datetime.timezone.utc), 'end_timestamp': 1731955797.627797, 'date': '2024-11-18', 'path': '/ramdisk/recordings/camera_1/2024-11-18/east_2024-11-18-18:49:28.mp4', 'filename': 'east_2024-11-18-18:49:28.mp4', 'thumbnail_path': '/ramdisk/thumbnails/camera_1/96.jpg', 'objects': [<viseron.domains.object_detector.detected_object.DetectedObject object at 0x7f39891ec460>]}
viseron | [2024-11-18 19:49:57] [DEBUG ] [viseron.states] - Setting state of sensor.camera_1_operation_state to state: scanning_for_objects, attributes {'name': 'east Operation State', 'domain': 'sensor'}
Note the log entry: 'path': '/ramdisk/recordings/camera_1/2024-11-18/east_2024-11-18-18:49:28.mp4'
Now looking inside of the container (docker exec -it container_id /bin/bash
) - nothing there:
root@d461e1aa7951:/ramdisk/recordings# find .
.
./camera_1
./camera_1/2024-11-18
Now, gracefully stopping the container, hoping that move_on_shutdown: true
would move it to the /recordings
mount point which maps to the host emmc:
viseron | [2024-11-18 20:17:43] [DEBUG ] [viseron.domains.camera.fragmenter.camera_1] - Checking for new segments to fragment in /tmp/viseron/ramdisk/segments/camera_1
[2024-11-18 20:17:44] [DEBUG ] [viseron.components.edgetpu.object_detector.camera_1] - Objects: [], message repeated 2 times
viseron | [2024-11-18 20:17:44] [DEBUG ] [viseron.domains.camera.fragmenter.camera_1] - Checking for new segments to fragment in /tmp/viseron/ramdisk/segments/camera_1
viseron | [2024-11-18 20:17:44] [DEBUG ] [viseron.domains.camera.fragmenter.camera_1] - Processing 1731957460.m4s
viseron | [2024-11-18 20:17:44] [DEBUG ] [viseron.components.storage.tier_handler.camera_1.tier_0] - File created: /ramdisk/segments/camera_1/1731957460.m4s
viseron | [2024-11-18 20:17:44] [DEBUG ] [viseron.components.edgetpu.object_detector.camera_1] - Objects: []
viseron | [2024-11-18 20:17:45] [DEBUG ] [viseron.domains.camera.fragmenter.camera_1] - Checking for new segments to fragment in /tmp/viseron/ramdisk/segments/camera_1
viseron | [2024-11-18 20:17:45] [DEBUG ] [viseron.components.storage.tier_handler.camera_1.tier_0] - File modified (delayed event): /ramdisk/segments/camera_1/1731957460.m4s
viseron | [2024-11-18 20:17:45] [DEBUG ] [viseron.components.edgetpu.object_detector.camera_1] - Objects: []
^CGracefully stopping... (press Ctrl+C again to force)
[+] Stopping 1/1
✔ Container viseron Stopped
Nothing in the logs indicates it was going to move the recording anywhere?
The recordings directory on the host shows the correct subdirectory structure, but has not .mp4
files, i.e. the file east_2024-11-18-18:49:28.mp4
is not there.
Is this a bug or what am I missing?
Sorry for the confusion! The dev
branch still has some kinks i need to workout.
In v3, the cameras recordings are stored as short m4s
segments to enable 24/7 recordings.
They are stored in the /segments
folder by default.
If you check the folder /ramdisk/segments
you should be seeing files there.
If you are in need of having the events as mp4 files, just like v2 did, you can set the create_event_clip: true
config option under the recorder
section for each camera.
That will create the files you are looking for in the recordings folder.
The logline and the folder creation is missleading since the file is never created, will fix that right away
Thank you for getting back to me! Yes indeed I saw the segments on ramdisk, but assumed they were something internal, especially seeing in the logs how they get regularly deleted.
I was indeed looking for one complete clip for the whole event, expecting it to end up in /recordings
at some point, allowing local high res review and evidence collecting if need be.
I now found it in the sources and have a question regarding this remark:
WARNING: Will store both the fragments AND the MP4 file, using more storage space.
Does that mean, that after the event fragments get concatenated into one event mp4, they will not get cleaned up? If so, why? My naive assumption would be, that it is not necessary to keep the same data twice or is there a technical limitation?
Thanks again for helping out, I'll test it shortly. My main issue at this point is the terrible detection quality of the Coral TPU regardless of the model used, but that is not a Viseron problem :)
Did a short test with the suggested setting and the complete .mp4
videos of individual events got created, however when I did a graceful shutdown, the .mp4
files got lost.
My assumption was, that move_on_shutdown: true
would move the videos to the next tier:
storage:
recorder:
tiers:
- path: /ramdisk
events:
max_age:
days: 1
max_size:
gb: 1.5
move_on_shutdown: true
- path: /recordings
events:
max_age:
days: 7
max_size:
gb: 80
Does this work differently or is my configuration not correct?
Thank you for getting back to me! Yes indeed I saw the segments on ramdisk, but assumed they were something internal, especially seeing in the logs how they get regularly deleted.
I was indeed looking for one complete clip for the whole event, expecting it to end up in
/recordings
at some point, allowing local high res review and evidence collecting if need be.I now found it in the sources and have a question regarding this remark:
WARNING: Will store both the fragments AND the MP4 file, using more storage space.
Does that mean, that after the event fragments get concatenated into one event mp4, they will not get cleaned up? If so, why? My naive assumption would be, that it is not necessary to keep the same data twice or is there a technical limitation?
Thanks again for helping out, I'll test it shortly. My main issue at this point is the terrible detection quality of the Coral TPU regardless of the model used, but that is not a Viseron problem :)
Correct, the fragments are not cleaned up since they are what Viseron uses mainly.
The Viseron UI does not utilize the MP4 recordings at all, it only displays HLS playlists using the m4s files. The event clip is there just to allow easy browsing outside of the Viseron UI. I could possibly allow the user to choose wether to use the MP4 files or the m4s files for playback in the browser, but i feel that would get very complex and its not something i have time to look into right now sadly.
Did a short test with the suggested setting and the complete
.mp4
videos of individual events got created, however when I did a graceful shutdown, the.mp4
files got lost.My assumption was, that
move_on_shutdown: true
would move the videos to the next tier:storage: recorder: tiers: - path: /ramdisk events: max_age: days: 1 max_size: gb: 1.5 move_on_shutdown: true - path: /recordings events: max_age: days: 7 max_size: gb: 80
Does this work differently or is my configuration not correct?
That should be working, let me investigate.
Just a note on your configuration, the second tier pointing to /recordings
will result in subfolders like /recordings/segments
, /recordings/recordings
etc
Correct, the fragments are not cleaned up since they are what Viseron uses mainly.
The Viseron UI does not utilize the MP4 recordings at all, it only displays HLS playlists using the m4s files. The event clip is there just to allow easy browsing outside of the Viseron UI. I could possibly allow the user to choose wether to use the MP4 files or the m4s files for playback in the browser, but i feel that would get very complex and its not something i have time to look into right now sadly.
OK, floating some ideas: perhaps a simpler workaround would be to have an option which forces the creation of .mp4 files on the next tier? As I understand the segments are being rotated frequently, so having them on a ramdisk makes sense in order to be kinder to the eMMC. The use case of the mp4 files is based on the need to store them for later review or evidence outside of Viseron, so it does not sound wrong creating them directly in the long-term i.e. tier2 storage? Goal is to offload the ramdisk spacewise, to ensure that it can continue handling new events without running out of space due to same data that will be present twice.
Just a note on your configuration, the second tier pointing to /recordings will result in subfolders like /recordings/segments, /recordings/recordings etc
Yes, thanks for the hint, I noticed 😅
I have more logs and a somewhat strange behaviour, I should probably update my dev image to make sure I am running the latest code.
So, after making some changes to the configuration (enabled more detection labels) I hit the restart button in the UI, which resulted in the following backtrace:
viseron | [2024-11-19 18:34:43] [INFO ] [viseron.core] - Initiating shutdown
viseron | [2024-11-19 18:34:43] [INFO ] [viseron.components.nvr.nvr.camera_1] - Stopping NVR thread
viseron | [2024-11-19 18:34:47] [ERROR ] [root] - Uncaught thread exception in thread Thread-105 (_shutdown)
viseron | Traceback (most recent call last):
viseron | File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/result.py", line 199, in _key_not_found
viseron | self._key_fallback(key, None)
viseron | File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/result.py", line 138, in _key_fallback
viseron | raise KeyError(key) from err
viseron | KeyError: 'path'
viseron |
viseron | The above exception was the direct cause of the following exception:
viseron |
viseron | Traceback (most recent call last):
viseron | File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
viseron | self.run()
viseron | File "/usr/lib/python3.10/threading.py", line 953, in run
viseron | self._target(*self._args, **self._kwargs)
viseron | File "/src/viseron/components/storage/tier_handler.py", line 346, in _shutdown
viseron | force_move_files(
viseron | File "/src/viseron/components/storage/tier_handler.py", line 1030, in force_move_files
viseron | file.path,
viseron | File "lib/sqlalchemy/cyextension/resultproxy.pyx", line 66, in sqlalchemy.cyextension.resultproxy.BaseRow.__getattr__
viseron | File "lib/sqlalchemy/cyextension/resultproxy.pyx", line 63, in sqlalchemy.cyextension.resultproxy.BaseRow._get_by_key_impl
viseron | File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/result.py", line 201, in _key_not_found
viseron | raise AttributeError(ke.args[0]) from ke
viseron | AttributeError: path
viseron | [2024-11-19 18:34:47] [ERROR ] [root] - Uncaught thread exception in thread Thread-104 (_shutdown)
viseron | Traceback (most recent call last):
viseron | File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/result.py", line 199, in _key_not_found
viseron | self._key_fallback(key, None)
viseron | File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/result.py", line 138, in _key_fallback
viseron | raise KeyError(key) from err
viseron | KeyError: 'path'
viseron |
viseron | The above exception was the direct cause of the following exception:
viseron |
viseron | Traceback (most recent call last):
viseron | File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
viseron | self.run()
viseron | File "/usr/lib/python3.10/threading.py", line 953, in run
viseron | self._target(*self._args, **self._kwargs)
viseron | File "/src/viseron/components/storage/tier_handler.py", line 346, in _shutdown
viseron | force_move_files(
viseron | File "/src/viseron/components/storage/tier_handler.py", line 1030, in force_move_files
viseron | file.path,
viseron | File "lib/sqlalchemy/cyextension/resultproxy.pyx", line 66, in sqlalchemy.cyextension.resultproxy.BaseRow.__getattr__
viseron | File "lib/sqlalchemy/cyextension/resultproxy.pyx", line 63, in sqlalchemy.cyextension.resultproxy.BaseRow._get_by_key_impl
viseron | File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/result.py", line 201, in _key_not_found
viseron | raise AttributeError(ke.args[0]) from ke
viseron | AttributeError: path
viseron | [2024-11-19 18:34:47] [ERROR ] [root] - Uncaught thread exception in thread Thread-106 (_shutdown)
viseron | Traceback (most recent call last):
viseron | File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/result.py", line 199, in _key_not_found
viseron | self._key_fallback(key, None)
viseron | File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/result.py", line 138, in _key_fallback
viseron | raise KeyError(key) from err
viseron | KeyError: 'path'
viseron |
viseron | The above exception was the direct cause of the following exception:
viseron |
viseron | Traceback (most recent call last):
viseron | File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
viseron | self.run()
viseron | File "/usr/lib/python3.10/threading.py", line 953, in run
viseron | self._target(*self._args, **self._kwargs)
viseron | File "/src/viseron/components/storage/tier_handler.py", line 346, in _shutdown
viseron | force_move_files(
viseron | File "/src/viseron/components/storage/tier_handler.py", line 1030, in force_move_files
viseron | file.path,
viseron | File "lib/sqlalchemy/cyextension/resultproxy.pyx", line 66, in sqlalchemy.cyextension.resultproxy.BaseRow.__getattr__
viseron | File "lib/sqlalchemy/cyextension/resultproxy.pyx", line 63, in sqlalchemy.cyextension.resultproxy.BaseRow._get_by_key_impl
viseron | File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/result.py", line 201, in _key_not_found
viseron | raise AttributeError(ke.args[0]) from ke
viseron | AttributeError: path
viseron | [2024-11-19 18:34:47] [ERROR ] [root] - Uncaught thread exception in thread Thread-110 (_shutdown)
viseron | Traceback (most recent call last):
viseron | File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/result.py", line 199, in _key_not_found
viseron | self._key_fallback(key, None)
viseron | File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/result.py", line 138, in _key_fallback
viseron | raise KeyError(key) from err
viseron | KeyError: 'path'
viseron |
viseron | The above exception was the direct cause of the following exception:
viseron |
viseron | Traceback (most recent call last):
viseron | File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
viseron | self.run()
viseron | File "/usr/lib/python3.10/threading.py", line 953, in run
viseron | self._target(*self._args, **self._kwargs)
viseron | File "/src/viseron/components/storage/tier_handler.py", line 346, in _shutdown
viseron | force_move_files(
viseron | File "/src/viseron/components/storage/tier_handler.py", line 1030, in force_move_files
viseron | file.path,
viseron | File "lib/sqlalchemy/cyextension/resultproxy.pyx", line 66, in sqlalchemy.cyextension.resultproxy.BaseRow.__getattr__
viseron | File "lib/sqlalchemy/cyextension/resultproxy.pyx", line 63, in sqlalchemy.cyextension.resultproxy.BaseRow._get_by_key_impl
viseron | File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/result.py", line 201, in _key_not_found
viseron | raise AttributeError(ke.args[0]) from ke
viseron | AttributeError: path
viseron | [2024-11-19 18:34:47] [ERROR ] [root] - Uncaught thread exception in thread Thread-111 (_shutdown)
viseron | Traceback (most recent call last):
viseron | File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/result.py", line 199, in _key_not_found
viseron | self._key_fallback(key, None)
viseron | File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/result.py", line 138, in _key_fallback
viseron | raise KeyError(key) from err
viseron | KeyError: 'path'
viseron |
viseron | The above exception was the direct cause of the following exception:
viseron |
viseron | Traceback (most recent call last):
viseron | File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
viseron | self.run()
viseron | File "/usr/lib/python3.10/threading.py", line 953, in run
viseron | self._target(*self._args, **self._kwargs)
viseron | File "/src/viseron/components/storage/tier_handler.py", line 346, in _shutdown
viseron | force_move_files(
viseron | File "/src/viseron/components/storage/tier_handler.py", line 1030, in force_move_files
viseron | file.path,
viseron | File "lib/sqlalchemy/cyextension/resultproxy.pyx", line 66, in sqlalchemy.cyextension.resultproxy.BaseRow.__getattr__
viseron | File "lib/sqlalchemy/cyextension/resultproxy.pyx", line 63, in sqlalchemy.cyextension.resultproxy.BaseRow._get_by_key_impl
viseron | File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/result.py", line 201, in _key_not_found
viseron | raise AttributeError(ke.args[0]) from ke
viseron | AttributeError: path
viseron | [2024-11-19 18:34:47] [ERROR ] [root] - Uncaught thread exception in thread Thread-112 (_shutdown)
viseron | Traceback (most recent call last):
viseron | File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/result.py", line 199, in _key_not_found
viseron | self._key_fallback(key, None)
viseron | File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/result.py", line 138, in _key_fallback
viseron | raise KeyError(key) from err
viseron | KeyError: 'path'
viseron |
viseron | The above exception was the direct cause of the following exception:
viseron |
viseron | Traceback (most recent call last):
viseron | File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
viseron | self.run()
viseron | File "/usr/lib/python3.10/threading.py", line 953, in run
viseron | self._target(*self._args, **self._kwargs)
viseron | File "/src/viseron/components/storage/tier_handler.py", line 346, in _shutdown
viseron | force_move_files(
viseron | File "/src/viseron/components/storage/tier_handler.py", line 1030, in force_move_files
viseron | file.path,
viseron | File "lib/sqlalchemy/cyextension/resultproxy.pyx", line 66, in sqlalchemy.cyextension.resultproxy.BaseRow.__getattr__
viseron | File "lib/sqlalchemy/cyextension/resultproxy.pyx", line 63, in sqlalchemy.cyextension.resultproxy.BaseRow._get_by_key_impl
viseron | File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/result.py", line 201, in _key_not_found
viseron | raise AttributeError(ke.args[0]) from ke
viseron | AttributeError: path
viseron | [2024-11-19 18:34:47] [ERROR ] [root] - Uncaught thread exception in thread Thread-113 (_shutdown)
viseron | Traceback (most recent call last):
viseron | File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/result.py", line 199, in _key_not_found
viseron | self._key_fallback(key, None)
viseron | File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/result.py", line 138, in _key_fallback
viseron | raise KeyError(key) from err
viseron | KeyError: 'path'
viseron |
viseron | The above exception was the direct cause of the following exception:
viseron |
viseron | Traceback (most recent call last):
viseron | File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
viseron | self.run()
viseron | File "/usr/lib/python3.10/threading.py", line 953, in run
viseron | self._target(*self._args, **self._kwargs)
viseron | File "/src/viseron/components/storage/tier_handler.py", line 346, in _shutdown
viseron | force_move_files(
viseron | File "/src/viseron/components/storage/tier_handler.py", line 1030, in force_move_files
viseron | file.path,
viseron | File "lib/sqlalchemy/cyextension/resultproxy.pyx", line 66, in sqlalchemy.cyextension.resultproxy.BaseRow.__getattr__
viseron | File "lib/sqlalchemy/cyextension/resultproxy.pyx", line 63, in sqlalchemy.cyextension.resultproxy.BaseRow._get_by_key_impl
viseron | File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/result.py", line 201, in _key_not_found
viseron | raise AttributeError(ke.args[0]) from ke
viseron | AttributeError: path
viseron | [2024-11-19 18:34:49] [INFO ] [viseron.core] - Shutdown complete
viseron | [viseron-finish] Viseron exit code 100
viseron | /var/run/postgresql:5432 - accepting connections
viseron | PostgreSQL Server has started!
viseron | [2024-11-19 18:34:50] [INFO ] [viseron.core] - -------------------------------------------
viseron | [2024-11-19 18:34:50] [INFO ] [viseron.core] - Initializing Viseron dev
the startup after that looked fine, so I assume all is good, but when I returned to check the events I saw that nothing new has been recorded, although I was outside to test detection.
I checked the logs and saw the following errors:
viseron | [2024-11-19 19:49:27] [INFO ] [viseron.components.ffmpeg.recorder.camera_1] - Starting recorder
viseron | [2024-11-19 19:49:27] [ERROR ] [viseron.components.storage.tier_handler.camera_1.tier_0] - Failed to delete file /ramdisk/snapshots/object_detector/camera_1/2024-11-18-17-02-28-917682d3-e657-4e84-b422-60645729b722.jpg: [Errno 2] No such file or directory: '/ramdisk/snapshots/object_detector/camera_1/2024-11-18-17-02-28-917682d3-e657-4e84-b422-60645729b722.jpg'
viseron | [2024-11-19 19:49:27] [ERROR ] [viseron.components.storage.tier_handler.camera_1.tier_0] - Failed to delete file /ramdisk/snapshots/object_detector/camera_1/2024-11-18-17-03-28-43abae53-3d51-41e1-83e2-bc228c7716f2.jpg: [Errno 2] No such file or directory: '/ramdisk/snapshots/object_detector/camera_1/2024-11-18-17-03-28-43abae53-3d51-41e1-83e2-bc228c7716f2.jpg'
viseron | [2024-11-19 19:49:27] [ERROR ] [viseron.components.storage.tier_handler.camera_1.tier_0] - Failed to delete file /ramdisk/snapshots/object_detector/camera_1/2024-11-18-18-48-55-d335c1b5-cc77-4e6c-afc8-d6499b12010e.jpg: [Errno 2] No such file or directory: '/ramdisk/snapshots/object_detector/camera_1/2024-11-18-18-48-55-d335c1b5-cc77-4e6c-afc8-d6499b12010e.jpg'
viseron | [2024-11-19 19:49:32] [INFO ] [viseron.components.nvr.nvr.camera_1] - Stopping recording in: 9
viseron | [2024-11-19 19:49:35] [INFO ] [viseron.components.nvr.nvr.camera_1] - Stopping recording in: 8
viseron | [2024-11-19 19:49:38] [INFO ] [viseron.components.nvr.nvr.camera_1] - Stopping recording in: 7
viseron | [2024-11-19 19:49:41] [INFO ] [viseron.components.nvr.nvr.camera_1] - Stopping recording in: 6
viseron | [2024-11-19 19:49:43] [INFO ] [viseron.components.nvr.nvr.camera_1] - Stopping recording in: 5
viseron | [2024-11-19 19:49:46] [INFO ] [viseron.components.nvr.nvr.camera_1] - Stopping recording in: 4
viseron | [2024-11-19 19:49:49] [INFO ] [viseron.components.nvr.nvr.camera_1] - Stopping recording in: 3
viseron | [2024-11-19 19:49:52] [INFO ] [viseron.components.nvr.nvr.camera_1] - Stopping recording in: 2
viseron | [2024-11-19 19:49:52] [ERROR ] [viseron.components.storage.tier_handler.camera_1.tier_0] - Failed to move file /ramdisk/segments/camera_1/1731955730.m4s to /recordings/segments/camera_1/1731955730.m4s: [Errno 2] No such file or directory: '/ramdisk/segments/camera_1/1731955730.m4s'
viseron | [2024-11-19 19:49:52] [ERROR ] [viseron.components.storage.tier_handler.camera_1.tier_0] - Failed to move file /ramdisk/segments/camera_1/1731955735.m4s to /recordings/segments/camera_1/1731955735.m4s: [Errno 2] No such file or directory: '/ramdisk/segments/camera_1/1731955735.m4s'
viseron | [2024-11-19 19:49:52] [ERROR ] [viseron.components.storage.tier_handler.camera_1.tier_0] - Failed to move file /ramdisk/segments/camera_1/1731955738.m4s to /recordings/segments/camera_1/1731955738.m4s: [Errno 2] No such file or directory: '/ramdisk/segments/camera_1/1731955738.m4s'
viseron | [2024-11-19 19:49:52] [ERROR ] [viseron.components.storage.tier_handler.camera_1.tier_0] - Failed to move file /ramdisk/segments/camera_1/1731955744.m4s to /recordings/segments/camera_1/1731955744.m4s: [Errno 2] No such file or directory: '/ramdisk/segments/camera_1/1731955744.m4s'
viseron | [2024-11-19 19:49:52] [ERROR ] [viseron.components.storage.tier_handler.camera_1.tier_0] - Failed to move file /ramdisk/segments/camera_1/1731955750.m4s to /recordings/segments/camera_1/1731955750.m4s: [Errno 2] No such file or directory: '/ramdisk/segments/camera_1/1731955750.m4s'
viseron | [2024-11-19 19:49:52] [ERROR ] [viseron.components.storage.tier_handler.camera_1.tier_0] - Failed to move file /ramdisk/segments/camera_1/1731955756.m4s to /recordings/segments/camera_1/1731955756.m4s: [Errno 2] No such file or directory: '/ramdisk/segments/camera_1/1731955756.m4s'
viseron | [2024-11-19 19:49:52] [ERROR ] [viseron.components.storage.tier_handler.camera_1.tier_0] - Failed to move file /ramdisk/segments/camera_1/1731955759.m4s to /recordings/segments/camera_1/1731955759.m4s: [Errno 2] No such file or directory: '/ramdisk/segments/camera_1/1731955759.m4s'
viseron | [2024-11-19 19:49:52] [ERROR ] [viseron.components.storage.tier_handler.camera_1.tier_0] - Failed to move file /ramdisk/segments/camera_1/1731955764.m4s to /recordings/segments/camera_1/1731955764.m4s: [Errno 2] No such file or directory: '/ramdisk/segments/camera_1/1731955764.m4s'
viseron | [2024-11-19 19:49:52] [ERROR ] [viseron.components.storage.tier_handler.camera_1.tier_0] - Failed to move file /ramdisk/segments/camera_1/1731955770.m4s to /recordings/segments/camera_1/1731955770.m4s: [Errno 2] No such file or directory: '/ramdisk/segments/camera_1/1731955770.m4s'
viseron | [2024-11-19 19:49:52] [ERROR ] [viseron.components.storage.tier_handler.camera_1.tier_0] - Failed to move file /ramdisk/segments/camera_1/1731955773.m4s to /recordings/segments/camera_1/1731955773.m4s: [Errno 2] No such file or directory: '/ramdisk/segments/camera_1/1731955773.m4s'
viseron | [2024-11-19 19:49:52] [ERROR ] [viseron.components.storage.tier_handler.camera_1.tier_0] - Failed to move file /ramdisk/segments/camera_1/1731955779.m4s to /recordings/segments/camera_1/1731955779.m4s: [Errno 2] No such file or directory: '/ramdisk/segments/camera_1/1731955779.m4s'
viseron | [2024-11-19 19:49:52] [ERROR ] [viseron.components.storage.tier_handler.camera_1.tier_0] - Failed to move file /ramdisk/segments/camera_1/1731955784.m4s to /recordings/segments/camera_1/1731955784.m4s: [Errno 2] No such file or directory: '/ramdisk/segments/camera_1/1731955784.m4s'
viseron | [2024-11-19 19:49:52] [ERROR ] [viseron.components.storage.tier_handler.camera_1.tier_0] - Failed to move file /ramdisk/segments/camera_1/1731955790.m4s to /recordings/segments/camera_1/1731955790.m4s: [Errno 2] No such file or directory: '/ramdisk/segments/camera_1/1731955790.m4s'
viseron | [2024-11-19 19:49:52] [ERROR ] [viseron.components.storage.tier_handler.camera_1.tier_0] - Failed to move file /ramdisk/segments/camera_1/1731955796.m4s to /recordings/segments/camera_1/1731955796.m4s: [Errno 2] No such file or directory: '/ramdisk/segments/camera_1/1731955796.m4s'
viseron | [2024-11-19 19:49:52] [ERROR ] [viseron.components.storage.tier_handler.camera_1.tier_0] - Failed to move file /ramdisk/segments/camera_1/1731955799.m4s to /recordings/segments/camera_1/1731955799.m4s: [Errno 2] No such file or directory: '/ramdisk/segments/camera_1/1731955799.m4s'
viseron | [2024-11-19 19:49:52] [ERROR ] [viseron.components.storage.tier_handler.camera_1.tier_0] - Failed to move file /ramdisk/thumbnails/camera_1/95.jpg to /recordings/thumbnails/camera_1/95.jpg: [Errno 2] No such file or directory: '/ramdisk/thumbnails/camera_1/95.jpg'
viseron | [2024-11-19 19:49:52] [ERROR ] [viseron.components.storage.tier_handler.camera_1.tier_0] - Failed to move file /ramdisk/thumbnails/camera_1/96.jpg to /recordings/thumbnails/camera_1/96.jpg: [Errno 2] No such file or directory: '/ramdisk/thumbnails/camera_1/96.jpg'
viseron | [2024-11-19 19:49:55] [INFO ] [viseron.components.nvr.nvr.camera_1] - Stopping recording in: 1
viseron | [2024-11-19 19:49:57] [INFO ] [viseron.components.nvr.nvr.camera_1] - Stopping recording in: 0
viseron | [2024-11-19 19:49:57] [INFO ] [viseron.components.ffmpeg.recorder.camera_1] - Stopping recorder
viseron | [2024-11-19 19:49:58] [ERROR ] [root] - Uncaught thread exception in thread tier_handler_camera_1
viseron | Traceback (most recent call last):
viseron | File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
viseron | self.run()
viseron | File "/usr/lib/python3.10/threading.py", line 953, in run
viseron | self._target(*self._args, **self._kwargs)
viseron | File "/src/viseron/components/storage/tier_handler.py", line 236, in _process_events
viseron | self._on_created(event)
viseron | File "/src/viseron/components/storage/tier_handler.py", line 768, in _on_created
viseron | super()._on_created(event)
viseron | File "/src/viseron/components/storage/tier_handler.py", line 262, in _on_created
viseron | session.execute(stmt)
viseron | File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/orm/session.py", line 2351, in execute
viseron | return self._execute_internal(
viseron | File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/orm/session.py", line 2236, in _execute_internal
viseron | result: Result[Any] = compile_state_cls.orm_execute_statement(
viseron | File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/orm/bulk_persistence.py", line 1283, in orm_execute_statement
viseron | result = conn.execute(
viseron | File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/base.py", line 1418, in execute
viseron | return meth(
viseron | File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/sql/elements.py", line 515, in _execute_on_connection
viseron | return connection._execute_clauseelement(
viseron | File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/base.py", line 1609, in _execute_clauseelement
viseron | ) = self._invoke_before_exec_event(
viseron | File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/base.py", line 1568, in _invoke_before_exec_event
viseron | elem, event_multiparams, event_params = fn(
viseron | File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/events.py", line 160, in wrap_before_execute
viseron | orig_fn(
viseron | File "/src/viseron/components/storage/triggers.py", line 24, in insert_into_files_meta
viseron | conn.execute(
viseron | File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/base.py", line 1418, in execute
viseron | return meth(
viseron | File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/sql/elements.py", line 515, in _execute_on_connection
viseron | return connection._execute_clauseelement(
viseron | File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/base.py", line 1640, in _execute_clauseelement
viseron | ret = self._execute_context(
viseron | File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/base.py", line 1846, in _execute_context
viseron | return self._exec_single_context(
viseron | File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/base.py", line 1986, in _exec_single_context
viseron | self._handle_dbapi_exception(
viseron | File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/base.py", line 2356, in _handle_dbapi_exception
viseron | raise exc_info[1].with_traceback(exc_info[2])
viseron | File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/base.py", line 1967, in _exec_single_context
viseron | self.dialect.do_execute(
viseron | File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/default.py", line 924, in do_execute
viseron | cursor.execute(statement, parameters)
viseron | UnicodeEncodeError: 'ascii' codec can't encode characters in position 40-45: ordinal not in range(128)
viseron | [2024-11-19 19:49:58] [ERROR ] [root] - Uncaught thread exception in thread Thread-10404 (_concatenate_fragments)
viseron | Traceback (most recent call last):
viseron | File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
viseron | self.run()
viseron | File "/usr/lib/python3.10/threading.py", line 953, in run
viseron | self._target(*self._args, **self._kwargs)
viseron | File "/src/viseron/domains/camera/recorder.py", line 405, in _concatenate_fragments
viseron | session.execute(stmt)
viseron | File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/orm/session.py", line 2351, in execute
viseron | return self._execute_internal(
viseron | File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/orm/session.py", line 2236, in _execute_internal
viseron | result: Result[Any] = compile_state_cls.orm_execute_statement(
viseron | File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/orm/bulk_persistence.py", line 1624, in orm_execute_statement
viseron | return super().orm_execute_statement(
viseron | File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/orm/context.py", line 293, in orm_execute_statement
viseron | result = conn.execute(
viseron | File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/base.py", line 1418, in execute
viseron | return meth(
viseron | File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/sql/elements.py", line 515, in _execute_on_connection
viseron | return connection._execute_clauseelement(
viseron | File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/base.py", line 1640, in _execute_clauseelement
viseron | ret = self._execute_context(
viseron | File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/base.py", line 1846, in _execute_context
viseron | return self._exec_single_context(
viseron | File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/base.py", line 1986, in _exec_single_context
viseron | self._handle_dbapi_exception(
viseron | File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/base.py", line 2356, in _handle_dbapi_exception
viseron | raise exc_info[1].with_traceback(exc_info[2])
viseron | File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/base.py", line 1967, in _exec_single_context
viseron | self.dialect.do_execute(
viseron | File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/default.py", line 924, in do_execute
viseron | cursor.execute(statement, parameters)
viseron | UnicodeEncodeError: 'ascii' codec can't encode characters in position 40-45: ordinal not in range(128)
viseron | [2024-11-19 19:50:05] [ERROR ] [viseron.watchdog.thread_watchdog] - Thread tier_handler_camera_1 is dead, restarting
viseron | [2024-11-19 19:50:06] [ERROR ] [root] - Uncaught thread exception in thread Thread-10429
viseron | Traceback (most recent call last):
viseron | File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
viseron | self.run()
viseron | File "/usr/lib/python3.10/threading.py", line 1378, in run
viseron | self.function(*self.args, **self.kwargs)
viseron | File "/src/viseron/components/storage/tier_handler.py", line 307, in _update_size
viseron | session.execute(stmt)
viseron | File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/orm/session.py", line 2351, in execute
viseron | return self._execute_internal(
viseron | File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/orm/session.py", line 2236, in _execute_internal
viseron | result: Result[Any] = compile_state_cls.orm_execute_statement(
viseron | File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/orm/bulk_persistence.py", line 1624, in orm_execute_statement
viseron | return super().orm_execute_statement(
viseron | File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/orm/context.py", line 293, in orm_execute_statement
viseron | result = conn.execute(
viseron | File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/base.py", line 1418, in execute
viseron | return meth(
viseron | File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/sql/elements.py", line 515, in _execute_on_connection
viseron | return connection._execute_clauseelement(
viseron | File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/base.py", line 1640, in _execute_clauseelement
viseron | ret = self._execute_context(
viseron | File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/base.py", line 1846, in _execute_context
viseron | return self._exec_single_context(
viseron | File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/base.py", line 1986, in _exec_single_context
viseron | self._handle_dbapi_exception(
viseron | File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/base.py", line 2356, in _handle_dbapi_exception
viseron | raise exc_info[1].with_traceback(exc_info[2])
viseron | File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/base.py", line 1967, in _exec_single_context
viseron | self.dialect.do_execute(
viseron | File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/default.py", line 924, in do_execute
viseron | cursor.execute(statement, parameters)
viseron | UnicodeEncodeError: 'ascii' codec can't encode characters in position 40-45: ordinal not in range(128)
Not sure what happened there and why the files were missing. I did not restart the whole container, I only restarted Viseron via the UI.
The encoding error is also quite strange, I do not use anything except UTF-8.
OK, floating some ideas: perhaps a simpler workaround would be to have an option which forces the creation of .mp4 files on the next tier? As I understand the segments are being rotated frequently, so having them on a ramdisk makes sense in order to be kinder to the eMMC. The use case of the mp4 files is based on the need to store them for later review or evidence outside of Viseron, so it does not sound wrong creating them directly in the long-term i.e. tier2 storage? Goal is to offload the ramdisk spacewise, to ensure that it can continue handling new events without running out of space due to same data that will be present twice.
The segments are stored for as long as they are wanted.
In your case, since you are only storing events, only the segments that relate to an event (triggered by trigger_recorder: true
will be kept.
I realize the naming here between events and recordings is a bit confusing, but it stems from v2 which functioned in a very different way.
I do see the need for storing the MP4 files, and you are not the first to have similar requirements.
What i could do is introduce event_clips
as a separate section in the tier config which would allow for more control of where they end up.
Something like this:
storage:
recorder:
tiers:
- path: /config/tier1
continuous:
max_size:
mb: 50
events:
max_size:
mb: 100
event_clips:
max_size:
mb: 1 # Setting a low value would essentially mean they get moved instantly
move_on_shutdown: true
- path: /config/tier2 # Store only event clips in this tier
event_clips:
max_size:
gb: 100
- path: /config/tier3
continuous:
max_size:
mb: 50
events:
max_size:
mb: 100
I will have a look at how complicated that would be. v3 has been in beta for ages now and i would like to get it released soon and i can work on polishing it further later on.
I have more logs and a somewhat strange behaviour, I should probably update my dev image to make sure I am running the latest code.
So, after making some changes to the configuration (enabled more detection labels) I hit the restart button in the UI, which resulted in the following backtrace:
Yes i noticed that as well, move_on_shutdown
is not working correctly.
I have it fixed locally, just need to clean it up before pushing to dev
.
Not sure what happened there and why the files were missing. I did not restart the whole container, I only restarted Viseron via the UI.
The missing files most likely a side effect of the unclean shutdown.
The encoding error is also quite strange, I do not use anything except UTF-8.
Indeed, never seen that before. Can you enable debug logging and see if the error keeps appearing?
logger:
default_level: debug
The segments are stored for as long as they are wanted. In your case, since you are only storing events, only the segments that relate to an event (triggered by
trigger_recorder: true
will be kept. I realize the naming here between events and recordings is a bit confusing, but it stems from v2 which functioned in a very different way.
I guess it is :) So, my understanding was that an "event" is something that triggers a recording when it occurs, so essentially in the end you get a recording of the event. Now, seeing your config proposal I think I did not get that right, it seems there are "events" and "event clips" which is not the same? Ah, I think I know what you mean, an "event clip" is the complete mp4? I am not sure if these names are not too confusing...
From the user perspective I consider the gazillion of segments something internal to how the application works, focus of those is to show up and be playable in the UI. They should get rotated according to the configured rules to ensure the system does not run out of space, but otherwise I do not think that anyone or anything except for the application itself is doing anything with those files?
The complete mp4 files, as you called them "event clips" are actually what a user would want to grab and copy out of the Viseron storage and save them permanently, the assumption here is, that these files are needed outside of Viseron.
Let me float yet another idea :) So, as long as I am within Viseron (i.e. using the UI) I do not really care what format the files are in, the UI plays everything nicely and I can look through the events to check what happened. The format only becomes interesting as soon as I see something caught on camera, that I need to store for myself permanently. This is actually the use case for having an mp4 clip of the event in the first place.
So, perhaps a more intuitive way would be to offer some sort of an export button in the UI, allowing to convert selected events to "event clips" and have them saved in a dedicated storage tier.
This way there is no automatic duplication as there is no real need for bulk-saving the mp4 clips - they are only needed when the user saw an event that needs to be saved permanently and since reviewing the events is anyway a manual step, hitting a button in the UI to generate an mp4 clip should be totally fine from a user's perspective. What do you think?
v3 has been in beta for ages now and i would like to get it released soon and i can work on polishing it further later on.
I totally get that, as a developer I also do not like when something does not come to an end and just drags on and on, better to get out a stable release with lesser features and then follow up by smaller releases, than polishing a scope-creeped beta release for years :>
Indeed, never seen that before. Can you enable debug logging and see if the error keeps appearing?
Unfortunately I was not able to reproduce it, i.e. the exceptions during the shutdown are there, but the exception at start did not occur anymore. Perhaps some UTF-8 characters got truncated when saving due to the shutdown exceptions and produced illegal UTF-8, which in turn caused an exception on startup? Just guessing..
I guess it is :) So, my understanding was that an "event" is something that triggers a recording when it occurs, so essentially in the end you get a recording of the event. Now, seeing your config proposal I think I did not get that right, it seems there are "events" and "event clips" which is not the same? Ah, I think I know what you mean, an "event clip" is the complete mp4? I am not sure if these names are not too confusing...
I think placing the segments in the /recordings
folder and storing the "event clips" in a new folder called /clips
would make more sense.
The problem is that v2 uses /recordings
to store "event clip" (the only kind of recording in v2) so for easier transition to v3 i had to go with different naming.
Do you think /recordings
for the short segments and /clips
for clips created by create_event_clip: true
would make sense?
Manually created clips probably has to live outside of the Viseron tiers so that they are not cleaned up by the system.
From the user perspective I consider the gazillion of segments something internal to how the application works, focus of those is to show up and be playable in the UI. They should get rotated according to the configured rules to ensure the system does not run out of space, but otherwise I do not think that anyone or anything except for the application itself is doing anything with those files?
Yes that is correct.
The complete mp4 files, as you called them "event clips" are actually what a user would want to grab and copy out of the Viseron storage and save them permanently, the assumption here is, that these files are needed outside of Viseron.
Let me float yet another idea :) So, as long as I am within Viseron (i.e. using the UI) I do not really care what format the files are in, the UI plays everything nicely and I can look through the events to check what happened. The format only becomes interesting as soon as I see something caught on camera, that I need to store for myself permanently. This is actually the use case for having an mp4 clip of the event in the first place.
So, perhaps a more intuitive way would be to offer some sort of an export button in the UI, allowing to convert selected events to "event clips" and have them saved in a dedicated storage tier.
This way there is no automatic duplication as there is no real need for bulk-saving the mp4 clips - they are only needed when the user saw an event that needs to be saved permanently and since reviewing the events is anyway a manual step, hitting a button in the UI to generate an mp4 clip should be totally fine from a user's perspective. What do you think?
Yes having a way to selectively export seems to be a good resolution.
It is already on the to-do list in the v3 PR actually: #619
Hopefully i can have a look at that soon! You are not the first to point out that its a needed feature.
The problem is that v2 uses
/recordings
to store "event clip" (the only kind of recording in v2) so for easier transition to v3 i had to go with different naming. Do you think/recordings
for the short segments and/clips
for clips created bycreate_event_clip: true
would make sense?
Hmm, as a user who is new to Viseron, "recordings" was the name where I expected to find the actual coherent mp4 recordings. One has to actually dive a bit deeper to understand the difference between "recordings", "clips" and "segments", actually if short segments are stored in "recordings" and if we also conclude, that those short segments are something internal to Viseron, then why isn't the directory where they are stored called accordingly?
If you have to keep the names as they are now, then indeed having /clips
(which are intuitively - recordings) would make sense in conjunction with create_event_clip: true
My later argument was, that if we can get the export feature, then having create_event_clip: true
isn't even needed, unless there is another use case which I am missing?
Yes having a way to selectively export seems to be a good resolution. It is already on the to-do list in the v3 PR actually: #619
Oh it's on the list? I did not see it mentioned anywhere in #619 which is already merged?
Hopefully i can have a look at that soon! You are not the first to point out that its a needed feature.
Awesome! If there is anything to test, please ping me. I am currently setting things up and waiting for the remaining cameras to arrive, so doing a lot of tests and experiments at the moment.
Hmm, as a user who is new to Viseron, "recordings" was the name where I expected to find the actual coherent mp4 recordings. One has to actually dive a bit deeper to understand the difference between "recordings", "clips" and "segments", actually if short segments are stored in "recordings" and if we also conclude, that those short segments are something internal to Viseron, then why isn't the directory where they are stored called accordingly?
If you have to keep the names as they are now, then indeed having
/clips
(which are intuitively - recordings) would make sense in conjunction withcreate_event_clip: true
My later argument was, that if we can get the export feature, then having
create_event_clip: true
isn't even needed, unless there is another use case which I am missing?
Yeah a lot of people seem to have the same expectations as you have. I think writing a proper glossary and also documentation on how Viseron handles recordings would clear up a lot of confusion, and also highlight to myself what to improve. I am the sole developer on this project and sometimes what makes sense to me might not make sense to others! So i appreciate you taking the time and writing down your thoughts.
Oh it's on the list? I did not see it mentioned anywhere in #619 which is already merged?
Sorry my bad, its this PR: #716
Yeah a lot of people seem to have the same expectations as you have. I think writing a proper glossary and also documentation on how Viseron handles recordings would clear up a lot of confusion, and also highlight to myself what to improve.
Indeed, I'd be interested to read and understand how you see these things, because the separation of "Events" and "Recordings", where both are playable videos of the same occurrence is not quite clear to me 😅
I am the sole developer on this project and sometimes what makes sense to me might not make sense to others! So i appreciate you taking the time and writing down your thoughts.
No problem, as a user I'm happy that you are interested in feedback and are considering to tune a thing or two. I've been running an open source project too back in the day, so I know how much work it actually is :)
Sorry my bad, its this PR: #716
Ah, this must be the point "Allow export of timespan"? How is this planned, export "all recordings within a given timespan"? If possible, please also add a possibility to "export this event/recording", i.e. the one that is being viewed by the user at the moment, that'd be really practical.
I am still battling the "clips" issue, so my config has:
recorder:
filename_pattern: east_%Y-%m-%d-%H:%M:%S
create_event_clip: true
So my expectation was that it would now create those mp4
clips that we have been talking about.
I see the event in the UI, but there are no mp4 files corresponding to this event. A few mp4 files with earlier timestamps have been created, but last I see is from 14:00 while I am looking for an event after 15:08
I assume it is available in the segments format since I can play it in the UI, but dozens of files named as 1732553822.m4s
are not easy to match :(
Also, when I go to "Recordings" in the UI, I only see one video from today, although I have dozens of events which I expected to end up as a "recording", I guess this brings me back to the earlier question on what the difference between "Events" and "Recordings" is and what it is good for?
I do not see any errors in the log, so I really have no clue why no mp4 "clips" have been created for these events, but I'd really like to download them. Any hints on how to figure out what segments belong to which event, without having to go through everything manually?
Ah, this must be the point "Allow export of timespan"? How is this planned, export "all recordings within a given timespan"? If possible, please also add a possibility to "export this event/recording", i.e. the one that is being viewed by the user at the moment, that'd be really practical.
Yes my plan is to allow export of individual events or by specifying a time range.
Could you paste your config?
I tried (with your filename_pattern just to be sure) and i get clips created properly.
abc@458814191c5f:/workspaces/viseron/config/tier3/recordings/viseron_vscode_camera/2024-11-25$ ls -al
total 5576
drwxr-xr-x 2 abc abc 4096 Nov 25 22:05 .
drwxr-xr-x 9 abc abc 4096 Nov 25 21:55 ..
-rw-r--r-- 1 abc abc 1977969 Nov 25 21:55 21:53:44.mp4
-rw-r--r-- 1 abc abc 1887373 Nov 25 22:01 22:01:04.mp4
-rw-r--r-- 1 abc abc 1831587 Nov 25 22:05 east_2024-11-25-22:05:04.mp4
Note that they are stored in UTC so depending on your timezone they might not appear where you think they are.
Note that they are stored in UTC so depending on your timezone they might not appear where you think they are.
OK, sorry, no need to check the config - this was exactly it, the recordings were made, but time was off, and since I did not see any recordings for after 3pm. I thought that they were missing. Which brings me to another interesting question: how can I, as a user, map the time I see in the UI to the timestamp of the filename (of course knowing this nuance one could "manually" convert the desired time to UTC and figure out what the clip name might be)? I would expect them to be the same, since I am interested in a recording of a particular event and the time when it happened is important. Is the issue with my pattern %Y-%m-%d-%H:%M:%S
or is it always UTC?
From a software engineering point of view I understand, that using UTC eliminates all sorts of problems (especially the daylight saving time jump horrors), but when exporting the clip I would expect it to be saved in the same time format that is being displayed in the UI, otherwise it's just confusing... Would you be open to the thought of tuning this (or perhaps adding an option for this) at some point?
Hi,
I am not quite sure if this is similar to https://github.com/roflcoopter/viseron/issues/797 but I simply can't find my recordings, although they are shown in the web interface.
I am using the dev version, the relevant docker compose config:
In my viseron config (I know it's default) I specified the recordings path anyway:
On the host the directories
recordings/camera_1/2024-11-18/
get created, but there are not files there.I do not see any errors in the logs:
However, the file:
/recordings/camera_1/2024-11-18/east_2024-11-18-14:08:47.mp4
is nowhere to be found. I tried attaching to a shell inside the docker container and the recordings directory is there as empty as on the host:I tried searching for the file in the container root, also in the host system root and I do not see it anywhere. Yet
http://nvr:8888/#/recordings/camera_1/2024-11-18
shows quite a number of files which can be played back in the browser.So... where are they located? :)