Closed MatthK closed 2 years ago
(Posting this in #308) I have had issues with getting photos to all scan (I had first added it as part of the reply to another issue (no photos show up under places #305) as scanning would never complete. I will also update #305 to keep things together there.
I have tried remove as many layers as possible in the docker setup by moving the test to a Debian9 VM (set with 4vcpu 4GB RAM) as opposed to MacOS which is just using a hidden Linux VM.
NOW I can see an error in the ownphotos.log where before I could not. It appears to just suck all of the memory and the whole VM becomes unresponsive...
---- HERE IS THE COMPLETE LOG FROM A CLEAN TRY ----
021-08-30 12:13:17,086 : views.py : list : 515 : INFO : querying & serializing took 0.01 seconds 2021-08-30 12:13:27,271 : serializers.py : update : 720 : INFO : Updated scan directory for user /data 2021-08-30 12:13:27,273 : serializers.py : update : 726 : INFO : Updated confidence for user 0.1 2021-08-30 12:13:27,274 : serializers.py : update : 736 : INFO : Updated semantic_search_topk for user 0 2021-08-30 12:13:27,275 : serializers.py : update : 742 : INFO : Updated favorite_min_rating for user 4 2021-08-30 12:13:43,409 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_3513.jpg 2021-08-30 12:13:45,702 : photo.py : _generate_captions : 143 : INFO : generated places365 captions for image /protected_media/thumbnails_big/dbd2bd73b451e048584f3082589e2ca71.webp. 2021-08-30 12:13:52,303 : photo.py : _generate_clip_embeddings : 117 : INFO : generated clip embeddings for image /protected_media/thumbnails_big/dbd2bd73b451e048584f3082589e2ca71.webp. 2021-08-30 12:13:53,592 : util.py : mapbox_reverse_geocode : 97 : INFO : mapbox returned status 200. 2021-08-30 12:13:54,889 : photo.py : _extract_faces : 362 : INFO : image dbd2bd73b451e048584f3082589e2ca71: 2 face(s) saved 2021-08-30 12:13:54,891 : directory_watcher.py : handle_new_image : 114 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: image processed: /data/IMG_3513.jpg, elapsed: 11.455016 2021-08-30 12:13:54,906 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_3065.jpg 2021-08-30 12:13:56,146 : photo.py : _generate_captions : 143 : INFO : generated places365 captions for image /protected_media/thumbnails_big/13f966e0d6ab83b3657878097a3ce1bc1.webp. 2021-08-30 12:14:03,017 : photo.py : _generate_clip_embeddings : 117 : INFO : generated clip embeddings for image /protected_media/thumbnails_big/13f966e0d6ab83b3657878097a3ce1bc1.webp. 2021-08-30 12:14:03,049 : directory_watcher.py : handle_new_image : 134 : ERROR : job 39b9ca47-340a-4927-8dff-8692774c8db8: could not load image /data/IMG_3065.jpg. reason: [Errno 12] Cannot allocate memory Traceback (most recent call last): File "/code/api/directory_watcher.py", line 109, in handle_new_image photo._extract_date_time_from_exif(True) File "/code/api/models/photo.py", line 211, in _extract_date_time_from_exif with exiftool.ExifTool() as et: File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 410, in enter self.start() File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 381, in start raise oe File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 372, in start self._process = subprocess.Popen( File "/usr/lib/python3.9/subprocess.py", line 951, in init self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/lib/python3.9/subprocess.py", line 1754, in _execute_child self.pid = _posixsubprocess.fork_exec( OSError: [Errno 12] Cannot allocate memory 2021-08-30 12:14:03,240 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_9411.HEIC.heif 2021-08-30 12:14:05,848 : directory_watcher.py : handle_new_image : 134 : ERROR : job 39b9ca47-340a-4927-8dff-8692774c8db8: could not load image /data/IMG_9411.HEIC.heif. reason: [Errno 12] Cannot allocate memory Traceback (most recent call last): File "/code/api/directory_watcher.py", line 106, in handle_new_image photo._calculate_aspect_ratio(False) File "/code/api/models/photo.py", line 200, in _calculate_aspect_ratio with exiftool.ExifTool() as et: File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 410, in enter self.start() File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 381, in start raise oe File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 372, in start self._process = subprocess.Popen( File "/usr/lib/python3.9/subprocess.py", line 951, in init self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/lib/python3.9/subprocess.py", line 1754, in _execute_child self.pid = _posixsubprocess.fork_exec( OSError: [Errno 12] Cannot allocate memory 2021-08-30 12:14:05,913 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_2143.HEIC.heif 2021-08-30 12:14:08,147 : directory_watcher.py : handle_new_image : 134 : ERROR : job 39b9ca47-340a-4927-8dff-8692774c8db8: could not load image /data/IMG_2143.HEIC.heif. reason: [Errno 12] Cannot allocate memory Traceback (most recent call last): File "/code/api/directory_watcher.py", line 106, in handle_new_image photo._calculate_aspect_ratio(False) File "/code/api/models/photo.py", line 200, in _calculate_aspect_ratio with exiftool.ExifTool() as et: File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 410, in enter self.start() File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 381, in start raise oe File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 372, in start self._process = subprocess.Popen( File "/usr/lib/python3.9/subprocess.py", line 951, in init self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/lib/python3.9/subprocess.py", line 1754, in _execute_child self.pid = _posixsubprocess.fork_exec( OSError: [Errno 12] Cannot allocate memory 2021-08-30 12:14:08,202 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/567712203.458368public.jpg 2021-08-30 12:14:08,713 : directory_watcher.py : handle_new_image : 134 : ERROR : job 39b9ca47-340a-4927-8dff-8692774c8db8: could not load image /data/567712203.458368public.jpg. reason: [Errno 12] Cannot allocate memory Traceback (most recent call last): File "/code/api/directory_watcher.py", line 106, in handle_new_image photo._calculate_aspect_ratio(False) File "/code/api/models/photo.py", line 200, in _calculate_aspect_ratio with exiftool.ExifTool() as et: File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 410, in enter self.start() File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 381, in start raise oe File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 372, in start self._process = subprocess.Popen( File "/usr/lib/python3.9/subprocess.py", line 951, in init self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/lib/python3.9/subprocess.py", line 1754, in _execute_child self.pid = _posixsubprocess.fork_exec( OSError: [Errno 12] Cannot allocate memory 2021-08-30 12:14:08,836 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_2386.heic 2021-08-30 12:14:11,886 : directory_watcher.py : handle_new_image : 134 : ERROR : job 39b9ca47-340a-4927-8dff-8692774c8db8: could not load image /data/IMG_2386.heic. reason: [Errno 12] Cannot allocate memory Traceback (most recent call last): File "/code/api/directory_watcher.py", line 106, in handle_new_image photo._calculate_aspect_ratio(False) File "/code/api/models/photo.py", line 200, in _calculate_aspect_ratio with exiftool.ExifTool() as et: File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 410, in enter self.start() File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 381, in start raise oe File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 372, in start self._process = subprocess.Popen( File "/usr/lib/python3.9/subprocess.py", line 951, in init self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/lib/python3.9/subprocess.py", line 1754, in _execute_child self.pid = _posixsubprocess.fork_exec( OSError: [Errno 12] Cannot allocate memory 2021-08-30 12:14:11,952 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_8176.jpg 2021-08-30 12:14:12,717 : directory_watcher.py : handle_new_image : 134 : ERROR : job 39b9ca47-340a-4927-8dff-8692774c8db8: could not load image /data/IMG_8176.jpg. reason: [Errno 12] Cannot allocate memory Traceback (most recent call last): File "/code/api/directory_watcher.py", line 106, in handle_new_image photo._calculate_aspect_ratio(False) File "/code/api/models/photo.py", line 200, in _calculate_aspect_ratio with exiftool.ExifTool() as et: File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 410, in enter self.start() File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 381, in start raise oe File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 372, in start self._process = subprocess.Popen( File "/usr/lib/python3.9/subprocess.py", line 951, in init self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/lib/python3.9/subprocess.py", line 1754, in _execute_child self.pid = _posixsubprocess.fork_exec( OSError: [Errno 12] Cannot allocate memory 2021-08-30 12:14:12,778 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_5830.heic 2021-08-30 12:14:14,758 : directory_watcher.py : handle_new_image : 134 : ERROR : job 39b9ca47-340a-4927-8dff-8692774c8db8: could not load image /data/IMG_5830.heic. reason: [Errno 12] Cannot allocate memory Traceback (most recent call last): File "/code/api/directory_watcher.py", line 106, in handle_new_image photo._calculate_aspect_ratio(False) File "/code/api/models/photo.py", line 200, in _calculate_aspect_ratio with exiftool.ExifTool() as et: File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 410, in enter self.start() File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 381, in start raise oe File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 372, in start self._process = subprocess.Popen( File "/usr/lib/python3.9/subprocess.py", line 951, in init self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/lib/python3.9/subprocess.py", line 1754, in _execute_child self.pid = _posixsubprocess.fork_exec( OSError: [Errno 12] Cannot allocate memory 2021-08-30 12:14:14,826 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_7527.heic 2021-08-30 12:14:17,863 : directory_watcher.py : handle_new_image : 134 : ERROR : job 39b9ca47-340a-4927-8dff-8692774c8db8: could not load image /data/IMG_7527.heic. reason: [Errno 12] Cannot allocate memory Traceback (most recent call last): File "/code/api/directory_watcher.py", line 106, in handle_new_image photo._calculate_aspect_ratio(False) File "/code/api/models/photo.py", line 200, in _calculate_aspect_ratio with exiftool.ExifTool() as et: File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 410, in enter self.start() File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 381, in start raise oe File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 372, in start self._process = subprocess.Popen( File "/usr/lib/python3.9/subprocess.py", line 951, in init self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/lib/python3.9/subprocess.py", line 1754, in _execute_child self.pid = _posixsubprocess.fork_exec( OSError: [Errno 12] Cannot allocate memory 2021-08-30 12:14:17,922 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_0224.jpg 2021-08-30 12:14:18,552 : directory_watcher.py : handle_new_image : 134 : ERROR : job 39b9ca47-340a-4927-8dff-8692774c8db8: could not load image /data/IMG_0224.jpg. reason: [Errno 12] Cannot allocate memory Traceback (most recent call last): File "/code/api/directory_watcher.py", line 106, in handle_new_image photo._calculate_aspect_ratio(False) File "/code/api/models/photo.py", line 200, in _calculate_aspect_ratio with exiftool.ExifTool() as et: File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 410, in enter self.start() File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 381, in start raise oe File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 372, in start self._process = subprocess.Popen( File "/usr/lib/python3.9/subprocess.py", line 951, in init self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/lib/python3.9/subprocess.py", line 1754, in _execute_child self.pid = _posixsubprocess.fork_exec( OSError: [Errno 12] Cannot allocate memory 2021-08-30 12:14:18,627 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_5896.heic 2021-08-30 12:14:21,493 : directory_watcher.py : handle_new_image : 134 : ERROR : job 39b9ca47-340a-4927-8dff-8692774c8db8: could not load image /data/IMG_5896.heic. reason: [Errno 12] Cannot allocate memory Traceback (most recent call last): File "/code/api/directory_watcher.py", line 106, in handle_new_image photo._calculate_aspect_ratio(False) File "/code/api/models/photo.py", line 200, in _calculate_aspect_ratio with exiftool.ExifTool() as et: File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 410, in enter self.start() File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 381, in start raise oe File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 372, in start self._process = subprocess.Popen( File "/usr/lib/python3.9/subprocess.py", line 951, in init self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/lib/python3.9/subprocess.py", line 1754, in _execute_child self.pid = _posixsubprocess.fork_exec( OSError: [Errno 12] Cannot allocate memory 2021-08-30 12:14:21,512 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/WP_20151218_13_30_28_Rawhighres.dng 2021-08-30 12:19:54,456 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_3297.heic 2021-08-30 12:19:58,838 : photo.py : _generate_captions : 143 : INFO : generated places365 captions for image /protected_media/thumbnails_big/c8c4c21ce392750f27807efc3e098bcd1.webp. 2021-08-30 12:20:05,393 : photo.py : _generate_clip_embeddings : 117 : INFO : generated clip embeddings for image /protected_media/thumbnails_big/c8c4c21ce392750f27807efc3e098bcd1.webp. 2021-08-30 12:20:06,945 : util.py : mapbox_reverse_geocode : 97 : INFO : mapbox returned status 200. 2021-08-30 12:20:08,988 : photo.py : _extract_faces : 362 : INFO : image c8c4c21ce392750f27807efc3e098bcd1: 1 face(s) saved 2021-08-30 12:20:08,990 : directory_watcher.py : handle_new_image : 114 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: image processed: /data/IMG_3297.heic, elapsed: 14.474443 2021-08-30 12:20:09,022 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/WP_20171127_21_17_10_Pro__highres.jpg 2021-08-30 12:20:10,789 : photo.py : _generate_captions : 143 : INFO : generated places365 captions for image /protected_media/thumbnails_big/576e7b15b9ca9a795945f70f2d6336c31.webp. 2021-08-30 12:20:16,222 : photo.py : _generate_clip_embeddings : 117 : INFO : generated clip embeddings for image /protected_media/thumbnails_big/576e7b15b9ca9a795945f70f2d6336c31.webp. 2021-08-30 12:20:17,072 : util.py : mapbox_reverse_geocode : 97 : INFO : mapbox returned status 200. 2021-08-30 12:20:19,459 : photo.py : _extract_faces : 362 : INFO : image 576e7b15b9ca9a795945f70f2d6336c31: 1 face(s) saved 2021-08-30 12:20:19,460 : directory_watcher.py : handle_new_image : 114 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: image processed: /data/WP_20171127_21_17_10_Prohighres.jpg, elapsed: 10.394164 2021-08-30 12:20:19,516 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_6338.heic 2021-08-30 12:20:22,407 : photo.py : _generate_captions : 143 : INFO : generated places365 captions for image /protected_media/thumbnails_big/a7569195075a165f4813dc3e6dc2b93a1.webp. 2021-08-30 12:20:27,582 : photo.py : _generate_clip_embeddings : 117 : INFO : generated clip embeddings for image /protected_media/thumbnails_big/a7569195075a165f4813dc3e6dc2b93a1.webp. 2021-08-30 12:20:28,813 : util.py : mapbox_reverse_geocode : 97 : INFO : mapbox returned status 200. 2021-08-30 12:20:29,922 : photo.py : _extract_faces : 362 : INFO : image a7569195075a165f4813dc3e6dc2b93a1: 1 face(s) saved 2021-08-30 12:20:29,923 : directory_watcher.py : handle_new_image : 114 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: image processed: /data/IMG_6338.heic, elapsed: 10.397034 2021-08-30 12:20:29,941 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/WP_20160823_19_25_11_Rawhighres.dng 2021-08-30 12:21:37,015 : photo.py : _generate_captions : 143 : INFO : generated places365 captions for image /protected_media/thumbnails_big/6e3aa849ab5f116973f9220a42f956ff1.webp. 2021-08-30 12:21:43,208 : photo.py : _generate_clip_embeddings : 117 : INFO : generated clip embeddings for image /protected_media/thumbnails_big/6e3aa849ab5f116973f9220a42f956ff1.webp. 2021-08-30 12:21:46,049 : directory_watcher.py : handle_new_image : 114 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: image processed: /data/WP_20160823_19_25_11_Rawhighres.dng, elapsed: 75.926134 2021-08-30 12:21:46,087 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_0404.jpg 2021-08-30 12:21:46,866 : photo.py : _generate_captions : 143 : INFO : generated places365 captions for image /protected_media/thumbnails_big/4f4e6ff4bc394df5d5222216e3fb28401.webp. 2021-08-30 12:21:52,186 : photo.py : _generate_clip_embeddings : 117 : INFO : generated clip embeddings for image /protected_media/thumbnails_big/4f4e6ff4bc394df5d5222216e3fb28401.webp. 2021-08-30 12:21:53,849 : photo.py : _extract_faces : 362 : INFO : image 4f4e6ff4bc394df5d5222216e3fb28401: 1 face(s) saved 2021-08-30 12:21:53,850 : directory_watcher.py : handle_new_image : 114 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: image processed: /data/IMG_0404.jpg, elapsed: 7.750198 2021-08-30 12:21:53,935 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_5584.heic 2021-08-30 12:21:56,280 : directory_watcher.py : handle_new_image : 134 : ERROR : job 39b9ca47-340a-4927-8dff-8692774c8db8: could not load image /data/IMG_5584.heic. reason: [Errno 12] Cannot allocate memory Traceback (most recent call last): File "/code/api/directory_watcher.py", line 106, in handle_new_image photo._calculate_aspect_ratio(False) File "/code/api/models/photo.py", line 200, in _calculate_aspect_ratio with exiftool.ExifTool() as et: File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 410, in enter__ self.start() File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 381, in start raise oe File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 372, in start self._process = subprocess.Popen( File "/usr/lib/python3.9/subprocess.py", line 951, in init self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/lib/python3.9/subprocess.py", line 1754, in _execute_child self.pid = _posixsubprocess.fork_exec( OSError: [Errno 12] Cannot allocate memory 2021-08-30 12:21:56,340 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/SarverHerringWedding-72.jpg 2021-08-30 12:21:56,946 : directory_watcher.py : handle_new_image : 134 : ERROR : job 39b9ca47-340a-4927-8dff-8692774c8db8: could not load image /data/SarverHerringWedding-72.jpg. reason: [Errno 12] Cannot allocate memory Traceback (most recent call last): File "/code/api/directory_watcher.py", line 106, in handle_new_image photo._calculate_aspect_ratio(False) File "/code/api/models/photo.py", line 200, in _calculate_aspect_ratio with exiftool.ExifTool() as et: File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 410, in enter self.start() File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 381, in start raise oe File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 372, in start self._process = subprocess.Popen( File "/usr/lib/python3.9/subprocess.py", line 951, in init self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/lib/python3.9/subprocess.py", line 1754, in _execute_child self.pid = _posixsubprocess.fork_exec( OSError: [Errno 12] Cannot allocate memory 2021-08-30 12:21:57,011 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_6369.heic 2021-08-30 12:21:58,906 : directory_watcher.py : handle_new_image : 134 : ERROR : job 39b9ca47-340a-4927-8dff-8692774c8db8: could not load image /data/IMG_6369.heic. reason: [Errno 12] Cannot allocate memory Traceback (most recent call last): File "/code/api/directory_watcher.py", line 106, in handle_new_image photo._calculate_aspect_ratio(False) File "/code/api/models/photo.py", line 200, in _calculate_aspect_ratio with exiftool.ExifTool() as et: File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 410, in enter self.start() File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 381, in start raise oe File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 372, in start self._process = subprocess.Popen( File "/usr/lib/python3.9/subprocess.py", line 951, in init self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/lib/python3.9/subprocess.py", line 1754, in _execute_child self.pid = _posixsubprocess.fork_exec( OSError: [Errno 12] Cannot allocate memory 2021-08-30 12:21:58,930 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/SarverHerringWedding-230.jpg 2021-08-30 12:21:59,486 : directory_watcher.py : handle_new_image : 134 : ERROR : job 39b9ca47-340a-4927-8dff-8692774c8db8: could not load image /data/SarverHerringWedding-230.jpg. reason: [Errno 12] Cannot allocate memory Traceback (most recent call last): File "/code/api/directory_watcher.py", line 106, in handle_new_image photo._calculate_aspect_ratio(False) File "/code/api/models/photo.py", line 200, in _calculate_aspect_ratio with exiftool.ExifTool() as et: File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 410, in enter self.start() File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 381, in start raise oe File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 372, in start self._process = subprocess.Popen( File "/usr/lib/python3.9/subprocess.py", line 951, in init self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/lib/python3.9/subprocess.py", line 1754, in _execute_child self.pid = _posixsubprocess.fork_exec( OSError: [Errno 12] Cannot allocate memory 2021-08-30 12:21:59,561 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_0433.jpeg 2021-08-30 12:22:00,267 : directory_watcher.py : handle_new_image : 134 : ERROR : job 39b9ca47-340a-4927-8dff-8692774c8db8: could not load image /data/IMG_0433.jpeg. reason: [Errno 12] Cannot allocate memory Traceback (most recent call last): File "/code/api/directory_watcher.py", line 106, in handle_new_image photo._calculate_aspect_ratio(False) File "/code/api/models/photo.py", line 200, in _calculate_aspect_ratio with exiftool.ExifTool() as et: File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 410, in enter self.start() File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 381, in start raise oe File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 372, in start self._process = subprocess.Popen( File "/usr/lib/python3.9/subprocess.py", line 951, in init self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/lib/python3.9/subprocess.py", line 1754, in _execute_child self.pid = _posixsubprocess.fork_exec( OSError: [Errno 12] Cannot allocate memory 2021-08-30 12:22:00,299 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_3516.jpg 2021-08-30 12:22:01,211 : directory_watcher.py : handle_new_image : 134 : ERROR : job 39b9ca47-340a-4927-8dff-8692774c8db8: could not load image /data/IMG_3516.jpg. reason: [Errno 12] Cannot allocate memory Traceback (most recent call last): File "/code/api/directory_watcher.py", line 106, in handle_new_image photo._calculate_aspect_ratio(False) File "/code/api/models/photo.py", line 200, in _calculate_aspect_ratio with exiftool.ExifTool() as et: File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 410, in enter self.start() File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 381, in start raise oe File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 372, in start self._process = subprocess.Popen( File "/usr/lib/python3.9/subprocess.py", line 951, in init self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/lib/python3.9/subprocess.py", line 1754, in _execute_child self.pid = _posixsubprocess.fork_exec( OSError: [Errno 12] Cannot allocate memory 2021-08-30 12:22:01,232 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_6670.jpg 2021-08-30 12:22:01,534 : directory_watcher.py : handle_new_image : 134 : ERROR : job 39b9ca47-340a-4927-8dff-8692774c8db8: could not load image /data/IMG_6670.jpg. reason: [Errno 12] Cannot allocate memory Traceback (most recent call last): File "/code/api/directory_watcher.py", line 106, in handle_new_image photo._calculate_aspect_ratio(False) File "/code/api/models/photo.py", line 200, in _calculate_aspect_ratio with exiftool.ExifTool() as et: File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 410, in enter self.start() File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 381, in start raise oe File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 372, in start self._process = subprocess.Popen( File "/usr/lib/python3.9/subprocess.py", line 951, in init self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/lib/python3.9/subprocess.py", line 1754, in _execute_child self.pid = _posixsubprocess.fork_exec( OSError: [Errno 12] Cannot allocate memory 2021-08-30 12:22:01,614 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_5720.heic 2021-08-30 12:22:03,791 : directory_watcher.py : handle_new_image : 134 : ERROR : job 39b9ca47-340a-4927-8dff-8692774c8db8: could not load image /data/IMG_5720.heic. reason: [Errno 12] Cannot allocate memory Traceback (most recent call last): File "/code/api/directory_watcher.py", line 106, in handle_new_image photo._calculate_aspect_ratio(False) File "/code/api/models/photo.py", line 200, in _calculate_aspect_ratio with exiftool.ExifTool() as et: File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 410, in enter self.start() File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 381, in start raise oe File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 372, in start self._process = subprocess.Popen( File "/usr/lib/python3.9/subprocess.py", line 951, in init self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/lib/python3.9/subprocess.py", line 1754, in _execute_child self.pid = _posixsubprocess.fork_exec( OSError: [Errno 12] Cannot allocate memory 2021-08-30 12:22:03,819 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_0372.jpg 2021-08-30 12:22:04,380 : directory_watcher.py : handle_new_image : 134 : ERROR : job 39b9ca47-340a-4927-8dff-8692774c8db8: could not load image /data/IMG_0372.jpg. reason: [Errno 12] Cannot allocate memory Traceback (most recent call last): File "/code/api/directory_watcher.py", line 106, in handle_new_image photo._calculate_aspect_ratio(False) File "/code/api/models/photo.py", line 200, in _calculate_aspect_ratio with exiftool.ExifTool() as et: File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 410, in enter self.start() File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 381, in start raise oe File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 372, in start self._process = subprocess.Popen( File "/usr/lib/python3.9/subprocess.py", line 951, in init self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/lib/python3.9/subprocess.py", line 1754, in _execute_child self.pid = _posixsubprocess.fork_exec( OSError: [Errno 12] Cannot allocate memory 2021-08-30 12:22:04,417 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_3310.jpg 2021-08-30 12:22:05,177 : directory_watcher.py : handle_new_image : 134 : ERROR : job 39b9ca47-340a-4927-8dff-8692774c8db8: could not load image /data/IMG_3310.jpg. reason: [Errno 12] Cannot allocate memory Traceback (most recent call last): File "/code/api/directory_watcher.py", line 106, in handle_new_image photo._calculate_aspect_ratio(False) File "/code/api/models/photo.py", line 200, in _calculate_aspect_ratio with exiftool.ExifTool() as et: File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 410, in enter self.start() File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 381, in start raise oe File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 372, in start self._process = subprocess.Popen( File "/usr/lib/python3.9/subprocess.py", line 951, in init__ self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/lib/python3.9/subprocess.py", line 1754, in _execute_child self.pid = _posixsubprocess.fork_exec( OSError: [Errno 12] Cannot allocate memory 2021-08-30 12:23:56,749 : api_util.py : get_location_clusters : 260 : INFO : location clustering took 0.01 seconds
Ahhh. I should have also noted that in this test I allowed the docker image to create the host folder for the volume mapping fo the /data map. It created it with root ownership and then I moved the photos to the folder instead of mapping to an existing folder.
Here is the state of some of the DB tables: longrunningjobs table and a csv os photo table.
Out of curiosity what exactly is/are clip embeddings? Just wondering what data this is capturing... api_photo.csv.txt
I was able to complete my test scan of 33 images by incrementally adding more RAM to the VM and starting clean each time. Once I reached 12GB it was able to process all 33 images and top showed a max usage of about 9GB. I have 1 heavy process worker and 2 regular workers configured in the .env file. That seems like a lot of RAM usage, but if it is expected please make a note for everyone.
@MatthK You could try creating a swapfile to "boost" your RAM. True it doesn't really boost it but it works. I went back to my setup and to conserve RAM from the host needed elsewhere I dropped my VM back to 4GB and added an 8GB swapfile and it worked. It wasn't as fast but it didn't error out with an out of memory like it does at 4GB or 8GB.
I'm not sure it's a RAM problem. The system uses less than 5GB, with almost 3GB still available. Is there somewhere in Docker that I can allocate more RAM to the container? And the Raspberry is not the fastest computer, so I don't expect it to be blazingly fast. Although the scan performance is quite slow.
@MatthK Docker by default should have access to all RAM/CPU capacity. As far as I know the only option available with docker is to limit resource usage. In my testing the docker container on my VM happily sucked up everything to the extent the VM was unresponsive to SSH or through the VM software 'monitor'... once I found the upper limit of what it would consume in my testing I was able to retain connectivity when it was scanning.
It'd be interesting to see if performance was better or worse if you added another worker and gave it extra swapfile capacity to offset the RAM the worker would use... Since you seem to have 3GB of capacity left you might be able to squeeze a little more out of the pi that way. It'd probably be CPU constrained but ...
@n8herring I pushed a new image on dev that should help with your memory issue.
@derneuere that made a huge difference! I saw the same exact test take max 3GB at it's peak and CPU usage was way down as well - about HALF actually!
-- I realize this is an off topic addition (to a closed topic). what determines if photos show in places? I can search places successfully but the map remains empty. The tables api_albumplace, api_albumplace_photos and api_photo are populated. Note that api_photo.exif_gps_lat and .exif_gps_lon are populated but .geolocation_json is just {} in case that matters.
"geolocation_json" has to be set. This is the mapbox response, where we get the cities/countries etc for a given lat/lon. We need this information to cluster the albums nicely.
I guess it doesn't save the photo but still creates the albums for places. I have to investigate this, because it does work for some people.
@derneuere Cleaned out test system did down/pull/up -d and places is working! Thanks for fixing that!
@derneuere Found an issue with how photos are displayed in places. Will create another issue to separate these threads.
🐛 Bug Report
What Operating system and version is LibrePhotos running on:
I just did a clean install on a Raspberry Pi4 with 8GB RAM. After some initial struggles, it eventually started up and I could set a directory with a few thousand photos (and videos). The initial scan was obviously slow, it took around 24 hours until the system was "responsive" again.
What architecture is LibrePhotos running on:
ARM
How is LibrePhotos installed:
Docker
If running via Docker or Kubernetes please list version including docker-compose:
How is you picture library mounted on the host (or in the virtual machine):
Local file system (Type), NFS, or SMB CIFS
Description of issue:
The defined Scan Directory has almost 2,300 items according to Windows Explorer, with 5 sub-directories containing a few more thousand photos (and videos). The library in Librephotos shows only 504 photos. I understand that some items might be no photos, but there are 1,877 JPG-files in the root directory alone. How come it does not include all the photos in the scan?