search

LibrePhotos / librephotos

A self-hosted open source photo management service. This is the repository of the backend.
MIT License
6.93k stars 302 forks source link

Only part of photos scanned?! #328

Closed MatthK closed 2 years ago

MatthK commented 3 years ago

🐛 Bug Report

What Operating system and version is LibrePhotos running on:

I just did a clean install on a Raspberry Pi4 with 8GB RAM. After some initial struggles, it eventually started up and I could set a directory with a few thousand photos (and videos). The initial scan was obviously slow, it took around 24 hours until the system was "responsive" again.

What architecture is LibrePhotos running on:

ARM

How is LibrePhotos installed:

Docker

If running via Docker or Kubernetes please list version including docker-compose:

ubuntu@docker:~$ docker version
Client:
 Version:           20.10.7
 API version:       1.41
 Go version:        go1.13.8
 Git commit:        20.10.7-0ubuntu1~20.04.1
 Built:             Wed Aug  4 22:53:01 2021
 OS/Arch:           linux/arm64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.7
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.8
  Git commit:       20.10.7-0ubuntu1~20.04.1
  Built:            Wed Aug  4 19:07:47 2021
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.5.2-0ubuntu1~20.04.2
  GitCommit:
 runc:
  Version:          1.0.0~rc95-0ubuntu1~20.04.2
  GitCommit:
 docker-init:
  Version:          0.19.0
  GitCommit:

How is you picture library mounted on the host (or in the virtual machine):

Local file system (Type), NFS, or SMB CIFS

Description of issue:

The defined Scan Directory has almost 2,300 items according to Windows Explorer, with 5 sub-directories containing a few more thousand photos (and videos). The library in Librephotos shows only 504 photos. I understand that some items might be no photos, but there are 1,877 JPG-files in the root directory alone. How come it does not include all the photos in the scan?

n8herring commented 3 years ago

(Posting this in #308) I have had issues with getting photos to all scan (I had first added it as part of the reply to another issue (no photos show up under places #305) as scanning would never complete. I will also update #305 to keep things together there.

I have tried remove as many layers as possible in the docker setup by moving the test to a Debian9 VM (set with 4vcpu 4GB RAM) as opposed to MacOS which is just using a hidden Linux VM.

NOW I can see an error in the ownphotos.log where before I could not. It appears to just suck all of the memory and the whole VM becomes unresponsive...

---- HERE IS THE COMPLETE LOG FROM A CLEAN TRY ----

021-08-30 12:13:17,086 : views.py : list : 515 : INFO : querying & serializing took 0.01 seconds 2021-08-30 12:13:27,271 : serializers.py : update : 720 : INFO : Updated scan directory for user /data 2021-08-30 12:13:27,273 : serializers.py : update : 726 : INFO : Updated confidence for user 0.1 2021-08-30 12:13:27,274 : serializers.py : update : 736 : INFO : Updated semantic_search_topk for user 0 2021-08-30 12:13:27,275 : serializers.py : update : 742 : INFO : Updated favorite_min_rating for user 4 2021-08-30 12:13:43,409 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_3513.jpg 2021-08-30 12:13:45,702 : photo.py : _generate_captions : 143 : INFO : generated places365 captions for image /protected_media/thumbnails_big/dbd2bd73b451e048584f3082589e2ca71.webp. 2021-08-30 12:13:52,303 : photo.py : _generate_clip_embeddings : 117 : INFO : generated clip embeddings for image /protected_media/thumbnails_big/dbd2bd73b451e048584f3082589e2ca71.webp. 2021-08-30 12:13:53,592 : util.py : mapbox_reverse_geocode : 97 : INFO : mapbox returned status 200. 2021-08-30 12:13:54,889 : photo.py : _extract_faces : 362 : INFO : image dbd2bd73b451e048584f3082589e2ca71: 2 face(s) saved 2021-08-30 12:13:54,891 : directory_watcher.py : handle_new_image : 114 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: image processed: /data/IMG_3513.jpg, elapsed: 11.455016 2021-08-30 12:13:54,906 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_3065.jpg 2021-08-30 12:13:56,146 : photo.py : _generate_captions : 143 : INFO : generated places365 captions for image /protected_media/thumbnails_big/13f966e0d6ab83b3657878097a3ce1bc1.webp. 2021-08-30 12:14:03,017 : photo.py : _generate_clip_embeddings : 117 : INFO : generated clip embeddings for image /protected_media/thumbnails_big/13f966e0d6ab83b3657878097a3ce1bc1.webp. 2021-08-30 12:14:03,049 : directory_watcher.py : handle_new_image : 134 : ERROR : job 39b9ca47-340a-4927-8dff-8692774c8db8: could not load image /data/IMG_3065.jpg. reason: [Errno 12] Cannot allocate memory Traceback (most recent call last): File "/code/api/directory_watcher.py", line 109, in handle_new_image photo._extract_date_time_from_exif(True) File "/code/api/models/photo.py", line 211, in _extract_date_time_from_exif with exiftool.ExifTool() as et: File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 410, in enter self.start() File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 381, in start raise oe File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 372, in start self._process = subprocess.Popen( File "/usr/lib/python3.9/subprocess.py", line 951, in init self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/lib/python3.9/subprocess.py", line 1754, in _execute_child self.pid = _posixsubprocess.fork_exec( OSError: [Errno 12] Cannot allocate memory 2021-08-30 12:14:03,240 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_9411.HEIC.heif 2021-08-30 12:14:05,848 : directory_watcher.py : handle_new_image : 134 : ERROR : job 39b9ca47-340a-4927-8dff-8692774c8db8: could not load image /data/IMG_9411.HEIC.heif. reason: [Errno 12] Cannot allocate memory Traceback (most recent call last): File "/code/api/directory_watcher.py", line 106, in handle_new_image photo._calculate_aspect_ratio(False) File "/code/api/models/photo.py", line 200, in _calculate_aspect_ratio with exiftool.ExifTool() as et: File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 410, in enter self.start() File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 381, in start raise oe File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 372, in start self._process = subprocess.Popen( File "/usr/lib/python3.9/subprocess.py", line 951, in init self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/lib/python3.9/subprocess.py", line 1754, in _execute_child self.pid = _posixsubprocess.fork_exec( OSError: [Errno 12] Cannot allocate memory 2021-08-30 12:14:05,913 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_2143.HEIC.heif 2021-08-30 12:14:08,147 : directory_watcher.py : handle_new_image : 134 : ERROR : job 39b9ca47-340a-4927-8dff-8692774c8db8: could not load image /data/IMG_2143.HEIC.heif. reason: [Errno 12] Cannot allocate memory Traceback (most recent call last): File "/code/api/directory_watcher.py", line 106, in handle_new_image photo._calculate_aspect_ratio(False) File "/code/api/models/photo.py", line 200, in _calculate_aspect_ratio with exiftool.ExifTool() as et: File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 410, in enter self.start() File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 381, in start raise oe File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 372, in start self._process = subprocess.Popen( File "/usr/lib/python3.9/subprocess.py", line 951, in init self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/lib/python3.9/subprocess.py", line 1754, in _execute_child self.pid = _posixsubprocess.fork_exec( OSError: [Errno 12] Cannot allocate memory 2021-08-30 12:14:08,202 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/567712203.458368public.jpg 2021-08-30 12:14:08,713 : directory_watcher.py : handle_new_image : 134 : ERROR : job 39b9ca47-340a-4927-8dff-8692774c8db8: could not load image /data/567712203.458368public.jpg. reason: [Errno 12] Cannot allocate memory Traceback (most recent call last): File "/code/api/directory_watcher.py", line 106, in handle_new_image photo._calculate_aspect_ratio(False) File "/code/api/models/photo.py", line 200, in _calculate_aspect_ratio with exiftool.ExifTool() as et: File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 410, in enter self.start() File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 381, in start raise oe File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 372, in start self._process = subprocess.Popen( File "/usr/lib/python3.9/subprocess.py", line 951, in init self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/lib/python3.9/subprocess.py", line 1754, in _execute_child self.pid = _posixsubprocess.fork_exec( OSError: [Errno 12] Cannot allocate memory 2021-08-30 12:14:08,836 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_2386.heic 2021-08-30 12:14:11,886 : directory_watcher.py : handle_new_image : 134 : ERROR : job 39b9ca47-340a-4927-8dff-8692774c8db8: could not load image /data/IMG_2386.heic. reason: [Errno 12] Cannot allocate memory Traceback (most recent call last): File "/code/api/directory_watcher.py", line 106, in handle_new_image photo._calculate_aspect_ratio(False) File "/code/api/models/photo.py", line 200, in _calculate_aspect_ratio with exiftool.ExifTool() as et: File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 410, in enter self.start() File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 381, in start raise oe File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 372, in start self._process = subprocess.Popen( File "/usr/lib/python3.9/subprocess.py", line 951, in init self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/lib/python3.9/subprocess.py", line 1754, in _execute_child self.pid = _posixsubprocess.fork_exec( OSError: [Errno 12] Cannot allocate memory 2021-08-30 12:14:11,952 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_8176.jpg 2021-08-30 12:14:12,717 : directory_watcher.py : handle_new_image : 134 : ERROR : job 39b9ca47-340a-4927-8dff-8692774c8db8: could not load image /data/IMG_8176.jpg. reason: [Errno 12] Cannot allocate memory Traceback (most recent call last): File "/code/api/directory_watcher.py", line 106, in handle_new_image photo._calculate_aspect_ratio(False) File "/code/api/models/photo.py", line 200, in _calculate_aspect_ratio with exiftool.ExifTool() as et: File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 410, in enter self.start() File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 381, in start raise oe File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 372, in start self._process = subprocess.Popen( File "/usr/lib/python3.9/subprocess.py", line 951, in init self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/lib/python3.9/subprocess.py", line 1754, in _execute_child self.pid = _posixsubprocess.fork_exec( OSError: [Errno 12] Cannot allocate memory 2021-08-30 12:14:12,778 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_5830.heic 2021-08-30 12:14:14,758 : directory_watcher.py : handle_new_image : 134 : ERROR : job 39b9ca47-340a-4927-8dff-8692774c8db8: could not load image /data/IMG_5830.heic. reason: [Errno 12] Cannot allocate memory Traceback (most recent call last): File "/code/api/directory_watcher.py", line 106, in handle_new_image photo._calculate_aspect_ratio(False) File "/code/api/models/photo.py", line 200, in _calculate_aspect_ratio with exiftool.ExifTool() as et: File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 410, in enter self.start() File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 381, in start raise oe File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 372, in start self._process = subprocess.Popen( File "/usr/lib/python3.9/subprocess.py", line 951, in init self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/lib/python3.9/subprocess.py", line 1754, in _execute_child self.pid = _posixsubprocess.fork_exec( OSError: [Errno 12] Cannot allocate memory 2021-08-30 12:14:14,826 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_7527.heic 2021-08-30 12:14:17,863 : directory_watcher.py : handle_new_image : 134 : ERROR : job 39b9ca47-340a-4927-8dff-8692774c8db8: could not load image /data/IMG_7527.heic. reason: [Errno 12] Cannot allocate memory Traceback (most recent call last): File "/code/api/directory_watcher.py", line 106, in handle_new_image photo._calculate_aspect_ratio(False) File "/code/api/models/photo.py", line 200, in _calculate_aspect_ratio with exiftool.ExifTool() as et: File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 410, in enter self.start() File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 381, in start raise oe File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 372, in start self._process = subprocess.Popen( File "/usr/lib/python3.9/subprocess.py", line 951, in init self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/lib/python3.9/subprocess.py", line 1754, in _execute_child self.pid = _posixsubprocess.fork_exec( OSError: [Errno 12] Cannot allocate memory 2021-08-30 12:14:17,922 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_0224.jpg 2021-08-30 12:14:18,552 : directory_watcher.py : handle_new_image : 134 : ERROR : job 39b9ca47-340a-4927-8dff-8692774c8db8: could not load image /data/IMG_0224.jpg. reason: [Errno 12] Cannot allocate memory Traceback (most recent call last): File "/code/api/directory_watcher.py", line 106, in handle_new_image photo._calculate_aspect_ratio(False) File "/code/api/models/photo.py", line 200, in _calculate_aspect_ratio with exiftool.ExifTool() as et: File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 410, in enter self.start() File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 381, in start raise oe File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 372, in start self._process = subprocess.Popen( File "/usr/lib/python3.9/subprocess.py", line 951, in init self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/lib/python3.9/subprocess.py", line 1754, in _execute_child self.pid = _posixsubprocess.fork_exec( OSError: [Errno 12] Cannot allocate memory 2021-08-30 12:14:18,627 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_5896.heic 2021-08-30 12:14:21,493 : directory_watcher.py : handle_new_image : 134 : ERROR : job 39b9ca47-340a-4927-8dff-8692774c8db8: could not load image /data/IMG_5896.heic. reason: [Errno 12] Cannot allocate memory Traceback (most recent call last): File "/code/api/directory_watcher.py", line 106, in handle_new_image photo._calculate_aspect_ratio(False) File "/code/api/models/photo.py", line 200, in _calculate_aspect_ratio with exiftool.ExifTool() as et: File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 410, in enter self.start() File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 381, in start raise oe File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 372, in start self._process = subprocess.Popen( File "/usr/lib/python3.9/subprocess.py", line 951, in init self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/lib/python3.9/subprocess.py", line 1754, in _execute_child self.pid = _posixsubprocess.fork_exec( OSError: [Errno 12] Cannot allocate memory 2021-08-30 12:14:21,512 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/WP_20151218_13_30_28_Rawhighres.dng 2021-08-30 12:19:54,456 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_3297.heic 2021-08-30 12:19:58,838 : photo.py : _generate_captions : 143 : INFO : generated places365 captions for image /protected_media/thumbnails_big/c8c4c21ce392750f27807efc3e098bcd1.webp. 2021-08-30 12:20:05,393 : photo.py : _generate_clip_embeddings : 117 : INFO : generated clip embeddings for image /protected_media/thumbnails_big/c8c4c21ce392750f27807efc3e098bcd1.webp. 2021-08-30 12:20:06,945 : util.py : mapbox_reverse_geocode : 97 : INFO : mapbox returned status 200. 2021-08-30 12:20:08,988 : photo.py : _extract_faces : 362 : INFO : image c8c4c21ce392750f27807efc3e098bcd1: 1 face(s) saved 2021-08-30 12:20:08,990 : directory_watcher.py : handle_new_image : 114 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: image processed: /data/IMG_3297.heic, elapsed: 14.474443 2021-08-30 12:20:09,022 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/WP_20171127_21_17_10_Pro__highres.jpg 2021-08-30 12:20:10,789 : photo.py : _generate_captions : 143 : INFO : generated places365 captions for image /protected_media/thumbnails_big/576e7b15b9ca9a795945f70f2d6336c31.webp. 2021-08-30 12:20:16,222 : photo.py : _generate_clip_embeddings : 117 : INFO : generated clip embeddings for image /protected_media/thumbnails_big/576e7b15b9ca9a795945f70f2d6336c31.webp. 2021-08-30 12:20:17,072 : util.py : mapbox_reverse_geocode : 97 : INFO : mapbox returned status 200. 2021-08-30 12:20:19,459 : photo.py : _extract_faces : 362 : INFO : image 576e7b15b9ca9a795945f70f2d6336c31: 1 face(s) saved 2021-08-30 12:20:19,460 : directory_watcher.py : handle_new_image : 114 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: image processed: /data/WP_20171127_21_17_10_Prohighres.jpg, elapsed: 10.394164 2021-08-30 12:20:19,516 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_6338.heic 2021-08-30 12:20:22,407 : photo.py : _generate_captions : 143 : INFO : generated places365 captions for image /protected_media/thumbnails_big/a7569195075a165f4813dc3e6dc2b93a1.webp. 2021-08-30 12:20:27,582 : photo.py : _generate_clip_embeddings : 117 : INFO : generated clip embeddings for image /protected_media/thumbnails_big/a7569195075a165f4813dc3e6dc2b93a1.webp. 2021-08-30 12:20:28,813 : util.py : mapbox_reverse_geocode : 97 : INFO : mapbox returned status 200. 2021-08-30 12:20:29,922 : photo.py : _extract_faces : 362 : INFO : image a7569195075a165f4813dc3e6dc2b93a1: 1 face(s) saved 2021-08-30 12:20:29,923 : directory_watcher.py : handle_new_image : 114 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: image processed: /data/IMG_6338.heic, elapsed: 10.397034 2021-08-30 12:20:29,941 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/WP_20160823_19_25_11_Rawhighres.dng 2021-08-30 12:21:37,015 : photo.py : _generate_captions : 143 : INFO : generated places365 captions for image /protected_media/thumbnails_big/6e3aa849ab5f116973f9220a42f956ff1.webp. 2021-08-30 12:21:43,208 : photo.py : _generate_clip_embeddings : 117 : INFO : generated clip embeddings for image /protected_media/thumbnails_big/6e3aa849ab5f116973f9220a42f956ff1.webp. 2021-08-30 12:21:46,049 : directory_watcher.py : handle_new_image : 114 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: image processed: /data/WP_20160823_19_25_11_Rawhighres.dng, elapsed: 75.926134 2021-08-30 12:21:46,087 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_0404.jpg 2021-08-30 12:21:46,866 : photo.py : _generate_captions : 143 : INFO : generated places365 captions for image /protected_media/thumbnails_big/4f4e6ff4bc394df5d5222216e3fb28401.webp. 2021-08-30 12:21:52,186 : photo.py : _generate_clip_embeddings : 117 : INFO : generated clip embeddings for image /protected_media/thumbnails_big/4f4e6ff4bc394df5d5222216e3fb28401.webp. 2021-08-30 12:21:53,849 : photo.py : _extract_faces : 362 : INFO : image 4f4e6ff4bc394df5d5222216e3fb28401: 1 face(s) saved 2021-08-30 12:21:53,850 : directory_watcher.py : handle_new_image : 114 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: image processed: /data/IMG_0404.jpg, elapsed: 7.750198 2021-08-30 12:21:53,935 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_5584.heic 2021-08-30 12:21:56,280 : directory_watcher.py : handle_new_image : 134 : ERROR : job 39b9ca47-340a-4927-8dff-8692774c8db8: could not load image /data/IMG_5584.heic. reason: [Errno 12] Cannot allocate memory Traceback (most recent call last): File "/code/api/directory_watcher.py", line 106, in handle_new_image photo._calculate_aspect_ratio(False) File "/code/api/models/photo.py", line 200, in _calculate_aspect_ratio with exiftool.ExifTool() as et: File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 410, in enter__ self.start() File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 381, in start raise oe File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 372, in start self._process = subprocess.Popen( File "/usr/lib/python3.9/subprocess.py", line 951, in init self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/lib/python3.9/subprocess.py", line 1754, in _execute_child self.pid = _posixsubprocess.fork_exec( OSError: [Errno 12] Cannot allocate memory 2021-08-30 12:21:56,340 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/SarverHerringWedding-72.jpg 2021-08-30 12:21:56,946 : directory_watcher.py : handle_new_image : 134 : ERROR : job 39b9ca47-340a-4927-8dff-8692774c8db8: could not load image /data/SarverHerringWedding-72.jpg. reason: [Errno 12] Cannot allocate memory Traceback (most recent call last): File "/code/api/directory_watcher.py", line 106, in handle_new_image photo._calculate_aspect_ratio(False) File "/code/api/models/photo.py", line 200, in _calculate_aspect_ratio with exiftool.ExifTool() as et: File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 410, in enter self.start() File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 381, in start raise oe File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 372, in start self._process = subprocess.Popen( File "/usr/lib/python3.9/subprocess.py", line 951, in init self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/lib/python3.9/subprocess.py", line 1754, in _execute_child self.pid = _posixsubprocess.fork_exec( OSError: [Errno 12] Cannot allocate memory 2021-08-30 12:21:57,011 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_6369.heic 2021-08-30 12:21:58,906 : directory_watcher.py : handle_new_image : 134 : ERROR : job 39b9ca47-340a-4927-8dff-8692774c8db8: could not load image /data/IMG_6369.heic. reason: [Errno 12] Cannot allocate memory Traceback (most recent call last): File "/code/api/directory_watcher.py", line 106, in handle_new_image photo._calculate_aspect_ratio(False) File "/code/api/models/photo.py", line 200, in _calculate_aspect_ratio with exiftool.ExifTool() as et: File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 410, in enter self.start() File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 381, in start raise oe File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 372, in start self._process = subprocess.Popen( File "/usr/lib/python3.9/subprocess.py", line 951, in init self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/lib/python3.9/subprocess.py", line 1754, in _execute_child self.pid = _posixsubprocess.fork_exec( OSError: [Errno 12] Cannot allocate memory 2021-08-30 12:21:58,930 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/SarverHerringWedding-230.jpg 2021-08-30 12:21:59,486 : directory_watcher.py : handle_new_image : 134 : ERROR : job 39b9ca47-340a-4927-8dff-8692774c8db8: could not load image /data/SarverHerringWedding-230.jpg. reason: [Errno 12] Cannot allocate memory Traceback (most recent call last): File "/code/api/directory_watcher.py", line 106, in handle_new_image photo._calculate_aspect_ratio(False) File "/code/api/models/photo.py", line 200, in _calculate_aspect_ratio with exiftool.ExifTool() as et: File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 410, in enter self.start() File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 381, in start raise oe File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 372, in start self._process = subprocess.Popen( File "/usr/lib/python3.9/subprocess.py", line 951, in init self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/lib/python3.9/subprocess.py", line 1754, in _execute_child self.pid = _posixsubprocess.fork_exec( OSError: [Errno 12] Cannot allocate memory 2021-08-30 12:21:59,561 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_0433.jpeg 2021-08-30 12:22:00,267 : directory_watcher.py : handle_new_image : 134 : ERROR : job 39b9ca47-340a-4927-8dff-8692774c8db8: could not load image /data/IMG_0433.jpeg. reason: [Errno 12] Cannot allocate memory Traceback (most recent call last): File "/code/api/directory_watcher.py", line 106, in handle_new_image photo._calculate_aspect_ratio(False) File "/code/api/models/photo.py", line 200, in _calculate_aspect_ratio with exiftool.ExifTool() as et: File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 410, in enter self.start() File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 381, in start raise oe File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 372, in start self._process = subprocess.Popen( File "/usr/lib/python3.9/subprocess.py", line 951, in init self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/lib/python3.9/subprocess.py", line 1754, in _execute_child self.pid = _posixsubprocess.fork_exec( OSError: [Errno 12] Cannot allocate memory 2021-08-30 12:22:00,299 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_3516.jpg 2021-08-30 12:22:01,211 : directory_watcher.py : handle_new_image : 134 : ERROR : job 39b9ca47-340a-4927-8dff-8692774c8db8: could not load image /data/IMG_3516.jpg. reason: [Errno 12] Cannot allocate memory Traceback (most recent call last): File "/code/api/directory_watcher.py", line 106, in handle_new_image photo._calculate_aspect_ratio(False) File "/code/api/models/photo.py", line 200, in _calculate_aspect_ratio with exiftool.ExifTool() as et: File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 410, in enter self.start() File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 381, in start raise oe File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 372, in start self._process = subprocess.Popen( File "/usr/lib/python3.9/subprocess.py", line 951, in init self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/lib/python3.9/subprocess.py", line 1754, in _execute_child self.pid = _posixsubprocess.fork_exec( OSError: [Errno 12] Cannot allocate memory 2021-08-30 12:22:01,232 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_6670.jpg 2021-08-30 12:22:01,534 : directory_watcher.py : handle_new_image : 134 : ERROR : job 39b9ca47-340a-4927-8dff-8692774c8db8: could not load image /data/IMG_6670.jpg. reason: [Errno 12] Cannot allocate memory Traceback (most recent call last): File "/code/api/directory_watcher.py", line 106, in handle_new_image photo._calculate_aspect_ratio(False) File "/code/api/models/photo.py", line 200, in _calculate_aspect_ratio with exiftool.ExifTool() as et: File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 410, in enter self.start() File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 381, in start raise oe File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 372, in start self._process = subprocess.Popen( File "/usr/lib/python3.9/subprocess.py", line 951, in init self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/lib/python3.9/subprocess.py", line 1754, in _execute_child self.pid = _posixsubprocess.fork_exec( OSError: [Errno 12] Cannot allocate memory 2021-08-30 12:22:01,614 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_5720.heic 2021-08-30 12:22:03,791 : directory_watcher.py : handle_new_image : 134 : ERROR : job 39b9ca47-340a-4927-8dff-8692774c8db8: could not load image /data/IMG_5720.heic. reason: [Errno 12] Cannot allocate memory Traceback (most recent call last): File "/code/api/directory_watcher.py", line 106, in handle_new_image photo._calculate_aspect_ratio(False) File "/code/api/models/photo.py", line 200, in _calculate_aspect_ratio with exiftool.ExifTool() as et: File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 410, in enter self.start() File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 381, in start raise oe File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 372, in start self._process = subprocess.Popen( File "/usr/lib/python3.9/subprocess.py", line 951, in init self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/lib/python3.9/subprocess.py", line 1754, in _execute_child self.pid = _posixsubprocess.fork_exec( OSError: [Errno 12] Cannot allocate memory 2021-08-30 12:22:03,819 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_0372.jpg 2021-08-30 12:22:04,380 : directory_watcher.py : handle_new_image : 134 : ERROR : job 39b9ca47-340a-4927-8dff-8692774c8db8: could not load image /data/IMG_0372.jpg. reason: [Errno 12] Cannot allocate memory Traceback (most recent call last): File "/code/api/directory_watcher.py", line 106, in handle_new_image photo._calculate_aspect_ratio(False) File "/code/api/models/photo.py", line 200, in _calculate_aspect_ratio with exiftool.ExifTool() as et: File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 410, in enter self.start() File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 381, in start raise oe File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 372, in start self._process = subprocess.Popen( File "/usr/lib/python3.9/subprocess.py", line 951, in init self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/lib/python3.9/subprocess.py", line 1754, in _execute_child self.pid = _posixsubprocess.fork_exec( OSError: [Errno 12] Cannot allocate memory 2021-08-30 12:22:04,417 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_3310.jpg 2021-08-30 12:22:05,177 : directory_watcher.py : handle_new_image : 134 : ERROR : job 39b9ca47-340a-4927-8dff-8692774c8db8: could not load image /data/IMG_3310.jpg. reason: [Errno 12] Cannot allocate memory Traceback (most recent call last): File "/code/api/directory_watcher.py", line 106, in handle_new_image photo._calculate_aspect_ratio(False) File "/code/api/models/photo.py", line 200, in _calculate_aspect_ratio with exiftool.ExifTool() as et: File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 410, in enter self.start() File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 381, in start raise oe File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 372, in start self._process = subprocess.Popen( File "/usr/lib/python3.9/subprocess.py", line 951, in init__ self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/lib/python3.9/subprocess.py", line 1754, in _execute_child self.pid = _posixsubprocess.fork_exec( OSError: [Errno 12] Cannot allocate memory 2021-08-30 12:23:56,749 : api_util.py : get_location_clusters : 260 : INFO : location clustering took 0.01 seconds

n8herring commented 3 years ago

Ahhh. I should have also noted that in this test I allowed the docker image to create the host folder for the volume mapping fo the /data map. It created it with root ownership and then I moved the photos to the folder instead of mapping to an existing folder.

n8herring commented 3 years ago

Here is the state of some of the DB tables: longrunningjobs table and a csv os photo table.

Out of curiosity what exactly is/are clip embeddings? Just wondering what data this is capturing... image api_photo.csv.txt

n8herring commented 3 years ago

I was able to complete my test scan of 33 images by incrementally adding more RAM to the VM and starting clean each time. Once I reached 12GB it was able to process all 33 images and top showed a max usage of about 9GB. I have 1 heavy process worker and 2 regular workers configured in the .env file. That seems like a lot of RAM usage, but if it is expected please make a note for everyone.

MatthK commented 3 years ago

I started scan process again and it is now still running. I am at 1,8xx photos, and probably taking another week or so.

Web-capture-31-8-2021-231149-192-168-7-17 PS: The numbers are still hidden as the red is not large enough. Maybe that can be improved?

It seems that a Raspberry Pi is not really up to the job to scan thousands of photos.

n8herring commented 3 years ago

@MatthK You could try creating a swapfile to "boost" your RAM. True it doesn't really boost it but it works. I went back to my setup and to conserve RAM from the host needed elsewhere I dropped my VM back to 4GB and added an 8GB swapfile and it worked. It wasn't as fast but it didn't error out with an out of memory like it does at 4GB or 8GB.

MatthK commented 3 years ago

I'm not sure it's a RAM problem. The system uses less than 5GB, with almost 3GB still available. Is there somewhere in Docker that I can allocate more RAM to the container? And the Raspberry is not the fastest computer, so I don't expect it to be blazingly fast. Although the scan performance is quite slow.

n8herring commented 3 years ago

@MatthK Docker by default should have access to all RAM/CPU capacity. As far as I know the only option available with docker is to limit resource usage. In my testing the docker container on my VM happily sucked up everything to the extent the VM was unresponsive to SSH or through the VM software 'monitor'... once I found the upper limit of what it would consume in my testing I was able to retain connectivity when it was scanning.

It'd be interesting to see if performance was better or worse if you added another worker and gave it extra swapfile capacity to offset the RAM the worker would use... Since you seem to have 3GB of capacity left you might be able to squeeze a little more out of the pi that way. It'd probably be CPU constrained but ...

derneuere commented 3 years ago

@n8herring I pushed a new image on dev that should help with your memory issue.

n8herring commented 3 years ago

@derneuere that made a huge difference! I saw the same exact test take max 3GB at it's peak and CPU usage was way down as well - about HALF actually!

-- I realize this is an off topic addition (to a closed topic). what determines if photos show in places? I can search places successfully but the map remains empty. The tables api_albumplace, api_albumplace_photos and api_photo are populated. Note that api_photo.exif_gps_lat and .exif_gps_lon are populated but .geolocation_json is just {} in case that matters.

derneuere commented 3 years ago

"geolocation_json" has to be set. This is the mapbox response, where we get the cities/countries etc for a given lat/lon. We need this information to cluster the albums nicely.

I guess it doesn't save the photo but still creates the albums for places. I have to investigate this, because it does work for some people.

n8herring commented 3 years ago

@derneuere Cleaned out test system did down/pull/up -d and places is working! Thanks for fixing that!

n8herring commented 3 years ago

@derneuere Found an issue with how photos are displayed in places. Will create another issue to separate these threads.