Extraction of EXIF data from non-jpg files is failing

kuzmos commented 3 years ago

🐛 Bug Report

[ ] I am including my log files in zip format.
[x] I have looked for similar issues among all the former issues including closed ones.

What Operating system and version is LibrePhotos running on:

VM Ubuntu lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.2 LTS Release: 20.04 Codename: focal

What architecture is LibrePhotos running on:

x64

How is LibrePhotos installed:

Docker

If running via Docker or Kubernetes please list version including docker-compose:

REPOSITORY TAG IMAGE ID CREATED SIZE reallibrephotos/librephotos dev 2 weeks ago 4.22GB redis latest 2 weeks ago 105MB postgres latest 2 weeks ago 314MB reallibrephotos/librephotos-frontend dev 5 weeks ago 1.32GB reallibrephotos/librephotos-proxy dev 5 weeks ago 133MB

docker -v Docker version 20.10.5, build 55c4c88 docker-compose -v docker-compose version 1.28.5, build c4eb3a1f

Are you running LibrePhotos on a virtual machine if so please list:

bhyve VM

How is you picture library mounted on the host (or in the virtual machine):

Local file system (Type), NFS, or SMB SMB

Description of issue:

When scanning for photos, non-jpeg files (Gif or BMP) produce the following errors in the log file: exifreader.py : rotate_image : 33 : ERROR : Error when grabbing exif data Traceback (most recent call last): File "/code/api/exifreader.py", line 11, in rotate_image image_exif = image._getexif() AttributeError: 'BmpImageFile' object has no attribute '_getexif'

or

exifreader.py : rotate_image : 33 : ERROR : Error when grabbing exif data Traceback (most recent call last): File "/code/api/exifreader.py", line 11, in rotate_image image_exif = image._getexif() AttributeError: 'GifImageFile' object has no attribute '_getexif'

The scan seems to continue after the errors.

How can we reproduce it:

Scan the directory with gif or bmp files

Additional Information:

Include a ZIP file containing your log files.
Provide pictures or links to videos that clearly demonstrate the issue.

tomamplius commented 3 years ago

This message is log after catch error when image not contains exif information. What is the issue?

kuzmos commented 3 years ago

Often the scanning crashes after those errors, however I'm not sure how to tell if the errors actually cause the crash.

tomamplius commented 3 years ago

Crash it's because process stop to progress?

kuzmos commented 3 years ago

I'm seeing this error, too - but after that the scan continued:

DSC_0015.JPG. reason: broken data stream when reading image file Traceback (most recent call last): File "/code/api/directory_watcher.py", line 154, in rescan_image photo._generate_thumbnail() File "/code/api/models/photo.py", line 159, in _generate_thumbnail image.thumbnail(ownphotos.settings.THUMBNAIL_SIZE_BIG, File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2336, in thumbnail im = self.resize(size, resample, box=box, reducing_gap=reducing_gap) File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 1916, in resize self.load() File "/usr/local/lib/python3.8/site-packages/PIL/ImageFile.py", line 284, in load raise_oserror(err_code) File "/usr/local/lib/python3.8/site-packages/PIL/ImageFile.py", line 67, in raise_oserror raise OSError(message + " when reading image file") OSError: broken data stream when reading image file

and then, after one of the errors AttributeError: 'GifImageFile' object has no attribute '_getexif' no new lines were appearing in the log file for a couple of hours until I started scanning again.

tomamplius commented 3 years ago

it's an old version of the backend? There is lot of update since this version. Can you upgrade?

kuzmos commented 3 years ago

Oh really? I'll do.

kuzmos commented 3 years ago

Maybe it's a lame question but how do I update a docker image without destroying my already created db? I've stopped all containers, updated docker-compose.yml and librephotos.env from git, added a newly appeared HEAVYWEIGHT_PROCESS option to my .env and ran docker-compose up -d. Now I have a python process eating around 4GB of RAM and the page is not up after 5 minutes. Anything particular to check in logs?

EDIT: the page eventually was up, I really have a lot of photos :) Just wanted to confirm that running docker-compose up -d is enough to update the images.

tomamplius commented 3 years ago

Sorry, i don't know docker. @Someone can't replay about docker upgrade? About memory, can you launch command "ps -eo pmem,pcpu,pid,args" ?

kuzmos commented 3 years ago

0.0 0.0 41328 /usr/bin/containerd-shim-runc-v2 -namespace moby -id <randomid> -address /run/containerd/containerd.sock 0.0 0.3 41350 redis-server *:6379 0.0 0.0 41381 /usr/bin/containerd-shim-runc-v2 -namespace moby -id <randomid> -address /run/containerd/containerd.sock 0.1 0.0 41405 postgres -c fsync=off -c synchronous_commit=off -c full_page_writes=off -c random_page_cost=1.0 2.2 0.0 41536 postgres: checkpointer 2.2 0.0 41537 postgres: background writer 0.0 0.0 41538 postgres: walwriter 0.0 0.0 41539 postgres: autovacuum launcher 0.0 0.0 41540 postgres: stats collector 0.0 0.0 41541 postgres: logical replication launcher 0.0 0.0 41558 /usr/bin/containerd-shim-runc-v2 -namespace moby -id <randomid> -address /run/containerd/containerd.sock 0.0 0.0 41579 /bin/bash /entrypoint.sh 13.0 1.0 41637 python image_similarity/main.py 0.0 0.0 41638 tee /logs/gunicorn_image_similarity.log 0.0 0.0 41666 /usr/bin/containerd-shim-runc-v2 -namespace moby -id <randomid> -address /run/containerd/containerd.sock 0.0 0.0 41688 bash /entrypoint.sh 0.0 0.0 41775 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port <my_port> -container-ip <ip> -container-port 80 0.0 0.0 41791 /usr/bin/containerd-shim-runc-v2 -namespace moby -id <randomid> -address /run/containerd/containerd.sock 0.0 0.0 41811 nginx: master process nginx -g daemon off; 0.0 0.0 41889 nginx: worker process 0.4 0.0 41893 node /usr/local/bin/serve build -d -l <my_port> 10.5 0.2 42063 python manage.py rqworker default 0.0 0.0 42064 tee /logs/rqworker.log 0.4 0.0 42065 /usr/local/bin/python /usr/local/bin/gunicorn --worker-class=gevent --timeout 3600 --bind backend:8001 --log-level=info ownphotos.wsgi 0.0 0.0 42066 tee /logs/gunicorn_django.log 11.8 4.4 42071 /usr/local/bin/python /usr/local/bin/gunicorn --worker-class=gevent --timeout 3600 --bind backend:8001 --log-level=info ownphotos.wsgi 0.0 0.0 42137 [kworker/u6:0+events_unbound] 0.0 0.0 42161 [kworker/u6:1-events_power_efficient] 2.4 2.3 42447 postgres: docker librephotos <ip>(60116) idle 0.0 0.0 42484 [kworker/u6:3-events_power_efficient] 0.0 0.0 42488 [kworker/0:0-mm_percpu_wq] 0.0 0.0 42529 [kworker/0:1-cifsiod] 0.0 0.0 42618 [kworker/1:0-cgroup_destroy] 0.0 0.0 42621 [kworker/1:1-mm_percpu_wq] 0.0 0.0 42697 ps -eo pmem,pcpu,pid,args

tomamplius commented 3 years ago

How much memory? photo already process ?

11.8 4.4 42071 /usr/local/bin/python /usr/local/bin/gunicorn --worker-class=gevent --timeout 3600 --bind backend:8001 --log-level=info ownphotos.wsgi
13.0 1.0 41637 python image_similarity/main.py
10.5 0.2 42063 python manage.py rqworker default

13% + 10% + 12% = 35% // 4G => ~12G?

13% for image_similarity it's lot i will check

kuzmos commented 3 years ago

At the moment of running ps it was taking around 2G. Yep, the VM has 6GB.

derneuere commented 3 years ago

This works now!

LibrePhotos / librephotos