Upgrade to Python 3.8

[derneuere]

Fixes #112 and makes building a docker image easier.

Some Jobs are no longer used, because they don't function together with places365.py and face_classify.py. I could not find a reason why. I had to refactor some calls so that it feels the same as before.

You can find up to date docker images under https://hub.docker.com/u/derneuere to confirm that it still works for you.

[MrColumbo]

Thanks for creating the docker images. Could you please add some hints about what port mappings and directory mapping somebody should do to run the docker images. I mean usually I do it like docker run p:80:80 -v /mydir:/dockerdir derneuere/ownphotos .....something like that. I am wondering if the original yml file still applies https://raw.githubusercontent.com/hooram/ownphotos/dev/docker-compose.yml.template and just the images from the frontend and backend needs to get replaced with your images?

[derneuere]

I didn't change any features regarding port mappings and directory mapping. The old docker-compose.yml.template should work. It does on my system. I found a couple of issues and updated the image :)

[sysoppl]

I know PR is not an issue thread, but I tried to run your docker but i'm stuck on login page. I'm using the old docker-compose template with backend host set to domain. I also tried ip:port, domain:port, domain, "domain" but nothing works :/ It always redirect to local ip. I really don't know from where is this IP, because docker containers have 172.23.* It was tested on fresh ubuntu server. Any ideas?

[tenent007963]

Ever tried reinstalling it?

[derneuere]

@MrColumbo Thanks for testing! 👍 I found the issue in the frontend. You don't need BACKEND_HOST anymore, linking is now done via ownphotos-proxy. Please download the latest docker image for the frontend.

[MrColumbo]

@zainlol it was actually sysoppl who reported the issue with the broken backend connection. But i got the same issue and just pulled the latest frontend. I do not get that backend connection error anymore but just a blank white page.

In the log directory I noticed that the most recent log is the command_build_similarity_index.log and there I found something interesting:

Traceback (most recent call last): File "/miniconda/lib/python3.8/site-packages/django/db/backends/utils.py", line 86, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "api_user" does not exist LINE 1: ...word", "api_user"."nextcloud_scan_directory" FROM "api_user"

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "manage.py", line 22, in execute_from_command_line(sys.argv) File "/miniconda/lib/python3.8/site-packages/django/core/management/init.py", line 401, in execute_from_command_line utility.execute()

.....

not sure if that is related or helpfull at all ;)

thanks for your quick support

[MrColumbo]

P.S.: The blank page actually is not that blank if I look at the source ....it contains thing like that 404

The requested path could not be found

[derneuere]

@MrColumbo I noticed, that if I called something like "localhost:3000/login" as the URL in a new tab, that I would always get a 404 error / a blank page. Could you maybe try to just call the your ownphotos url without pathing?

The error in command_build_similarity_index.log seems like a new error. Could you check if the docker container ownphotos still runs after the error and send the other logs like ownphotos.log?

[MrColumbo]

I get forwarded to /login automatically when I go to /

ownphotos.log is zero bytes so far ;)

rqworker.log gunicorn_django.log command_build_similarity_index.log command_migrate.log command_makemigrations.log show_migrate.log

[MrColumbo]

P.S.: the error does not stop any container ...they have been and are still all running

[derneuere]

@MrColumbo I can't seem to reproduce the error. Could you run docker-compose up without the -d? It should print all the logs of all the containers. Could you paste the whole console content after you hit the error?

[MrColumbo]

docker-compose output.txt

I have to confess that I tried to login twice since I noticed a lot of action on the output window and wanted to check if the page is still white but did not found the tab ;)

[derneuere]

@MrColumbo Did you already ran ownphotos with an old version or are you trying to set it up for the first time?

[MrColumbo]

i started setting it up today .....never used it before ...

[sysoppl]

Same, blank page after login page Chrome says:

Fresh ubuntu 20.04, unfortunately without logs because the server was deleted

[derneuere]

@MrColumbo @sysoppl The original dev never pushed some files needed for initialization. Took me some time to figure that one out. I took them from an old docker build. Now it works.

[MrColumbo]

looks much better now :+1: .....just a bit confused about the photo directory. In the admin area it says in the Scan Directory field "Not Set" but it feels like I can not set it there. In the "normal" setting there is field to set the scan directory but there it says "only an admin can set it" ....funny thing is that so far there is only the admin user (the one from the docker compose file) I copied some photos in the directory which is mapped via docker compose to /data but they did not seem to get picked by the crawler so far ......

[derneuere]

@MrColumbo Yep, the current flow is a bit janky. If something doesn't update, it is probably because you have no authorisation. Open the webconsole and look for a 401 (Unauthorized) Error. Just logout and login again and the things should the show current progress. I will investigate that issue, because it wasn't there a couple of commits ago.

Go to the admin area and click on the red icon:

Choose the folder listed below and click on update:

Go then to the settings page and click on "Start Scanning":

In the admin area should be a process. After a while you will see how many pictures it is trying to scan:

[MrColumbo]

thanks - logging off and logging back in did it

[MrColumbo]

How long should it take before i see a percentage on the job in the admin area? So far I just see the spinning sign on the left but the progress bar is zero.

The last few lines in the ownphotos.log are those

2020-11-16 07:53:40,819 : directory_watcher.py : handle_new_image : 93 : INFO : adding to AlbumDate took 0.02 2020-11-16 07:53:41,253 : directory_watcher.py : handle_new_image : 99 : INFO : adding to AlbumThing took 0.43 2020-11-16 07:53:41,562 : directory_watcher.py : handle_new_image : 105 : INFO : im2vec took 0.31 2020-11-16 07:53:41,562 : directory_watcher.py : handle_new_image : 107 : INFO : job b4169a78-f6cd-4e8d-8bc8-6ed80b565bae: image processed: /data/2020-07-24_172235DC-GX9.JPG, elapsed: {"md5": 0.156857, "thumbnails": 0.946377, "captions": 0.250041, "image_save": null, "exif": null, "geolocation": null, "faces": 0.175098, "album_place": null, "album_date": 0.024898, "album_thing": 0.434043, "im2vec": 0.308912} 2020-11-16 07:53:41,569 : directory_watcher.py : handle_new_image : 33 : INFO : job b4169a78-f6cd-4e8d-8bc8-6ed80b565bae: handling image /data/2020-07-24_172236DC-GX9.JPG 2020-11-16 07:53:42,579 : directory_watcher.py : handle_new_image : 64 : INFO : thumbnail get took 0.92 2020-11-16 07:53:42,828 : models.py : _generate_captions : 225 : INFO : generated places365 captions for image /code/protected_media/thumbnails/a7dccab6c52fbbe35287265268d60f8e1.jpg. 2020-11-16 07:53:42,828 : directory_watcher.py : handle_new_image : 70 : INFO : caption generation took 0.25 2020-11-16 07:53:42,828 : directory_watcher.py : handle_new_image : 76 : INFO : image save took 0.25 2020-11-16 07:53:42,838 : directory_watcher.py : handle_new_image : 81 : INFO : add to AlbumPlace took 0.25 2020-11-16 07:53:43,017 : directory_watcher.py : handle_new_image : 87 : INFO : face extraction took 0.18 2020-11-16 07:53:43,037 : directory_watcher.py : handle_new_image : 93 : INFO : adding to AlbumDate took 0.02 2020-11-16 07:53:43,663 : directory_watcher.py : handle_new_image : 99 : INFO : adding to AlbumThing took 0.63 2020-11-16 07:53:43,929 : directory_watcher.py : handle_new_image : 105 : INFO : im2vec took 0.27 2020-11-16 07:53:43,929 : directory_watcher.py : handle_new_image : 107 : INFO : job b4169a78-f6cd-4e8d-8bc8-6ed80b565bae: image processed: /data/2020-07-24_172236DC-GX9.JPG, elapsed: {"md5": 0.068334, "thumbnails": 0.918049, "captions": 0.248971, "image_save": null, "exif": null, "geolocation": null, "faces": 0.179427, "album_place": null, "album_date": 0.019563, "album_thing": 0.625552, "im2vec": 0.265566} 2020-11-16 07:53:43,937 : directory_watcher.py : scan_photos : 174 : INFO : Added 40 photos 2020-11-16 07:53:43,937 : image_similarity.py : build_image_similarity_index : 27 : INFO : builing similarity index for user admin 2020-11-16 07:53:47,454 : views.py : list : 963 : INFO : querying & serializing took 0.15 seconds

the rworker.log looks like that

21:48:22 Worker rq:worker:9758af7c63ec472cb9a77aa6dd4065df: started, version 1.6.1 21:48:22 Subscribing to channel rq:pubsub:9758af7c63ec472cb9a77aa6dd4065df 21:48:22 *** Listening on default... 21:48:22 Cleaning registries for queue: default 22:15:22 Cleaning registries for queue: default ... (same line many times with different time) 08:36:28 Cleaning registries for queue: default 09:03:28 Cleaning registries for queue: default

the image similar log looks like that

2020-11-15 15:18:44,144 : main.py : : 50 : INFO : starting server 2020-11-15 17:11:20,605 : main.py : : 50 : INFO : starting server 2020-11-15 19:03:45,100 : main.py : : 50 : INFO : starting server 2020-11-15 19:52:20,955 : retrieval_index.py : build_index_for_user : 17 : INFO : building index for user 1 - got 0 photos to process 2020-11-15 19:52:21,018 : retrieval_index.py : build_index_for_user : 27 : INFO : finished building index for user 1 - took 0.05 seconds 2020-11-15 21:47:20,050 : main.py : : 50 : INFO : starting server 2020-11-15 21:48:06,794 : retrieval_index.py : build_index_for_user : 17 : INFO : building index for user 1 - got 0 photos to process 2020-11-15 21:48:06,810 : retrieval_index.py : build_index_for_user : 27 : INFO : finished building index for user 1 - took 0.01 seconds 2020-11-15 21:49:23,882 : retrieval_index.py : build_index_for_user : 17 : INFO : building index for user 1 - got 0 photos to process 2020-11-15 21:49:23,934 : retrieval_index.py : build_index_for_user : 27 : INFO : finished building index for user 1 - took 0.03 seconds 2020-11-16 07:53:45,103 : retrieval_index.py : build_index_for_user : 17 : INFO : building index for user 1 - got 39 photos to process 2020-11-16 07:53:45,397 : retrieval_index.py : build_index_for_user : 27 : INFO : finished building index for user 1 - took 0.28 seconds

and that is the gunicorn similarity log

127.0.0.1 - - [2020-11-15 21:48:06] "POST /build/ HTTP/1.1" 200 139 0.017821 127.0.0.1 - - [2020-11-15 21:49:23] "POST /build/ HTTP/1.1" 200 139 0.053638 127.0.0.1 - - [2020-11-16 07:53:45] "POST /build/ HTTP/1.1" 200 140 0.376323

the other log files have not been updated 30801933 0 -rw-r--r-- 1 root root 0 Nov 15 22:48 command_build_similarity_index.log 30801931 4 -rw-r--r-- 1 root root 20 Nov 15 22:47 command_makemigrations.log 30801932 4 -rw-r--r-- 1 root root 144 Nov 15 22:47 command_migrate.log 30801935 4 -rw-r--r-- 1 root root 349 Nov 15 22:48 gunicorn_django.log 30801928 4 -rw-r--r-- 1 root root 231 Nov 16 08:53 gunicorn_image_similarity.log 30801929 4 -rw-r--r-- 1 root root 1385 Nov 16 08:53 image_similarity.log 30801930 72 -rw-r--r-- 1 root root 69577 Nov 16 08:53 ownphotos.log 30801934 4 -rw-r--r-- 1 root root 1491 Nov 16 10:30 rqworker.log 30801927 4 -rw-r--r-- 1 root root 687 Nov 15 22:48 show_migrate.log

In the photo section i see blank thumbnails when i go for photos with timestamps (there are 39 photos)

[derneuere]

You have to click the reload button to get the current status. The log says that your scan is done. The blank thumbnails seem like an authorization problem. I created a new image yesterday to improve this behavior. Maybe try logoff and login again 👍

[MrColumbo]

logoff and back login again did it again. Next time I will try that before asking ;) I noticed that no places are identified even each picture has GPS info and on a single picture view it shows the correct map. Since i used the access token in the composer file where is asks MAPBOX_API_KEY I am thinking that maybe the access token is something completely different. Should the album places function work? Is it dependent the mapbox key?

[derneuere]

@MrColumbo I found the error. I have no idea if it ever worked, but it works now :D It should now read the coordinates on new pictures if you have a valid maxbox key. There is also a geocoder based on openstreet maps, where you don't need an api key, but the json response is different. But that could be a nice feature for the future :)

There is currently no way to start geocoding for pictures already scanned.