Scanning photos does not work at all (0.1.8)

beccon4 commented 4 years ago

In the hope to end my photo scanning woes and to be up to date, I raised my Nextcloud to 20.0.0 and with that Maps to 0.1.8.

Then I

cleared my file-cache
and scanned it again. (not without pruning the file tree a bit - getting rid of obsolete files) occ:files:scan
then run occ maps:scan-photos beccon

There are around 300.000 files in my tree mainly consists of two SMB/CIFS external shares of which one is of particular interest as it contains all my images uploaded from the phone - so they all contain geotags. Both filescan and scan-photos result in the file trees get traversed - the photo scan shows that it has read quite a few entries.

But nothing appears in the app. Why?

Further analysis shows: select count(*) from oc_maps_photos where user_id='beccon' -> 191 (which is a tiny fraction)

select count(*) from oc_maps_photos where user_id='beccon' and lat is not null; -> at least they should appear

There is no discrepancy between the filecache and the photo cache anymore: select count(*) from oc_filecache fc join oc_maps_photos mp on mp.file_id=fc.fileid where mp.user_id='beccon'; -> 191

and it shows my four geotagged images which my UI denies showing select count(*) from oc_filecache fc join oc_maps_photos mp on mp.file_id=fc.fileid where mp.user_id='beccon' and mp.lat is not null; ->4

tacruc commented 4 years ago

So there are two possible issues:

First scanning is not programmed or tested against external storages. Mighed work might not.
Second maps:scan foto needs time as it just schedules the real scan.

beccon4 commented 4 years ago

Well - it should work shouldn't it as I can see it in the log messages originating from the scan-photo command. So I suppose they get recognised:

...
[13417] Photo "/beccon/files/MobilBild/2020/02/IMG_20200204_143751_1.jpg" added
[13418] Photo "/beccon/files/MobilBild/2020/02/IMG_20200204_144715_0.jpg" added
[13419] Photo "/beccon/files/MobilBild/2020/02/IMG_20200204_144715_1.jpg" added
[13420] Photo "/beccon/files/MobilBild/2020/02/IMG_20200204_144715_2.jpg" added
[13421] Photo "/beccon/files/MobilBild/2020/02/IMG_20200206_122727_0.jpg" added
[13422] Photo "/beccon/files/MobilBild/2020/02/IMG_20200206_122727_1.jpg" added
[13423] Photo "/beccon/files/MobilBild/2020/02/IMG_20200206_122727_2.jpg" added
[13424] Photo "/beccon/files/MobilBild/2020/02/IMG_20200209_124205_0.jpg" added
[13425] Photo "/beccon/files/MobilBild/2020/02/IMG_20200209_124205_1.jpg" added
[13426] Photo "/beccon/files/MobilBild/2020/02/IMG_20200209_124205_2.jpg" added
[13427] Photo "/beccon/files/MobilBild/2020/02/IMG_20200211_083432_0.jpg" added
[13428] Photo "/beccon/files/MobilBild/2020/02/IMG_20200211_083432_1.jpg" added
[13429] Photo "/beccon/files/MobilBild/2020/02/IMG_20200211_083432_2.jpg" added
[13430] Photo "/beccon/files/MobilBild/2020/02/IMG_20200212_184447_0.jpg" added
[13431] Photo "/beccon/files/MobilBild/2020/02/IMG_20200212_184447_1.jpg" added
[13432] Photo "/beccon/files/MobilBild/2020/02/IMG_20200212_184447_2.jpg" added
[13433] Photo "/beccon/files/MobilBild/2020/02/IMG_20200214_151142_0.jpg" added
[13434] Photo "/beccon/files/MobilBild/2020/02/IMG_20200214_151142_1.jpg" added
[13435] Photo "/beccon/files/MobilBild/2020/02/IMG_20200214_151142_2.jpg" added
[13436] Photo "/beccon/files/MobilBild/2020/02/IMG_20200214_193222_2.jpg" added
[13437] Photo "/beccon/files/MobilBild/2020/02/IMG_20200214_193244_1.jpg" added
[13438] Photo "/beccon/files/MobilBild/2020/02/IMG_20200214_193343_0.jpg" added
[13439] Photo "/beccon/files/MobilBild/2020/02/IMG_20200214_193343_1.jpg" added
[13440] Photo "/beccon/files/MobilBild/2020/02/IMG_20200214_193430_0.jpg" added
....

Neither of these files appear in the database though:

MariaDB [nextcloud]> select * from oc_filecache fc join oc_maps_photos mp on mp.file_id=fc.fileid where fc.name like '%IMG_2020%';
Empty set (0.04 sec)

So what can happen that prevents the scan command to actually include the image files into its table oc_maps_photos?

These files are in the oc_filecache:

MariaDB [nextcloud]> select fileid,storage,path,mimetype,mimepart,storage_mtime from oc_filecache fc where name like '%IMG_20200214_1934%';
+---------+---------+------------------------------------------+----------+----------+---------------+
| fileid  | storage | path                                     | mimetype | mimepart | storage_mtime |
+---------+---------+------------------------------------------+----------+----------+---------------+
| 2790421 |      40 | 2020/02/Camera/IMG_20200214_193430_2.jpg |       12 |       11 |    1581701870 |
| 2790427 |      40 | 2020/02/Camera/IMG_20200214_193430_1.jpg |       12 |       11 |    1581701893 |
| 2790433 |      40 | 2020/02/Camera/IMG_20200214_193430_0.jpg |       12 |       11 |    1581701914 |
| 2790438 |      40 | 2020/02/IMG_20200214_193430_2.jpg        |       12 |       11 |    1581701931 |
| 2790445 |      40 | 2020/02/IMG_20200214_193430_1.jpg        |       12 |       11 |    1581701953 |
| 2790451 |      40 | 2020/02/IMG_20200214_193430_0.jpg        |       12 |       11 |    1581701979 |
+---------+---------+------------------------------------------+----------+----------+---------------+
6 rows in set (0.13 sec)

so missing scan in the first place can't be the cause.

And it did work. It initially scanned my MobilBild external storage folder - but then got stuck as running out of disk space. Then I increased my virtual hard disk - moved the machine to another host (IP changed) - scanned again - the filescanner manually and then the scan-photos. And yes, there are files of other users and files from another external storage. What else can I do?

Any ideas? Thanks for help.

tacruc commented 4 years ago

AS mentioned multiple times maps scan phots is not filling up the maps database, but creating backgoubdjoba to do so. I fyou check the Jobs table there should be plenty oft Jobs.

beccon4 commented 4 years ago

occ maps:scan-photos puts entries into the scheduler table oc_jobs - they got processed later and disappear. Why won't they got included into the oc_maps_photos? And why do entries in that table do not appear in the UI?

j-ed commented 4 years ago

AS mentioned multiple times maps scan phots is not filling up the maps database, but creating backgoubdjoba to do so. I fyou check the Jobs table there should be plenty oft Jobs.

I can confirm what @tacruc wrote, that the photo scan don't fill-up the database directly, but adds background jobs. In my case e.g. around 12.000 background jobs had been created and the system have started to execute them one-by-one. All my photos are stored on an external share and were processed without any problem.

I have to admit, that the displayed messages raised a different expectation at the beginning too. Additionally it was a little bit disturbing that the whole picture counter was dropped to e.g. 30 write after executing the occ command instead of just adding the new files, but I think this is a different story.

beccon4 commented 4 years ago

Additionally it was a little bit disturbing that the whole picture counter was dropped to e.g. 30 write after executing the occ command instead of just adding the new files, but I think this is a different story. Nice to hear that you managed to import your photos. Meanwhile I found out myself that scanning is triggered in the background and an executed later. I could reproduce by querying oc_jobs constantly. There, in the parameter column all my images appear- Looking forward to see them in the UI under "your photos" led to a disappointment. Somewhere they disappear - but where?

I rebuilt all indices various times - no avail. What could be the cause? Which routine is started from the scheduler (hard to find out due to that much indirection) to actually enter the data? I even have entries in my oc_maps_photos table which have geotags and are attached to my user account but do not appear in the UI. What's wrong here? Any ideas? Thanks for any hint in advance.

tacruc commented 4 years ago

https://github.com/nextcloud/maps/blob/master/lib/BackgroundJob/AddPhotoJob.php

Is the cron.php configured correctly, or are you using ajax cron?

beccon4 commented 4 years ago

https://github.com/nextcloud/maps/blob/master/lib/BackgroundJob/AddPhotoJob.php

Is the cron.php configured correctly, or are you using ajax cron?

Cron works fine. Mails arrive. Files get scanned too.

My Crontab looks like this:

*/10 * * * * php -f /var/www/nextcloud/cron.php

bendilobi commented 4 years ago

I've got the same problem. "occ maps:scan-photos" says it added 366 photos but they never appear (oc_maps_photos is empty).

At the same time I have now 18073 jobs in oc_jobs, almost all of them "OCA\Maps\BackgroundJob\UpdatePhotoByFileJob". Running "occ maps:scan-photos" doesn't increase the number of jobs here...

Nextcloud complains there are jobs that are not executed for days. I'm using webcron with one call every 5 Minutes.

My installation is very fresh, installed one week ago, with just one user. Not using any external storage.

beccon4 commented 4 years ago

So I'm not the only one suffering

MrHappy commented 3 years ago

I had a lot of jobs that were created, but never were run (Nextcloud on a Asustor in Docker), so I ran the cron.php in Docker. Now the jobs have a 'last run' timestamp, but the table oc_maps_photos was not filled...

beccon4 commented 3 years ago

And the code isn't that straight forward to find the issue quickly

adrhc commented 2 months ago

yes, it's confusing; I thought scan adds the photos to the map but instead:

SELECT class, count(*) FROM oc_jobs GROUP BY class;
OCA\Maps\BackgroundJob\AddPhotoJob                                |   149108

tacruc commented 3 weeks ago

The extraction is done by a background per photo. As there was issues that certain images crashed the php exit lib and thereby uncatchable the php process (one images implied others where never scanned.) Similarly memory leaks in the used libraries, accumulated over the images and lead to out of memory errors. ( Implied again no way to scan all photos). If this happens now, the Background jobs and cron is taking care to continue scanning at the next image.

If I remember correctly there is an option to force the scan command to scan directly. Since a few versions nextclout extracts the metadata itself. I didn't have time to migrate the maps app to that source for metadata.

nextcloud / maps

Scanning photos does not work at all (0.1.8) #478