Indexing of @eadir folders

curnacat commented 10 years ago

First let me thank you Didier for this Script as it is very useful.

As I see the script is indexing all the folders under the "photo" directory. However, if you are using "photo Station", Synology creates some folders with the name @eadir where it puts all the thumbnails to access the photos.

As I understand those folders should not be in the database.

bonidier commented 10 years ago

Hi curnacat,

thanks for your feedback,

Yes, indexing process should ignore eaDir foldersin all shared folders. Did you find an eaDir folder in your database ?

curnacat commented 10 years ago

Thanks,

I haven't looked directly to the database, but checking the log (/opt/etc/init.d/S99synodlna-reindex-inotify log) I find plenty of "adding new file: .../@eadir/Img_***.jpg/SYNOPHOTO_THUMB_PREVIEW.JPG.

It seems as synodlna-reindex is adding the contents of the @eadir directories.

bonidier commented 10 years ago

current script ignore only eaDir folder, now fixed

see commit 69944ed9c8e23aaf84d6f5898d562420071acbae

refresh your copy,
restart S99synodlna-reindex-inotify service if running

Thanks !

curnacat commented 10 years ago

Thanks Didier.

First of all let me tell you that I'm not an expert in linux nor programming although I may have some basic knowledge. Sorry if sometimes there are some things too basic that I don't catch.

In the previous post I made a mistake while writing, the directory was "@eaDir" and not "@eadir" as I wrote". However, I keep having some strange behavior with the service. I will try to post the problem.

As you can see, although the service is installed, it tells me as it is not started, nor I can start it. However, if I ask for the log, it seems it is running and it's when it tells me that is adding some files from @eaDir directories.

ServidorCentral> /opt/etc/init.d/S99synodlna-reindex-inotify status
bad or empty PID
status: not started
ServidorCentral> /opt/etc/init.d/S99synodlna-reindex-inotify start
bad or empty PID
PID=20228
/opt/etc/init.d/S99synodlna-reindex-inotify: line 92: can't create /opt/var/run/S99synodlna-reindex-inotify.pid: nonexistent directory                                                                                              
bad or empty PID
something bad happened, check /var/tmp/synodlna-reindex.log log
ServidorCentral> /opt/etc/init.d/S99synodlna-reindex-inotify stop
bad or empty PID
service not started
ServidorCentral> /opt/etc/init.d/S99synodlna-reindex-inotify log
Hum OK, let's catch your activity...
Setting up watches.  Beware: since -r was given, this may take a while!
Adding new file: /volume1/photo/Fotografies/Marga/holanda 02/Den Bosch/@eaDir/DSC02392.JPG/SYNOPHOTO_THUMB_PREVIEW.jpg                                                                                                 
Adding new file: /volume1/photo/Fotografies/Marga/holanda 02/Den Bosch/@eaDir/DSC02393.JPG/SYNOPHOTO_THUMB_PREVIEW.jpg                                                                                                 
Adding new file: /volume1/photo/Fotografies/Marga/holanda 02/Den Bosch/@eaDir/DSC02394.JPG/SYNOPHOTO_THUMB_PREVIEW.jpg                                                                                                 
Adding new file: /volume1/photo/Fotografies/Marga/holanda 02/Den Bosch/@eaDir/DSC02397.JPG/SYNOPHOTO_THUMB_PREVIEW.jpg                                                                                                 
Adding new file: /volume1/photo/Fotografies/Marga/holanda 02/Den Bosch/@eaDir/DSC02398.JPG/SYNOPHOTO_THUMB_PREVIEW.jpg                                                                                                 
Adding new file: /volume1/photo/Fotografies/Marga/holanda 02/Den Bosch/@eaDir/DSC02399.JPG/SYNOPHOTO_THUMB_PREVIEW.jpg

Although on the log was telling me that it was adding some files from "eaDir directories", when I make a query to the database those do not appear, as they really where not added. I show you the code I used to make the query so you can see it.

ServidorCentral> psql -U postgres
psql (9.3.4)
Type "help" for help.
postgres=# \c mediaserver
You are now connected to database "mediaserver" as user "postgres".
mediaserver=# select * from photo where path like '%@ea%';
 id | path | title | filesize | album | resolutionx | resolutiony | camera_make | camera_model | exposure | aperture | iso | date | timetaken | mdate | fs_uuid | fs_online
----+------+-------+----------+-------+-------------+-------------+-------------+--------------+----------+----------+-----+------+-----------+-------+---------+-----------
(0 rows)
mediaserver=#
mediaserver=# select * from photo where path like '%Mar%';
   id   |                                                              path                                                              |                             title                             | filesize |                           album                            | resolutionx | resolutiony |       camera_make       |           camera_model           |     exposure      | aperture | iso  |        date         |      timetaken      |        mdate        | fs_uuid | fs_online
--------+--------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------+----------+------------------------------------------------------------+-------------+-------------+-------------------------+----------------------------------+-------------------+----------+------+---------------------+---------------------+---------------------+---------+-----------
 305646 | /volume1/photo/Fotografies/Comunes/2013-09 Roger amb 1 mes/Margalida's _IMG_1031.JPG                                           | Margalida's _IMG_1031                                         |   120184 | 2013-09 Roger amb 1 mes                                    |         800 |         600 |                         |                                  |                   |          |    0 | 2013-12-11 01:33:47 | 1970-01-01 00:00:00 |

bonidier commented 10 years ago

Hi curnacat,

I think you've found a 2 bugs, thanks for your help !

S99synodlna-reindex-inotify

I've re-check my README file process about requirements : the PID path for service is missing

I've updated Makefile for :

create PID path if needed
always copy init.d script from sources, better if I need to update launch script in future

the eaDir exclusion

note : the '@' before a string is mapped to a user in GitHub, that's why your eaDir goes to lowercase eadir

I've removed lowercase eadir exclusion :-)
I think I've found the eaDir exclusion bug for synodlna-reindex-inotify, only directory were excluded

process to apply all this fix

as admin user,

1/ kill current bad inotifywait process if running :

sudo ps w | grep inotifywait
# if return something,
sudo killall inotifywait

2/ refresh your copy of synodlna-index 3/ fix service requirements

make service

4/ start S99synodlna-reindex-inotify

bonidier commented 10 years ago

just a precision I'll add into README file : when launching the service, you should wait for this message (some seconds, minutes, depends of the directories count) into service's log, just after "Setting up watches"

Watches established

curnacat commented 10 years ago

Thank you Didier.

Now the service it's working and I do not find anymore strange log messages with the @eaDir directories.

However I'm finding something strange, sorry to bother you once again...

I have run the script synodlna-reindex.sh and it showed to me there were some changes to make. Removing some obsolete files and adding new ones. At least that is what it tells the screen log.

After finishing, I ran once againg the script thinking it would find that the database was completely synchronized and it would tell me there were no changes to make. However my surprise was to find that the database was not synchronized and told me to remove the same number of files and add the same number too.

In summary, although the log tells me is adding and removing files it is not really doing it, so the database is not really changed. I don't know if it may have anything to do with the fact that I had to add the admin superuser to postgres or anything else.

bonidier commented 10 years ago

Hi curnacat,

I'm aware of this behavior,

In fact the script is just calling synoindex's Synology binary, which add files to sync with DB in a queue file, Each file analyze from synoindex binaries may take a little time before DB inserts. So, when my script says that a file is processed, it's purely asynchronous with real DB state

(see /var/spool/syno_indexing_queue for current queue when the script is launched)

may be I'll found a way to get the diff between which is sync in DB and queue file

bonidier / synodlna-index