Metadata / identifier strategy / Avoid to build same identifier

[fxprunayre]

Current strategy is to build metadata identifier based on file name. If 2 files have same name, this will produce same record identifier. This was done on purpose in order to be able to update metadata record on future scan.

In some case, there will be duplicate filename and another strategy should be used.

Options: 1) Build on filepath (but if the file location change, a new identifier will be generated) 2) ?

[mcoudert]

For some RASTER products, the filename is always the same but the parent folder allows to identify the product. It could be an alternative for such products?

[sppigot]

Yep - currently uuid of file is generated from sha of filename. Would be nice if sha included date of modification as well (since some files will have the same name). Currently title is filename, but directory path often includes useful metadata - so directory path (minus context data_dir) would be better to use as title of metadata record.

Ultimately it might be better to have the crawler running as a thread which monitors the context data_dir using java7 watchdir and:

• cache file state in a jcs instance (stashes the jcs on a filesystem when it is stopped, restores it when it starts) - could also use another persistence like a h2 db I suppose but maybe a bit too heavyweight for this in terms of config
• updates or generates new metadata records according to the changes returned by watchdir
• some uncertainty over whether watchdir can handle file move operations correctly? Use jpathwatch instead as per http://jpathwatch.wordpress.com/

[fxprunayre]

Add an option to configure the strategy between the following 2 options:

• based on filename (current implementation and keep for backward compatibility)
• based on filepath (after the datadir)

Setting name is : uuidCreatedFromFilePath (if true, the default, the strategy is based on filepath)

Eg.

• data dir is E:/data/
• Filepath is E:/data/3_ORTHO/_ORTHOPHOTO_SION/10CM/demo.tiff
• UUID is defined on
  • demo.tiff if uuidCreatedFromFilePath = false
  • 3_ORTHO/_ORTHOPHOTO_SION/10CM/demo.tiff if uuidCreatedFromFilePath = true

[fxprunayre]

Closing even if more advanced strategy could be adopted.