LibreCat / Catmandu

Catmandu - a data processing toolkit
https://librecat.org
175 stars 31 forks source link

Path generator #349

Closed nicolasfranck closed 6 years ago

coveralls commented 6 years ago

Coverage Status

Coverage increased (+0.4%) to 94.0% when pulling a26aef72a24ff5e8a6666f5b22309165f1150209 on path_generator into 572da246ef07cc8c8e556ab58b4fcb3fd89fcb39 on dev.

coveralls commented 6 years ago

Coverage Status

Coverage decreased (-1.7%) to 93.128% when pulling 98687658ca8a39f3a5621f8aec1498ff45d4a47f on path_generator into 8012106b3a1b842fffbb9c4e7d41a20cd2881c14 on dev.

nicolasfranck commented 6 years ago

ok changes committed

phochste commented 6 years ago

Thanks, I have a final remark about creating an MD5 out of an _id in the Map implementation. I can understand you want to support _id containing spaces, slashes and other weird characters which would mess up the directory structure when _id would be use verbatim as directory names. What concerns me a bit it that from the MD5 one can't figure out what the original _id was, it is a one-way solution. If by some accident the mapping SQLite database gets corrupted/lost, then one looses access to all these _id/directory mappings.

In other words, wouldn't it be a nice feature to recreate the SQLite mapping database from the data found in the filesystem? E.g. not use a hashing function but an encryption function like Crypt::DES or something that can work two ways?

nics commented 6 years ago

Another option is to constrain _id to non problematic characters. Added advantage is that the path is navigetable from the command line without tooling.

nicolasfranck commented 6 years ago

Indeed. Anyway, the tool will continue to work for those records that are already mapped in the database (the path is never recomputed), but it should be indeed possible to "restore" the mapping (only stored in the bag) from the command line. I'll fix that.

phochste commented 6 years ago

Changes moved to : https://github.com/LibreCat/Catmandu-FileStore