Open graphixillusion opened 3 weeks ago
alucryd
commented
3 weeks ago Thanks for the suggestion. Do you mean you'd like to manually move some files outside of oxyromon? I'd rather have the archiving mechanic be a part of oxyromon, maybe supporting some popular cloud storage protocols would be a nice first step. I wouldn't be able to distinguish archived files from accidentally deleted files and it would totally break purge-roms.
graphixillusion
commented
3 weeks ago Thanks to you for the support! Exactly, let's assume that i'm starting from a green dat so 100% complete. This option should mark the dat and all the content as archived. From now on, you can move/archive this roms whichever you want, on external hard drives, bluray, tapes, cloud: whenever a new update of the dat comes out wich adds/remove/renames something, all the new stuff will be stored in the current scanning folder and what's archived is not touched becouse the files are not there but the scan should acts as they are in it. So i can still updating this archived set even if i have cold stored all the files somewhere else: at this point if at a later time i will merge the archived files previously stored with the current scan folder which has just what's new (somesort of diff update) i can reconstruct the full 100% set. Conceptually speaking i was thinking to a method which shoudn't break too much in the current architecture and this method should use sparse files. Sparse files allows us to make dummy files which are logically of the same size of the archived counterpart but which has 0 byte of physical space occupied on the hdd. So everything should works as before, we just need to ignore checksums on all the files which have the archived flag. Talking about the archived flag, a simple solution should be something like this:
| Dat Rom Room Folder\ | -.Archived\ <--- every archived files should have a sparse dummy copy in this folder with a 0 byte physical size on the hdd (skip hash checksum becouse the dat doesn't recognize them) | |
|---|---|---|
| -roms <--- what's new/updated/changed is putted here |
of course this is just an idea. What do you think about it?
alucryd
commented
2 weeks ago That's a good idea. I'd still want the archiving process to be part of oxyromon, via a archive-roms or archive-systems sub command. It would take a list of systems, and a destination, local or remote. That way the process can get tracked, and it could be applied even to partial sets. Roms would get flagged as archived in the database, and importing a changed DAT would automatically remove the flag from changed files and mark them as missing for further reimport. I need to investigate how sparse files are handled by rsync-like commands, be it regular rsync, gs rsync, etc... Using rsync would simplify the archiving and diff process quite a lot. An option to keep local files would be nice too for people who just want to have a backup and keep files locally to keep using them.
While I write that I realize there's some overlap with the existing export-roms sub command, maybe I could extend that one.
I think this feature should help who wants/needs to cold archive roms, expecially the big sets which are not update very often. Let's say that i want to archive a big set somewhere (tape archive, cloud, backups, etc...) but this set is currently being update/modified: the archived files are not physically present anymore in the scan folder but the program should treat them as they were still present, so only what's new/update/modified is currently written in the current folder. If an archived roms needs a rename, only the archived reference is renamed; if an archived set with multiple roms inside a single zip (think to MAME for example) needs to be update with new roms, in the current folder a zip with just the new modifications/additions is made.