Tool to monitor and manage local disk space

[dkirkby]

This issue is to discuss and then implement a new script (called bosslocal ?) that will scan your $DATA_LOCAL_ROOT and report things like:

• Total number of files
• Total number of Mb
• Number of files & bytes by file type (spec, spec-lite, frame, cframe, metadata)

Ideally, there would be options to only include files that have (or have not) been accessed in X days, etc, similar to the unix find command.

The script should also have options to do some cleanup of the largest files that have not been accessed recently, etc.

[dcunning11235]

I think https://github.com/dkirkby/bossdata/issues/52 has bearing on this. Reversing part of my comment there, I'm thinking e.g. bosslocal is expanded to a general e.g. bossmgr util that allows for management of the local 'raw' files, the DB's (addition, removal, listing of indexes), archiving/deletion, etc. from one cmd line util.

[dkirkby]

I agree that the new command-line tool should know about the raw files that backup each sqlite db and be able to manage them intelligently. For example, a --prune option might delete the raw file as long as the db file is present. Perhaps bosslocalmgr for the name?

[dcunning11235]

Waiting on my virtual machine drive to backup so I can resize it is giving me ample free time; here is a rough rundown on functionality I can think of, part plan, part question, part wishlist:

• File system
  • Info/Display
  • Total number of files (all, by catalog, SDSS file type, date range)
    • Include as possible criteria PLATE, MJD?
  • Total file size (all, by catalog, file type, date range)
    • Include as possible criteria PLATE, MJD?
  • BOSS environment/path settings
  • Management
  • Remove files ((?)all, by catalog, SDSS file type, date range)
    • Should exclude DB files from this?
    • ...or include extra "Are you sure?" kind of prompts?
    • But we should have an "uninstall" kind of option that removes ALL data files
  • (?) Archive files (by catalog, file type, date range)
    • Files are always available from sdss servers, so needed?
    • This introduces further layers of complication: Do we use one big gzip, try to ensure no duplication of files? What if someone requests a file that is part of archive?
    • Make this part of a second iteration, if at all.
  • Prune (remove) files that have been converted to DB's
• DB (All below: per catalog)
  • Info/Display
  • Print out indexes
  • Filesys sizes of DBs
  • Management
  • Drop all indexes (other than default/PK)
  • Create new indexes, specified from file
  • Drop individual indexes
    • ...or only "drop all" and then add back in as needed?
  • Compact DB (because removing indexes doesn't automatically free filesystem space)
• (?) Environment
  • Set environment/path variable
  • Including appending to e.g. .bashrc?
  • Allow for switching between 'presets' for e.g. QUASAR, BOSS, etc.
  • Display BOSS environment/path settings

[dcunning11235]

Features as actually implemented:

• File Management
  • Total number and size of file
  • via --total-files
  • displays results grouped by file type (spec, spec-lite, plate, frame, cframe, meta, other)
  • Delete files
  • via --delete
  • files targeted can be selected via UNIX find-like options: -amin, -atime, -cmin, -ctime
  • files targeted can also be selected via simple wildcard (?,*,[char_list]) filename matching via -filename
  • A dry-run is available via --delete-test (perhaps change to --delete-dry-run?)
  • --delete will quite happily delete all your many hours/GB's of downloaded data files. Be careful!
  • Functionality that needs to be added before initial release
  • make --total-files work with same targeting options as --delete
• DB Management
  • Performance/Indexes
  • View indexes via --db-index-print
  • Add indexes via --db-index-update FILENAME, where FILENAME points at a file consisting of lines formatted like: catalog_name.index_name(column_name_1,column_name_2,...,column_name_n)
  • Delete ALL indexes (except PK's) via --db-index-reset
  • DB size/diskspace
  • --db-compact causes DB files to be resized (useful if indexes have been deleted and no new equivalent ones recreated; this frees space that would otherwise not be automatically freed.)
  • --prune removed metadata files that have already been imported into DB's
• bosslocalmgr settings
  • --verbose, as usual
  • --catalogs is a comma delimited list specifying the catalogs (DBs) the DB commands will work on, e.g. --catalogs FULL,QUASAR
  • Right now, only supported strings are FULL, LITE, QUASAR.
  • Conflating catalog with DB/metadata type which are not precisely the same thing (LITE vs FULL vs. QUASAR.) Should change this before first release?
  • As noted above, could make this work with file ops like --delete in future version
  • --env dumps environment vars related bossdata for those who don't like grep
  • set-env temporarily overrides environment settings with the defaults for the specified CATALOG name. Originally intended to allow setting of parent process/shell environment, but that is on hold (maybe indefinitely). Kept option to allow change of process environment before any other commands are processed.

Next Iteration

• --delete does not remove empty directories; it should
• --delete could probably use a little more protection: "Are you sure?"-style checks, double confirms on deleting DB files; or, display a summary (a la --delete-test in compact form) and only delete on confirm?
• --delete will happily delete everything under a path. This should probably not be the default behavior!!!
• Include alternate file search (and display) criteria
  • selection by catalog (only e.g. QUASAR or LITE)
  • selection by metadata (e.g. PLATE or MJD)
• Try to include environment updating?
  • Should this include (or instead) be writing new environment settings to e.g. a ~/.bossdatarc file that is called from ~/.bashrc... Or is this boiling the ocean?
• Left index deletion out on purpose, but perhaps next iteration includes this (?)

Thrown out

• Archiving
• Separate DB file size display

[dkirkby]

A few quick comments based on your notes (but I haven't looked at any code yet):

• I would rename --delete-test to --dry-run since a lot of unix commands already use this convention.
• Total file size in different categories is usually what we care about rather than total number of files or the size of individual files (in case this helps simplify or prioritize the API).
• We should check how -amin, -atime, -cmin, -ctime translate to actual usage:
  • Does a newly downloaded file get its creation time set to the download time?
  • Does opening a file (via the SpecFile ctor, for example) actually update the last access time?
  • Since bossdata treats all downloaded files as read-only, I assume that modification time is not useful, but we should check this.
  • All of the above might be different on OSX/Windows/Ubuntu/... !
• Is this tool limited to the files associated with the current settings of $BOSS_DATA_URL and $BOSS_SAS_PATH? The obvious answer is yes, but then you might miss a much larger data volume associated with different variables and delete the wrong files.
• On meta db indexing, I don't think we want to let users pick their own indices, since you can easily make performance worse if you don't know what you are doing and custom indices significantly complicate our code. Instead, we should decide on a good set of indices and (somehow) require that all databases use them.
• For the --catalogs arg, we already have another one (PLATELIST) and will probably be adding more, so you want an API that scales well to more types of catalog.

[dcunning11235]

• Okay
• Yup
• Yeah, hadn't thought about those details (other than just assuming mtime didn't matter); I'll investigate on Ubuntu and Win7, pull someone in to help with OSX.
• File ops start at $BOSS_LOCAL_ROOT. I went back and forth on this, whether it should always start at root, further down under some 'catalog' specific directory, including REDUX or not; settled on the most global option. I wanted to add in restricting to some catalog in a future revision.
• I agree that users could create some headaches for their local performance with the indices... but they can loose all their data by deleting it with the same tool. Or 'lose' it by mucking with their ENV settings. Supporting this does add complexity to bosslocalmgr, but it shouldn't affect any other code.
• Yeah, catalogs are a bit of a pain... think we're getting to the point where we need a bossdata.conf and associated config class that everything uses. Right now I'm not doing anything particularly graceful or future-proof.

[dcunning11235]

Creation date is a filesystem dependent feature; which makes sense, but isn't something I'd considered, always assuming it to be there.

In any case: on the ext4 filesystem this is available, but not on ext2 or ext3; for these cases, modification date is the closest we can get (date from server is not preserved, at least as far as HTTP downloads go... not sure about globus.) FAT?? and NTFS support this, as well. In any case, the short answer is that -mtime and -mmin options are going to be needed.

Points a file has its date set:

event	ctime*	mtime	atime	Note
Download	1	1	1
Access	0	0	1	Only with latest db_manage branch
Index Update	0	1	1	If we keep this functionality

ctime*: If supported, obviously

The table makes a good argument for just throwing -ctime and -cmin and replacing with -mtime etc. None of this takes into account how OSX works however.