Client command list

[ketiltrout]

A proposal for the client command list, to see what we need. Per #122, we will stick to <noun> <verb>:

• [x] db - database maintenance #231
  • [x] db init - create tables, if necessary
  • Additional possibilities: See #232
  • db status. Status of what?
  • db check. Check integrity (table interconnections)
• [x] group - for StorageGroup: Implementation #204
  • [x] group autosync GROUP NODE : #228
  • [x] group create - create new group
  • [x] group list - list groups
  • [x] group modify GROUP - edit group metadata
  • [x] group nodes GROUP - list nodes in group
  • [x] group rename GROUP - rename group
  • [x] group show GROUP - show StorageGroup record
  • [x] --actions - to show auto-actions: #228
  • [x] --transfers - to summarise pending transfers in #238
  • [x] group sync GROUP NODE #233
  • An alias for node sync NODE GROUP
  • Additional possibilities:
  • group add-node. See #205
• [x] node - for StorageNode: Partial implementation #221
  • [x] node activate NODE - activate node
  • [x] node autoclean NODE GROUP : #228
  • [x] node clean - clean node of file copies #223
  • [x] --include-bad - also clean suspect/corrupt files. #226
    • See #225
  • See #65
  • See #89
  • See #92
  • [x] node create - create new node.
  • See #39
  • [x] node deactivate NODE - deactivate node
  • node files NODE... list files on node See file status
  • [x] node init #237
  • See #227
  • [x] node list - list nodes (curently called "status")
  • See #121
  • [x] node modify NODE - edit node metadata
  • [x] node rename NODE - rename node
  • [x] node scan - scan the node for known files to add (in alpenhorn-1 this was called "import-files", but in alpenhorn-2 client currently called "scan") #236
  • [x] node show NODE - show StorageNode record: #224
  • [x] --actions - to show auto-actions: #228
  • [x] --transfers - to summarise pending transfers out #238
  • [x] node stats NODE - show file counts/sizes on node
  • [x] node sync NODE GROUP #233
  • See #106
  • See #172
  • [x] node verify NODE PATH... re-verify file(s) on node. #229
  • See #133
  • See #141
• [x] acq - for ArchiveAcq: Implementation #206
  • [x] acq create NAME - create new ArchiveAcq record #239
  • See #252
  • [x] acq files ACQ - list ArchiveFiles in ACQ
  • [x] acq list --node=<NODE> --group=<GROUP> - list acquisitions, optionally limited to node/group
  • [x] acq show ACQ - show ArchiveAcq record for ACQ
  • acq where ACQ - which nodes contain files from ACQ? merged into show
• [ ] file - for ArchiveFile and ArchiveFileCopy
  • [x] file clean FILE... set ArchiveFileCopy.has_file for existing record #257
  • [x] file create NAME ACQ create new ArchiveFIle record directly #251
  • See #252
  • [ ] file find list ArchiveFileCopys #261
  • See #45
  • See #248
  • [x] file import PATH request import of PATH by daemon #249
  • [x] file list - list ArchiveFiles #247
  • [x] file modify - modify ArchiveFile data (MD5 and size) #255
  • [x] file show FILE show ArchiveFile record #246
  • [ ] file state FILE NODE get/update/create ArchiveFileCopy record for FILE on NODE #259
  • See #165
  • file status FILE summarise ArchiveFileCopy status across storage nodes
  • Merged into file show
  • [x] file sync FILE NODE GROUP - transfer file from node to group #256
  • See #64
  • See #126
  • file where FILE - list nodes containing file see file show --nodes
• transfer - for ArchiveFileCopyRequest Dispersed to group/node/file. See comment below
  • transfer request NODE GROUP FILE... - create transfer request for file(s) from node to group
    • Replaced by file transfer
  • transfer cancel NODE GROUP FILE - cancel pending request for FILE from NODE to GROUP
    • Replaced by file transfer --cancel
  • transfer list - list transfers
  • Replaced by file status --show-transfers
  • transfer pending - list pending transfer stats
  • Replaced by group show --show-transfers
  • Replaced by node show --show-transfers
  • Replaced by file status --show-transfers
  • transfer sync NODE GROUP - sync node to group (transfer all file not already present in group)
  • Replaced by node sync
  • Replaced by group sync
• action - for StorageTransferAction Dispersed to group/node. See comment below
  • action list - list StorageTransferAction
  • Replaced by node show --show-actions
  • Replaced by group show --show-actions
  • action show NODE GROUP - show StorageTransferAction for node-to-group
  • Replaced by node show --show-actions
  • Replaced by group show --show-actions
  • For modifying the StorageTransferActionTable:
  • group auto-sync
  • node auto-clean
• ~transport - transport disk stuff.~ Not sure about this. It's very different to the rest of the alpenhorn commands.
  • Proposal to remove this command group completely: #240
  • ~transport create - yikes Formerly called format-transport.~
  • See #8
  • See #115
  • See #124
  • ~transport mount - mount and activate~
  • ~transport unmount - deactivate and unmount~
  • transport list - list transport disks
  • Use: group show TRANSPORT_GROUP_NAME --show-nodes

[tjrennie]

This looks like a complete list of commands, but I'd like to see a way of seeing the last n transfers that were cancelled or ignored as well as the last n transfers that were run (which i assume is action list) . - You mentioned this was more server metrics but it would be useful regardless

[ketiltrout]

I've added a transfer list command which I think might do what you want, with the right set of filtering options

[ketiltrout]

After getting some familiarity working on the client for a while, I've removed the transfer and action "noun" groups.

I think it's better to limit the groups to "concrete" objects: Groups, Node, Acqs, Files (including File Copies).

I've dispersed the functionality of the removed groups to the other ones. Let me know what you think, and if you can see any gap in functionality that I've missed.

[ketiltrout]

After developing the on-demand import framework for the daemon (#235) I've been thinking about the need to have the ability to manually modify Archive entries in the database with the CLI. With that in mind, I've added the following commands to the list above to permit that:

ArchiveAcq

• acq create -- create a new ArchiveAcq record acq create NAME --notes=COMMEMNT

ArchiveFile

• file create -- create a new ArchiveFile record

  • Directly: file create NAME ACQ_NAME --md5=<SUM> --size=<SIZE>
  • Via reading a file: file create NAME ACQ_NAME --from-file --prefix=PREFIX i.e. stats/md5s local file [PREFIX/]ACQ_NAME/NAME to generate MD5 and size (PREFIX is optional, to tell alpenhorn where to find the file)

  Neither of these methods involve using an import-detect plugin, meaning no third-party callback is invoke. See file import if you want to have that functionality

• file import -- request import of a file on a node (via ArchiveFileImportRequest) file import PATH NODE --register-new

• file modify -- can be used to change MD5 / size file modify PATH --md5=<sum> --size=<SIZE> file modify NAME ACQ_NAME --from-file --prefix=PREFIX

ArchiveFileCopy

These command also use the "file" command group for simplicity)

• file state -- set the state of a file (copy) on a node file state PATH NODE_NAME --<STATE> --create --ready/-not-ready

  <STATE> is one of:

  • --present: has_file='Y' wants_file='Y'
  • --cleaned: has_file='Y' wants_file='M'
  • --released: has_file='Y' wants_file='N'
  • --suspect: has_file='M' wants_file='Y'
  • --corrupt: has_file='X' wants_file='Y'
  • --missing: has_file='N' wants_file='Y'
  • --removed: has_file='N' wants_file='N'

  This can be used to create new ArchiveFileCopy records (--create-new)

• file clean -- manipulate ArchiveFileCopy.wants_file for existing records