jthornber / blk-archive

Dedup and compress your device mapper devices. Works especially well with thin provisioning.
8 stars 3 forks source link

Add optional JSON output #3

Closed tasleson closed 1 year ago

tasleson commented 1 year ago

Currently blk-archive outputs human readable output, some examples.

A pack:

 # blk-archive pack -a /home/tasleson/debug_a/ /dev/test/dedupeable
 elapsed          : 7.296
 stream id        : efd2fd886d600cdb
 file size        : 200M
 mapped size      : 94.94M
 total read       : 94.94M
 fills size       : 654.48K
 duplicate data   : 69.03M
 data written     : 5.92M
 hashes written   : 38.19K
 stream written   : 371
 ratio            : 15.94
 speed            : 13.01M/s

A pack, re-directing stdout & stderr to a file which is different as the code has tty detection:

Building packer /dev/dm-5 ...
Packing /dev/dm-5 ...
resized index to 104857
Progress: 40%
Progress: 80%
elapsed          : 14.26
stream id        : 268540f9aa80c145
file size        : 200M
mapped size      : 200M
total read       : 200M
fills size       : 0
duplicate data   : 200M
data written     : 0
hashes written   : 0
stream written   : 749
ratio            : 279993.59
speed            : 14.03M/s

a list:

 # blk-archive list -a /home/tasleson/debug_a/
 9e5ef5af1ff9f469 209715200 May 01 23 14:49 thick
 268540f9aa80c145 209715200 May 01 23 14:50 thick
 efd2fd886d600cdb 209715200 May 01 23 18:03 dedupeable

To make is easier to script blk-archive, having JSON output is helpful, eg.

 # blk-archive -j pack -a /home/tasleson/debug_a/ /tmp/file_to_add 
{
  "stream_id": "3ff19828970f2683"
}

# blk-archive -j list -a /home/tasleson/debug_a/
[
  {
    "size": 209715200,
    "source": "thick",
    "stream_id": "9e5ef5af1ff9f469",
    "time": "2023-05-01T14:49:52.911862103+00:00"
  },
  {
    "size": 209715200,
    "source": "thick",
    "stream_id": "268540f9aa80c145",
    "time": "2023-05-01T14:50:43.325372187+00:00"
  },
  {
    "size": 209715200,
    "source": "dedupeable",
    "stream_id": "efd2fd886d600cdb",
    "time": "2023-05-01T18:03:06.833113381+00:00"
  },
  {
    "size": 104857600,
    "source": "file_to_add",
    "stream_id": "3ff19828970f2683",
    "time": "2023-05-01T20:11:08.920709494+00:00"
  }
]

This change uses the 'quiet_report', so most of the time there is no output when the user specifies -j or'--json on the command line. A new structure was added, to encapsulate the report for list and pack, perhaps this can be done in a more elegant way.

Note: The main reason for adding this was for the purpose of automated testing.