Znapzendzetup list recursive

jimklimov commented 4 years ago

So I was debugging my laptop backup setup today, and was annoyed that I must either znapzendzetup list source datasets that I know to have been configured by name, one by one, or suffer the whole-system dataset listing (with USB HDD taking many minutes to give the list)...

And so I scratched a few itches:

added ability to list several named dataset configs in one command call
added ability to recurse into children of named dataset(s)
added a fallback for systems whose memory vs ZFS tree sizing won't permit listing all the child dataset attributes :-)
extended ZFS.pm routines getDataSetProperties() and listDataSets() with ability to inspect not only a single string $dataSet value, but also arrays of strings, to reduce the number of (slow) calls into zfs command
added optional debugging and features (for lowmemRecurse) support in znapzendzetup

Tested with various combinations locally, seems to work for stuff I typed :)

But CI selftests not yet updated, probably zfs mockup needs some love :(

coveralls commented 4 years ago

Coverage increased (+0.9%) to 88.414% when pulling 8b20412a36edffd154feb9306c00a7ed197af8fa on jimklimov:znapzendzetup-list-recursive into 95f9e5ab81ae83989ffa6c233e79c7e292fa7b1c on oetiker:master.

jimklimov commented 4 years ago

Oh wow, seeing green CI builds at https://travis-ci.org/jimklimov/znapzend/builds/597797056 now :)

Gotta go get some sleep to not break anything further :)

jimklimov commented 4 years ago

Can't make sense of some logs... says tests are green but by coverage seems something failed mid-way.

And also this:

# zfs destroy remote/destination@2019-10-14-184000,2019-10-14-184200,2019-10-14-184300,2019-10-14-184400,2019-10-14-184500,2019-10-14-184600,2019-10-14-184700,2019-10-14-184800,2019-10-14-184900,2019-10-14-185100,2019-10-14-185200,2019-10-14-185300,2019-10-14-185400,2019-10-14-185500,2019-10-14-185600,2019-10-14-185700,2019-10-14-185800,2019-10-14-185900,2019-10-14-190200,2019-10-14-190300,2019-10-14-190400,2019-10-14-190600,2019-10-14-190700,2019-10-14-190800,2019-10-14-190900,2019-10-14-191100,2019-10-14-191200,2019-10-14-191300,2019-10-14-191400,2019-10-14-191600,2019-10-14-191700,2019-10-14-191800,2019-10-14-191900

# zfs list -H -o name -t filesystem,volume backup/destfail

[Mon Oct 14 19:40:13 2019] [warn] destination 'backup/destfail' does not exist or is offline. ignoring it for this round...
[Mon Oct 14 19:40:13 2019] [warn] ERROR: suspending cleanup source dataset because at least one send task failed
[Mon Oct 14 19:40:13 2019] [info] done with backupset tank/source in 3 seconds
[Mon Oct 14 19:40:13 2019] [info] znapzend (PID=6792) initialized -- resuming normal operations.
not ok 4 - znapzend dies with no backup sets defined or enabled at startup
#   Failed test 'znapzend dies with no backup sets defined or enabled at startup'
#   at ./t/znapzend.t line 69.
# expecting: Regexp ((?^:No backup set defined or enabled))
# found: normal exit

So was it supposed to fail but didn't (and have I broken something?) or not?

jimklimov commented 4 years ago

Seems I did...

=== getDataSetProperties(): Is array...
# zfs list -H -o name -t filesystem,volume -r tank/source
cannot open 'tank/source': dataset does not exist
=== getDataSetProperties(): List all...
# zfs list -H -o name -t filesystem,volume

at least somtehing

jimklimov commented 4 years ago

Seems the CI gods are satisfied now... though in the build log I see

ok 15 - znapzendzetup import --write
# Tests were run but no plan was declared and done_testing() was not seen.
# Looks like your test exited with 1 just after 15.

and locally the test 16 passes:

ok 15 - znapzendzetup import --write
=== getDataSetProperties():
...
ok 16 - znapzendzetup list --features=lowmemRecurse --debug --recursive tank/source
1..16

oetiker commented 4 years ago

make sure to update the test suit for your new features.

jimklimov commented 4 years ago

Yes, on it. Essentially, there is one "real" codepath that is not called and I'm fixing (the recursion over several datasets listed by zfs in one shot), and the rest of coverage misses are mostly error-cases that we do not trigger. I'll try to add some faults :)

jimklimov commented 4 years ago

Thanks for that other merge, glued it to this one now, waiting for re-testing :)

oetiker / znapzend

Znapzendzetup list recursive #439