New Feature: add `--type=directory` to `onyo get`

[aqw]

The parent issue is #673

Allow onyo get to return data about directories. The type should be called directory.

• all directories that are not under .git and .onyo should be returned
  • this includes Asset Directories. users who wish to exclude can use --match is_asset_directory=False

This addresses #672.

Clearly, directories do not have keys, so only pseudo keys will be outputable. Querying non-pseudo-keys should result in <unset>. The primary use case is to output a list of directories (i.e. path). But perhaps some people will also find interest in the git pseudo keys.

The advanced use-case I imagine is being able to determine if directories are empty. Adding a pseudo key (is_empty) does not appeal to me, and feels too specific to directory. I see two reasonable solutions:

1) new pseudo key: children literally count the number of children (directories, assets, etc). I'm not sold on this, and sounds expensive. 2) a user simply writes a for loop:

    for d in $(onyo get --type directory); do
        onyo get --type directory,asset --include d || echo "empty dir: $d"
    done

This particular approach (others would inspect the output) would require modifying the return codes of `onyo get` to match that of `grep` ("exit status is 0 if a line is selected, 1 if no lines were selected, and 2 if an error occurred."). Currently `onyo get` returns `1` on error and `0` if no results are found.

I currently favor option 2. If others agree, we should open an issue to track the new onyo get return codes.

• [ ] add to onyo_get
• [ ] add to onyo get
• [ ] shell completion
• [ ] help text
• [ ] tests

[bpoldrack]

Just adding another open question:

Directories could be "matchable" in various ways. They do have some metadata themselves (empty, not empty, glob, creation/modification time, etc.) So, how is --match interpreted, when --type=directory?

• not allow it at all?
• have special expressions that --match would accept in that mode: some notion of contains (including no asset)?
• match assets within via --match (making the use of --match an implicit contains) and have a separate option to be used for directory metadata matching
• make up a bunch of "reserved keys" for dirs and do the matching with these. Would still have one degree of freedom: Either --match is only valid w/ these keys and we only match dir metadata, or it matches assets within and we have separate but very similar option, that does dir-metadata matching.

[bpoldrack]

I'm not sold on this, and sounds expensive.

Not sold on this either, but: I do think option 2 is actually more expensive. If only because you'd fire onyo up again and again. Counting children within an onyo process is cheaper, because it's actually not I/O - we check git ls-tree (as opposed to traversing FS), turning that task into in-memory Path analysis.

[aqw]

Directories could be "matchable" in various ways. They do have some metadata themselves (empty, not empty, glob, creation/modification time, etc.) So, how is --match interpreted, when --type=directory?

In my mind, it works the exact same way as for asset: you can match, sort, etc for keys. Anything else would feel arbitrary and would limit to usefulness of the feature. And such matching is already demonstrated in the above example: --type directory --match is_asset_directory=False.

have special expressions that --match would accept in that mode: some notion of contains (including no asset)?

Yeah, this is a different conversation: are there pseudo keys that we should add as a result of the new type directory?

I'm certainly open to that (I suggested children).

match assets within via --match (making the use of --match an implicit contains) and have a separate option to be used for directory metadata matching

How do you see contains working? Is it a fixed vocabulary of things (assets, directories, etc). Or is it accepting a specific path (which could actually be done the other way around; if you know your asset, you could match based on directory regex).

make up a bunch of "reserved keys" for dirs and do the matching with these. Would still have one degree of freedom: Either --match is only valid w/ these keys and we only match dir metadata, or it matches assets within and we have separate but very similar option, that does dir-metadata matching.

This feels like it's going off in a very bespoke direction.

What I'm aiming for is onyo get to behave the same as it does now. All the abilities are the same. And one can query for non-directory keys. They'll just all be <unset> (or possibly a new type of <NA> or something).

So pretty much all the above questions evaporate.

The question that remains is:

• are there pseudo keys that should be added to expose useful information about directories?
  • I think that the existing pseudo keys are useful
  • I think the git pseudo keys could be semi-useful (which we already want to add)
  • the remaining use-case I see is: how can we make it easy to determine if a dir is empty?
  • I provided a for-loop example that requires no additional pseudo keys, in case we're averse to adding more

Not sold on this either, but: I do think option 2 is actually more expensive. If only because you'd fire onyo up again and again.

I wasn't saying that option 2 was not expensive. I was saying that option 1 might be more expensive than we'd like for the feature.

The issue that I have with children, is that it's in effect bundling a subsequent onyo get call (literally what option 2 is). It's a small leap to say "but what kind of children"? Which is a filter. And we recurse down the rabbit hole.

[bpoldrack]

Just FTR: With #678 the question of how to address "empty" is kinda solved.

[aqw]

I agree, it seems we've coalesced around is.empty (or onyo.is.empty).

Also, FWIW, the for-loop method is (as of now) possible due to #683 being merged.