New Feature: add `--type=directory` to `onyo get`

[aqw]

(Note: I have edited this description to reflect decisions made in #678 and #680. In the future, we will have separate design and feature issues to avoid rewriting discussion history like this.)

---

The parent issue is #673

Allow onyo get to return data about directories. The type should be called directory.

• all directories that are not under .git and .onyo should be returned
  • this includes Asset Directories. users who wish to exclude can use --match onyo.is.asset=False

This addresses #672.

Clearly, directories do not have keys, so only pseudo keys will be outputable. Querying non-pseudo-keys should result in <unset>. The primary use case is to output a list of directories (i.e. onyo.path.relative). But perhaps some people will also find interest in the git pseudo keys.

The advanced use-case I imagine is being able to determine if directories are empty. This can be done using a for-loop.

for d in $(onyo get --type directory); do
    onyo get --type directory,asset --include d || echo "empty dir: $d"
done

Even better is to add a pseudo key:

• onyo.is.empty: True or False (Directories are empty if they have no children (dir or assets). Assets are never empty. Templates could in theory be empty.)

Pseudo Keys:

• onyo.is.asset: True or False depending on if it is an asset directory
• onyo.is.directory: True
• onyo.is.template: False
• onyo.is.empty: True or False depending if the directory has children (dirs or assets)
  • an asset directory with no child assets or directories shall return True (despite containing the hidden asset file)
• onyo.path.relative: path relative to the repo root
• onyo.path.file: False (open for debate if <unset> should be used, but I like False)
• onyo.path.parent: path to parent directory relative to the repo root

Corner case:

• Be sure to honor .onyoignore, and not include directories that are ignored (semi-related: #548)

Tasks:

• [ ] add to onyo_get
• [ ] add to onyo get
• [ ] shell completion
• [ ] help text
• [ ] tests

[bpoldrack]

Just adding another open question:

Directories could be "matchable" in various ways. They do have some metadata themselves (empty, not empty, glob, creation/modification time, etc.) So, how is --match interpreted, when --type=directory?

• not allow it at all?
• have special expressions that --match would accept in that mode: some notion of contains (including no asset)?
• match assets within via --match (making the use of --match an implicit contains) and have a separate option to be used for directory metadata matching
• make up a bunch of "reserved keys" for dirs and do the matching with these. Would still have one degree of freedom: Either --match is only valid w/ these keys and we only match dir metadata, or it matches assets within and we have separate but very similar option, that does dir-metadata matching.

[bpoldrack]

I'm not sold on this, and sounds expensive.

Not sold on this either, but: I do think option 2 is actually more expensive. If only because you'd fire onyo up again and again. Counting children within an onyo process is cheaper, because it's actually not I/O - we check git ls-tree (as opposed to traversing FS), turning that task into in-memory Path analysis.

[aqw]

Directories could be "matchable" in various ways. They do have some metadata themselves (empty, not empty, glob, creation/modification time, etc.) So, how is --match interpreted, when --type=directory?

In my mind, it works the exact same way as for asset: you can match, sort, etc for keys. Anything else would feel arbitrary and would limit to usefulness of the feature. And such matching is already demonstrated in the above example: --type directory --match is_asset_directory=False.

have special expressions that --match would accept in that mode: some notion of contains (including no asset)?

Yeah, this is a different conversation: are there pseudo keys that we should add as a result of the new type directory?

I'm certainly open to that (I suggested children).

match assets within via --match (making the use of --match an implicit contains) and have a separate option to be used for directory metadata matching

How do you see contains working? Is it a fixed vocabulary of things (assets, directories, etc). Or is it accepting a specific path (which could actually be done the other way around; if you know your asset, you could match based on directory regex).

make up a bunch of "reserved keys" for dirs and do the matching with these. Would still have one degree of freedom: Either --match is only valid w/ these keys and we only match dir metadata, or it matches assets within and we have separate but very similar option, that does dir-metadata matching.

This feels like it's going off in a very bespoke direction.

What I'm aiming for is onyo get to behave the same as it does now. All the abilities are the same. And one can query for non-directory keys. They'll just all be <unset> (or possibly a new type of <NA> or something).

So pretty much all the above questions evaporate.

The question that remains is:

• are there pseudo keys that should be added to expose useful information about directories?
  • I think that the existing pseudo keys are useful
  • I think the git pseudo keys could be semi-useful (which we already want to add)
  • the remaining use-case I see is: how can we make it easy to determine if a dir is empty?
  • I provided a for-loop example that requires no additional pseudo keys, in case we're averse to adding more

Not sold on this either, but: I do think option 2 is actually more expensive. If only because you'd fire onyo up again and again.

I wasn't saying that option 2 was not expensive. I was saying that option 1 might be more expensive than we'd like for the feature.

The issue that I have with children, is that it's in effect bundling a subsequent onyo get call (literally what option 2 is). It's a small leap to say "but what kind of children"? Which is a filter. And we recurse down the rabbit hole.

[bpoldrack]

Just FTR: With #678 the question of how to address "empty" is kinda solved.

[aqw]

I agree, it seems we've coalesced around is.empty (or onyo.is.empty).

Also, FWIW, the for-loop method is (as of now) possible due to #683 being merged.