mktree: support more flexible usage

vdye commented 4 months ago

The goal of this series is to make 'git mktree' a much more flexible and powerful tool for constructing arbitrary trees in memory without the use of an index or worktree. The main additions are:

Using an optional "base tree" to add or replace entries in an existing tree rather than creating a new one from scratch.
- Building off of this, having entries with mode "0" indicate "remove this entry, if it exists, from the tree"
Handling tree entries inside of subtrees (e.g., folder1/my-file.txt)

It also introduces some quality-of-life updates:

Using the same input parsing as 'update-index' to allow a wider variety of tree entry formats.
Adding deduplication of input entries & more thorough validation of inputs (with an option to disable both - plus input sorting - if desired with '--literally').

The implementation change underpinning the new features is completely revamping how the tree is constructed in memory. Instead of writing a single tree object into a strbuf and hashing it into the object database, we construct an in-core sparse index and write out the root tree, as well as any new subtrees, using the cache tree infrastructure.

The series is organized as follows:

Commits 1-3 contain miscellaneous small renames/refactors to make the code more readable & prepare for larger refactoring later.
Commits 4-7 generalize the input parsing performed by 'read_index_info()' in 'update-index' and update 'mktree' to use it.
Commit 8 removes the check on object existence & type from submodule entries.
Commit 9 adds the '--literally' option to 'mktree'. Practically, this option allows tests that currently use 'mktree' to generate corrupt trees to continue functioning after we strengthen input validations.
Commits 10 & 11 add input path validation & entry deduplication, respectively.
Commit 12 replaces the strbuf-to-object tree creation with construction of an in-core index & writing out the cache tree.
Commits 13-15 add the ability to add tree entries to an existing "base" tree. Takes 3 commits to do it because it requires a bit of finesse around directory/file deduplication and iterating over a tree with 'read_tree()' with a parallel iteration over the input tree entries.
Commit 16 allows for deeper paths in the input.
Commit 17 adds handling for mode '0' as "removal" entries.

I also plan to add a '--strict' option that runs 'fsck' checks on the new tree(s) before writing to the object database (similar to 'mkttag --strict'), but this series is pretty long as it is and that part can easily be separated out into its own series.

Changes since V1

Renamed '_tree_entry_array' & '_tree_entry_iterator' functions to 'tree_entryarray' & 'tree_entryiterator', respectively.
Removed the return value from 'tree_entry_iterator_advance'.
Added call to 'tree_entry_array_clear' in 'tree_entry_array_release', updated both methods to optionally free tree entries based on a 'free_entries' arg.
Updated 'read_index_info()':
- Replaced INDEX_INFO_EMPTY_LINE with INDEX_INFO_UNRECOGNIZED_LINE which is returned when any "malformed" line is found, setting an input strbuf to the line contents for the caller to deal with.
- Updated default object type value (when no type info is specified) to 'OBJ_ANY' from 'OBJ_NONE'.
Updated 'mktree' to 'die()' on malformed line with "input format error", rather than 'error()' with "malformed input line", to avoid unnecessarily changing the error message.
Removed check for object existence & type when the entry type is a submodule, rearranged checks for readability.
Replaced use of 'grep' with 'test_grep' in new/updated tests.
Wrapped lines in documentation updates to 76 characters.
Applied documentation refactor patch in [1].
Dropped '[--]' from the 'mktree' usage string.

[1] https://lore.kernel.org/git/xmqqle3aovpq.fsf@gitster.g/

Thanks!

Victoria

cc: Eric Sunshine sunshine@sunshineco.com cc: Patrick Steinhardt ps@pks.im

vdye commented 4 months ago

/submit