Tests: create programmatically, add files in sub-directory

[ethomson]

Hmm. We've considered this in the past in libgit2 and LibGit2Sharp and decided that it's much easier to ship actual repositories and working directories than it is to try to create them on-the-fly. It's very hard, for example, to know that you're testing what's expected.

Here's an example: what do you do when you encounter an "invalid" commit in a repository? Maybe this commit has an invalid timestamp, or maybe it's got an invalid signature. You probably want to test your ability to handle this case. But libgit2 will not let you write one of these commits, because, well it's invalid.

Here's another example: libgit2 had the ability to read merge commits before it had the ability to do a merge...

Seeing as git itself is basically its own specification, it's very useful to be able to create repositories with git itself, and then make sure that libgit2 can parse this data correctly. Or to check in a corpus of known good data with git and ensure that libgit2 creates identical data.

What's the motivation for removing these and creating them on-the-fly?

[mstrap]

@ethomson I was starting this, because I noticed that status was broken on the included repository (all files had "modified" state), besides that I feel it's the "right" approach :)

We've considered this in the past in libgit2 and LibGit2Sharp and decided that it's much easier to ship actual repositories and working directories than it is to try to create them on-the-fly.

I would agree that libgit2 shouldn't be used to create libgit2 test cases, however libjagged (or LibGit2Sharp) is one level higher: the goal is not to test libgit, but libjagged. In this sense libgit2 is the reference (as Git is for libgit tests). My ultimate goal is to get rid of NativeTestMethods.createTestRepository() and use libjagged API to create test setups. Until all necessary functionality is implemented (I've started with Index API, Commit API is still missing entirely), this is the workaround.

Just some examples:

• to test status API with its various flags, it will be interesting to create untracked, added, removed, missing, ignored, ... files and directories. Setting up such repositories will require just a few lines of code vs. preparing a couple of predefined repositories
• one of my medium-term goals is to write the API for rebase; most likely running rebase scenarios will require to invoke commands like add/remove/discarding conflicts. I can't think of a reasonable solution without doing that programmatically in the test itself

[ethomson]

Setting up such repositories will require just a few lines of code vs. preparing a couple of predefined repositories

I don't agree with that at all. This pull request is already not just a few lines of code, it's a bunch of C and JNI to do the test setup. It's already hard to debug code that goes through JNI. But now debugging test building becomes subject to those same frustrations. Worse, it's not portable (notice the test failures on anything that's not Windows) and there are buffer overflows in the statically sized strings.

You're going to spend an insane amount of time duplicating code like libgit2's POSIX emulation layer just to write the unit tests when you could just bang out a repository, check it in and go. This is also what LibGit2Sharp and rugged do, not just libgit2. It's proven and effective. Plus you can easily borrow the test data that libgit2 already has for a variety of purposes (like rebase).

If you really want to build test data programmatically, I think that you actually want to write shell scripts that talk to git to do their heavy lifting. But I'm not particularly enthused about that, either.

[mstrap]

I don't agree with that at all. This pull request is already not just a few lines of code, it's a bunch of C and JNI to do the test setup. It's already hard to debug code that goes through JNI. But now debugging test building becomes subject to those same frustrations. Worse, it's not portable (notice the test failures on anything that's not Windows) and there are buffer overflows in the statically sized strings.

OK, my bad. I should have checked on other platforms, too.

But as I had mentioned, the native test code is just a temporary solution until there is sufficient jagged API to build test setups using only the API. So I will use Index API in Status tests and vice versa. Would you agree on such a self-contained solution?

You're going to spend an insane amount of time duplicating code like libgit2's POSIX emulation layer just to write the unit tests when you could just bang out a repository, check it in and go. This is also what LibGit2Sharp and rugged do, not just libgit2. It's proven and effective. Plus you can easily borrow the test data that libgit2 already has for a variety of purposes (like rebase).

Reusing libgit2's test data sounds quite promising. I guess this test data is quite stable, so there wouldn't be any overhead in adjusting libjagged tests due to changed libgit2 test data when updating libgit2? I'll look into this for the Status API which is work-in-progress. I'd still try to get rid of your test repository and replace it by the corresponding libgit2 test repository. Is that OK for you?