Update the redis backend

charypar commented 10 years ago

Redis backend overhaul to enable real use as storage for bare repositories.

This is my first pull request to libgit2 and I'm not entirely familiar with all the conventions so I may have missed better options to do things. I'll welcome all comments and suggestions of improvement.

What's been done

Updated Odb backend to the current libgit2 API
Added Refdb backend
Changed the key structure to enable multiple repositories in the same redis instance
Enabled a static build of the redis backend and linking to a local libgit2

I understand that a redis backend will be slow, especially when used over the network, but I still think it's worth it. My single most important use-case is to enable using git repositories on Heroku or similar app container, which doesn't have a writable file system (not coincidentally, that's also my use-case). It also enables the use of ephemeral, in-memory repositories for cases like undo history without going all out with in-memory storage, etc.

My end goal is to enable switching backends in rugged for ruby/rails apps. I will open a PR to that end soon(-ish) but this can go in first and independently (almost, see below).

For now, the backend is tested through rugged and the tests depend on my hacked-in support for redis backend (see here if interested, but it's probably not worth it). You'll basically have to trust me it works :).

It would be really good to add testing support to libgit2-backends itself. I'm willing to give it a shot if anyone can provide pointers on how best to do it.

Example repo

127.0.0.1:6379> keys rugged:storage/*
 1) "rugged:storage/b1ff909250a5fda83042abc86f7033f9:odb:35bf36aaa1cb0a8e50c2087a18b5cbfc52a92d3"
 2) "rugged:storage/b1ff909250a5fda83042abc86f7033f9:odb:3fc49d757edb6c1add766fbb34f3995810a146c"
 3) "rugged:storage/b1ff909250a5fda83042abc86f7033f9:odb:21e2587d773419407b63992b1425857c7311a99"
 4) "rugged:storage/b1ff909250a5fda83042abc86f7033f9:odb:22001f573a8e7a0768f6116932f0e4456aa529d"
 5) "rugged:storage/b1ff909250a5fda83042abc86f7033f9:refdb:refs/heads/published"
 6) "rugged:storage/b1ff909250a5fda83042abc86f7033f9:odb:d0955f5f1fb29c2dfef1f6957e78458dd9ffd0e"
 7) "rugged:storage/b1ff909250a5fda83042abc86f7033f9:odb:75b7fe1e6981891882f9eb14a123696be4123a8"
 8) "rugged:storage/b1ff909250a5fda83042abc86f7033f9:odb:c8e9c8d242d22db6ed07cac812769d27be20df5"
 9) "rugged:storage/b1ff909250a5fda83042abc86f7033f9:odb:b6dbb5c6c696965e64634cfcb956e84e1a43d2e"
10) "rugged:storage/b1ff909250a5fda83042abc86f7033f9:refdb:HEAD"
11) "rugged:storage/b1ff909250a5fda83042abc86f7033f9:refdb:refs/heads/master"
12) "rugged:storage/b1ff909250a5fda83042abc86f7033f9:odb:fb8d8b4369c084668ab8c62cc50dbc184ff23cc"
13) "rugged:storage/b1ff909250a5fda83042abc86f7033f9:odb:6e457a4e82ede40c4b41dec6df0c90772de8903"
14) "rugged:storage/b1ff909250a5fda83042abc86f7033f9:odb:331a16c1dcb6e2bfeb06aa17aad774c58830a6a"
15) "rugged:storage/b1ff909250a5fda83042abc86f7033f9:odb:5e271d2c8606b8358434ac6fcb461c1c90aa4c2"
16) "rugged:storage/b1ff909250a5fda83042abc86f7033f9:odb:5d95e7bf9ef10fc440fd3e8e687f3173cdf0852"
17) "rugged:storage/b1ff909250a5fda83042abc86f7033f9:odb:5b5b9e43d8eeba5de0ac8e0ec2a26047eaf79fb"
18) "rugged:storage/b1ff909250a5fda83042abc86f7033f9:odb:3d24d025529cb619e8b80c7ce04534c22fbbbc3"
19) "rugged:storage/b1ff909250a5fda83042abc86f7033f9:odb:4efa77fdf5470a5aa7d26a3d6b63738d29470ca"
20) "rugged:storage/b1ff909250a5fda83042abc86f7033f9:odb:f7f9a3c3add8acf7209634c62a777861141cd03"
21) "rugged:storage/b1ff909250a5fda83042abc86f7033f9:odb:cc9096d0d236f765844b75700a72620f959c810"

Issues

I see the following problems with the implementation:

No reflog support (because I don't need it)
No tests
Cmake changes are most likely really bad
Built against a slightly older refdb API because of the testing through rugged which isn't entirely up to date with it's vendored libgit2 (this could be a blocker)

/cc @mhodgson, @arthurschreiber

charypar commented 10 years ago

I made a few changes to this, namely

Fixed reference iteration to only list refs under refs/
Added 'AUTH' command support
Improved redis connection management to reuse a single connection across the backend instances

charypar commented 10 years ago

This is now pretty solid. It's confirmed to work (on Heroku) for our basic usecase. I'd still like to do a couple benchmarks to see how this performs locally and over the network.

But for now, any comments on the implementation itself?

charypar commented 10 years ago

@mhodgson @carlosmn any chance for some feedback on this?

charypar commented 9 years ago

bump.

libgit2 / libgit2-backends