search

vitei / git-media

Handling large media files in Git
MIT License
10 stars 3 forks source link

any plans to merge upstream #2

Open mikofski opened 9 years ago

mikofski commented 9 years ago

Curious why you haven't sent a pull request to @alebedev. I really like the changes you've made but it irks me to have 2 repositories for the same application.

ggiles commented 9 years ago

I think mainly because we're quite far behind his master branch now.. also, we keep finding bugs :)

TimMensch commented 9 years ago

I just spent WAY too much time working on a full merge: Merge of albedev and vitei branches. It seems to work on SCP transport. In testing I also discovered several other things in the vitei fork that didn't seem to work: git merge clear for instance.

As I said, I've spent way too much time on it already and need to get back to "real" work. If someone wants to pick up where I left off, please do. The other transports should be tested; I suspect the change to enable subdirectories in the local cache was never tested on the SCP transport, for instance. It doesn't look like it should have ever worked. So it probably won't work on the WebDAV transport that I pulled from the @alebedev branch.

At the moment, given the state of all of the major big-media plug-ins for git, I'm going to give up and leave my big files in git. At least in the case of the SCP transport, git-media is pathetically slow at doing a git media sync for any nontrivial number of files. Heck, it's pathetically slow for one file; it needs some form of caching so that it can know what versions have been pushed without pinging the server for every file on every sync. And it looks like the other transports use the same approach. :(

I'm sure there are other transports that work better than SCP, but even the existence of one that worked so poorly gives me a bad taste for git-media -- and it seems like none could be fast if you have, say, 200 media files in a repo. Too much overhead for every query.

By the way, what IS a "hashstash"? I can't find a server anywhere that is called hashstash; when I Google it I mostly get "marijuana storage solutions". :(

dpwright commented 9 years ago

Wow, thanks for this! I've been meaning to do a similar job for a while, but as you've just discovered it can get pretty hairy! I'll answer some of your questions here:

Firstly, yeah, there's a lot of stuff that probably "doesn't quite work" in the vitei branch. A lot of our commits are "quick fixes" for specific issues we were having, and geared toward the hash stash which is what we use, so may have caused issues for other transports -- the subdirectory thing is a perfect example.

Re. the hash stash -- that's actually one of the reasons I hadn't got around to cleaning up our git-media fork ready for merging yet. We also noticed the SCP transport was slow -- especially on Windows -- so we knocked together our own little server which allows you to store and retrieve files based on their hash. The first version we knocked together is mentioned on our blog, but that's not the version we're currently using and isn't exactly easy for other people to get up and running with! One of the things I've had on a backlog for ages is to clean that up and make a gem for it, so that we could submit it as a first-class citizen to git-media, but I just haven't got around to that yet.

Heck, it's pathetically slow for one file; it needs some form of caching so that it can know what versions have been pushed without pinging the server for every file on every sync.

I suspect this is an issue we don't suffer from as much as we almost never run git media sync. I can't remember the details, but I think we check as part of our clean filter (when adding files etc) whether the file exists in our local cache. If it does we do nothing; otherwise we copy the file to the local cache and push it up to the hash stash at the same time, so it's very rare to find that the file already existed on the hash stash. Obviously this makes working offline basically impossible.

Anyway, thanks for putting the effort into this merge. It's a great first step -- if we can polish up the hash stash and tidy up some of our fixes so that they make sense on all available transports we might have something worth submitting upstream.