Consider move images to git-lfs

[avaika]

Images consume a lot of space in this repo.

Please consider an option to move it outside e.g. to git-lfs.

Though in order to reduce current repo size history cleanup will also be required.

[jantic]

Images consume a lot of space in this repo.

Please consider an option to move it outside e.g. to git-lfs.

Though in order to reduce current repo size history cleanup will also be required.

Good suggestion. I'm not in a hurry to do it because the images are currently taking ~110MB which while a lot bigger than code, it isn't horrible imho. I put it up that way consciously because it was quick, easy, and convenient. But I do have ambitions of adding more images so yeah...need to tame it.

[jantic]

If I do this, I'll probably move the weights there as well.

[avaika]

I'm not in a hurry to do it because the images are currently taking ~110MB

After commit https://github.com/jantic/DeOldify/commit/71c772922bba175e53e4ebe9fe783f8311cea37a it takes 200Mb in history :) .git is already 289Mb at my computer even after git gc.

Moreover, if you decide to rewrite the history (to reduce it) it's better to do it as soon as possible, cause time goes by and it will be more harder to rewrite the history (especially in case more people will be interested in the project).

[jantic]

I'm not in a hurry to do it because the images are currently taking ~110MB

After commit 71c7729 it takes 200Mb in history :) .git is already 289Mb at my computer even after git gc.

Moreover, if you decide to rewrite the history (to reduce it) it's better to do it as soon as possible, cause time goes by and it will be more harder to rewrite the history (especially in case more people will be interested in the project).

Craaaaaap... Good point. Sorry, I fucked this one up then. If you want to go ahead and do it please do. I'll gladly take the pull request. Otherwise I'll try to get to it within a few days.

[avaika]

Unfortunately it doesn't seem it would be possible to perform history clean up via pull request :)

But here's a step-by-step on what is required:

• download bfg:

  wget -O bfg.jar "https://search.maven.org/classic/remote_content?g=com.madgag&a=bfg&v=LATEST"

• mirror the repo as bare git:

  git clone --mirror git@github.com:jantic/DeOldify deoldify_bare

• go to deoldify_bare and cleanup image history:

  cd deoldify_bare
  java -jar /path/to/bfg.jar --no-blob-protection --delete-files "*.jpg"
  java -jar /path/to/bfg.jar --no-blob-protection --delete-files "*.jpeg"
  java -jar /path/to/bfg.jar --no-blob-protection --delete-files "*.[Pp][Nn][Gg]"
  git reflog expire --expire=now --all 
  git gc --prune=now --aggressive
  git push

  That would cleanup history and reduce repo size to around 77M (already 3 times smaller).

By that point there will be only one huge file left in history -- ColorizeVisualization.ipynb. Not sure on why it got that big in history (though it's still pretty small). Didn't trace back what was going on with it.

You also can completely remove bare repo here. Clone a new one normal repo (or pull the changes to existing clone, but be careful with history changes, so you won't endup pushing old history again).

In normal repo we need to bring back the images, install lfs and push the images there.

git lfs install
git lfs track "*images/*"
cp -r /path/to/test_images/ /path/to/result_images/ /repo/
# as we already set lfs track on images it should be automatically treated as lfs
git add test_images result_images
git commit -m "comment" test_images result_images
git push origin master

Meanwhile I can write a mini-howto and add it to README on how to checkout and work with LFS. That can be done via pull request :)

PS. Here is cleaned repo. Though I couldn't add git-lfs images due to bug which doesn't allow adding new lfs objects to public forks.

[jantic]

Wow man that's fantastic. Yeah, this is new to me so yeah- wasn't sure what all was required to get this going. And I was distracted with day job lol. Please do add to the README- that would be so appreciated. I'll merge it in when I get the steps above done. I'll try working on this today.

[jantic]

Ok after a minor hiccup I think I got this in place now. I didn't get the history down to the size you reported but it's definitely smaller. I'll close for now but if I screwed up please let me know! I'll add weights to this (making a separate issue for that) once I see that the dust has settled on this.

I'm -so- glad you found this and found this early!

[avaika]

@jantic doesn't look that images were migrated to lfs. I can still see it in the main repo. Are you sure you did the bfg part on raw mirror?

[jantic]

Definitely did the raw mirror steps verbatim. I must be misunderstanding something here.

I'll come back to this later. I don't think it's a pressing need given the images are pretty static at this point.

[jantic]

Reopening for future consideration

[jantic]

I'm a bit confused what you're seeing actually. If you look at the images, it says this on GitHub

Stored with Git LFS

I think we're good to go...right...?