Reduce repo size ? Squash Commit ?

napari / napari.github.io

website for the napari image viewer

https://napari.org/

44 stars 43 forks source link

Reduce repo size ? Squash Commit ? #313

Open Carreau opened 2 years ago

Carreau commented 2 years ago

I'm currently on a "slow" connection (~1MiB/s, fluctuating) and it takes forever to clone, and regularly fail (connection reset by peer), my guess github has a timeout.

Can I suggest to have a napari.github.io-full that potentially contain the full history, and this being only a squashed version ? Or maybe there is a way to work only with a shallow clones, but last time I checked you can't push from shallow clone.

It may also speedup CI operation as this full repo is ~900Mb, and the worktree is "only" 170Mb.

I could also suggest to see if we can convert most of the gif to webm to get a ~10x decrease in animation size ? Webm is quite well supported.

3.4M  painting.gif
223K  painting.webm
 20M  pathology.gif
2.8M  pathology.webm
 18M  point_annotator_demo.gif
100K  point_annotator_demo.webm

(ffmpeg -i painting.gif -c vp9 -b:v 0 -crf 40 painting.webm)

melissawm commented 6 months ago

This would be very helpful if we follow the strategy on https://github.com/napari/docs/pull/230, so that checking out this repo under that action takes less time.

melissawm commented 1 month ago

I feel like we just did it BUT I'd like to raise another point for future consideration. Copying over my message from zulip:

Looking at the recent versions it looks like the fallback videos are adding way too much weight to the artifact. So I wonder if there's a way to correct that. Especially since these videos are not autogenerated, meaning that we know exactly when they are updated so it would be fine to only keep one copy of each and not have it duplicated across different versions.

jni commented 1 month ago

So, I think we can fix things at deploy time by writing a script that:

finds duplicate videos and images
figures out the most recent folder containing those
changes all links for that file to point to that one
deletes the older versions

melissawm commented 1 month ago

I believe we can close this now, unless we want to further discuss compressing the images or finding other solutions.

jni commented 1 month ago

The bulk of it is done indeed, thank you @melissawm! That's very exciting, as you see this is a very old issue. 😃

I would say though we should keep this open until we:

enable and automate image compression (e.g. by a post-commit action)
enable either periodic or continuous repo squash (rather than having it be manual)