DioxusLabs / taffy

A high performance rust-powered UI layout library
https://docs.rs/taffy
Other
2.12k stars 107 forks source link

Git history contains large unneeded files #233

Closed Speykious closed 2 years ago

Speykious commented 2 years ago

The git history contains large unneeded files. When cloning the repository, it copies about ~53 MiB of data, which is unusual for a Rust crate about layouting.

So I investigated the largest objects and found that there was previously a folder bindings in which a lot of binaries were inserted in the git history:

git rev-list --objects --all |\
  git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' |\
  sed -n 's/^blob //p' |\
  sort --numeric-sort --key=2 |\
  cut -c 1-12,41- |\
  $(command -v gnumfmt || echo numfmt) --field=2 --to=iec-i --suffix=B --padding=7 --round=nearest |\
  tail -n 42
6d3cc77f162e  512KiB tests/generated.rs
05cb789092ab  512KiB tests/generated.rs
8c7e3ce27ddd  516KiB bindings/kotlin/stretch/src/main/rust/target/debug/deps/liblog-3af23aa4c7c67e2a.rlib
df89457fc5d6  516KiB tests/generated.rs
2a0491e1fd5d  516KiB tests/generated.rs
18ac5e513ef2  528KiB docs/yarn.lock
ff31193dcf07  559KiB bindings/kotlin/stretch/src/main/rust/target/debug/deps/liberror_chain-a5baabbc578187ac.rlib
cafe08d1af9f  582KiB bindings/kotlin/stretch/src/main/rust/target/debug/deps/libcesu8-e477268f74187945.rlib
9687b4c15a42  593KiB bindings/kotlin/stretch/src/main/rust/target/debug/deps/libbyteorder-d9f384ad4814faa0.rlib
421a7717343d  614KiB bindings/kotlin/stretch/src/main/rust/target/debug/build/byteorder-45f01b6b3f9adf1f/build-script-build
a5b426933f6f  624KiB bindings/kotlin/stretch/src/main/rust/target/debug/build/memchr-93a5c8e4430ff8a3/build-script-build
beba97e4722b  652KiB bindings/kotlin/stretch/src/main/rust/target/debug/deps/libmemchr-9d91e78ec73233a7.rlib
ba82ca16ec27  775KiB bindings/kotlin/stretch/src/main/rust/target/debug/build/jni-b3ae1fb6623bdd28/build_script_build-b3ae1fb6623bdd28.dSYM/Contents/Resources/DWARF/build_script_build-b3ae1fb6623bdd28
50b6c7582975  961KiB bindings/kotlin/stretch/src/main/rust/target/debug/deps/libascii-9ca039f21602eeeb.rlib
8b82df6444ec  1.4MiB bindings/kotlin/stretch/src/main/rust/target/debug/deps/libstretch.dylib.dSYM/Contents/Resources/DWARF/libstretch.dylib
4a9921fb1525  1.6MiB bindings/kotlin/stretch/src/main/rust/target/debug/deps/libwalkdir-2fb148241b85af72.rlib
3a813f0a876f  1.7MiB bindings/kotlin/stretch/src/main/rust/target/debug/build/memchr-93a5c8e4430ff8a3/build_script_build-93a5c8e4430ff8a3.dSYM/Contents/Resources/DWARF/build_script_build-93a5c8e4430ff8a3
28b9bbdd992a  1.7MiB bindings/kotlin/stretch/src/main/rust/target/debug/build/byteorder-45f01b6b3f9adf1f/build_script_build-45f01b6b3f9adf1f.dSYM/Contents/Resources/DWARF/build_script_build-45f01b6b3f9adf1f
12822b4f1ad6  2.3MiB bindings/kotlin/stretch/src/main/libs/x86/libstretch.so
916cf790c691  2.3MiB bindings/kotlin/stretch/src/main/libs/armeabi-v7a/libstretch.so
6b9df4547351  2.4MiB bindings/kotlin/stretch/src/main/libs/armeabi-v7a/libstretch.so
bcca778e9df6  2.4MiB bindings/kotlin/stretch/src/main/libs/armeabi-v7a/libstretch.so
3f9127f13038  2.5MiB bindings/kotlin/stretch/src/main/libs/armeabi-v7a/libstretch.so
12aa9a059393  2.5MiB bindings/kotlin/stretch/src/main/libs/arm64-v8a/libstretch.so
4a2fca50fc34  2.5MiB bindings/kotlin/stretch/src/main/libs/x86/libstretch.so
f39f1cdba312  2.5MiB bindings/kotlin/stretch/src/main/libs/x86/libstretch.so
0e5f0c96ab21  2.6MiB bindings/kotlin/stretch/src/main/libs/x86/libstretch.so
4d05d07d5bd3  2.8MiB bindings/kotlin/stretch/src/main/libs/arm64-v8a/libstretch.so
62b879c13eea  2.8MiB bindings/kotlin/stretch/src/main/libs/arm64-v8a/libstretch.so
c0f318ecf5d0  2.8MiB bindings/kotlin/stretch/src/main/libs/arm64-v8a/libstretch.so
b20ef20d041f  6.1MiB bindings/kotlin/stretch/src/main/rust/target/i686-linux-android/release/deps/libcombine-c0dbcbc94374836d.rlib
2c8883043c2c  6.1MiB bindings/kotlin/stretch/src/main/rust/target/armv7-linux-androideabi/release/deps/libcombine-c0dbcbc94374836d.rlib
30f05ec5f4f0  6.1MiB bindings/kotlin/stretch/src/main/rust/target/aarch64-linux-android/release/deps/libcombine-c0dbcbc94374836d.rlib
8275736f6473  6.4MiB bindings/kotlin/stretch/src/main/rust/target/debug/deps/libcombine-d7adb7fab46dc97a.rlib
6f256e65743d  6.5MiB bindings/kotlin/stretch/src/main/rust/target/armv7-linux-androideabi/release/deps/libjni-e150d04fb1ab2894.rlib
7a021f1485ef  6.5MiB bindings/kotlin/stretch/src/main/rust/target/i686-linux-android/release/deps/libjni-e150d04fb1ab2894.rlib
a0c5f7319cec  6.8MiB bindings/kotlin/stretch/src/main/rust/target/aarch64-linux-android/release/deps/libjni-e150d04fb1ab2894.rlib
8410e9565a09   10MiB scripts/gentest/selenium.jar
21a3b1a809a1   11MiB bindings/kotlin/stretch/src/main/rust/target/debug/deps/libjni-38faaa70ee598caf.rlib
ea2ecb13a778   19MiB bindings/swift/StretchKit/Libraries/libstretch.a
db8bee31ebdd   19MiB bindings/swift/StretchKit/Libraries/libstretch.a
acbf40f1ba16   33MiB bindings/swift/StretchKit/Libraries/libstretch.a

Fixing this would require rewriting the git history, although I think it's worth it if that means people won't get these unused files.

alice-i-cecile commented 2 years ago

Yep, I'm fine to rewrite history here. Are you able to do that via a PR?

Speykious commented 2 years ago

I'm not sure honestly... I could try though. I'll see what I can do.

mockersf commented 2 years ago

I think you need write access to the main branch

Speykious commented 2 years ago

So I tried to do it today, and it was actually rather easy to do with the BFG repo cleaner.

Here are the commands that I ran. I did that on a folder named taffy.git, which is obtained from git clone --mirror git@github.com:DioxusLabs/taffy.git.

# Delete the bindings/ folder from history where almost every big blob is
bfg --delete-folders bindings taffy.git

# Delete the only blob that was left outside of the bindings/ folder
bfg --delete-files selenium.jar taffy.git

# BFG only rewrites commits but doesn't actually remove the blobs, so we do this now
cd taffy.git
git reflog expire --expire=now --all && git gc --prune=now --aggressive

Here's the end result:

git rev-list --objects --all |\
  git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' |\
  sed -n 's/^blob //p' |\
  sort --numeric-sort --key=2 |\
  cut -c 1-12,41- |\
  $(command -v gnumfmt || echo numfmt) --field=2 --to=iec-i --suffix=B --padding=7 --round=nearest |\
  tail -n 10
e36ac7aba85b  455KiB tests/generated.rs
8d68c589925c  466KiB docs/yarn.lock
15e1acdeafdc  467KiB tests/generated.rs
03a18ea25baa  468KiB tests/generated.rs
d22058ee45fb  503KiB docs/yarn.lock
6d3cc77f162e  512KiB tests/generated.rs
05cb789092ab  512KiB tests/generated.rs
df89457fc5d6  516KiB tests/generated.rs
2a0491e1fd5d  516KiB tests/generated.rs
18ac5e513ef2  528KiB docs/yarn.lock

However, I think it would only be useful if I had write access. So I think it's best if a maintainer/owner does it. :) In which case, I highly recommend doing experiments on a --mirror clone (like I said at the beginning) so that you can make sure to not delete something that wasn't meant to be deleted.

alice-i-cecile commented 2 years ago

Thank you very much for the detailed write-up and advice. I'm not a master of git, so it really will help me. @jkelleyrtp are you okay if I rewrite history to do this?

jkelleyrtp commented 2 years ago

@Speykious thanks for figuring out the commands!

@alice-i-cecile yes please! We might need to temporarily turn off main branch protections - LMK if you need action from me there

jkelleyrtp commented 2 years ago

Done, thanks for figuring out the way forward with this one :)