Consider FaTul for storing blueprints

[nyurik]

Over the last weekend this repo inspired me to write FaTul - a way to store blueprints with minimal changes between revisions, and being able to easily inspect them. I ran my tool on the whole history of this repo, resulting in https://github.com/nyurik/brians-blueprints/tree/rebase -- a branch with the significantly smaller history and easy to diff each revision.

I ran git log --shortstat "--pretty=format:%ad | %s [%an]\n%h" --date=short --reverse to see the difference between the old and the new (even this command took 5s on the new branch, and 20s on the old one):

 2022-05-30 | Initial commit [bcwhite0]
 fa54e09 2 files changed, 123 insertions(+)

+2022-07-11 | Add FaTul (history rewrite) to optimize storage [Yuri Astrakhan]
+ab5b7ed 4 files changed, 683 insertions(+)
+
 2022-06-01 | Initial commit of simple split/build tool. [Brian White]
-e21919d 3 files changed, 199 insertions(+)
+5ff1a50 3 files changed, 199 insertions(+)

 2022-06-01 | Import v4.20 of Brian's Trains [Brian White]
-7db951a 218 files changed, 461055 insertions(+)
+8fdaf73 110 files changed, 82667 insertions(+)

 2022-06-01 | Create brians-trains.md [Brian White]
-46901de 1 file changed, 220 insertions(+)
+f7889f1 1 file changed, 220 insertions(+)

 2022-06-01 | Updated embedded lincense information. [Brian White]
-c81ba74 2 files changed, 4 insertions(+), 1 deletion(-)
+43498c6 2 files changed, 4 insertions(+), 1 deletion(-)

 2022-06-01 | Changed file format of individual blueprints to allow working on singles. Added pack/unpack actions to main script. [Brian White]
-860cba4 99 files changed, 449184 insertions(+), 448965 deletions(-)
+cc5f5b9 1 file changed, 34 insertions(+), 11 deletions(-)

 2022-06-01 | Added command examples to README [Brian White]
-fd69111 1 file changed, 27 insertions(+), 1 deletion(-)
+b8bcf3b 1 file changed, 27 insertions(+), 1 deletion(-)

 2022-06-03 | Support for upgrade planners. [Brian White]
-ec7cf95 1 file changed, 7 insertions(+), 4 deletions(-)
+e39952f 1 file changed, 7 insertions(+), 4 deletions(-)

 2022-06-03 | Import v2.0 of self-building-angelbob blueprints [Brian White]
-06e40d9 103 files changed, 3271387 insertions(+)
+8a8638c 52 files changed, 685218 insertions(+)

 2022-06-03 | Fixed duplicate name of rail block 2x1 in angelbob. [Brian White]
-d14a3f5 4 files changed, 19432 insertions(+), 5249 deletions(-)
+67da8f3 2 files changed, 3115 insertions(+), 981 deletions(-)

 2022-06-03 | Added note to visit the "all releases" page. [Brian White]
-32551d3 1 file changed, 3 insertions(+), 1 deletion(-)
+4f68531 1 file changed, 3 insertions(+), 1 deletion(-)

 2022-06-03 | Imported Brian's Bootstrap v1.2.0 [Brian White]
-1b77def 11 files changed, 24726 insertions(+)
+5e3bb57 6 files changed, 3352 insertions(+)

 2022-06-03 | Imported Finite State Machive v1.2 [Brian White]
-4d81db5 1 file changed, 10150 insertions(+)
+2b52db2 1 file changed, 3315 insertions(+)

 2022-06-03 | Import Self Building Rocket v1.1.5 [Brian White]
-14b3004 25 files changed, 290512 insertions(+)
+74bd2be 13 files changed, 46918 insertions(+)

 2022-06-03 | Imported Brian's Trains (Auxiliary) v4.9 [Brian White]
-ccd6337 51 files changed, 68603 insertions(+)
+4decf2a 26 files changed, 16773 insertions(+)

 2022-06-03 | Imported Tileable Reactor v4.3.2 [Brian White]
-d31cdb6 17 files changed, 179904 insertions(+)
+6bff00d 9 files changed, 26233 insertions(+)

 2022-06-03 | Added .md documentation for blueprints. [Brian White]
-a7045c2 5 files changed, 714 insertions(+)
+4a18635 5 files changed, 714 insertions(+)

 2022-06-14 | Updated AngelBob mall for new recipes requiring bronze. [Brian White]
-5f64395 53 files changed, 28889 insertions(+), 41864 deletions(-)
+3b96bf5 53 files changed, 3291 insertions(+), 8898 deletions(-)

 2022-06-15 | Un-merged chests in AngelBob Mall. [Brian White]
-8be2dc3 2 files changed, 38388 insertions(+), 22272 deletions(-)
+341aa73 2 files changed, 6804 insertions(+), 705 deletions(-)

 2022-06-15 | Fixed Angel/Bob purple science for new recipes. [Brian White]
-f482309 3 files changed, 174 insertions(+), 172 deletions(-)
+2790059 3 files changed, 72 insertions(+), 72 deletions(-)

 2022-06-15 | Fixed bad belt direction in updated purple science. [Brian White]
-84d7a33 2 files changed, 150 insertions(+), 150 deletions(-)
+70e2200 2 files changed, 77 insertions(+), 77 deletions(-)

 2022-06-16 | Fixed Acid & Catalysts which was somehow reverted. [Brian White]
-0b52f7a 3 files changed, 8779 insertions(+), 8393 deletions(-)
+8316811 3 files changed, 483 insertions(+), 420 deletions(-)

 2022-06-21 | Acids/Catalysts improvements; tungesten gears to rocket engines. [Brian White]
-57a6ed6 4 files changed, 41458 insertions(+), 38239 deletions(-)
+a14e650 4 files changed, 613 insertions(+), 73 deletions(-)

 2022-06-22 | Changed requester station divisor back to actual chest count (no performance improvement from off-by-one) Changed station-rail exit signal to chain-signal so trains can't exit and immediately block the main track. [Brian White]
-7bf34f0 109 files changed, 24908 insertions(+), 25064 deletions(-)
+4566f61 109 files changed, 8841 insertions(+), 9327 deletions(-)

 2022-06-23 | Print message if object has no label. [Brian White]
-fc8aad6 1 file changed, 3 insertions(+), 1 deletion(-)
+544d8af 1 file changed, 3 insertions(+), 1 deletion(-)

 2022-06-23 | Imported self-building-factory v3.5.7 [Brian White]
-1d59da5 159 files changed, 6507475 insertions(+)
+b980b7c 80 files changed, 1073933 insertions(+)

 2022-06-23 | Updated numerous station names to new style. [Brian White]
-68d9bac 6 files changed, 65 insertions(+), 65 deletions(-)
+d0d5954 6 files changed, 64 insertions(+), 64 deletions(-)

 2022-06-23 | Fixed some station names. [Brian White]
-fd214d0 4 files changed, 9 insertions(+), 9 deletions(-)
+29dc723 4 files changed, 5 insertions(+), 5 deletions(-)

 2022-06-23 | Added missing station names to many blocks. [Brian White]
-9173dd8 31 files changed, 696713 insertions(+), 696865 deletions(-)
+f785e04 30 files changed, 150774 insertions(+), 150806 deletions(-)

 2022-06-23 | Removed unused variable. [Brian White]
-98eac46 1 file changed, 1 deletion(-)
+2355b92 1 file changed, 1 deletion(-)

 2022-06-24 | Allow unicode in json (#2) [Yuri Astrakhan]
-7600883 1 file changed, 2 insertions(+), 2 deletions(-)
+0ef5160 1 file changed, 2 insertions(+), 2 deletions(-)

 2022-06-24 | Cleanup json before changes (#3) [Yuri Astrakhan]
-83db559 264 files changed, 749 insertions(+), 749 deletions(-)
+031dc12 1 file changed, 1 insertion(+), 1 deletion(-)

 2022-06-26 | Added embedded descriptions for multi stations. [Brian White]
-33330a5 26 files changed, 7891 insertions(+), 7743 deletions(-)
+11e9ebe 26 files changed, 2572 insertions(+), 2344 deletions(-)

 2022-06-26 | Set station names to all city BPs (#5) [Yuri Astrakhan]
-ec958f5 21 files changed, 162 insertions(+), 162 deletions(-)
+93bc73e 21 files changed, 162 insertions(+), 162 deletions(-)

 2022-06-26 | Updated description for 3.5.8. [Brian White]
-8b0c9f3 1 file changed, 2 insertions(+), 2 deletions(-)
+06cbe60 1 file changed, 2 insertions(+), 2 deletions(-)

 2022-06-26 | Added landfill under all BT rail blueprints. [Brian White]
-56b7165 48 files changed, 743736 insertions(+), 49 deletions(-)
+b5402a6 36 files changed, 106337 insertions(+), 36 deletions(-)

 2022-07-01 | First pieces of a SpaceX blueprint book. [Brian White]
-babc8ed 30 files changed, 453785 insertions(+)
+71e78ba 16 files changed, 89190 insertions(+)

 2022-07-01 | Better performance when exporting blueprint strings. [Brian White]
-2ceff96 1 file changed, 1 insertion(+), 1 deletion(-)
+cef2e88 1 file changed, 1 insertion(+), 1 deletion(-)

 2022-07-01 | Added some more links to documentation. [Brian White]
-776a50b 1 file changed, 3 insertions(+), 1 deletion(-)
+077424d 1 file changed, 3 insertions(+), 1 deletion(-)

 2022-07-01 | Fixed steam power is SpaceX build. [Brian White]
-2ecf5a0 1 file changed, 3989 insertions(+), 3943 deletions(-)
+520ce2b 1 file changed, 231 insertions(+), 196 deletions(-)

 2022-07-01 | Small documentation improvements. [Brian White]
-07a2c06 1 file changed, 4 insertions(+), 4 deletions(-)
+c5d00b1 1 file changed, 4 insertions(+), 4 deletions(-)

 2022-07-01 | Added small amounts of sulfur and sulfuric acid production to Basic Science. [Brian White]
-63f3337 1 file changed, 27524 insertions(+), 26752 deletions(-)
+66a06fe 1 file changed, 193 insertions(+), 41 deletions(-)

 2022-07-02 | Improvments to Basic Science and Mall. [Brian White]
-188405d 4 files changed, 3157 insertions(+), 3194 deletions(-)
+e6697b8 4 files changed, 9 insertions(+), 13 deletions(-)

 2022-07-03 | New 'belt' block; fixed BS loaders; added Mall solid fuel. [Brian White]
-be3bbd8 6 files changed, 45766 insertions(+), 29951 deletions(-)
+c014e8b 5 files changed, 3542 insertions(+), 34 deletions(-)

 2022-07-05 | SE: refinery, steel, pumpjack, belts [Brian White]
-2c428aa 14 files changed, 288106 insertions(+), 9371 deletions(-)
+b11fdd7 9 files changed, 54020 insertions(+), 47 deletions(-)

 2022-07-05 | SE: depot hub, mine [Brian White]
-def7bc5 8 files changed, 180980 insertions(+), 4809 deletions(-)
+8baa4fc 6 files changed, 30522 insertions(+), 12 deletions(-)

 2022-07-06 | SE: kovarex/nuclear, modules, fixes for belts & refinery [Brian White]
-8f18331 24 files changed, 292211 insertions(+), 57980 deletions(-)
+8e9295a 16 files changed, 42534 insertions(+), 552 deletions(-)

 2022-07-06 | SBF: fixed Construction Supply, restored missing Q1 rail template [Brian White]
-4035a0b 77 files changed, 437167 insertions(+), 433132 deletions(-)
+43ae5a4 22 files changed, 190715 insertions(+), 190239 deletions(-)

 2022-07-06 | SE: fixed light oil cracking in Refinery [Brian White]
-5a5fcd2 4 files changed, 45658 insertions(+), 45711 deletions(-)
+2d30d15 4 files changed, 34 insertions(+), 46 deletions(-)

 2022-07-07 | AUX: updated signals & locked to grid [Brian White]
-441ce88 7 files changed, 7770 insertions(+), 7829 deletions(-)
+ff3da9a 7 files changed, 3285 insertions(+), 3275 deletions(-)

 2022-07-07 | AUX: added landfill under tracks [Brian White]
-0ea7c09 6 files changed, 91760 insertions(+), 1 deletion(-)
+fef55c7 6 files changed, 13123 insertions(+), 6 deletions(-)

 2022-07-08 | SE: nuclear, circuits, bricks&glass, lds [Brian White]
-4f4e5c6 23 files changed, 736839 insertions(+), 119118 deletions(-)
+b4cd55e 16 files changed, 110629 insertions(+), 70 deletions(-)

 2022-07-08 | SBF: re-export with all necessary mods installed [Brian White]
-76b5c34 82 files changed, 1148 insertions(+), 931 deletions(-)
+23a3a7c 82 files changed, 261 insertions(+), 119 deletions(-)

How I got it

To test how this repo would look if stored using fatul, I wrote a simple script to convert it. To try this rebasing yourself:

• make sure your repo is clean, and up to date with github. You may want to create another clone dir to be certain.
• create a new branch git checkout -b fatul fead642 (this will only have the initial files without any data)
• copy fatul.py, encode, decode, and dump files to the /tools dir and commit it with git commit -a -m "adding fatul tools"
• create a new branch off of the main branch for the rebase, e.g. git checkout -b rebase main
• place this script (inside the details section below) one level above the brians-blueprints git repo, e.g. as ../recode.sh and make it executable chmod +x ../recode.sh
• run automatic rebase with git rebase fatul --exec ../recode.sh (will take 10-15 min)
• Make sure you have filter-repo installed -- this is a massively popular tool for complex git changes.
• Delete all of the old data, keeping various top level files and the tools/ dir: git filter-repo --path-glob '*.md' --path LICENSE --path .gitignore --path tools/
• Move all dirs from tools/results to root dir, and keep all top level files and the tools/ dir: git filter-repo --path-glob '*.md' --path LICENSE --path .gitignore --path tools/ --subdirectory-filter=tools/results/
• At this point I had some issues with the repo possibly due to messing up the filter-repo tool, so I created another clone, fetched the data from the first one, created another branch rebase2 set to the initial commit git reset --hard fa54 and used git cherry-pick <revision_of_the_fatul_addition>^..rebase to apply all changes from the original rebase branch.

The rebase branch now has all the same data as the main, except it is stored in fatul format, and without any giant data blobs in history.

```bash #!/usr/bin/env bash set -e -o pipefail fatul=tools/fatul.py fbp=tools/fbp.py output=tools/results tmp_file=../tmp.txt tmp_output1=../output1.txt tmp_err1=../err1.txt tmp_output2=../output2.txt tmp_err2=../err2.txt rm -rf "${output}" for d in *; do if [ -d "$d" ] && [ "$d" != "tools" ]; then dir="$(basename "$d")" mkdir -p "$output/$dir" if [ -f "$d/.metadata" ]; then echo "Processing dir $d ($dir)" rm -rf "$tmp_file" python3 "$fbp" build --outfile="$tmp_file" --indir "$d" > "$tmp_output1" 2> "$tmp_err1" python3 "$fatul" decode "$output/$dir" "$tmp_file" > "$tmp_output2" 2> "$tmp_err2" else echo "processing files in $d" for d2 in "$d/"*; do if [ -d "$d2" ]; then echo "It's a dir! $d2" exet 1 fi filename="$(basename "$d2")" echo "processing file $d2 ($filename)" rm -rf "$tmp_file" python3 "$fbp" pack --outfile="$tmp_file" --infile "$d2" > "$tmp_output1" 2> "$tmp_err1" python3 "$fatul" decode "$output/$dir/$filename" "$tmp_file" > "$tmp_output2" 2> "$tmp_err2" done fi fi done git add . git commit -C HEAD --amend ```

[nyurik]

Hi @bcwhite-code , any thoughts on this? Thx!

[bcwhite-code]

I do plan to use your tool for future releases. I won't be replacing the existing history, though, just switching going forward.

[bcwhite-code]

All repositories have been migrated.