Repository size

[navidcy]

I noticed that the repository has been growing in size... At the moment is 131M.

(base) $ git clone --mirror git@github.com:CliMA/Oceananigans.jl.git Oceananigans-6.jl
Cloning into bare repository 'Oceananigans-6.jl'...
remote: Enumerating objects: 103248, done.
remote: Counting objects: 100% (2544/2544), done.
remote: Compressing objects: 100% (1078/1078), done.
remote: Total 103248 (delta 1852), reused 2071 (delta 1466), pack-reused 100704
Receiving objects: 100% (103248/103248), 119.80 MiB | 1.54 MiB/s, done.
Resolving deltas: 100% (78348/78348), done.
(base) $ du -sh Oceananigans-6.jl
131M    Oceananigans-6.jl

I run a script to find the big files. There are some .jld2 files in the GitHub repo... and also some report-....nsys-rep files.... (@simone-silvestri?)

All sizes are in kB's. The pack column is the size of the object, compressed, inside the pack file.
size   pack   SHA                                       location
48992  16829  1dd3f4706ebc79549624bee40f78b5aa2c0d644d  report-last.sqlite
48916  16814  839a363a9dd693e35abb48f4822be360ac313a4c  report-shared-memory.sqlite
20694  12212  950954a5d3ef4fac9d6d0480886b95e66d7ec510  report-shufflin.nsys-rep
20664  12149  283618488234bf4285558c4da6617a6c9583482c  report-last.nsys-rep
20605  12036  1f746e5f46363e656115984a84ddae8003572135  report-shared-memory.nsys-rep
2465   1984   eeeca1f2b394f43e2547462dc8bbc09d8b2e379d  test/deep_convection_golden_master_model_checkpoint_10.jld
2229   628    f72f4d2641339222c75da6404a1808dd8e4a8952  logger
1413   1303   7ef3d2c84f36e91ee7c4988629454c05cc451a6b  docs/src/appendix/convergence_plots/cosine_advection_diffusion_solutions.png
1407   1399   2f48fac8a7f5f0202abd72c6d90a2603c42b3430  paper/free_convection_and_baroclinic_instability.png
1319   1220   061ab36b8d442c52e2564826d9c5cbb72462654a  docs/src/verification/convergence_plots/cosine_advection_diffusion_solutions.png
1062   972    841a7461932f5ceefa996d3f993fdda7b8e042ea  docs/src/appendix/convergence_plots/gaussian_advection_diffusion_solutions.png
990    905    937939cc1ef2ec5ef69a08c6c79b79ac45715af4  docs/src/verification/convergence_plots/gaussian_advection_diffusion_solutions.png
924    674    bddab0c2f59062e7013b49c582242d338e6cb735  test/regression_tests/data/data_rayleigh_benard_regression.jld2
796    794    e2cf3b5a147885d73a4262590bf440e35337a8e5  examples/Inertial_Instability_2D_old.mp4
730    550    0ee7298c84ad9e9fa223824ef3ed9f1584c0695a  test/thermal_bubble_golden_master_model_checkpoint_10.jld
718    505    41f8e56c345f11959fcfc70a545f5abb8cbe0a02  test/regression_tests/data/ocean_large_eddy_simulation_AnisotropicMinimumDissipation_iteration10000.jld2
718    506    a7a57fa8fdc70604f81db7062541f065119d44b5  test/regression_tests/data/ocean_large_eddy_simulation_AnisotropicMinimumDissipation_iteration10010.jld2
718    506    51891abf2cd18ddbc2b2b77af54dd7d9bfd2e71c  test/regression_tests/data/ocean_large_eddy_simulation_SmagorinskyLilly_iteration10010.jld2
718    506    c48525b35c1b6870a2a3594cab85f73dec71787d  test/regression_tests/data/ocean_large_eddy_simulation_SmagorinskyLilly_iteration10000.jld2
718    479    3519eeb0dea062209f7940c7d5cb6442d2fcdd37  test/regression_tests/data/ocean_large_eddy_simulation_VerstappenAnisotropicMinimumDissipation_iteration10010.jld2
718    479    fbf720bf84dcb9a2d2f429187851a7f891989da7  test/regression_tests/data/ocean_large_eddy_simulation_VerstappenAnisotropicMinimumDissipation_iteration10000.jld2
718    479    5b796cdfdf8e792506ea49835ecb81aa5769f52d  test/regression_tests/data/ocean_large_eddy_simulation_SmagorinskyLilly_iteration10010.jld2
718    479    ba4645921310e028a606481e85f20c2529f3909d  test/regression_tests/data/ocean_large_eddy_simulation_SmagorinskyLilly_iteration10000.jld2
712    496    9765742b042b9df25192c4ddbaf6de02d858b770  test/regression_tests/data/rayleigh_benard_iteration1000.jld2
712    496    d6932dc59613019ec19000d84700fd8e20fb7ee7  test/regression_tests/data/rayleigh_benard_iteration1100.jld2

I can use BFG repo-cleaner to remove those files from the git history of the repo. But NOTE that everyone would have to delete their local clones after that and reclone. Otherwise you'd push them back at next git push.

[navidcy]

it might be tricky to do this... I don't know how to coordinate to do it and then kaboom! everyone deleting their local clones...

cc @glwagner, @simone-silvestri, @francispoulin, @tomchor, @jagoosw, @sandreza, @vchuravy

[simone-silvestri]

oh, yeah those reports might have slipped in sometime. Can't we track back the commit that pushed them and remove them from the history?

[navidcy]

Yeap. But I believe we'd need to make sure that we all reclone after.

cc @Angus-g in case they have in mind a solution that doesn't require everyone deleting their local repo clones

[francispoulin]

I am happy to do whatever people feel is best.

[angus-g]

I think because at least the sqlite and nsys-rep files are isolated in the ss/splace_filling branch, it's probably sufficient to just delete that branch? If they're unreferenced, they'll eventually get cleaned up (or you can run git gc). I'm not sure how aggressively GitHub runs that, i.e. if you delete the branch from GitHub's side, when will the associated objects disappear? It'll happen "eventually", but at least that way everybody doesn't need history modification of main and a reclone.

[simone-silvestri]

Thanks @angus-g, we can definitely delete that branch!

[navidcy]

And with this opportunity perhaps we should all go through the branches and delete any old stale branches we own...

[glwagner]

Can you add those files to .gitignore to prevent this from happening in the future?

[navidcy]

@simone-silvestri should we delete branch ss/splace_filling?

[glwagner]

def delete if its taking up size.

For heavy branches, better to use forks.