search

Devsh-Graphics-Programming / Nabla

Vulkan, OptiX and CUDA Interoperation Modular Rendering Library and Framework for PC/Linux/Android
http://devsh.eu
Apache License 2.0
444 stars 48 forks source link

The Nabla repository is a little too large to casually include in projects (due to unused git objects) #623

Closed expenses closed 3 months ago

expenses commented 7 months ago

Description

Hi, I did a fresh clone of Nabla and it required downloading 1.88 GB:

Cloning into 'Nabla'...
remote: Enumerating objects: 114400, done.
remote: Counting objects: 100% (1568/1568), done.
remote: Compressing objects: 100% (645/645), done.
remote: Total 114400 (delta 988), reused 1317 (delta 826), pack-reused 112832
Receiving objects: 100% (114400/114400), 1.88 GiB | 18.20 MiB/s, done.
Resolving deltas: 100% (85301/85301), done.

I'd love to be able to include it in my project for some of the glsl built-ins, but this is a little too large for me to casually include it as a git submodule if I just want to use one or two functions.

I ran a script from https://stackoverflow.com/a/46085465 to get the sizes of objects in the repository and it came up with example_tests:

eb171b419446   64MiB examples_tests/13.Silverlining/bin/.pbtmpc3PzpB
f25a8fce24f4   64MiB examples_tests/13.Silverlining/bin/.pbtmphgywp3
b73caa80802b   68MiB examples_tests/media/noises/spp_benchmark_4k_512.exr
416977c7e0d2   76MiB examples_tests/media/sponza.zip
84e57b3c513d   77MiB examples_tests/media/sponza.zip
7cd3b5db497e   77MiB examples_tests/media/sponza.zip
66e69dc8cd36   77MiB examples_tests/media/sponza.zip
6b0b55959fcd   77MiB examples_tests/media/sponza.zip
776c942fe2c2   78MiB examples_tests/41.VisibilityBuffer/bin/visibilitybuffer_rwdi.iobj
963cad7c3095   88MiB examples_tests/media/sponza.zip

I ran git filter-repo --invert-paths --path examples_tests/ --force to see what things would look like once that directory was removed from git history and the size of .git went down to 500MB. Running the script again results in site_media being the next culprit:

17d0dd1da35c   19MiB site_media/readme/gifs/autoexposure/exposure.gif
c5978c572a40   20MiB site_media/readme/gifs/denoisingalbedoandnormals/denoising 3.gif
d21db36eb090   22MiB site_media/readme/gifs/denoisingalbedoandnormals/denoising 4.gif
3e1e88317ee2   22MiB site_media/readme/gifs/raytracingflythrough/raytracing 4.gif
b180c2ef6602   23MiB site_media/readme/gifs/denoisingalbedoandnormals/denoising 2.gif
b78d5c002d14   24MiB site_media/readme/gifs/raytracingflythrough/raytracing 5.gif
87f630d9d65d   24MiB site_media/readme/gifs/denoisingalbedoandnormals/denoising 1.gif
7615cd16a19a   24MiB site_media/readme/gifs/denoisingalbedoandnormals/denoising 5.gif
6b3e61d6a6f9   24MiB site_media/readme/gifs/raytracingflythrough/raytracing 2.gif
6b1e3fa94e47   57MiB site_media/readme/gifs/raytracingflythrough/raytracing 6.gif

After running git filter-repo --invert-paths --path site_media --force the size went down again to a very acceptable 50MB.

Solution proposal

If you're comfortable with removing these two directories from git history then I say go for it. Obviously that's necessarily an easy decision to make though. examples_tests is now a submodule but (I might be mistaken) it seems like it was part of this repository in the past.

AnastaZIuk commented 7 months ago

Unfortunately we realize the problem exists and we will filter this history at some point, thanks for pointing it out. I've tried to cut the size first here https://github.com/Devsh-Graphics-Programming/Nabla/pull/600. Shall we sort this out @devshgraphicsprogramming?

@expenses current nice solution would be to do shallow clone, you will reduce the size to 6.19 MiB

$ git clone --depth=1 git@github.com:Devsh-Graphics-Programming/Nabla.git .
Cloning into '.'...
remote: Enumerating objects: 2663, done.
remote: Counting objects: 100% (2663/2663), done.
remote: Compressing objects: 100% (2275/2275), done.
remote: Total 2663 (delta 386), reused 1789 (delta 310), pack-reused 0
Receiving objects: 100% (2663/2663), 6.19 MiB | 6.11 MiB/s, done.
Resolving deltas: 100% (386/386), done.

If Nabla is your submodule you can shallow init and update it too

git submodule update --init --depth=1 <module_path_to_nabla>

If you build Nabla I recommend to let CMake do submodule updates for you, to request shallow updates we have NBL_CI_GIT_SUBMODULES_SHALLOW option, you can also exclude whole examples_tests (and parents) from the submodule request init & update by turning off NBL_BUILD_EXAMPLES option, to toggle the request you can use NBL_UPDATE_GIT_SUBMODULE option

AnastaZIuk commented 7 months ago

but (I might be mistaken) it seems like it was part of this repository in the past.

indeed, got filtered with git-filter-repo :D

devshgraphicsprogramming commented 7 months ago

I'd love to be able to include it in my project for some of the glsl built-ins, but this is a little too large for me to casually include it as a git submodule if I just want to use one or two functions.

@expenses you're probably better off justcopying the GLSL over into your project, because we're about to delete it in favour of HLSL.

If you're comfortable with removing these two directories from git history then I say go for it. Obviously that's necessarily an easy decision to make though. examples_tests is now a submodule but (I might be mistaken) it seems like it was part of this repository in the past.

The ditt branch is pretty far behind, and we need to keep it alive for some time (until a new Vulkan Path Tracer replaces the OpenGL+OpenCL+CUDA abomination we made).

Once that's done I'm somewhat open to "truncating" the history.

However I'll probably move out nbl/builtin/hlsl to a submodule first.

expenses commented 7 months ago

@expenses current nice solution would be to do shallow clone, you will reduce the size to 6.19 MiB

Ah, that's a great solution, I totally forgot about shallow clones!

@expenses you're probably better off justcopying the GLSL over into your project, because we're about to delete it in favour of HLSL.

Yep fair enough, I want to switch (back) to HLSL but I'm stuck with GLSL until I can use 8-bit ints, 64-bit atomics and BDA nicely.

devshgraphicsprogramming commented 7 months ago

@expenses current nice solution would be to do shallow clone, you will reduce the size to 6.19 MiB

Ah, that's a great solution, I totally forgot about shallow clones!

@expenses you're probably better off justcopying the GLSL over into your project, because we're about to delete it in favour of HLSL.

Yep fair enough, I want to switch (back) to HLSL but I'm stuck with GLSL until I can use 8-bit ints, 64-bit atomics and BDA nicely.

8bit is a problem, 64bit work just fine, BDA incoming

devshgraphicsprogramming commented 3 months ago

ok there's a lot of options in our CMake now to control shallow vs deep submodule checkout also we don't recursively clone all submodules anyway