search

BlueBrain / nmodl

Code Generation Framework For NEURON MODeling Language
https://bluebrain.github.io/nmodl/
Apache License 2.0
56 stars 15 forks source link

Shallow clone of submodules #90

Open pramodk opened 5 years ago

pramodk commented 5 years ago

If we do clone recursive, we get : 

$ du -h nmodl
577M    nmodl/

I think the culprit is nmodl/ext/cli11/extern/json'. But we should do shallow clones : https://stackoverflow.com/questions/2144406/how-to-make-shallow-git-submodules

lkeegan commented 5 years ago

git clone --recurse-submodules git@github.com:BlueBrain/nmodl.git

gives 

error: Server does not allow request for unadvertised object 6b2f7346db5cf5d438aa35ee16d4cb7fe5a71e46
Fetched in submodule path 'cmake/hpc-coding-conventions', but it did not contain 6b2f7346db5cf5d438aa35ee16d4cb7fe5a71e46. Direct fetching of that commit failed.

I think because we are currently several commits behind the hpc-coding-conventions repo, and the shallow clone only fetches the last commit, which doesn't match the one requested.

git submodule update --depth 10 worked

pramodk commented 5 years ago

Interesting. Older git version on BB5 gives : 

kumbhar@bbpv2:~/tmp$ git --version
git version 1.8.3.1
kumbhar@bbpv2:~/tmp$ git clone --recurse-submodules git@github.com:BlueBrain/nmodl.git nmodl_new
Cloning into 'nmodl_new'...
remote: Enumerating objects: 276, done.
remote: Counting objects: 100% (276/276), done.
remote: Compressing objects: 100% (73/73), done.
remote: Total 3423 (delta 227), reused 211 (delta 201), pack-reused 3147
Receiving objects: 100% (3423/3423), 1.55 MiB | 0 bytes/s, done.
Resolving deltas: 100% (2351/2351), done.
Submodule 'cmake/hpc-coding-conventions' (https://github.com/BlueBrain/hpc-coding-conventions.git) registered for path 'cmake/hpc-coding-conventions'
Submodule 'ext/cli11' (https://github.com/CLIUtils/CLI11.git) registered for path 'ext/cli11'
Submodule 'ext/eigen' (https://github.com/eigenteam/eigen-git-mirror.git) registered for path 'ext/eigen'
Submodule 'ext/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'ext/pybind11'
Cloning into 'cmake/hpc-coding-conventions'...
remote: Enumerating objects: 177, done.
remote: Counting objects: 100% (177/177), done.
remote: Compressing objects: 100% (121/121), done.
remote: Total 452 (delta 120), reused 94 (delta 56), pack-reused 275
Receiving objects: 100% (452/452), 106.04 KiB | 0 bytes/s, done.
Resolving deltas: 100% (252/252), done.
Submodule path 'cmake/hpc-coding-conventions': checked out '6b2f7346db5cf5d438aa35ee16d4cb7fe5a71e46'
Cloning into 'ext/cli11'...
remote: Enumerating objects: 8568, done.
remote: Total 8568 (delta 0), reused 0 (delta 0), pack-reused 8568
Receiving objects: 100% (8568/8568), 5.95 MiB | 3.24 MiB/s, done.
Resolving deltas: 100% (6511/6511), done.
Submodule path 'ext/cli11': checked out '2cd58ef8cf185c74a7263b7e0e3b37aadd396f6e'
Submodule 'extern/googletest' (https://github.com/google/googletest.git) registered for path 'extern/googletest'
Submodule 'extern/json' (https://github.com/nlohmann/json.git) registered for path 'extern/json'
Submodule 'extern/sanitizers' (https://github.com/arsenm/sanitizers-cmake) registered for path 'extern/sanitizers'
Cloning into 'extern/googletest'...
remote: Enumerating objects: 16, done.
remote: Counting objects: 100% (16/16), done.
remote: Compressing objects: 100% (10/10), done.
remote: Total 16471 (delta 6), reused 15 (delta 6), pack-reused 16455
Receiving objects: 100% (16471/16471), 5.73 MiB | 1.93 MiB/s, done.
Resolving deltas: 100% (12139/12139), done.
Submodule path 'ext/cli11/extern/googletest': checked out 'ec44c6c1675c25b9827aacd08c02433cccde7780'
Cloning into 'extern/json'...
remote: Enumerating objects: 148, done.
remote: Counting objects: 100% (148/148), done.
remote: Compressing objects: 100% (89/89), done.
remote: Total 46579 (delta 82), reused 100 (delta 45), pack-reused 46431
Receiving objects: 100% (46579/46579), 173.25 MiB | 9.47 MiB/s, done.
Resolving deltas: 100% (37737/37737), done.
Submodule path 'ext/cli11/extern/json': checked out 'db53bdac1926d1baebcb459b685dcd2e4608c355'
Cloning into 'extern/sanitizers'...
remote: Enumerating objects: 214, done.
remote: Total 214 (delta 0), reused 0 (delta 0), pack-reused 214
Receiving objects: 100% (214/214), 47.80 KiB | 0 bytes/s, done.
Resolving deltas: 100% (136/136), done.
Submodule path 'ext/cli11/extern/sanitizers': checked out '6947cff3a9c9305eb9c16135dd81da3feb4bf87f'
Cloning into 'ext/eigen'...
remote: Enumerating objects: 101040, done.
remote: Total 101040 (delta 0), reused 0 (delta 0), pack-reused 101040
Receiving objects: 100% (101040/101040), 87.33 MiB | 27.97 MiB/s, done.
Resolving deltas: 100% (80291/80291), done.
Submodule path 'ext/eigen': checked out 'b138753b1007d1eb0aeafddb558c1f4805c77003'
Cloning into 'ext/pybind11'...
remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1), done.
remote: Total 10522 (delta 0), reused 0 (delta 0), pack-reused 10521
Receiving objects: 100% (10522/10522), 3.85 MiB | 2.29 MiB/s, done.
Resolving deltas: 100% (7107/7107), done.
Submodule path 'ext/pybind11': checked out '9a19306fbf30642ca331d0ec88e7da54a96860f9'
Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'tools/clang'
Cloning into 'tools/clang'...
remote: Enumerating objects: 353, done.
remote: Total 353 (delta 0), reused 0 (delta 0), pack-reused 353
Receiving objects: 100% (353/353), 119.74 KiB | 0 bytes/s, done.
Resolving deltas: 100% (149/149), done.
Submodule path 'ext/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'

On my mac :

 2019-03-25 20:58:57 ☆  bluebrain355 in ~/Downloads
○ → git --version
git version 2.21.0
 2019-03-25 20:59:01 ☆  bluebrain355 in ~/Downloads
○ → git clone --recurse-submodules git@github.com:BlueBrain/nmodl.git nmodl_new
Cloning into 'nmodl_new'...
remote: Enumerating objects: 276, done.
remote: Counting objects: 100% (276/276), done.
remote: Compressing objects: 100% (73/73), done.
remote: Total 3423 (delta 227), reused 211 (delta 201), pack-reused 3147
Receiving objects: 100% (3423/3423), 1.55 MiB | 1.12 MiB/s, done.
Resolving deltas: 100% (2351/2351), done.
Submodule 'cmake/hpc-coding-conventions' (https://github.com/BlueBrain/hpc-coding-conventions.git) registered for path 'cmake/hpc-coding-conventions'
Submodule 'ext/cli11' (https://github.com/CLIUtils/CLI11.git) registered for path 'ext/cli11'
Submodule 'ext/eigen' (https://github.com/eigenteam/eigen-git-mirror.git) registered for path 'ext/eigen'
Submodule 'ext/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'ext/pybind11'
Cloning into '/Users/kumbhar/Downloads/nmodl_new/cmake/hpc-coding-conventions'...
remote: Enumerating objects: 73, done.
remote: Counting objects: 100% (73/73), done.
remote: Compressing objects: 100% (71/71), done.
remote: Total 73 (delta 2), reused 45 (delta 0), pack-reused 0
Cloning into '/Users/kumbhar/Downloads/nmodl_new/ext/cli11'...
remote: Enumerating objects: 116, done.
remote: Counting objects: 100% (116/116), done.
remote: Compressing objects: 100% (112/112), done.
remote: Total 116 (delta 1), reused 41 (delta 0), pack-reused 0
Receiving objects: 100% (116/116), 206.78 KiB | 830.00 KiB/s, done.
Resolving deltas: 100% (1/1), done.
Cloning into '/Users/kumbhar/Downloads/nmodl_new/ext/eigen'...
remote: Enumerating objects: 1862, done.
remote: Counting objects: 100% (1862/1862), done.
remote: Compressing objects: 100% (1455/1455), done.
remote: Total 1862 (delta 411), reused 1009 (delta 350), pack-reused 0
Receiving objects: 100% (1862/1862), 2.87 MiB | 1014.00 KiB/s, done.
Resolving deltas: 100% (411/411), done.
Cloning into '/Users/kumbhar/Downloads/nmodl_new/ext/pybind11'...
remote: Enumerating objects: 203, done.
remote: Counting objects: 100% (203/203), done.
remote: Compressing objects: 100% (192/192), done.
remote: Total 203 (delta 4), reused 46 (delta 2), pack-reused 0
Receiving objects: 100% (203/203), 587.23 KiB | 1.02 MiB/s, done.
Resolving deltas: 100% (4/4), done.
error: Server does not allow request for unadvertised object 6b2f7346db5cf5d438aa35ee16d4cb7fe5a71e46
Fetched in submodule path 'cmake/hpc-coding-conventions', but it did not contain 6b2f7346db5cf5d438aa35ee16d4cb7fe5a71e46. Direct fetching of that commit failed.
pramodk commented 5 years ago

I think because we are currently several commits behind the hpc-coding-conventions repo, and the shallow clone only fetches the last commit, which doesn't match the one requested.

but we do that also with eigen

pramodk commented 5 years ago

After reading through https://github.com/rust-lang/rust/pull/24521 I agree and understood what's going on.

May be better to include standalone headers for heavy repos like CLI11. I asked upstream about creating header from latest master (https://github.com/CLIUtils/CLI11/issues/258).

pramodk commented 5 years ago

@ohm314 @tristan0x : do you have any recommendations about this?

We want to use libraries like Eigen, CLI11 as submodule but avoid recursive clone of their own submodules. I don't see any option in submodules for this.

tristan0x commented 5 years ago

@ohm314 @tristan0x : do you have any recommendations about this?

We want to use libraries like Eigen, CLI11 as submodule but avoid recursive clone of their own submodules. I don't see any option in submodules for this.

pramodk commented 5 years ago

Most small header-only libraries usually provide a single header file that you can include in your repository.

this is true for CLI11 where I added single header file in this PR.

For bigger header-only libraries like Eigen, you can put the releases directly in the repo but I would rather use the system installation instead.

Unfortunately Eigen doesnt have latest release that we need with CUDA backend support. We have to use few months old commit.

tristan0x commented 5 years ago

Most small header-only libraries usually provide a single header file that you can include in your repository.

this is true for CLI11 where I added single header file in this PR.

For bigger header-only libraries like Eigen, you can put the releases directly in the repo but I would rather use the system installation instead.

Unfortunately Eigen doesn't have latest release that we need with CUDA backend support. We have to use few months old commit.

du -sh .git/modules/ext/eigen ext/eigen 
91M     .git/modules/ext/eigen
19M     ext/eigen

If the development version of Eigen is required, then I don't think you can avoid the 19M in this repository. But the 91M git history doesn't provide any additional value. If you want to use a submodule, consider using a fork of Eigen as submodule so that you don't have to deal with --depth option. Otherwise, you can add to the repository the zip of the development branch from GitHub or Bitbucket, for instance eigen-git-mirror-b138753b1007d1eb0aeafddb558c1f4805c77003.zip I prefer the first solution because you will easily get rid of the submodule when Eigen will release a new version. With the second solution, git history will keep Eigen forever unless rewritten.

ohm314 commented 5 years ago

That's actually an excellent suggestion, @tristan0x ! I like the idea to fork the repo. At regular intervals, when we see it is necessary or we need new features, we sync the fork but otherwise have a stable version that we know to be good and can stay with a shallow submodule

alkino commented 3 years ago

And so? How should we do today? The submodules are still huge