Cellar system: Improve usage of HDD by Hunter

[ruslo]

Instead of unpacking cache/source archives to the root directory it's possible to unpack it once and create links instead. This is similar to homebrew.

[ruslo]

Results of my test with android-ndk-r10e-api-19-armeabi-v7a-neon-hid-sections-lto + Release configuration.

Total build time after introducing change in ogles_gpgpu submodule is 34 minutes 55 seconds (cache hit everywhere).

Next timings from build log:

• Unpack time of the regular 3rd parties is low, biggest is OpenCV (10 sec) and Boost (2 sec)
• ogles_gpgpu build from scratch (21 sec)
• Various dependencies of Android SDK (7 packages) unpacked in 29 sec in total
• The biggest part is Android SDK itself - 110 sec

All unpack operations: 10 + 2 + 29 + 110 = 151 sec

I've copied Android SDK cache archive for testing. Extract by tar xf takes the same amount of time (about 110 sec). If instead of creating archive we will use links time will be:

• 2.4 sec create directories (mkdir <dir>)
• 22.2 sec create files (ln <cellar/file> <root/file>)

So instead of 110 sec we will have 24.6 sec, i.e. minus 80%.

Expected new timing after this feature introduced will be: 151 x 0.2 = 30.2 sec

In total update of the ogles_gpgpu will cost 30.2 (unpack) + 21 (build) = 51.2 sec

But since we are changing the root directory (config-id changed) we have to update -I/path/to/headers flags and all Drishti binaries will be rebuild even if there is no changes in ogles_gpgpu headers.

Note: LTO is an expensive feature, same toolchain without LTO shows 12 minutes (time for Drishti build only).

[ruslo]

Planning to use Bash script on Linux and OSX for now. For Windows it will remain the same (copy file). CMake feature: https://gitlab.kitware.com/cmake/cmake/issues/16926 can be used in future, hopefully will work on Windows too.

[ruslo]

https://github.com/ruslo/hunter/releases/tag/v0.19.0

[ruslo]

On my test Boost link takes longer than unpack (it's now 6 sec instead of 2). Android SDK as expected 19 sec.

Real time:

-
Generate: 0:01:12.102801s
Build: 0:11:59.754139s
-
Total: 0:13:11.857547s
-
SUCCESS

[ruslo]

https://github.com/elucideye/drishti/pull/460

[ruslo]

CMake feature: https://gitlab.kitware.com/cmake/cmake/issues/16926 can be used in future, hopefully will work on Windows too

I forgot about it, reopen :)

[headupinclouds]

Thanks!

[ruslo]

CMake feature: https://gitlab.kitware.com/cmake/cmake/issues/16926 can be used in future, hopefully will work on Windows too

I forgot about it, reopen :)

Waiting for CMake 3.9:

• https://gitlab.kitware.com/cmake/cmake/issues/16926#note_275176

Moving to 0.6 milestone

[headupinclouds]

This seems to be working well for me so far.

[headupinclouds]

https://github.com/ruslo/polly/pull/159

Close?

[ruslo]

Close?

No, the file(LINK ...) feature can only be discussed after 3.9 release. Nothing is implemented at this moment in CMake.

[headupinclouds]

This is a matter of replacing the current internal (opitonal) hunter python scripts with CMake's file(LINK ...) command, right?

[ruslo]

This is a matter of replacing the current internal (opitonal) hunter python scripts with CMake's file(LINK ...) command, right?

Yes