search

tensorflow / build

Build-related tools for TensorFlow
Apache License 2.0
281 stars 116 forks source link

Process for building Tensorflow that maximises use of system libraries? #42

Open jdpipe opened 3 years ago

jdpipe commented 3 years ago

Hi there, I raised a question about this on the Discord server, but was advised to come here and ask instead.

Are there any instructions/guidelines on how to build a smaller, lighter Tensorflow shared library (C bindings) and also Python module that maximises the use of standard system libraries, eg in Ubuntu 18.04, 20.04, CentOS or MSYS2?

With 29 dependencies listed in the THIRD_PARTY_TC_C_LICENSES, it seems to me that this 320 MB shared library could be potentially much smaller if it made use of system libraries instead of statically linked versions, and if there is a decent TensorFlow test suite then it should not be too hard to assure that the library is still functioning correctly.

I note https://github.com/tensorflow/build/tree/master/official_build_environments ('They should have as few external dependencies as possible'). With the dockers and statically linked shared libraries, it seems that TensorFlow community has a strong bias against this approach, preferring more self-contained builds with less risk of platform-specific divergence. I just wonder if, as the code's user base grows, whether there might be a case for more 'standard' OS packaging of this library (eg .deb, .rpm) and then also the associated Python bindings, which presumably in the 'standard install' also contains a separate copy of the shared library?

bhack commented 3 years ago

I suppose that is just a side effect of the monorepo approach:

https://jmmv.dev/2021/02/google-monorepos-and-caching.html