[WIP] TensorFlow 2.0

[riga]

The purpose if this issue is to document the steps required to have a running TensorFlow 2.0 installation. Commits are pushed to the tf-2.0 branch.

• [ ] Let TensorFlow download custom versions (i.e. the ones we use in cmsdist) of protobuf and eigen.
• [ ] Check if cuda library stubs can be used for the TF installation. At the moment, the lib/*.so* files are used (cuda.spec).
• [ ] Check if the correct symbols are linked, i.e., not the cxx11 ones.
• [ ] Run the test_deepjet.py test to validate the python module.
• [ ] Move changes from the custom onnxruntime branch to cms-externals.
• [ ] Update keras version in pip/requirements.txt file.

@smuzaffar @mrodozov

[riga]

Testing different --cxxopts:

(test_deepjet.py)

• No opts at all:

  • Software compilation works.
  • DeepJet test works.
  • CMSSW TF interface linking fails:

    undefined reference to `tensorflow::io::internal::JoinPathImpl[abi:cxx11](std::initializer_list<std::basic_string_view<char, std::char_traits<char> > >)'

• --cxxopt=-std=c++1z

  • Software compilation fails:

    tensorflow/core/lib/gif/gif_io.cc: In function 'int tensorflow::gif::input_callback(GifFileType*, GifByteType*, int)':
    tensorflow/core/lib/gif/gif_io.cc:40:5: error: 'memcpy' was not declared in this scope
    memcpy(buf, info->buf, size);
    ^~~~~~
    tensorflow/core/lib/gif/gif_io.cc:40:5: note: 'memcpy' is defined in header '<cstring>'; did you forget to '#include <cstring>'?

  • Included <cstring>. Software compilation still fails:

    'make_unique' is not a member of 'absl'

• --cxxopt=-std=c++1z --cxxopt=-D_GLIBCXX_USE_CXX11_ABI=0

  • Software compilation fails:

    ./tensorflow/stream_executor/stream_executor_pimpl.h:779:28: error: 'make_unique' is not a member of 'absl'

[mrodozov]

The memcpy issue I solved adding cstring, see the last commit on https://github.com/cms-externals/tensorflow/tree/cms/v1.13.1

[mrodozov]

@riga the https://github.com/riga/cmsdist/tree/tf-2.0 branch is copy of 820 , right ? We wil also need the cmssw code to link against (maybe post a link here to branch or something)

[riga]

@mrodozov Yes, I started from https://github.com/cms-sw/cmsdist/tree/tf-2.0 which @smuzaffar prepared to include the changes he proposed.

For the CMSSW code: except for the custom sessions, the code we have in the release already compiles correctly (but linking fails of course). Therefore I didn't create a branch yet, but you can do

# the usual checkout
export SCRAM_ARCH=slc7_amd64_gcc820
cmsrel CMSSW_11_0_0_pre9
cd cmsrel CMSSW_11_0_0_pre9/src
cmsenv

# checkout tensorflow and remove custom code
git cms-init
git cms-addpkg PhysicsTools/TensorFlow
rm -f PhysicsTools/TensorFlow/{src,interface}/{NTSession.*,TBBSession.*}

scram b