CI with Tensorflow 2.1

[WeichenXu123]

Now CI run on following env:

• PYARROW_VERSION=0.12.1 TF_VERSION=1.15.0 DEPLOYABLE=0 PY=2.7 PYSPARK_VERSION=2.4.4
• PYARROW_VERSION=0.15.1 TF_VERSION=1.15.0 DEPLOYABLE=0 PY=2.7 PYSPARK_VERSION=2.4.4 ARROW_PRE_0_15_IPC_FORMAT=1
• PYARROW_VERSION=0.12.1 TF_VERSION=1.15.0 DEPLOYABLE=0 PY=3.7 PYSPARK_VERSION=2.4.4
• PYARROW_VERSION=0.15.1 TF_VERSION=1.15.0 DEPLOYABLE=1 PY=3.7 PYSPARK_VERSION=2.4.4 ARROW_PRE_0_15_IPC_FORMAT=1
• PYARROW_VERSION=0.15.1 TF_VERSION=1.15.0 DEPLOYABLE=0 PY=3.7 PYSPARK_VERSION=3.0.0 ARROW_PRE_0_15_IPC_FORMAT=0
• PYARROW_VERSION=0.15.1 TF_VERSION=2.1.0 DEPLOYABLE=0 PY=3.7 PYSPARK_VERSION=3.0.0 ARROW_PRE_0_15_IPC_FORMAT=0

I replace all import tensorflow as tf with import tensorflow.compat.v1 as tf, so that all TF v1 API can be run on tensorflow 2.

Some changes on test:

• I wrap all TF test with @create_tf_graph_if_tf2 (Creating default tf.graph() explicitly is required by TF2 with v1 compatible API)

Some change on source code:

• I update petastorm.tf_utils._set_shape, add a check only when fields_as_dict[k].get_shape().dims is None then set shape for tensor. Because TF2 (default eager mode) sometimes will provide accurate runtime shape so we do not need to set it.

[codecov[bot]]

Codecov Report

Merging #536 into master will decrease coverage by 0.05%. The diff coverage is 94.44%.

@@            Coverage Diff             @@
##           master     #536      +/-   ##
==========================================
- Coverage   86.53%   86.48%   -0.06%     
==========================================
  Files          85       85              
  Lines        4717     4721       +4     
  Branches      743      744       +1     
==========================================
+ Hits         4082     4083       +1     
- Misses        516      517       +1     
- Partials      119      121       +2     

Impacted Files	Coverage Δ
setup.py	0.00% <ø> (ø)
petastorm/tf_utils.py	80.30% <83.33%> (-0.62%)	:arrow_down:
...o_world/external_dataset/tensorflow_hello_world.py	100.00% <100.00%> (ø)
...al_dataset/tests/test_generate_external_dataset.py	96.29% <100.00%> (+0.14%)	:arrow_up:
..._world/petastorm_dataset/tensorflow_hello_world.py	100.00% <100.00%> (ø)
...m_dataset/tests/test_generate_petastorm_dataset.py	96.55% <100.00%> (+0.12%)	:arrow_up:
examples/mnist/tests/test_tf_mnist.py	100.00% <100.00%> (ø)
examples/mnist/tf_example.py	79.59% <100.00%> (ø)
..._dataset_converter/tensorflow_converter_example.py	85.10% <100.00%> (ø)
petastorm/benchmark/throughput.py	80.55% <100.00%> (ø)
... and 2 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 52c8c9f...8077d97. Read the comment docs.

[WeichenXu123]

@selitvin Updated. One issue is, when I do this replace it with a new decorator create_tf_graph (without differentiating between TF2 and TF1), one error occur in Py2:

__________ ERROR collecting petastorm/tests/test_ngram_end_to_end.py ___________
In test_ngram_length_1_tf: function uses no argument 'reader_factory'
_____________ ERROR collecting petastorm/tests/test_tf_dataset.py ______________
In test_with_one_shot_iterator: function uses no argument 'reader_factory'

It should be pytest issue in py2. How to fix it ? Or we just remove CI for py2 ?