tensorflow / tensorboard

TensorFlow's Visualization Toolkit
Apache License 2.0
6.68k stars 1.65k forks source link

CI for `audio:summary_test` broken on tf2.0+py3 #2224

Open wchargin opened 5 years ago

wchargin commented 5 years ago

Reported by @jameswex (thanks!); seems to be affecting all PRs that have a dependency path to this target.

The //tensorboard/plugins/audio:summary_test test target is consistently timing out after 300s on only the Python 3 + TensorFlow 2.0 preview build configuration. Sample log:

https://travis-ci.com/tensorflow/tensorboard/jobs/198991418

Excerpt:

[ RUN      ] SummaryV1OpTest.test_audio_count_less_than_max_outputs
[  SKIPPED ] SummaryV1OpTest.test_audio_count_less_than_max_outputs
[ RUN      ] SummaryV1OpTest.test_audio_count_when_more_than_max
[  SKIPPED ] SummaryV1OpTest.test_audio_count_when_more_than_max
--- %< --- snip: more test cases --- %< ---
[ RUN      ] SummaryV1PbTest.test_wav_format_roundtrip
[  SKIPPED ] SummaryV1PbTest.test_wav_format_roundtrip
[ RUN      ] SummaryV2OpTest.test_audio_count_less_than_max_outputs
W0509 19:41:36.559446 140250432620352 deprecation.py:323] From /home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/tensorflow/python/summary/summary_iterator.py:68: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version.
Instructions for updating:
Use eager execution and: 
`tf.data.TFRecordDataset(path)`
Terminated

That test case:

https://github.com/tensorflow/tensorboard/blob/c23b5652812aa4efbdbe20a4be1ddd6bed624bec/tensorboard/plugins/audio/summary_test.py#L120-L126

I cannot reproduce locally, using:

I have not investigated.

stephanwlee commented 5 years ago

For history: could not reproduce this when opened another PR from the HEAD. After struggling a bit, we decided to clear the Travis cache for #2192 and #2194 and it magically fixed the issue.

https://travis-ci.com/tensorflow/tensorboard/caches