Yuhao Zhang, Yifan Chen, Shing-Chi Cheung, Yingfei Xiong, Lu Zhang
Year of Publication
2018
Summary
This paper focuses on studying deep learning applications built on TensorFlow. First, they construct a dataset of Github bugfix commits and Stack Overflow posts. Second, it identifies the symptoms and root causes of these deep-learning bugs according to QA Pages, commit messages, pull request messages and issue discussions. Third, they also study the strategies deployed by the TensorFlow user for bug detection and localization.
Their research questions are as follows:
RQ1: What are the symptoms and root causes of the bugs?
RQ2: What new challenges exist to detect the bugs and how do TF users handle them?
RQ3: What new challenges exist to localize the bugs and how do TF users handle them?
Contributions of The Paper
The paper has 3 significant contributions, as highlighted below:
A dataset of TensorFlow bugs collected from StackOverflow and GitHub.
A study of the symptoms and root causes of the bugs, which could assist future studies on TensorFlow application testing and debugging techniques.
A study of the new challenges in detecting and localizing the bugs and the current strategies to address them, which opens new problems for future research
Comments
This paper was the first taxonomy of deep learning bugs, highlighting 5 challenges for bug localization and reproduction in DL-based Systems.
Challenge 1: Probabilistic Correctness.
Challenge 2: Coincidental Correctness.
Challenge 3: Stochastic Execution.
Challenge 4: The densely interdependent neural network.
Challenge 5: The unknown behavior of neural networks.
We could use these challenges to motivate further research or as a support/explainability for our findings whenever possible.
Publisher
ISSTA
Link to The Paper
https://dl.acm.org/doi/10.1145/3213846.3213866
Name of The Authors
Yuhao Zhang, Yifan Chen, Shing-Chi Cheung, Yingfei Xiong, Lu Zhang
Year of Publication
2018
Summary
This paper focuses on studying deep learning applications built on TensorFlow. First, they construct a dataset of Github bugfix commits and Stack Overflow posts. Second, it identifies the symptoms and root causes of these deep-learning bugs according to QA Pages, commit messages, pull request messages and issue discussions. Third, they also study the strategies deployed by the TensorFlow user for bug detection and localization.
Their research questions are as follows:
Contributions of The Paper
The paper has 3 significant contributions, as highlighted below:
Comments
This paper was the first taxonomy of deep learning bugs, highlighting 5 challenges for bug localization and reproduction in DL-based Systems.
Challenge 1: Probabilistic Correctness. Challenge 2: Coincidental Correctness. Challenge 3: Stochastic Execution. Challenge 4: The densely interdependent neural network. Challenge 5: The unknown behavior of neural networks.
We could use these challenges to motivate further research or as a support/explainability for our findings whenever possible.