neulab / ExplainaBoard

Interpretable Evaluation for AI Systems
MIT License
359 stars 36 forks source link

Set the time limit for integration tests #551

Closed tetsuok closed 1 year ago

tetsuok commented 1 year ago

Sometimes integration tests are taking more than 1 hour (example). That's a symptom that there is something wrong with ExplainaBoard, resources ExplainaBoard fetches for testing, or ExplainaBoard's dependencies. The fact that the tests are taking time is not economical, and becomes a drain on productivity. We should set the time limit for the tests, and raise a timeout error if the tests exceed the time limit to prevent the tests from getting slower and slower. This PR explicitly sets the time limit of 30 minutes for the step to run integration tests. The default setting is 360 minutes, which is too long.