CI: need to restart tests

LeventErkok / sbv

SMT Based Verification in Haskell. Express properties about Haskell programs and automatically prove them using SMT solvers.

https://github.com/LeventErkok/sbv

Other

243 stars 34 forks source link

CI: need to restart tests #479

Closed LeventErkok closed 4 years ago

LeventErkok commented 5 years ago

I just can't get Travis or Appveyor to run the tests and finish in the allotted time and have a consistent feedback.

Multiple issues:

Building takes too long
Z3 downloads are not realiable (the master doesn't match nightly)
Arbitrary test failures due to random issues

So, giving up on this for the time being; using CI for build only. Should really instantiate tests back, or restructure CI completely. Help would be appreciated.

ghost commented 4 years ago

Drive by comment: lately, I have been most interested in circleci, gitlab for features, although I have shifted to buildbot for FLOSS purity.

Glance at the circle-ci ability to automatically split test suites into parallel subgroups, or manually split them into parallel subgroups using gitlabs pipeline syntax. For gitlab ci, you would probably have to mirror this repo to gitlab. Github-actions or azure-pipelines might have the similar features for manually splitting into parallel jobs

LeventErkok commented 4 years ago

@kanishka-azimi

I'm really agnostic to what test/build-framework we use; whatever works best I'm happy to switch.

The biggest problem has been reliably downloading z3 and running the tests within the allowed resource bounds of these services. If you can lend a hand to automate and make that reliable, your contributions would be most welcome!

LeventErkok commented 4 years ago

This will really never be an option. Tests are long, so many failure modes, so, this one is really no ROI.