search

microsoft / ga4gh-tes

C# implementation of the GA4GH TES API; provides distributed batch task execution on Microsoft Azure
MIT License
34 stars 27 forks source link

Add support for Tes Compliance Tests to run on both 1.0.0 and 1.1.0 #459

Open yuliadub opened 1 year ago

yuliadub commented 1 year ago

Problem: Today Tes compliance tests only run tests for TES version 1.0.0

Solution: Today tes compliance tests only run on version 1.0.0 after a TES integration test is run. After this issue is complete tes compliance tests should run on version 1.1.0. Both results should be recorded appropriately and report a warning on the release if they fail. This can be run as a separate step in the release as 1.0.0 or in the same step.

Code dependencies Will this require code changes in:

Additional context Original task that set up compliance tests: https://github.com/microsoft/ga4gh-tes/issues/26

vsmalladi commented 10 months ago

New repo to use is: https://github.com/ga4gh/compliance-tests-ga4gh-tes/tree/feature/tests

yuliadub commented 7 months ago

ability to run tests on both frameworks is being added in this PR: https://github.com/microsoft/ga4gh-tes/pull/653/files

in this case, the openapi-test-runner will run on one, then another version back to back, and then the results will be copied into a file. The complexity with pulling this data out is due to the fact that today the script runs inside a docker image. All we do today is we check to see if any results have failed, and they do, produce a warning on the ADO specific pipeline run. The next step to this is to display the results as badges inside the ga4gh-tes repo, that would show if the tests are passing or not. The thoughts on how to do this are going to captured in the following issue: 660

vsmalladi commented 1 month ago

@yuliadub can we revist this?

yuliadub commented 1 month ago

@vsmalladi didnt you add functionality to run on both frameworks already? Looking here.

or are you looking to add the badge to show if they are passing or not?

vsmalladi commented 1 month ago

ah yes @yuliadub i think we can close this then and move to the badge issue right?

yuliadub commented 1 month ago

@vsmalladi up to you (or Nikita and Matt?), I dont have a strong opinion -- the badges will be valuable once we actually start passing the compliance tests, not sure if we are yet or not.