search

TriBITSPub / TriBITS

TriBITS: Tribal Build, Integrate, and Test System,
http://tribits.org
Other
36 stars 46 forks source link

cdash_analyze_and_report.py: Add --builds-filter-fields and --exclude-tests-filter-fields #348

Open bartlettroscoe opened 3 years ago

bartlettroscoe commented 3 years ago

Description

This story is to scope out and possibly the following new cdash_analyze_and_report.py arguments:

This would eliminate the usage of the arguments --cdash-builds-filters and --cdash-nonpassed-tests-filters. This eliminates duplication that exists in specifying these fields and also allows the filters in --cdash-exclude-tests-filter-fields=<fields> to be used when getting the history for each of the tests in filtering out these test failures.

With this, one would specify the minimal options:

--cdash-project-testing-day-start-time="04:01" \
--cdash-project-name="Trilinos" \
--cdash-site-url="https://testing-dev.sandia.gov/cdash" \
--cdash-builds-filter-fields="field1=groupname&compare1=62&value1=Experimental&field2=buildname&compare2=65&value2=Trilinos-atdm-" \
--cdash-exclude-tests-filter-fields="<fields>" \

Those options can be used to construct the full URLs to get build and test data from cdash/index.php and cdash/queryTests.php.

bartlettroscoe commented 3 years ago

Some background analysis and requirements analysis ....

One way to improve things for this intended use case is to see if we can split out the filters for excluding tests from the filters for the subset of builds to cover. And then really CDash should have different CDash groups for "Primary", "Secondary" and other builds (likely in "Specialized" and "Experimental"). Then the set of builds and tests should be determined by the cdash/index.php and cdash/queryTests.php queries. And it would be great if you could just provide the list of filters for that subset set of builds and the use them for both the cdash/index.php and the cdash/queryTests.php queries. Is that even possible? Did they make CDash equivalent in that way? Let's see. The two queries for the full set of ATDM Trilinos builds for testing day 2021-01-15 is:

and

The filter fields selecting the set of builds for the cdash/index.php and cdash/queryTests.php queries are:

respectively.

Wow, those look identical (thank goodness)! So I think knowing this we could have the input arguments to cdash_analyze_and_report.py:

--cdash-project-testing-day-start-time="04:01" \
--cdash-project-name="Trilinos" \
--cdash-site-url="https://testing-dev.sandia.gov/cdash" \
--builds-filter-fields="field1=groupname&compare1=62&value1=Experimental&field2=buildname&compare2=65&value2=Trilinos-atdm-" \
--exclude-tests-filter-fields="field4=testoutput&compare4=94&value4=Error%20initializing%20RM%20connection.%20Exiting&field5=testoutput&compare5=96&value5=srun%3A%20error%3A%20s_p_parse_file%3A%20unable%20to%20read%20.%2Fetc%2Fslurm%2Fslurm.conf.%3A%20Permission%20denied&field6=testoutput&compare6=96&value6=cudaGetDeviceCount.*cudaErrorUnknown.*unknown%20error.*Kokkos_Cuda_Instance.cpp&field7=testoutput&compare7=96&value7=cudaMallocManaged.*cudaErrorUnknown.*unknown%20error.*Sacado_DynamicArrayTraits.hpp&field8=testoutput&compare8=96&value8=srun%3A%20error.*launch%20failed%3A%20Error%20configuring%20interconnect&field9=testoutput&compare9=94&value9=An%20ORTE%20daemon%20has%20unexpectedly%20failed%20after%20launch&field10=testoutput&compare10=94&value10=It%20looks%20like%20orte_init%20failed%20for%20some%20reason%3B%20your%20parallel%20process%20is" \

We would then automatically put in the cdash/queryTests.php filter field 'field3=status&compare3=62&value3=passed' so by default we show only non-passing tests. And if the user wanted to also exclude say "not run" tests, they could add that to the filters in --exclude-tests-filter-fields. (NOTE: We can adjust the indices to build the full correct URLs for the cdash/index.php and cdash/queryTests.php fields and fill in the rest of the URL automatically.)

With this change, the drivers like trilinos_atdm_builds_status_primary.sh and trilinos_atdm_builds_status_secondary.sh could have their own unique --builds-filter-fields fields and not have to filter out results based on the input set of expected builds (which brings up a whole set of problems as noted above).

NOTE: We need to see how this would work with the AND/OR blocks of filter fields. So it looks like that could be handled too as the format looks like:

What his shows is that a top-level 'field' can a 'block' and the rest comes from there. That looks pretty easy to parse.

bartlettroscoe commented 3 years ago

Related to: