search

Open-Systems-Pharmacology / RE-Test-Reports

Assess consistency of simple reports generated using `{ospsuite.reportingengine}` package
0 stars 1 forks source link

Information on failed tests #18

Closed Yuri05 closed 5 days ago

Yuri05 commented 1 week ago

At the moment, many tests fail. (How) is it possible to see which tests fail and why? When I click on the "failed tests" image - I just see the image; in the log of the GH action no information about failed tests is available.

grafik

pchelle commented 1 week ago

The table may be not as clear as I thought. The tests here only compare if the reports in the folder "tests/Reports" provide same outputs as the reports in the folder "Reports". The comparison checks consistency in created files, text within the markdown report, content of the simulation, PK analysis, and sensitivity analysis results, content of the generated tables and plots (if any pixel is different between each image).

For instance, here 2 tests are failing for the Aciclovir-Mean workflow: 1- The 4 plots in the report are different from reference plots 2- Simulation Results are different from reference simulation results

The reason why they failed is unfortunately for us to investigate and fix. From what I have checked, the reason why the simulation results fail seems to come using the new {ospsuite} and {rSharp} versions: The saved results have one more significant digit compared to the previous version.

Likewise, the plots are slightly different, potentially due to updates in {tlf}, {ggplot2} or other graphical packages.

Currently, the way to fix this is 1- review manually that the changes are not significant 2- update the reference reports with the latest version

pchelle commented 1 week ago

This is also the reason why I would like to track the package versions in the ReadMe file defined in issue #17

Yuri05 commented 1 week ago

Thx for the detailed explanation.

Question:

For instance, here 2 tests are failing for the Aciclovir-Mean workflow: 1- The 4 plots in the report are different from reference plots 2- Simulation Results are different from reference simulation results

Can I see this information somewhere on GitHub (logfile of the GH-action, or logfiles of created reports or ...)?

pchelle commented 1 week ago

Currently, no. I format tests results/logs as a table in the readme.

But totally doable, I can create a log or a zip artifact to store the the raw results of the tests.

Yuri05 commented 1 week ago

But totally doable, I can create a log or a zip artifact to store the the raw results of the tests.

This would be great!