search

fedora-ci / installability-pipeline

installability pipeline
0 stars 8 forks source link

Run all checks together and upload custom result viewer #25

Closed AdamWill closed 8 months ago

AdamWill commented 8 months ago

Inspired by https://github.com/fedora-ci/rpminspect-runner/pull/80 this does much the same for installability. Instead of running four separate tests, run all four checks in a single test, then (using the stuff from https://github.com/fedora-ci/mini-tps/pull/18 ) generate the result JSON, put it and the viewer HTML into the test data dir, and write a custom results.yaml pointing to it. This should result in the viewer being easily accessible from the TF results, giving packagers a much nicer view on the results.

AdamWill commented 8 months ago

Note that after https://github.com/fedora-ci/rpminspect-runner/pull/80 was merged, https://github.com/fedora-ci/rpminspect-runner/commit/0460a33a40ca0ffe105465afe5fa828da25643db dropped the bit at the end that exits with the overall result code, which I copied in this PR, at the end of installability_runner.sh:

echo "mtps-run-tests overall result: $tmtresult (status code: $highrc)"
exit $highrc

but I'm confused as to why those lines were removed over in rpminspect. I tested this PR without those two lines, and when running locally via tmt, the result gets reported as passed even if some of the checks failed. With those lines left in, the result is correctly failed when one of the checks failed. So I'm leaving those in.

@martinpitt do you see anything wrong or missing here? Are changes needed anywhere else, or is just following the same layout you got working for rpminspect enough? Thanks!

AdamWill commented 8 months ago

https://happyassassin.net/temp/viewer.html shows the output from a sample run of this PR (backed by https://happyassassin.net/temp/result.json ). I copied the files out of the installability/execute/data/guest/default-0/installability-2/data/ dir after a run via tmt.

thrix commented 8 months ago

trying to sanity test via https://osci-jenkins-1.ci.fedoraproject.org/job/fedora-ci/job/installability-pipeline/view/change-requests/job/PR-25/

thrix commented 8 months ago

https://osci-jenkins-1.ci.fedoraproject.org/job/fedora-ci/job/installability-pipeline/view/change-requests/job/PR-25/7/console

AdamWill commented 8 months ago

well, er, that seems to have failed somehow, but I have no idea what the failure means? is it my fault? :D

thrix commented 8 months ago

@AdamWill nope, I will try to run it manually, no idea why the REF is incorrect now

thrix commented 8 months ago

Testing here:

https://artifacts.dev.testing-farm.io/4f9b5dd4-9e3f-4775-af85-9e9a98176820/

thrix commented 8 months ago

@AdamWill hmm, maybe I run it wrong? seems it did nto do anything I would expect, checking more

jpopelka commented 8 months ago

@AdamWill I rebased your commit since there was #26 merged meanwhile and now it works OK (no idea why it didn't before though). It, however, still doesn't show the viewer and I even don't see it in the Log links, but have no idea what's wrong. Maybe @thrix would know?

In fact, I'm quite puzzled how this works because in the RHEL installability pipeline we have it like:

  log:
    - ../output.txt
    - viewer.html

and it shows the viewer by default (e.g. here).

AdamWill commented 8 months ago

@jpopelka it is there, down several levels: https://artifacts.dev.testing-farm.io/57a488ca-5d0e-4fed-a659-cfc1048670c3/work-installabilityiruqmecv/installability/execute/data/guest/default-0/installability-2/data/ . If you open https://artifacts.dev.testing-farm.io/57a488ca-5d0e-4fed-a659-cfc1048670c3/work-installabilityiruqmecv/installability/execute/data/guest/default-0/installability-2/data/viewer.html you get the expected view. No idea why it's not showing up at the top level, though.

AdamWill commented 8 months ago

BTW, are you sure you tested the latest version? The results.yaml in that tree has:

- name: /installability
  result: pass
  log:
    - ../output.txt
    - viewer.html
    - result.json

i.e. still the old ordering with output.txt on top...

jpopelka commented 8 months ago

I see the viewer is there in data/ and I'm pretty sure I'm testing the latest because the results.yaml has the viewer on top.

I'm seeing the same problem in my rpmdeplint pipeline PR where I'm adding another log file and I see it in the results.yaml and it's also in the data/ but yet it's not displayed in the Log links.

I'm merging this (and my PR) and will ask Testing Farm people in chat.

AdamWill commented 8 months ago

https://github.com/fedora-ci/installability-pipeline/pull/27