search

kernelci / kcidb

kernelci.org common database tools
GNU General Public License v2.0
27 stars 34 forks source link

Codec summary report #566

Closed JenySadadia closed 3 weeks ago

JenySadadia commented 1 month ago

Generate codec summary report and email notifications for the same.

Created on top of https://github.com/kernelci/kcidb/pull/565 to use common.j2.

JenySadadia commented 1 month ago

Hello @denisyuji @helen-fornazier This would generate an email notification like the below:

Subject: KernelCI report for fluster tests: master@v6.11-rc4

REVISION

    Commit
        name: v6.11-rc4
        hash: 3d5f968a177d468cd13568ef901c5be84d83d32b
     Checked out from
        https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master

FLUSTER TESTS

      sc7180-trogdor-kingoftown:
      -v4l2-decoder-conformance-av1:                  passed: 0, failed: 0
      -v4l2-decoder-conformance-av1-chromium-10bit:   passed: 0, failed: 0
      -v4l2-decoder-conformance-h264:                 passed: 0, failed: 0
      -v4l2-decoder-conformance-h264-frext:           passed: 0, failed: 0
      -v4l2-decoder-conformance-h265:                 passed: 1, failed: 1
      -v4l2-decoder-conformance-vp8:                  passed: 0, failed: 0
      -v4l2-decoder-conformance-vp9:                  passed: 0, failed: 0

      sc7180-trogdor-lazor-limozeen:
      -v4l2-decoder-conformance-av1:                  passed: 0, failed: 0
      -v4l2-decoder-conformance-av1-chromium-10bit:   passed: 0, failed: 0
      -v4l2-decoder-conformance-h264:                 passed: 0, failed: 0
      -v4l2-decoder-conformance-h264-frext:           passed: 0, failed: 0
      -v4l2-decoder-conformance-h265:                 passed: 0, failed: 0
      -v4l2-decoder-conformance-vp8:                  passed: 0, failed: 0
      -v4l2-decoder-conformance-vp9:                  passed: 0, failed: 2

See complete and up-to-date report at:

    https://kcidb.kernelci.org/d/revision/revision?orgId=1&var-git_commit_hash=3d5f968a177d468cd13568ef901c5be84d83d32b&var-patchset_hash=

Tested-by: kernelci.org bot <bot@kernelci.org>

Thanks,
KernelCI team

Please let me know your feedback on this. Note: I have generated report on dummy data so you won't be able to see these results on KCIDB dashboard.

helen-fornazier commented 1 month ago

@JenySadadia thanks for this work.

I was just wondering if it wouldn't be better to point to this instead:

https://fresh2-kcidb-grafana-m6io3uhhiq-uc.a.run.app/d/bdviluqy6gv7kc/codecs?orgId=1&var-platform=All&%20var-datasource=default&var-git_commit_hash=3ec3f5fc4a91e389ea56b111a73d97ffc94f19c6

I also noticed that it takes some time for the tests to be run after the checkout. For instance, by the time of this post we still don't have tests for the last checkout, so I was wondering how you handle this.

JenySadadia commented 1 month ago

@JenySadadia thanks for this work.

I was just wondering if it wouldn't be better to point to this instead:

https://fresh2-kcidb-grafana-m6io3uhhiq-uc.a.run.app/d/bdviluqy6gv7kc/codecs?orgId=1&var-platform=All&%20var-datasource=default&var-git_commit_hash=3ec3f5fc4a91e389ea56b111a73d97ffc94f19c6

Sure, I'll update the report to point it to the new dashboard.

I also noticed that it takes some time for the tests to be run after the checkout. For instance, by the time of this post we still don't have tests for the last checkout, so I was wondering how you handle this.

We are sending out notifications after 3 hours when a revision create/update event is received. Tests will be executed in that waiting period.

JenySadadia commented 1 month ago

https://fresh2-kcidb-grafana-m6io3uhhiq-uc.a.run.app/d/bdviluqy6gv7kc/codecs?orgId=1&var-platform=All&%20var-datasource=default&var-git_commit_hash=3ec3f5fc4a91e389ea56b111a73d97ffc94f19c6

Do you consider MISS/ERROR/SKIP tests in total count?

JenySadadia commented 1 month ago

Updated templates to display compatible[0] instead of platform.

helen-fornazier commented 1 month ago

https://fresh2-kcidb-grafana-m6io3uhhiq-uc.a.run.app/d/bdviluqy6gv7kc/codecs?orgId=1&var-platform=All&%20var-datasource=default&var-git_commit_hash=3ec3f5fc4a91e389ea56b111a73d97ffc94f19c6

Do you consider MISS/ERROR/SKIP tests in total count?

Yes

JenySadadia commented 1 month ago

Report v2:

REVISION

    Commit
        name: v6.11-rc4
        hash: 3d5f968a177d468cd13568ef901c5be84d83d32b
     Checked out from
        https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master

FLUSTER TESTS

      google,kingoftown:
      -v4l2-decoder-conformance-av1:                  passed: 0, failed: 0, total: 0
      -v4l2-decoder-conformance-av1-chromium-10bit:   passed: 0, failed: 0, total: 0
      -v4l2-decoder-conformance-h264:                 passed: 0, failed: 0, total: 0
      -v4l2-decoder-conformance-h264-frext:           passed: 0, failed: 0, total: 0
      -v4l2-decoder-conformance-h265:                 passed: 1, failed: 1, total: 2
      -v4l2-decoder-conformance-vp8:                  passed: 0, failed: 0, total: 0
      -v4l2-decoder-conformance-vp9:                  passed: 0, failed: 0, total: 0

      google,lazor-rev5-sku5:
      -v4l2-decoder-conformance-av1:                  passed: 0, failed: 0, total: 0
      -v4l2-decoder-conformance-av1-chromium-10bit:   passed: 0, failed: 0, total: 0
      -v4l2-decoder-conformance-h264:                 passed: 0, failed: 0, total: 0
      -v4l2-decoder-conformance-h264-frext:           passed: 0, failed: 0, total: 0
      -v4l2-decoder-conformance-h265:                 passed: 0, failed: 0, total: 0
      -v4l2-decoder-conformance-vp8:                  passed: 0, failed: 0, total: 0
      -v4l2-decoder-conformance-vp9:                  passed: 0, failed: 2, total: 3

See complete and up-to-date report at:

    https://fresh2-kcidb-grafana-m6io3uhhiq-uc.a.run.app/d/bdviluqy6gv7kc/codecs?orgId=1&var-platform=All&var-datasource=default&var-git_commit_hash=3d5f968a177d468cd13568ef901c5be84d83d32b

Tested-by: kernelci.org bot <bot@kernelci.org>

Thanks,
KernelCI team
JenySadadia commented 1 month ago

@helen-fornazier @denisyuji Once the report looks good to you, I can work on improving formatting e.g. table-like view for stats if required.

denisyuji commented 1 month ago

@JenySadadia , this looks good, good work! It would be great if we can also add some information about the regression. Like it is produced by the summary script:

$ git clone https://github.com/kernelci/kernelci-pipeline.git && cd kernelci-pipeline
$ touch .env
$ sed -i 's/api_config = "docker-host"/api_config = "staging"/' config/kernelci.toml
$ docker compose run result_summary --preset=summary-v4l2-decoder-conformance-regressions --last-updated-from=$(date --date='yesterday' --rfc-3339=date)

The result is displayed at data/output/v4l2-decoder-conformance-regressions.html

JenySadadia commented 1 month ago

@JenySadadia , this looks good, good work! It would be great if we can also add some information about the regression. Like it is produced by the summary script:

$ git clone https://github.com/kernelci/kernelci-pipeline.git && cd kernelci-pipeline
$ touch .env
$ sed -i 's/api_config = "docker-host"/api_config = "staging"/' config/kernelci.toml
$ docker compose run result_summary --preset=summary-v4l2-decoder-conformance-regressions --last-updated-from=$(date --date='yesterday' --rfc-3339=date)

The result is displayed at data/output/v4l2-decoder-conformance-regressions.html

At the moment, maestro is not sending regressions to KCIDB. Hence, I don't think that is possible along with this report. However, we can send this information directly from maestro pipeline. @r-c-n Could you please point me out if there is another way that I'm not aware of?

spbnick commented 3 weeks ago

Please move your logic for fluster tests from Python to Jinja template code, either the way I did it, or whatever way you prefer. Do not keep my DRAFT: commit, but make sure to keep the max body size amendment, as a separate commit.

Thank you!

Also, I think we need to make the notification generation a bit more flexible, so you could prepare data to be rendered inside the subscription, and then pass it to Jinja.

JenySadadia commented 3 weeks ago

Please move your logic for fluster tests from Python to Jinja template code, either the way I did it, or whatever way you prefer. Do not keep my DRAFT: commit, but make sure to keep the max body size amendment, as a separate commit.

As you have contributed to the PR by converting python code to Jinja2, I would like to keep your name in commits with Co-authored-by tag. Thanks again for all the help.

JenySadadia commented 3 weeks ago

Report v3:

Subject: KernelCI report for fluster tests: master@v6.11-rc4
REVISION

    Commit
        name: v6.11-rc4
        hash: 3d5f968a177d468cd13568ef901c5be84d83d32b
     Checked out from
        https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master

FLUSTER TESTS

      google,kingoftown:
       av1:                  ✅ 0  ❌ 0  Σ 0
       av1-chromium-10bit:   ✅ 0  ❌ 0  Σ 0
       h264:                 ✅ 0  ❌ 0  Σ 0
       h264-frext:           ✅ 0  ❌ 0  Σ 0
       h265:                 ✅ 1  ❌ 1  Σ 2
       vp8:                  ✅ 0  ❌ 0  Σ 0
       vp9:                  ✅ 0  ❌ 0  Σ 0

      google,lazor-rev5-sku5:
       av1:                  ✅ 0  ❌ 0  Σ 0
       av1-chromium-10bit:   ✅ 0  ❌ 0  Σ 0
       h264:                 ✅ 0  ❌ 0  Σ 0
       h264-frext:           ✅ 0  ❌ 0  Σ 0
       h265:                 ✅ 0  ❌ 0  Σ 0
       vp8:                  ✅ 0  ❌ 0  Σ 0
       vp9:                  ✅ 0  ❌ 2  Σ 3

See complete and up-to-date report at:

    https://fresh2-kcidb-grafana-m6io3uhhiq-uc.a.run.app/d/bdviluqy6gv7kc/codecs?orgId=1&var-platform=All&var-datasource=default&var-git_commit_hash=3d5f968a177d468cd13568ef901c5be84d83d32b

Tested-by: kernelci.org bot <bot@kernelci.org>

Thanks,
KernelCI team