openshift-eng / elliott

Apache License 2.0
15 stars 28 forks source link

[ART-3479] more validations for OLM operators/operands #534

Closed sosiouxme closed 1 year ago

sosiouxme commented 1 year ago

Check for some common gotchas and problems.

$ elliott -g openshift-4.12 verify-attached-operators 104603 --exclude-attached --exclude-shipped
$ elliott -g openshift-4.13 verify-attached-operators 111422

This finds that there were a ton of problems with the 4.12 GA operator bundles (yep), and only one with current 4.13 bundles:

Bundle ose-ptp-operator-metadata-container-v4.13.0.202303180002.p0.g49d521a.assembly.stream-1 reference cloud-event-proxy-container-v4.13.0-202303180002.p0.gf0ea5cf.assembly.stream:
    needs CDN repo 'openshift4/ose-cloud-event-proxy' but advisory only has {'openshift4/cloud-event-proxy-rhel8', 'openshift4/ose-cloud-event-proxy-rhel8'}
openshift-bot commented 1 year ago

Build #5

.package recreate: /mnt/workspace/jenkins/working/art-tools_elliott_PR-534/.tox/.package
.package installdeps: setuptools>=65.5.1, setuptools_scm[toml]>=6.2
/mnt/workspace/jenkins/working/art-tools_elliott_PR-534/.tox/.package/lib/python3.8/site-packages/setuptools/config/pyprojecttoml.py:108: _BetaConfiguration: Support for `[tool.setuptools]` in `pyproject.toml` is still *beta*.
  warnings.warn(msg, _BetaConfiguration)
/mnt/workspace/jenkins/working/art-tools_elliott_PR-534/.tox/.package/lib/python3.8/site-packages/setuptools/config/pyprojecttoml.py:108: _BetaConfiguration: Support for `[tool.setuptools]` in `pyproject.toml` is still *beta*.
  warnings.warn(msg, _BetaConfiguration)
py38 recreate: /mnt/workspace/jenkins/working/art-tools_elliott_PR-534/.tox/py38
py38 installdeps: .[tests]
py38 inst: /mnt/workspace/jenkins/working/art-tools_elliott_PR-534/.tox/.tmp/package/1/rh-elliott-2.0.15.dev41+g67d0414.tar.gz
py38 installed: aiodns==3.0.0,aiohttp==3.8.4,aiosignal==1.3.1,astroid==2.15.1,async-timeout==4.0.2,asynctest==0.13.0,attrs==22.2.0,Brotli==1.0.9,cachetools==5.3.0,cchardet==2.1.7,certifi==2022.12.7,cffi==1.15.1,chardet==5.1.0,charset-normalizer==3.1.0,click==8.1.3,colorama==0.4.6,coverage==7.2.2,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.6,distlib==0.3.6,errata-tool==1.30.0,exceptiongroup==1.1.1,filelock==3.10.7,flake8==6.0.0,flexmock==0.11.3,frozenlist==1.3.3,future==0.18.3,gssapi==1.8.2,idna==3.4,iniconfig==2.0.0,isort==5.12.0,jira==3.5.0,jsonpath-rw==1.4.0,koji==1.32.0,lazy-object-proxy==1.9.0,mccabe==0.7.0,mock==5.0.1,multidict==6.0.4,mypy==1.1.1,mypy-extensions==1.0.0,oauthlib==3.2.2,packaging==23.0,platformdirs==3.2.0,pluggy==1.0.0,ply==3.11,pycares==4.3.0,pycodestyle==2.10.0,pycparser==2.21,pyflakes==3.0.1,pygit2==1.10.1,pylint==2.17.1,pyproject-api==1.5.1,pytest==7.2.2,python-bugzilla==3.2.0,python-dateutil==2.8.2,PyYAML==6.0,requests==2.28.2,requests-gssapi==1.2.3,requests-oauthlib==1.3.1,requests-toolbelt==0.10.1,rh-elliott @ file:///mnt/workspace/jenkins/working/art-tools_elliott_PR-534/.tox/.tmp/package/1/rh-elliott-2.0.15.dev41%2Bg67d0414.tar.gz,ruamel.yaml==0.17.21,ruamel.yaml.clib==0.2.7,semver==2.13.0,six==1.16.0,tenacity==8.2.2,tomli==2.0.1,tomlkit==0.11.7,tox==4.4.8,typing-extensions==4.5.0,urllib3==1.26.15,virtualenv==20.21.0,wrapt==1.15.0,yarl==1.8.2
py38 run-test-pre: PYTHONHASHSEED='2316113586'
py38 run-test: commands[0] | coverage run --branch --source elliottlib -m unittest discover -t . -s tests/
..............................................Cannot find any corresponding flaw bugs for these trackers: [13]. These trackers do not have a valid whiteboard component value: [12].
..BZ 9 is ignored because its status was MODIFIED at the moment of sweep cutoff (2021-06-30 12:30:00), however its status changed back to ['ASSIGNED'] afterwards
.....................changed 123 from status1 to status2
.123 is already on status1
.changed 123 from status1 to status2
..changed 123 from status1 to status2
.123 is already on status1
.changed 123 from status1 to status2
.................................................Bug(s) ['OCPBUGS-4', 'OCPBUGS-6', 'OCPBUGS-5'] have CVE in description but do not have TrackerBug labels. Please investigate.
............................................/mnt/workspace/jenkins/working/art-tools_elliott_PR-534/.tox/py38/lib64/python3.8/site-packages/aiohttp/client.py:342: ResourceWarning: Unclosed client session <aiohttp.client.ClientSession object at 0x7fc2200215e0>
  _warnings.warn(
ResourceWarning: Enable tracemalloc to get the object allocation traceback
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7fc2200215e0>
......
----------------------------------------------------------------------
Ran 174 tests in 0.942s

OK
py38 run-test: commands[1] | coverage report
Name                                              Stmts   Miss Branch BrPart  Cover
-----------------------------------------------------------------------------------
elliottlib/__init__.py                               12      7      2      1    43%
elliottlib/_version.py                                2      2      0      0     0%
elliottlib/assembly.py                               92      9     54      7    89%
elliottlib/assertion.py                              11      0      6      0   100%
elliottlib/brew.py                                  336    195    134      5    36%
elliottlib/bug/__init__.py                            0      0      0      0   100%
elliottlib/build_finder.py                          114     12     74     15    86%
elliottlib/bzutil.py                                807    329    336     26    55%
elliottlib/cincinnati.py                             21     21      4      0     0%
elliottlib/cli/__init__.py                            0      0      0      0   100%
elliottlib/cli/__main__.py                          270    270     75      0     0%
elliottlib/cli/add_metadata_cli.py                   28     28      4      0     0%
elliottlib/cli/advisory_commons_cli.py               90     90     38      0     0%
elliottlib/cli/advisory_drop_cli.py                  37     37     14      0     0%
elliottlib/cli/advisory_images_cli.py                14     14      2      0     0%
elliottlib/cli/advisory_impetus_cli.py               22     22      6      0     0%
elliottlib/cli/attach_bugs_cli.py                    37     37     18      0     0%
elliottlib/cli/attach_cve_flaws_cli.py              130     66     58      9    45%
elliottlib/cli/change_state_cli.py                   44     44     16      0     0%
elliottlib/cli/cli_opts.py                           15      0     17      0   100%
elliottlib/cli/common.py                             48     12      6      1    69%
elliottlib/cli/create_cli.py                         70     70     20      0     0%
elliottlib/cli/create_placeholder_cli.py             36     36     12      0     0%
elliottlib/cli/create_textonly_cli.py                51     51      8      0     0%
elliottlib/cli/find_bugs_blocker_cli.py              43      4     10      3    87%
elliottlib/cli/find_bugs_qe_cli.py                   36      4      6      0    90%
elliottlib/cli/find_bugs_sweep_cli.py               218     33    124     21    82%
elliottlib/cli/find_builds_cli.py                   268    198    179      2    19%
elliottlib/cli/get_golang_versions_cli.py            59     14     32      9    73%
elliottlib/cli/list_cli.py                           14     14      2      0     0%
elliottlib/cli/puddle_advisories_cli.py              34     34      8      0     0%
elliottlib/cli/remove_bugs_cli.py                    46      6     14      6    80%
elliottlib/cli/repair_bugs_cli.py                    61      5     26      6    87%
elliottlib/cli/rhcos_cli.py                          93     93     52      0     0%
elliottlib/cli/rpmdiff_cli.py                       102    102     34      0     0%
elliottlib/cli/tag_builds_cli.py                    128    128     62      0     0%
elliottlib/cli/tarball_sources_cli.py                83     83     26      0     0%
elliottlib/cli/validate_rhsa.py                      45     45     16      0     0%
elliottlib/cli/verify_attached_bugs_cli.py          257     91    167     20    57%
elliottlib/cli/verify_attached_operators_cli.py     148     56     68      4    56%
elliottlib/cli/verify_cvp_cli.py                    165    165     78      0     0%
elliottlib/constants.py                              52      0      0      0   100%
elliottlib/cvp.py                                   216    216    113      0     0%
elliottlib/dotconfig.py                              54     11     30      6    75%
elliottlib/errata.py                                329    208    142      4    33%
elliottlib/errata_async.py                          151     14     58      9    89%
elliottlib/exceptions.py                             12      0      0      0   100%
elliottlib/exectools.py                              84      7     24      3    89%
elliottlib/gitdata.py                               171    137     76      0    14%
elliottlib/imagecfg.py                               21      2      0      0    90%
elliottlib/logutil.py                                 9      1      2      1    82%
elliottlib/metadata.py                              154     43     76     18    67%
elliottlib/model.py                                 105     22     30      2    79%
elliottlib/openshiftclient.py                        34     34      8      0     0%
elliottlib/pushd.py                                  21      0      2      0   100%
elliottlib/resultsdb.py                              28     28      4      0     0%
elliottlib/rhcos.py                                  63     63     16      0     0%
elliottlib/rpm_utils.py                              30     14     16      5    46%
elliottlib/rpmcfg.py                                  5      2      0      0    60%
elliottlib/rpmdiff.py                                31      0      0      0   100%
elliottlib/runtime.py                               278    202    106      2    22%
elliottlib/tarball_sources.py                        90     61     34      0    30%
elliottlib/util.py                                  297    139    110     15    50%
-----------------------------------------------------------------------------------
TOTAL                                              6322   3631   2655    200    40%
py38 run-test: commands[2] | flake8
___________________________________ summary ____________________________________
  py38: commands succeeded
  congratulations :)
openshift-bot commented 1 year ago

Build #6

.package recreate: /mnt/workspace/jenkins/working/art-tools_elliott_PR-534/.tox/.package
.package installdeps: setuptools>=65.5.1, setuptools_scm[toml]>=6.2
/mnt/workspace/jenkins/working/art-tools_elliott_PR-534/.tox/.package/lib/python3.8/site-packages/setuptools/config/pyprojecttoml.py:108: _BetaConfiguration: Support for `[tool.setuptools]` in `pyproject.toml` is still *beta*.
  warnings.warn(msg, _BetaConfiguration)
/mnt/workspace/jenkins/working/art-tools_elliott_PR-534/.tox/.package/lib/python3.8/site-packages/setuptools/config/pyprojecttoml.py:108: _BetaConfiguration: Support for `[tool.setuptools]` in `pyproject.toml` is still *beta*.
  warnings.warn(msg, _BetaConfiguration)
py38 recreate: /mnt/workspace/jenkins/working/art-tools_elliott_PR-534/.tox/py38
py38 installdeps: .[tests]
py38 inst: /mnt/workspace/jenkins/working/art-tools_elliott_PR-534/.tox/.tmp/package/1/rh-elliott-2.0.15.dev43+g19f392c.tar.gz
py38 installed: aiodns==3.0.0,aiohttp==3.8.4,aiosignal==1.3.1,astroid==2.15.1,async-timeout==4.0.2,asynctest==0.13.0,attrs==22.2.0,Brotli==1.0.9,cachetools==5.3.0,cchardet==2.1.7,certifi==2022.12.7,cffi==1.15.1,chardet==5.1.0,charset-normalizer==3.1.0,click==8.1.3,colorama==0.4.6,coverage==7.2.2,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.6,distlib==0.3.6,errata-tool==1.30.0,exceptiongroup==1.1.1,filelock==3.10.7,flake8==6.0.0,flexmock==0.11.3,frozenlist==1.3.3,future==0.18.3,gssapi==1.8.2,idna==3.4,iniconfig==2.0.0,isort==5.12.0,jira==3.5.0,jsonpath-rw==1.4.0,koji==1.32.0,lazy-object-proxy==1.9.0,mccabe==0.7.0,mock==5.0.1,multidict==6.0.4,mypy==1.1.1,mypy-extensions==1.0.0,oauthlib==3.2.2,packaging==23.0,platformdirs==3.2.0,pluggy==1.0.0,ply==3.11,pycares==4.3.0,pycodestyle==2.10.0,pycparser==2.21,pyflakes==3.0.1,pygit2==1.10.1,pylint==2.17.1,pyproject-api==1.5.1,pytest==7.2.2,python-bugzilla==3.2.0,python-dateutil==2.8.2,PyYAML==6.0,requests==2.28.2,requests-gssapi==1.2.3,requests-oauthlib==1.3.1,requests-toolbelt==0.10.1,rh-elliott @ file:///mnt/workspace/jenkins/working/art-tools_elliott_PR-534/.tox/.tmp/package/1/rh-elliott-2.0.15.dev43%2Bg19f392c.tar.gz,ruamel.yaml==0.17.21,ruamel.yaml.clib==0.2.7,semver==2.13.0,six==1.16.0,tenacity==8.2.2,tomli==2.0.1,tomlkit==0.11.7,tox==4.4.8,typing-extensions==4.5.0,urllib3==1.26.15,virtualenv==20.21.0,wrapt==1.15.0,yarl==1.8.2
py38 run-test-pre: PYTHONHASHSEED='3426693219'
py38 run-test: commands[0] | coverage run --branch --source elliottlib -m unittest discover -t . -s tests/
..............................................Cannot find any corresponding flaw bugs for these trackers: [13]. These trackers do not have a valid whiteboard component value: [12].
..BZ 9 is ignored because its status was MODIFIED at the moment of sweep cutoff (2021-06-30 12:30:00), however its status changed back to ['ASSIGNED'] afterwards
.....................changed 123 from status1 to status2
.123 is already on status1
.changed 123 from status1 to status2
..changed 123 from status1 to status2
.123 is already on status1
.changed 123 from status1 to status2
.................................................Bug(s) ['OCPBUGS-4', 'OCPBUGS-5', 'OCPBUGS-6'] have CVE in description but do not have TrackerBug labels. Please investigate.
............................................/mnt/workspace/jenkins/working/art-tools_elliott_PR-534/.tox/py38/lib64/python3.8/site-packages/aiohttp/client.py:342: ResourceWarning: Unclosed client session <aiohttp.client.ClientSession object at 0x7ffa930be5b0>
  _warnings.warn(
ResourceWarning: Enable tracemalloc to get the object allocation traceback
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7ffa930be5b0>
......
----------------------------------------------------------------------
Ran 174 tests in 1.058s

OK
py38 run-test: commands[1] | coverage report
Name                                              Stmts   Miss Branch BrPart  Cover
-----------------------------------------------------------------------------------
elliottlib/__init__.py                               12      7      2      1    43%
elliottlib/_version.py                                2      2      0      0     0%
elliottlib/assembly.py                               92      9     54      7    89%
elliottlib/assertion.py                              11      0      6      0   100%
elliottlib/brew.py                                  336    195    134      5    36%
elliottlib/bug/__init__.py                            0      0      0      0   100%
elliottlib/build_finder.py                          114     12     74     15    86%
elliottlib/bzutil.py                                807    329    336     26    55%
elliottlib/cincinnati.py                             21     21      4      0     0%
elliottlib/cli/__init__.py                            0      0      0      0   100%
elliottlib/cli/__main__.py                          270    270     75      0     0%
elliottlib/cli/add_metadata_cli.py                   28     28      4      0     0%
elliottlib/cli/advisory_commons_cli.py               90     90     38      0     0%
elliottlib/cli/advisory_drop_cli.py                  37     37     14      0     0%
elliottlib/cli/advisory_images_cli.py                14     14      2      0     0%
elliottlib/cli/advisory_impetus_cli.py               22     22      6      0     0%
elliottlib/cli/attach_bugs_cli.py                    37     37     18      0     0%
elliottlib/cli/attach_cve_flaws_cli.py              130     66     58      9    45%
elliottlib/cli/change_state_cli.py                   44     44     16      0     0%
elliottlib/cli/cli_opts.py                           15      0     17      0   100%
elliottlib/cli/common.py                             48     12      6      1    69%
elliottlib/cli/create_cli.py                         70     70     20      0     0%
elliottlib/cli/create_placeholder_cli.py             36     36     12      0     0%
elliottlib/cli/create_textonly_cli.py                51     51      8      0     0%
elliottlib/cli/find_bugs_blocker_cli.py              43      4     10      3    87%
elliottlib/cli/find_bugs_qe_cli.py                   36      4      6      0    90%
elliottlib/cli/find_bugs_sweep_cli.py               218     33    124     21    82%
elliottlib/cli/find_builds_cli.py                   268    198    179      2    19%
elliottlib/cli/get_golang_versions_cli.py            59     14     32      9    73%
elliottlib/cli/list_cli.py                           14     14      2      0     0%
elliottlib/cli/puddle_advisories_cli.py              34     34      8      0     0%
elliottlib/cli/remove_bugs_cli.py                    46      6     14      6    80%
elliottlib/cli/repair_bugs_cli.py                    61      5     26      6    87%
elliottlib/cli/rhcos_cli.py                          93     93     52      0     0%
elliottlib/cli/rpmdiff_cli.py                       102    102     34      0     0%
elliottlib/cli/tag_builds_cli.py                    128    128     62      0     0%
elliottlib/cli/tarball_sources_cli.py                83     83     26      0     0%
elliottlib/cli/validate_rhsa.py                      45     45     16      0     0%
elliottlib/cli/verify_attached_bugs_cli.py          257     91    167     20    57%
elliottlib/cli/verify_attached_operators_cli.py     148     56     68      4    56%
elliottlib/cli/verify_cvp_cli.py                    165    165     78      0     0%
elliottlib/constants.py                              52      0      0      0   100%
elliottlib/cvp.py                                   216    216    113      0     0%
elliottlib/dotconfig.py                              54     11     30      6    75%
elliottlib/errata.py                                329    208    142      4    33%
elliottlib/errata_async.py                          158     20     60      9    86%
elliottlib/exceptions.py                             12      0      0      0   100%
elliottlib/exectools.py                              84      7     24      3    89%
elliottlib/gitdata.py                               171    137     76      0    14%
elliottlib/imagecfg.py                               21      2      0      0    90%
elliottlib/logutil.py                                 9      1      2      1    82%
elliottlib/metadata.py                              154     43     76     18    67%
elliottlib/model.py                                 105     22     30      2    79%
elliottlib/openshiftclient.py                        34     34      8      0     0%
elliottlib/pushd.py                                  21      0      2      0   100%
elliottlib/resultsdb.py                              28     28      4      0     0%
elliottlib/rhcos.py                                  63     63     16      0     0%
elliottlib/rpm_utils.py                              30     14     16      5    46%
elliottlib/rpmcfg.py                                  5      2      0      0    60%
elliottlib/rpmdiff.py                                31      0      0      0   100%
elliottlib/runtime.py                               278    202    106      2    22%
elliottlib/tarball_sources.py                        90     61     34      0    30%
elliottlib/util.py                                  297    139    110     15    50%
-----------------------------------------------------------------------------------
TOTAL                                              6329   3637   2657    200    40%
py38 run-test: commands[2] | flake8
./elliottlib/errata_async.py:100:78: W291 trailing whitespace
./elliottlib/errata_async.py:109:5: E303 too many blank lines (2)
ERROR: InvocationError for command /mnt/workspace/jenkins/working/art-tools_elliott_PR-534/.tox/py38/bin/flake8 (exited with code 1)
___________________________________ summary ____________________________________
ERROR:   py38: commands failed