WIP Complain when tracker/flaw bugs are inconsistent

[openshift-bot]

Build #1

GLOB sdist-make: /mnt/workspace/jenkins/working/art-tools_elliott_PR-480/setup.py
py38 create: /mnt/workspace/jenkins/working/art-tools_elliott_PR-480/.tox/py38
py38 installdeps: -rrequirements-dev.txt
py38 inst: /mnt/workspace/jenkins/working/art-tools_elliott_PR-480/.tox/.tmp/package/1/rh-elliott-2.0.14.dev201+gd405082.zip
py38 installed: aiodns==3.0.0,aiohttp==3.8.3,aiosignal==1.3.1,astroid==2.12.13,async-timeout==4.0.2,attrs==22.1.0,Brotli==1.0.9,cachetools==5.2.0,cchardet==2.1.7,certifi==2022.12.7,cffi==1.15.1,chardet==5.1.0,charset-normalizer==2.1.1,click==8.1.3,colorama==0.4.6,coverage==6.5.0,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.6,distlib==0.3.6,errata-tool==1.30.0,exceptiongroup==1.0.4,filelock==3.8.2,flake8==6.0.0,flexmock==0.11.3,frozenlist==1.3.3,future==0.18.2,gssapi==1.8.2,idna==3.4,iniconfig==1.1.1,isort==5.11.0,jira==3.4.1,jsonpath-rw==1.4.0,koji==1.31.0,lazy-object-proxy==1.8.0,mccabe==0.7.0,mock==4.0.3,multidict==6.0.3,mypy==0.991,mypy-extensions==0.4.3,oauthlib==3.2.2,packaging==22.0,platformdirs==2.6.0,pluggy==1.0.0,ply==3.11,pycares==4.3.0,pycodestyle==2.10.0,pycparser==2.21,pyflakes==3.0.1,pygit2==1.10.1,pylint==2.15.8,pyproject_api==1.2.1,pytest==7.2.0,python-bugzilla==3.2.0,python-dateutil==2.8.2,PyYAML==6.0,requests==2.28.1,requests-gssapi==1.2.3,requests-oauthlib==1.3.1,requests-toolbelt==0.10.1,rh-elliott @ file:///mnt/workspace/jenkins/working/art-tools_elliott_PR-480/.tox/.tmp/package/1/rh-elliott-2.0.14.dev201%2Bgd405082.zip,ruamel.yaml==0.17.21,ruamel.yaml.clib==0.2.7,semver==2.13.0,setuptools-scm==7.0.5,six==1.16.0,tenacity==8.1.0,tomli==2.0.1,tomlkit==0.11.6,tox==4.0.8,typing_extensions==4.4.0,urllib3==1.26.13,virtualenv==20.17.1,wrapt==1.14.1,yarl==1.8.2
py38 run-test-pre: PYTHONHASHSEED='2897977123'
py38 run-test: commands[0] | coverage run --branch --source elliottlib -m unittest discover -t . -s tests/
................................................BZ 9 is ignored because its status was MODIFIED at the moment of sweep cutoff (2021-06-30 12:30:00), however its status changed back to ['ASSIGNED'] afterwards
......E...............E...changed 123 from status1 to status2
.123 is already on status1
.changed 123 from status1 to status2
..changed 123 from status1 to status2
.123 is already on status1
.changed 123 from status1 to status2
........................../mnt/workspace/jenkins/working/art-tools_elliott_PR-480/elliottlib/errata_async.py:39: RuntimeWarning: coroutine 'AsyncMockMixin._execute_mock_call' was never awaited
  resp.raise_for_status()
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
................F.....Bugs ['OCPBUGS-4', 'OCPBUGS-5'] have CVE number in summary but do not have tracker labels. Please fix this
...........................................
======================================================================
ERROR: test_get_corresponding_flaw_bugs_bz_strict (tests.test_bzutil.TestBugzillaBugTracker)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/mnt/workspace/jenkins/working/art-tools_elliott_PR-480/tests/test_bzutil.py", line 114, in test_get_corresponding_flaw_bugs_bz_strict
    self.assertRaisesRegex(
  File "/usr/lib64/python3.8/unittest/case.py", line 1357, in assertRaisesRegex
    return context.handle('assertRaisesRegex', args, kwargs)
  File "/usr/lib64/python3.8/unittest/case.py", line 202, in handle
    callable_obj(*args, **kwargs)
  File "/mnt/workspace/jenkins/working/art-tools_elliott_PR-480/elliottlib/bzutil.py", line 430, in get_corresponding_flaw_bugs
    flaw_bugs = bug_tracker.get_flaw_bugs(
  File "/mnt/workspace/jenkins/working/art-tools_elliott_PR-480/elliottlib/bzutil.py", line 881, in get_flaw_bugs
    raise ValueError(f"Unexpected: {not_flaws} were found to not be qualified flaw bugs")
ValueError: Unexpected: [1, 2] were found to not be qualified flaw bugs

======================================================================
ERROR: test_get_corresponding_flaw_bugs_jira (tests.test_bzutil.TestJIRABugTracker)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/mnt/workspace/jenkins/working/art-tools_elliott_PR-480/tests/test_bzutil.py", line 57, in test_get_corresponding_flaw_bugs_jira
    actual = BugTracker.get_corresponding_flaw_bugs(tracker_bugs, flaw_bug_tracker)
  File "/mnt/workspace/jenkins/working/art-tools_elliott_PR-480/elliottlib/bzutil.py", line 430, in get_corresponding_flaw_bugs
    flaw_bugs = bug_tracker.get_flaw_bugs(
  File "/mnt/workspace/jenkins/working/art-tools_elliott_PR-480/elliottlib/bzutil.py", line 881, in get_flaw_bugs
    raise ValueError(f"Unexpected: {not_flaws} were found to not be qualified flaw bugs")
ValueError: Unexpected: [9998] were found to not be qualified flaw bugs

======================================================================
FAIL: test_find_bugs_sweep_advisory_jira (tests.test_find_bugs_sweep_cli.FindBugsSweepTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/mnt/workspace/jenkins/working/art-tools_elliott_PR-480/tests/test_find_bugs_sweep_cli.py", line 140, in test_find_bugs_sweep_advisory_jira
    self.fail(t)
AssertionError: Traceback (most recent call last):

  File "/mnt/workspace/jenkins/working/art-tools_elliott_PR-480/.tox/py38/lib/python3.8/site-packages/click/testing.py", line 408, in invoke
    return_value = cli.main(args=args or (), prog_name=prog_name, **extra)

  File "/mnt/workspace/jenkins/working/art-tools_elliott_PR-480/.tox/py38/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)

  File "/mnt/workspace/jenkins/working/art-tools_elliott_PR-480/.tox/py38/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))

  File "/mnt/workspace/jenkins/working/art-tools_elliott_PR-480/.tox/py38/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)

  File "/mnt/workspace/jenkins/working/art-tools_elliott_PR-480/.tox/py38/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)

  File "/mnt/workspace/jenkins/working/art-tools_elliott_PR-480/.tox/py38/lib/python3.8/site-packages/click/decorators.py", line 38, in new_func
    return f(get_current_context().obj, *args, **kwargs)

  File "/mnt/workspace/jenkins/working/art-tools_elliott_PR-480/elliottlib/cli/find_bugs_sweep_cli.py", line 140, in find_bugs_sweep_cli
    raise ElliottFatalError(f"Error finding or attaching bugs: {errors}. See logs for more information.")

elliottlib.exceptions.ElliottFatalError: Error finding or attaching bugs: [AttributeError("'MockClass' object has no attribute 'is_cve_in_summary'"), AttributeError("'MockClass' object has no attribute 'is_cve_in_summary'")]. See logs for more information.

----------------------------------------------------------------------
Ran 168 tests in 0.696s

FAILED (failures=1, errors=2)
ERROR: InvocationError for command /mnt/workspace/jenkins/working/art-tools_elliott_PR-480/.tox/py38/bin/coverage run --branch --source elliottlib -m unittest discover -t . -s tests/ (exited with code 1)
___________________________________ summary ____________________________________
ERROR:   py38: commands failed

[joepvd]

Nice! I like it to shift-left this error to also hit while preparing the advisories, and not hit this error when promoting.

We have fewer methods to override failures in prepare-release, though, we do not have promotion_permits. Do you foresee circumstances where we would have put a promotion_permit, and now would be stuck? Or how do you think the concrete handling would look like?

[openshift-merge-robot]

@thegreyd: PR needs rebase.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.