error when formatting HTTPError

[brondsem]

HTTPError has some weird behavior which causes exceptiongroup to error out when rendering the traceback. There is a cpython issue at https://github.com/python/cpython/issues/98778 about HTTPError's weirdness. But it'd be great if exceptiongroup could help handle this better, so error handling works as expected, and not waiting for a cpython fix and releases.

I found this while using pytest, which uses exceptiongroup.

The setup:

$ cat test_httperror.py
from urllib.error import HTTPError

def test():
    raise HTTPError('url', 408, 'timeout', None, None)

$ pip freeze | egrep 'pytest|exceptiongroup'
exceptiongroup==1.0.1
pytest==7.2.0

The error:

$ pytest test_httperror.py
================================================================================================================================================== test session starts ==================================================================================================================================================
platform linux -- Python 3.7.10, pytest-7.2.0, pluggy-1.0.0
rootdir: /src/allura, configfile: pytest.ini
collected 1 item

test_httperror.py
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/var/local/env-allura/lib/python3.7/site-packages/_pytest/main.py", line 270, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/var/local/env-allura/lib/python3.7/site-packages/_pytest/main.py", line 324, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/var/local/env-allura/lib/python3.7/site-packages/pluggy/_hooks.py", line 265, in __call__
INTERNALERROR>     return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
INTERNALERROR>   File "/var/local/env-allura/lib/python3.7/site-packages/pluggy/_manager.py", line 80, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/var/local/env-allura/lib/python3.7/site-packages/pluggy/_callers.py", line 60, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/var/local/env-allura/lib/python3.7/site-packages/pluggy/_result.py", line 60, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/var/local/env-allura/lib/python3.7/site-packages/pluggy/_callers.py", line 39, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/var/local/env-allura/lib/python3.7/site-packages/_pytest/main.py", line 349, in pytest_runtestloop
INTERNALERROR>     item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
INTERNALERROR>   File "/var/local/env-allura/lib/python3.7/site-packages/pluggy/_hooks.py", line 265, in __call__
INTERNALERROR>     return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
INTERNALERROR>   File "/var/local/env-allura/lib/python3.7/site-packages/pluggy/_manager.py", line 80, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/var/local/env-allura/lib/python3.7/site-packages/pluggy/_callers.py", line 60, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/var/local/env-allura/lib/python3.7/site-packages/pluggy/_result.py", line 60, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/var/local/env-allura/lib/python3.7/site-packages/pluggy/_callers.py", line 39, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/var/local/env-allura/lib/python3.7/site-packages/_pytest/runner.py", line 112, in pytest_runtest_protocol
INTERNALERROR>     runtestprotocol(item, nextitem=nextitem)
INTERNALERROR>   File "/var/local/env-allura/lib/python3.7/site-packages/_pytest/runner.py", line 131, in runtestprotocol
INTERNALERROR>     reports.append(call_and_report(item, "call", log))
INTERNALERROR>   File "/var/local/env-allura/lib/python3.7/site-packages/_pytest/runner.py", line 222, in call_and_report
INTERNALERROR>     report: TestReport = hook.pytest_runtest_makereport(item=item, call=call)
INTERNALERROR>   File "/var/local/env-allura/lib/python3.7/site-packages/pluggy/_hooks.py", line 265, in __call__
INTERNALERROR>     return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
INTERNALERROR>   File "/var/local/env-allura/lib/python3.7/site-packages/pluggy/_manager.py", line 80, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/var/local/env-allura/lib/python3.7/site-packages/pluggy/_callers.py", line 55, in _multicall
INTERNALERROR>     gen.send(outcome)
INTERNALERROR>   File "/var/local/env-allura/lib/python3.7/site-packages/_pytest/skipping.py", line 265, in pytest_runtest_makereport
INTERNALERROR>     rep = outcome.get_result()
INTERNALERROR>   File "/var/local/env-allura/lib/python3.7/site-packages/pluggy/_result.py", line 60, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/var/local/env-allura/lib/python3.7/site-packages/pluggy/_callers.py", line 39, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/var/local/env-allura/lib/python3.7/site-packages/_pytest/runner.py", line 366, in pytest_runtest_makereport
INTERNALERROR>     return TestReport.from_item_and_call(item, call)
INTERNALERROR>   File "/var/local/env-allura/lib/python3.7/site-packages/_pytest/reports.py", line 349, in from_item_and_call
INTERNALERROR>     longrepr = item.repr_failure(excinfo)
INTERNALERROR>   File "/var/local/env-allura/lib/python3.7/site-packages/_pytest/python.py", line 1823, in repr_failure
INTERNALERROR>     return self._repr_failure_py(excinfo, style=style)
INTERNALERROR>   File "/var/local/env-allura/lib/python3.7/site-packages/_pytest/nodes.py", line 490, in _repr_failure_py
INTERNALERROR>     truncate_locals=truncate_locals,
INTERNALERROR>   File "/var/local/env-allura/lib/python3.7/site-packages/_pytest/_code/code.py", line 669, in getrepr
INTERNALERROR>     return fmt.repr_excinfo(self)
INTERNALERROR>   File "/var/local/env-allura/lib/python3.7/site-packages/_pytest/_code/code.py", line 944, in repr_excinfo
INTERNALERROR>     reprtraceback = self.repr_traceback(excinfo_)
INTERNALERROR>   File "/var/local/env-allura/lib/python3.7/site-packages/_pytest/_code/code.py", line 871, in repr_traceback
INTERNALERROR>     reprentry = self.repr_traceback_entry(entry, einfo)
INTERNALERROR>   File "/var/local/env-allura/lib/python3.7/site-packages/_pytest/_code/code.py", line 822, in repr_traceback_entry
INTERNALERROR>     s = self.get_source(source, line_index, excinfo, short=short)
INTERNALERROR>   File "/var/local/env-allura/lib/python3.7/site-packages/_pytest/_code/code.py", line 760, in get_source
INTERNALERROR>     lines.extend(self.get_exconly(excinfo, indent=indent, markall=True))
INTERNALERROR>   File "/var/local/env-allura/lib/python3.7/site-packages/_pytest/_code/code.py", line 772, in get_exconly
INTERNALERROR>     exlines = excinfo.exconly(tryshort=True).split("\n")
INTERNALERROR>   File "/var/local/env-allura/lib/python3.7/site-packages/_pytest/_code/code.py", line 588, in exconly
INTERNALERROR>     lines = format_exception_only(self.type, self.value)
INTERNALERROR>   File "/usr/local/lib/python3.7/traceback.py", line 140, in format_exception_only
INTERNALERROR>     return list(TracebackException(etype, value, None).format_exception_only())
INTERNALERROR>   File "/var/local/env-allura/lib/python3.7/site-packages/exceptiongroup/_formatting.py", line 106, in __init__
INTERNALERROR>     self.__notes__ = getattr(exc_value, "__notes__", None)
INTERNALERROR>   File "/usr/local/lib/python3.7/tempfile.py", line 475, in __getattr__
INTERNALERROR>     file = self.__dict__['file']
INTERNALERROR> KeyError: 'file'

================================================================================================================================================= no tests ran in 0.10s =================================================================================================================================================

[agronholm]

Why is getattr(exc_value, "__notes__", None) triggering a KeyError? What is HTTPError doing to cause that?

[brondsem]

https://github.com/python/cpython/issues/98778 discuss that issue. I just re-read that, and realized I can fix this by creating the HTTPError differently (more correctly, I suppose)

-raise HTTPError('url', 408, 'timeout', None, None)
+raise HTTPError('url', 408, 'timeout', None, io.BytesIO())

So feel free to close this, unless you really want to do extra work :)

[agronholm]

This is where it crashes: https://github.com/python/cpython/blob/c3c3871415c86088d45abcf73ccd2c2b09dc5772/Lib/tempfile.py#L477

The problem seems to be that HTTPError does not call the superclass initializer, but the __getattr__() method in the superclass requires the file attribute to be present. When it's not, it crashes with a KeyError.

[agronholm]

Closing since this is a Python stdlib issue.