While working on #92 (typedoc) and running tests with nose, I got pretty annoyed with 'unhelpful' output of nose, which basically just told me "Something isn't working!". Which is why tweaked the test a bit and ran it with pytest, wich gave me a much better overview on what the problem was.
Benefits of pytest
It has a more readable output
It can run nosetests and unittests (nose is still needed as dependency until all tests are refactored)
It has a lot of plugins (i.e. pytest-cov to generate code coverage reports)
with pytest.raises(CustomError, match="error_msg_pattern"):
func_obj(*args, **kwargs)
Example outputs
The test tests/test_typedoc.py::test_references, basically compares the saved output of a doclet (desired result), with the generated one fpr quality. So the nose output is basically massiv_list_of_dicts1 != massiv_list_of_dicts2, which doesn't help (see example below).
As an example to compare the given Assertions, I changed the longname name in tests/typescript/class.ts.jsdoc from external:class~ClassDefinition to external:class~ClassDefinition1:
Pytest shows the failing test from the definition to the line that asserted and tuctates the
compared objects of they are too big.
Note for the output to change I replaced eq_(jsdoc, jsdoc_ref) (from nose) with assert jsdoc == jsdoc_ref, which is the pytest way to compare equality and IMO the prettier python code.
======================================================== FAILURES =========================================================
__________________________________________________ Tests.test_references __________________________________________________
self = <tests.test_typedoc.Tests testMethod=test_references>
def test_references(self):
for source in os.listdir(self.source_dir):
if source.endswith('.ts'):
with open(self.typedoc(source), 'r') as typedocfile:
jsdoc = parse_typedoc(typedocfile)
jsdoc_ref_file = os.path.join(self.source_dir, source + '.jsdoc')
if not os.path.exists(jsdoc_ref_file):
# When a reference file is missing, this is probably a new test.
# Generate a reference file for the developer to review. If the
# generated reference is good, just remove the '.ref' extension.
with open(jsdoc_ref_file + '.ref', 'w') as jsdocfile:
json.dump(
jsdoc,
jsdocfile,
sort_keys=True,
indent=4,
separators=(',', ': '))
print('wrote %s' % jsdoc_ref_file)
else:
with open(jsdoc_ref_file, 'r') as jsdocfile:
jsdoc_ref = json.load(jsdocfile)
# eq_(jsdoc, jsdoc_ref)
> assert jsdoc == jsdoc_ref
E AssertionError: assert [{'comment': ...3', ...}, ...] == [{'comment': '...3', ...}, ...]
E At index 1 diff: {'kind': 'class', 'comment': '<empty>', 'meta': {'path': './',
'filename': 'class.ts', 'lineno': 4, 'code': {}}, 'name': 'ClassDefinition', 'longname': 'external:class~ClassDefinition',
'memberof': 'external:class', 'description': 'A definition of a class\n\n', 'classdesc': '', 'params': [], 'extends': []}
!= {'classdesc': '', 'comment': '<empty>', 'description': 'A definition of a class\n\n', 'extends': [], 'kind': 'class',
'longname': 'external:class~ClassDefinition1', 'memberof': 'external:class', 'meta': {'code': {}, 'filename': 'class.ts', 'lineno...
E
E ...Full output truncated (2 lines hidden), use '-vv' to show
tests\test_typedoc.py:66: AssertionError
-------------------------------------------------- Captured stdout call ---------------------------------------------------
pytest output** (pytest -vv)
Note the +, - and ? at the accual difference.
======================================================== FAILURES =========================================================
__________________________________________________ Tests.test_references __________________________________________________
self = <tests.test_typedoc.Tests testMethod=test_references>
def test_references(self):
for source in os.listdir(self.source_dir):
if source.endswith('.ts'):
with open(self.typedoc(source), 'r') as typedocfile:
jsdoc = parse_typedoc(typedocfile)
jsdoc_ref_file = os.path.join(self.source_dir, source + '.jsdoc')
if not os.path.exists(jsdoc_ref_file):
# When a reference file is missing, this is probably a new test.
# Generate a reference file for the developer to review. If the
# generated reference is good, just remove the '.ref' extension.
with open(jsdoc_ref_file + '.ref', 'w') as jsdocfile:
json.dump(
jsdoc,
jsdocfile,
sort_keys=True,
indent=4,
separators=(',', ': '))
print('wrote %s' % jsdoc_ref_file)
else:
with open(jsdoc_ref_file, 'r') as jsdocfile:
jsdoc_ref = json.load(jsdocfile)
# eq_(jsdoc, jsdoc_ref)
> assert jsdoc == jsdoc_ref
E AssertionError: assert [{'comment': ...3', ...}, ...] == [{'comment': '...3', ...}, ...]
E At index 1 diff: {'kind': 'class', 'comment': '<empty>', 'meta': {'path': './', 'filename': 'class.ts
, 'lineno': 4, 'code': {}}, 'name': 'ClassDefinition', 'longname': 'external:class~ClassDefinition', 'memberof': 'external:
lass', 'description': 'A definition of a class\n\n', 'classdesc': '', 'params': [], 'extends': []} != {'classdesc': '', 'co
ment': '<empty>', 'description': 'A definition of a class\n\n', 'extends': [], 'kind': 'class', 'longname': 'external:class
ClassDefinition1', 'memberof': 'external:class', 'meta': {'code': {}, 'filename': 'class.ts', 'lineno': 4, 'path': './'}, '
ame': 'ClassDefinition', 'params': []}
E Full diff:
E [{'comment': '<empty>',
E 'description': '',
E 'kind': 'external',
E 'longname': 'external:class',
E 'meta': {'code': {}, 'filename': 'class.ts', 'lineno': 1, 'path': './'},
E 'name': '"class"'},
E {'classdesc': '',
E 'comment': '<empty>',
E 'description': 'A definition of a class\n\n',
E 'extends': [],
E 'kind': 'class',
E - 'longname': 'external:class~ClassDefinition',
E + 'longname': 'external:class~ClassDefinition1',
E ? +
E 'memberof': 'external:class',
E 'meta': {'code': {}, 'filename': 'class.ts', 'lineno': 4, 'path': './'},
E 'name': 'ClassDefinition',
E 'params': []},
E {'comment': '<empty>',
E 'description': 'ClassDefinition constructor\n\n',
E 'kind': 'function',
E 'longname': 'external:class~ClassDefinition~new ClassDefinition',
E 'memberof': 'external:class~ClassDefinition',
E 'meta': {'code': {'paramnames': ['simple', 'complex', 'args']},
E 'filename': 'class.ts',
E 'lineno': 4,
E 'path': './'},
E 'name': 'new ClassDefinition',
E 'params': [{'description': 'A parameter with a simple type\n\n',
E 'name': 'simple',
E 'type': {'names': ['number']}},
E {'description': 'A parameter with a complex type\n\n',
E 'name': 'complex',
E 'type': {'names': ['<TODO>']}},
E {'description': 'rest arguments\n\n\n',
E 'name': 'args',
E 'type': {'names': ['any[]']}}],
E 'returns': [{'name': 'new ClassDefinition',
E 'type': {'names': ['class.ClassDefinition']}}]},
E {'comment': '<empty>',
E 'description': 'Read/write property/variable\n\n',
E 'kind': 'member',
E 'longname': 'external:class~ClassDefinition#property',
E 'memberof': 'external:class~ClassDefinition',
E 'meta': {'code': {}, 'filename': 'class.ts', 'lineno': 35, 'path': './'},
E 'name': 'property',
E 'type': {'names': ['any']}},
E {'comment': '<empty>',
E 'description': 'Read-only or getter property\n\n',
E 'kind': 'member',
E 'longname': 'external:class~ClassDefinition#property2',
E 'memberof': 'external:class~ClassDefinition',
E 'meta': {'code': {}, 'filename': 'class.ts', 'lineno': 38, 'path': './'},
E 'name': 'property2',
E 'type': {'names': ['any']}},
E {'comment': '<empty>',
E 'description': 'Property getter\n\n',
E 'kind': 'member',
E 'longname': 'external:class~ClassDefinition#property3',
E 'memberof': 'external:class~ClassDefinition',
E 'meta': {'code': {}, 'filename': 'class.ts', 'lineno': 44, 'path': './'},
E 'name': 'property3',
E 'type': {'names': ['any']}},
E {'comment': '<empty>',
E 'description': 'Property setter\n\n',
E 'kind': 'member',
E 'longname': 'external:class~ClassDefinition#property4',
E 'memberof': 'external:class~ClassDefinition',
E 'meta': {'code': {}, 'filename': 'class.ts', 'lineno': 52, 'path': './'},
E 'name': 'property4',
E 'type': {'names': ['any']}},
E {'comment': '<empty>',
E 'description': 'This is a method without return type\n\n',
E 'kind': 'function',
E 'longname': 'external:class~ClassDefinition.method1',
E 'memberof': 'external:class~ClassDefinition',
E 'meta': {'code': {'paramnames': ['simple', 'complex', 'args']},
E 'filename': 'class.ts',
E 'lineno': 20,
E 'path': './'},
E 'name': 'method1',
E 'params': [{'description': '\n\nA parameter with a simple type',
E 'name': 'simple',
E 'type': {'names': ['number']}},
E {'description': '\n\nA parameter with a complex type',
E 'name': 'complex',
E 'type': {'names': ['<TODO>']}},
E {'description': '\n\nrest arguments\n',
E 'name': 'args',
E 'type': {'names': ['any[]']}}],
E 'returns': []},
E {'comment': '<empty>',
E 'description': 'This is a method with a simple return type\n\n',
E 'kind': 'function',
E 'longname': 'external:class~ClassDefinition.method2',
E 'memberof': 'external:class~ClassDefinition',
E 'meta': {'code': {'paramnames': ['simple', 'complex', 'args']},
E 'filename': 'class.ts',
E 'lineno': 30,
E 'path': './'},
E 'name': 'method2',
E 'params': [{'description': '\n\nA parameter with a simple type',
E 'name': 'simple',
E 'type': {'names': ['number']}},
E {'description': '\n\nA parameter with a complex type',
E 'name': 'complex',
E 'type': {'names': ['<TODO>']}},
E {'description': '\n\nrest arguments',
E 'name': 'args',
E 'type': {'names': ['any[]']}}],
E 'returns': [{'description': '42\n',
E 'name': 'method2',
E 'type': {'names': ['number']}}]}]
tests\test_typedoc.py:66: AssertionError
erikrose
commented
5 years ago
While working on #92 (typedoc) and running tests with
nose, I got pretty annoyed with 'unhelpful' output ofnose, which basically just told me "Something isn't working!". Which is why tweaked the test a bit and ran it withpytest, wich gave me a much better overview on what the problem was.Benefits of pytest
nosetestsandunittests(noseis still needed as dependency until all tests are refactored)pytest-covto generate code coverage reports)eq_(a, b)=>assert a == bassert_in(a, b)=>assert a in bassert_not_in(a, b)=>assert a not in bassert_raises(CustomError, func_obj, args)=>Example outputs
The test
tests/test_typedoc.py::test_references, basically compares the saved output of adoclet(desired result), with the generated one fpr quality. So thenoseoutput is basicallymassiv_list_of_dicts1 != massiv_list_of_dicts2, which doesn't help (see example below).As an example to compare the given Assertions, I changed the longname name in
tests/typescript/class.ts.jsdocfromexternal:class~ClassDefinitiontoexternal:class~ClassDefinition1:noseoutput (python setup.py test)pytestoutput (pytest)Pytestshows the failing test from the definition to the line that asserted and tuctates the compared objects of they are too big. Note for the output to change I replacedeq_(jsdoc, jsdoc_ref)(fromnose) withassert jsdoc == jsdoc_ref, which is thepytestway to compare equality and IMO the prettier python code.pytestoutput** (pytest -vv)Note the
+,-and?at the accual difference.