python / cpython

The Python programming language
https://www.python.org
Other
63.54k stars 30.44k forks source link

Roundtrip-test tokenize.untokenize(iterable_of_5_tuples) #64949

Open terryjreedy opened 10 years ago

terryjreedy commented 10 years ago
BPO 20750
Nosy @terryjreedy, @iritkatriel

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields: ```python assignee = 'https://github.com/terryjreedy' closed_at = None created_at = labels = ['tests', 'type-bug', 'library', '3.11'] title = 'Roundtrip-test tokenize.untokenize(iterable_of_5_tuples)' updated_at = user = 'https://github.com/terryjreedy' ``` bugs.python.org fields: ```python activity = actor = 'iritkatriel' assignee = 'terry.reedy' closed = False closed_date = None closer = None components = ['Library (Lib)', 'Tests'] creation = creator = 'terry.reedy' dependencies = [] files = [] hgrepos = [] issue_num = 20750 keywords = [] message_count = 4.0 messages = ['212034', '212040', '212043', '407578'] nosy_count = 3.0 nosy_names = ['terry.reedy', 'python-dev', 'iritkatriel'] pr_nums = [] priority = 'normal' resolution = None stage = 'needs patch' status = 'open' superseder = None type = 'behavior' url = 'https://bugs.python.org/issue20750' versions = ['Python 3.11'] ```

terryjreedy commented 10 years ago

Currently, test_tokenize.roundtrip only tests the 2-tuple mode of untokenize. The currently buggy 5-tuple mode needs to be tested also, as far as possible, to reduce the chance of introducing more bugs when fixing current bugs. When this is done, all the constructed tests pass. Good so far.

The doctest normally tests 10 files selected from test/test_*.py and a few tries showed no problem. However, when testing all with -ucpu. F:\Python\dev> 3\py33\pcbuild\python_d -m test -ucpu test_tokenize 7 failed. So we must remove these from the list of candidates until the failure cause is determined and either fixed or determined not fixable (as with test_3131.py).

As suggested in the revised docstring, finding the failure items for these 7 would be much easier if the file roundtrip test were removed from the doctest and made into a unittest with assertEqual.

1762cc99-3127-4a62-9baf-30c3d0f51ef7 commented 10 years ago

New changeset 8d6dd02a973f by Terry Jan Reedy in branch '3.3': Issue bpo-20750, Enable roundtrip tests for new 5-tuple untokenize. The http://hg.python.org/cpython/rev/8d6dd02a973f

New changeset 73aa6d672b81 by Terry Jan Reedy in branch 'default': Merge with 3.3, bpo-20750 http://hg.python.org/cpython/rev/73aa6d672b81

terryjreedy commented 10 years ago

The 2.7 tokenize tests do not have roundtrip tests.

Excluding the 7 files is a temporary hack. The issue should stay open until each either passes and is removed from the exclusion list or is determined to be a permanent exclusion, like test_pep3131.

iritkatriel commented 2 years ago

The 7 tests are still removed and need to be checked:

https://github.com/python/cpython/blob/03768c4d139df46212a091ed931aad03bec18b57/Lib/test/test_tokenize.py#L1620