trace module cannot produce coverage reports for zipped modules

[abalkin]

BPO	10342
Nosy	@terryjreedy, @abalkin, @vstinner, @PCManticore
Dependencies	bpo-10371: Deprecate trace module undocumented API
Files	testtrace.zip: run as "python testtrace.zip" issue10342.diff issue10342a.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = 'https://github.com/abalkin' closed_at = None created_at = labels = ['3.7', 'type-bug', 'library'] title = 'trace module cannot produce coverage reports for zipped modules' updated_at = user = 'https://github.com/abalkin' ``` bugs.python.org fields: ```python activity = actor = 'belopolsky' assignee = 'belopolsky' closed = False closed_date = None closer = None components = ['Library (Lib)'] creation = creator = 'belopolsky' dependencies = ['10371'] files = ['19522', '19537', '35803'] hgrepos = [] issue_num = 10342 keywords = ['patch', 'needs review'] message_count = 12.0 messages = ['120626', '120691', '120701', '120703', '120704', '120707', '120708', '121051', '121056', '121067', '221928', '221944'] nosy_count = 4.0 nosy_names = ['terry.reedy', 'belopolsky', 'vstinner', 'Claudiu.Popa'] pr_nums = [] priority = 'normal' resolution = None stage = 'patch review' status = 'open' superseder = None type = 'behavior' url = 'https://bugs.python.org/issue10342' versions = ['Python 3.7'] ```

[abalkin]

Please run attached zip archive as a python script. Note that the problem is not specific for using __main__.py - any module that comes from a zip archive or loaded by a custom loader would show the same bug.

$ unzip -l test.zip 
Archive:  test.zip
  Length     Date   Time    Name
 --------    ----   ----    

---

   79  10-24-08 18:26   \_\_main__.py

-------- ------- 79 1 file $ $ cat __main__.py from trace import Trace

def traced(flag):
    if flag:
        return 1
    else:
        return 2

tracer = Trace()
tracer.runfunc(traced, False)
results = tracer.results()
results.write_results(coverdir='.')

$ python testtrace.zip 
 --- modulename: __main__, funcname: traced
Not printing coverage data for 'testtrace.zip/__main__.py': [Errno 20] Not a directory: 'testtrace.zip/__main__.py'
__main__.py(4): __main__.py(7): $

[vstinner]

I commited Alexander's fix for bpo-10329: r86303. The trace module now uses the input Python script encoding, instead of the locale encoding.

[abalkin]

I am attaching a "proof of concept" patch. The trace code is in a dire need of restructuring to eliminate repeated reading of source files.

[vstinner]

+ try: + with open(filename, 'rb') as fp: + encoding, _ = tokenize.detect_encoding(fp.readline) + except IOError: + encoding = None

You should use 'utf-8' instead of None (which will fall back to the locale encoding) here.

[abalkin]

On Sun, Nov 7, 2010 at 8:47 PM, STINNER Victor \report@bugs.python.org\ wrote:

STINNER Victor \victor.stinner@haypocalc.com\ added the comment:

•            try:
•                with open(filename, 'rb') as fp:
•                    encoding, _ = tokenize.detect_encoding(fp.readline)
•            except IOError:
•                encoding = None

You should use 'utf-8' instead of None (which will fall back to the locale encoding) here.

I know. I was too lazy to look up the correct spelling for the proof of concept. I am really posting this patch to show how this bug can be fixed in theory and discuss how much of refactoring is acceptable. For example, do we need to preserve trace.find_strings() function? What is the best way to pass around source code? - file-like objects, line iterators, readline-like function? Also compile(prog, filename, "exec") to find module's bytecode is a hack. There must be a standard way to achieve that which would use a .pyc file if available.

[brettcannon]

The handling of __loader__ looks fine to me, although I don't quite see the point of the get_source call as it isn't returned or used.

[abalkin]

On Sun, Nov 7, 2010 at 8:59 PM, Brett Cannon \report@bugs.python.org\ wrote:

.. I don't quite see the point of the get_source call as it isn't returned or used.

It is passed to find_docstrings() to produce the "strs" dictionary.

[terryjreedy]

What is the best way to pass around source code?

• file-like objects, line iterators, readline-like function?

Line iterator (list of lines) as returned by open().readlines. Memory should not be an issue. Read disk once and close.

with open('file', ...) as f:
  src = f.readlines()

[abalkin]

On Fri, Nov 12, 2010 at 12:54 PM, Terry J. Reedy \report@bugs.python.org\ wrote: ..

>What is the best way to pass around source code? > - file-like objects, line iterators, readline-like function?

Line iterator (list of lines) as returned by open().readlines. Memory should not be an issue. Read disk once and close.

with open('file', ...) as f:  src = f.readlines()

Line iterator does not carry the information about source encoding which may be important for annotating the source code.

[terryjreedy]

Line iterator does not carry the information about source encoding which may be important for annotating the source code.

I would pass around both encoding and lines, possibly as a tuple.

A person heavily into OO might define a _Source class and turn the private functions into methods. But that would not play well with the deprecation process until it is done with.

[abalkin]

I updated the patch for 3.5.

[Brett]

I don't quite see the point of the get_source call as it isn't returned or used.

It *is* used: it is passed to _find_strings_stream:

+ source = loader.get_source(modulename) + strs = _find_strings_stream(io.StringIO(source))

[5df057ac-c83d-447e-8c45-910556b17608]

Hi, I left a couple of comments on Rietveld.