nose coverage breaks with "compile() expected string without null bytes"

[p00j4]

it used to work and suddenly started breaking, didn't change any package version

  notestes 1.3.7
  python :Python 2.7.6

using --with-coverage it fails right after the Total calculation line and doesn't generate index.html file and also if used --with-xcover, doesn't generate the coverage.xml file

i'm running like:

nosetests -vv testcases_path --with-id  --with-xunit --exclude-dir=blah --cover-package=pkg1,pkg2 --cover-erase --cover-inclusive --with-coverage -s

Stacktrace

------------------------------------------------------------------------------------------------
TOTAL                                                              50869  23891    53%   

14:29:27 Traceback (most recent call last):
14:29:27   File "/usr/local/bin/nosetests", line 11, in <module>
14:29:27     sys.exit(run_exit())
14:29:27   File "/usr/local/lib/python2.7/dist-packages/nose/core.py", line 121, in __init__
14:29:27     **extra_args)
14:29:27   File "/usr/lib/python2.7/unittest/main.py", line 95, in __init__
14:29:27     self.runTests()
14:29:27   File "/usr/local/lib/python2.7/dist-packages/nose/core.py", line 207, in runTests
14:29:27     result = self.testRunner.run(self.test)
14:29:27   File "/usr/local/lib/python2.7/dist-packages/nose/core.py", line 66, in run
14:29:27     result.printErrors()
14:29:27   File "/usr/local/lib/python2.7/dist-packages/rednose.py", line 430, in printErrors
14:29:27     self.config.plugins.report(self.stream)
14:29:27   File "/usr/local/lib/python2.7/dist-packages/nose/plugins/manager.py", line 99, in __call__
14:29:27     return self.call(*arg, **kw)
14:29:27   File "/usr/local/lib/python2.7/dist-packages/nose/plugins/manager.py", line 167, in simple
14:29:27     result = meth(*arg, **kw)
14:29:27   File "/usr/local/lib/python2.7/dist-packages/nosexcover/nosexcover.py", line 69, in report
14:29:27     super(XCoverage, self).report(stream)
14:29:27   File "/usr/local/lib/python2.7/dist-packages/nose/plugins/cover.py", line 196, in report
14:29:27     self.coverInstance.html_report(modules, self.coverHtmlDir)
14:29:27   File "/usr/local/lib/python2.7/dist-packages/coverage/control.py", line 951, in html_report
14:29:27     return reporter.report(morfs)
14:29:27   File "/usr/local/lib/python2.7/dist-packages/coverage/html.py", line 120, in report
14:29:27     self.report_files(self.html_file, morfs, self.config.html_dir)
14:29:27   File "/usr/local/lib/python2.7/dist-packages/coverage/report.py", line 80, in report_files
14:29:27     report_fn(cu, self.coverage._analyze(cu))
14:29:27   File "/usr/local/lib/python2.7/dist-packages/coverage/control.py", line 831, in _analyze
14:29:27     return Analysis(self, it)
14:29:27   File "/usr/local/lib/python2.7/dist-packages/coverage/results.py", line 16, in __init__
14:29:27     self.statements = self.file_reporter.statements()
14:29:27   File "/usr/local/lib/python2.7/dist-packages/coverage/python.py", line 113, in statements
14:29:27     self._statements, self._excluded = self.parser.parse_source()
14:29:27   File "/usr/local/lib/python2.7/dist-packages/coverage/parser.py", line 224, in parse_source
14:29:27     self._raw_parse()
14:29:27   File "/usr/local/lib/python2.7/dist-packages/coverage/parser.py", line 185, in _raw_parse
14:29:27     self.statement_starts.update(self.byte_parser._find_statements())
14:29:27   File "/usr/local/lib/python2.7/dist-packages/coverage/parser.py", line 93, in byte_parser
14:29:27     self._byte_parser = ByteParser(self.text, filename=self.filename)
14:29:27   File "/usr/local/lib/python2.7/dist-packages/coverage/parser.py", line 351, in __init__
14:29:27     self.code = compile(text, filename, "exec")
14:29:27 TypeError: compile() expected string without null bytes

[p00j4]

ah, looks like, a unicode char in filename is breaking it. though in python 2.7.11, in parser.py file compile method is replaced with compile_unicode method which makes it work with 2.7.11 but due to production requirement, I can't update python version. is there any way i can work around from nose plugin when the call returns to nose ?

[p00j4]

1. update of coverage package alone (which is safe too) fixes this
2. there was a syntactically wrong uni-code is a main python code file, had to remove it to make it work. (strangely the syntax error is ignored by python pyramid compiler and never being found in testing phase.)

[acefei]

I try to update the latest coverage version (4.4.1) which using compile_unicode method, this problem is still exist.

error 15-Jul-2017 19:29:29 Traceback (most recent call last): error 15-Jul-2017 19:29:29 File "/opt/ActivePython-2.7/bin/coverage", line 11, in error 15-Jul-2017 19:29:29 sys.exit(main()) error 15-Jul-2017 19:29:29 File "/opt/ActivePython-2.7/lib/python2.7/site-packages/coverage/cmdline.py", line 756, in main error 15-Jul-2017 19:29:29 status = CoverageScript().command_line(argv) error 15-Jul-2017 19:29:29 File "/opt/ActivePython-2.7/lib/python2.7/site-packages/coverage/cmdline.py", line 517, in command_line error 15-Jul-2017 19:29:29 total = self.coverage.xml_report(outfile=outfile, **report_args) error 15-Jul-2017 19:29:29 File "/opt/ActivePython-2.7/lib/python2.7/site-packages/coverage/control.py", line 1070, in xml_report error 15-Jul-2017 19:29:29 return reporter.report(morfs, outfile=outfile) error 15-Jul-2017 19:29:29 File "/opt/ActivePython-2.7/lib/python2.7/site-packages/coverage/xmlreport.py", line 76, in report error 15-Jul-2017 19:29:29 self.report_files(self.xml_file, morfs) error 15-Jul-2017 19:29:29 File "/opt/ActivePython-2.7/lib/python2.7/site-packages/coverage/report.py", line 91, in report_files error 15-Jul-2017 19:29:29 report_fn(fr, self.coverage._analyze(fr)) error 15-Jul-2017 19:29:29 File "/opt/ActivePython-2.7/lib/python2.7/site-packages/coverage/control.py", line 899, in _analyze error 15-Jul-2017 19:29:29 return Analysis(self.data, it) error 15-Jul-2017 19:29:29 File "/opt/ActivePython-2.7/lib/python2.7/site-packages/coverage/results.py", line 19, in init error 15-Jul-2017 19:29:29 self.statements = self.file_reporter.lines() error 15-Jul-2017 19:29:29 File "/opt/ActivePython-2.7/lib/python2.7/site-packages/coverage/python.py", line 176, in lines error 15-Jul-2017 19:29:29 return self.parser.statements error 15-Jul-2017 19:29:29 File "/opt/ActivePython-2.7/lib/python2.7/site-packages/coverage/python.py", line 171, in parser error 15-Jul-2017 19:29:29 self._parser.parse_source() error 15-Jul-2017 19:29:29 File "/opt/ActivePython-2.7/lib/python2.7/site-packages/coverage/parser.py", line 238, in parse_source error 15-Jul-2017 19:29:29 self._raw_parse() error 15-Jul-2017 19:29:29 File "/opt/ActivePython-2.7/lib/python2.7/site-packages/coverage/parser.py", line 207, in _raw_parse error 15-Jul-2017 19:29:29 self.raw_statements.update(self.byte_parser._find_statements()) error 15-Jul-2017 19:29:29 File "/opt/ActivePython-2.7/lib/python2.7/site-packages/coverage/parser.py", line 96, in byte_parser error 15-Jul-2017 19:29:29 self._byte_parser = ByteParser(self.text, filename=self.filename) error 15-Jul-2017 19:29:29 File "/opt/ActivePython-2.7/lib/python2.7/site-packages/coverage/parser.py", line 364, in init error 15-Jul-2017 19:29:29 self.code = compile_unicode(text, filename, "exec") error 15-Jul-2017 19:29:29 File "/opt/ActivePython-2.7/lib/python2.7/site-packages/coverage/phystokens.py", line 286, in compile_unicode error 15-Jul-2017 19:29:29 code = compile(source, filename, mode) error 15-Jul-2017 19:29:29 TypeError: compile() expected string without null bytes

I add print filename in site-packages/coverage/phystokens.py line 285 to indicate the filename before exception occurred. Then I use a script with the filename to identify null byte.

import sys
# input filename found by site-packages/coverage/phystokens.py
filename = sys.argv[1]
with open(filename, 'r') as f:
   content = f.readlines()

for i,l in enumerate(content):
    if b'\x00' in l:
        print i,l

Finally, remove null bytes, nose coverage works fine.

BTW, I found the null bytes were in the doc string, it most likely caused compile_unicode method invalid.