Unable to get coverage when subprocess uses a different Python version

[taranlu-houzz]

Describe the bug Hello, I have a fairly unique problem: I have a new tool package that I am writing in Python 3. Within this package, there are some stand-alone scripts written in Python 2 (they use the mayapy utility script to invoke the Python interpreter in a valid environment for Autodesk Maya to run headless). In order to get coverage for these scripts, I have tried to follow the information on https://coverage.readthedocs.io/en/coverage-5.5/subprocess.html, but it does not seem to work for me. In order to get the sitecustomize.py to load in the subprocess, I pass a modified environment to the subprocess command that looks like this:

MAYAPY_ENV_HACKY = {
    **os.environ,
    "COVERAGE_PROCESS_START": str(pyproject_file_path()),
    "PYTHONPATH": os.pathsep.join([str(pyproject_file_path().parent)] + site.getsitepackages()),
}

I am doing this for the following reasons:

• The virtualenv for the project is Python 3, thus the Python 2 subprocess has no idea about the coverage package that has been installed into the virtualenv.
• I have the sitecustomize.py at the root of my project structure (thus, the need for pyproject_file_path().parent).

I have coverage configured via the pyproject.toml:

[tool.coverage.paths]
source = ["src", "*/site-packages"]

[tool.coverage.run]
branch = true
parallel = true
source = ["----"]

[tool.coverage.report]
show_missing = true
fail_under = 100

And this is my sitecustomize.py:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

print(__name__, "A")
try:
    print(__name__, "B")
    import coverage

    coverage.process_startup()

    print("STARTED COVERAGE")
except ImportError:
    print(__name__, "C")
    pass

I have tested running the subprocess with just invoking the interpreter via mayapy, and it works and gives the expected output from my sitecustomize.py. However, when I try to run the actual scripts and then combine them, they always show 0% coverage for the files.

Other information about my setup:

• The package that I am writing is a namespace package.
• I am using ward to run tests: https://github.com/darrenburns/ward.
• The first stand-alone script runs the other via a subprocess, but I would think this should be handled by the modified environment.
• I am on macOS 10.14.6

To Reproduce How can we reproduce the problem? Please be specific. Don't just link to a failing CI job. Answer the questions below:

• What version of Python are you using? 3.8.8
• What version of coverage.py are you using? 5.5
• What versions of what packages do you have installed?

  The currently activated Python version 3.9.2 is not supported by the project (>=3.7,<3.9).
  Trying to find and use a compatible version. 
  Using python3.8 (3.8.8)
  alabaster                             0.7.12         A configurable sidebar-enabled Sphinx theme
  appdirs                               1.4.4          A small Python module for determining appropriate platform-specific dirs, e.g. a "user data dir".
  astor                                 0.8.1          Read/rewrite/write Python ASTs
  attrs                                 21.2.0         Classes Without Boilerplate
  babel                                 2.9.1          Internationalization utilities
  bandit                                1.7.0          Security oriented static analyser for python code.
  beautifulsoup4                        4.9.3          Screen-scraping library
  better-exceptions                     0.3.3          Pretty and helpful exceptions, automatically
  black                                 20.8b1         The uncompromising code formatter.
  boto3                                 1.17.101       The AWS SDK for Python
  botocore                              1.20.101       Low-level, data-driven core of boto 3.
  certifi                               2021.5.30      Python package for providing Mozilla's CA Bundle.
  chardet                               4.0.0          Universal encoding detector for Python 2 and 3
  click                                 8.0.1          Composable command line interface toolkit
  click-completion                      0.5.2          Fish, Bash, Zsh and PowerShell completion for Click
  click-default-group                   1.2.2          Extends click.Group to invoke a command without explicit subcommand name
  colorama                              0.4.4          Cross-platform colored terminal text.
  commonmark                            0.9.1          Python parser for the CommonMark Markdown spec
  coverage                              5.5            Code coverage measurement for Python
  cucumber-tag-expressions              2.0.4          Provides tag-expression parser for cucumber/behave
  darglint                              1.8.0          A utility for ensuring Google-style docstrings stay up to date with the source code.
  doc2dash                              2.3.0          Convert docs to Dash.app
  docutils                              0.17.1         Docutils -- Python Documentation Utilities
  dparse                                0.5.1          A parser for Python dependency files
  fancycompleter                        0.9.1          colorful TAB completion for Python prompt
  flake8                                3.9.2          the modular source code checker: pep8 pyflakes and co
  flake8-annotations                    2.6.2          Flake8 Type Annotation Checks
  flake8-bandit                         2.1.2          Automated security testing with bandit and flake8.
  flake8-black                          0.2.1          flake8 plugin to call black as a code style validator
  flake8-bugbear                        21.4.3         A plugin for flake8 finding likely bugs and design problems in your program. Contains warnings that don't belong in pyflakes and pyc...
  flake8-docstrings                     1.6.0          Extension for flake8 which uses pydocstyle to check docstrings
  flake8-polyfill                       1.0.2          Polyfill package for Flake8 plugins
  flynt                                 0.61           CLI tool to convert a python project's %-formatted strings to f-strings.
  gitdb                                 4.0.7          Git Object Database
  gitpython                             3.1.18         Python Git Library
  ----                                  2.2.0          Standardized logging setup.
  ----                                  1.0.1          Import utility lib.
  ----                                  1.0.2          Subprocess utility lib.
  idna                                  2.10           Internationalized Domain Names in Applications (IDNA)
  imagesize                             1.2.0          Getting image size from png/jpeg/jpeg2000/gif file
  jinja2                                3.0.1          A very fast and expressive template engine.
  jmespath                              0.10.0         JSON Matching Expressions
  loguru                                0.5.3          Python logging made (stupidly) simple
  markupsafe                            2.0.1          Safely add untrusted strings to HTML/XML markup.
  mccabe                                0.6.1          McCabe checker, plugin for flake8
  mypy-extensions                       0.4.3          Experimental type system extensions for programs checked with the mypy typechecker.
  p4python                              2020.1.2056111 P4Python - Python interface to Perforce API
  packaging                             20.9           Core utilities for Python packages
  pathspec                              0.8.1          Utility library for gitignore style pattern matching of file paths.
  pbr                                   5.6.0          Python Build Reasonableness
  pdbpp                                 0.10.2         pdb++, a drop-in replacement for pdb
  pluggy                                0.13.1         plugin and hook calling mechanisms for python
  pprintpp                              0.4.0          A drop-in replacement for pprint that's actually pretty
  psutil                                5.8.0          Cross-platform lib for process and system monitoring in Python.
  pycodestyle                           2.7.0          Python style guide checker
  pydocstyle                            6.1.1          Python docstring style checker
  pyflakes                              2.3.1          passive checker of Python programs
  pygments                              2.9.0          Pygments is a syntax highlighting package written in Python.
  pyparsing                             2.4.7          Python parsing module
  pyrepl                                0.9.0          A library for building flexible command line interfaces
  python-dateutil                       2.8.1          Extensions to the standard Python datetime module
  pytz                                  2021.1         World timezone definitions, modern and historical
  pyyaml                                5.4.1          YAML parser and emitter for Python
  regex                                 2021.4.4       Alternative regular expression module, to replace re.
  requests                              2.25.1         Python HTTP for Humans.
  rich                                  10.4.0         Render rich text, tables, progress bars, syntax highlighting, markdown and more to the terminal
  s3transfer                            0.4.2          An Amazon S3 Transfer Manager
  safety                                1.10.3         Checks installed dependencies for known vulnerabilities.
  shellingham                           1.4.0          Tool to Detect Surrounding Shell
  six                                   1.16.0         Python 2 and 3 compatibility utilities
  smmap                                 4.0.0          A pure Python implementation of a sliding window memory map manager
  snowballstemmer                       2.1.0          This package provides 29 stemmers for 28 languages generated from Snowball algorithms.
  soupsieve                             2.2.1          A modern CSS selector implementation for Beautiful Soup.
  sphinx                                3.5.2          Python documentation generator
  sphinx-rtd-theme                      0.5.1          Read the Docs theme for Sphinx
  sphinxcontrib-applehelp               1.0.2          sphinxcontrib-applehelp is a sphinx extension which outputs Apple help books
  sphinxcontrib-devhelp                 1.0.2          sphinxcontrib-devhelp is a sphinx extension which outputs Devhelp document.
  sphinxcontrib-htmlhelp                2.0.0          sphinxcontrib-htmlhelp is a sphinx extension which renders HTML help files
  sphinxcontrib-jsmath                  1.0.1          A sphinx extension which renders display math in HTML via JavaScript
  sphinxcontrib-qthelp                  1.0.3          sphinxcontrib-qthelp is a sphinx extension which outputs QtHelp document.
  sphinxcontrib-serializinghtml         1.1.5          sphinxcontrib-serializinghtml is a sphinx extension which outputs "serialized" HTML files (json and pickle).
  stevedore                             3.3.0          Manage dynamic plugins for Python applications
  toml                                  0.10.2         Python Library for Tom's Obvious, Minimal Language
  tomli                                 1.0.2          A lil' TOML parser
  typed-ast                             1.4.3          a fork of Python 2 and 3 ast modules with type comment support
  typing-extensions                     3.10.0.0       Backported and Experimental Type Hints for Python 3.5+
  urllib3                               1.26.6         HTTP library with thread-safe connection pooling, file post, and more.
  ward                                  0.61.1b0       A modern Python testing framework
  wmctrl                                0.4            A tool to programmatically control windows inside X
  wrapt                                 1.12.1         Module for decorators, wrappers and monkey patching.
  zope.interface                        5.4.0          Interfaces for Python

Expected behavior I would expect the coverage to be logged for the code run via the subprocesses.

[taranlu-houzz]

I think that this issue seems like it might be related: #655

[taranlu-houzz]

So, I found the solution to my issue: Adding the path to the subdir in the package that contains the stand-alone scripts to the source list under tool.coverage.run in my pyproject.toml got things to work.

This was not immediately obvious as an issue, since coverage was detecting the files, it just seems that it was not collecting data on them. Initially, in order to get coverage to notice the subdir, I added __init__.py as needed along the path (Python 3 should not need them). After adding the path to the subdir to my pyproject.toml I was able to delete the extraneous __init__.py files and coverage still reported on everything as expected.