Test coverage wrong when importing large library

[MattTheCuber]

Type: Bug

1. Create a simple project that imports MONAI with a test
2. Open the Testing tab and run tests with coverage
3. Module-level lines (e.g. imports, class definitions, and function definitions) will show as missed in the coverage report

Extension version: 2024.16.0 VS Code version: Code 1.94.0 (d78a74bcdfad14d5d3b1b782f87255d802b57511, 2024-10-02T13:08:12.626Z) OS version: Windows_NT x64 10.0.19045 Modes: Remote OS version: Linux x64 4.18.0-348.23.1.el8_5.x86_64 Remote OS version: Linux x64 4.18.0-348.23.1.el8_5.x86_64 Remote OS version: Linux x64 4.18.0-348.23.1.el8_5.x86_64 Remote OS version: Linux x64 4.18.0-348.23.1.el8_5.x86_64

System Info

|Item|Value| |---|---| |CPUs|AMD Ryzen 7 5825U with Radeon Graphics (16 x 1996)| |GPU Status|2d_canvas: enabled
canvas_oop_rasterization: enabled_on
direct_rendering_display_compositor: disabled_off_ok
gpu_compositing: enabled
multiple_raster_threads: enabled_on
opengl: enabled_on
rasterization: enabled
raw_draw: disabled_off_ok
skia_graphite: disabled_off
video_decode: enabled
video_encode: enabled
vulkan: disabled_off
webgl: enabled
webgl2: enabled
webgpu: enabled
webnn: disabled_off| |Load (avg)|undefined| |Memory (System)|38.83GB (21.36GB free)| |Process Argv|--file-uri file:///c%3A/Users/mvine/Documents/Programming/goliath-fragct.code-workspace| |Screen Reader|no| |VM|0%| |Item|Value| |---|---| |Remote|SSH: 10.10.10.137| |OS|Linux x64 4.18.0-348.23.1.el8_5.x86_64| |CPUs|AMD Ryzen 9 5950X 16-Core Processor (32 x 3650)| |Memory (System)|125.82GB (89.95GB free)| |VM|0%| |Item|Value| |---|---| |Remote|SSH: 10.10.10.137| |OS|Linux x64 4.18.0-348.23.1.el8_5.x86_64| |CPUs|AMD Ryzen 9 5950X 16-Core Processor (32 x 3650)| |Memory (System)|125.82GB (89.95GB free)| |VM|0%| |Item|Value| |---|---| |Remote|SSH: 10.10.10.137| |OS|Linux x64 4.18.0-348.23.1.el8_5.x86_64| |CPUs|AMD Ryzen 9 5950X 16-Core Processor (32 x 3650)| |Memory (System)|125.82GB (89.95GB free)| |VM|0%| |Item|Value| |---|---| |Remote|SSH: 10.10.10.137| |OS|Linux x64 4.18.0-348.23.1.el8_5.x86_64| |CPUs|AMD Ryzen 9 5950X 16-Core Processor (32 x 3589)| |Memory (System)|125.82GB (89.95GB free)| |VM|0%|

[eleanorjboyd]

thank you for the bug report! Could you include what is shown if you run coverage from the terminal as comparison?

[MattTheCuber]

thank you for the bug report! Could you include what is shown if you run coverage from the terminal as comparison?

Sure thing:

---------- coverage: platform linux, python 3.11.6-final-0 -----------
Name                     Stmts   Miss  Cover   Missing
------------------------------------------------------
app/__init__.py              0      0   100%
app/main.py                  1      0   100%
app/tests/__init__.py        0      0   100%
app/tests/test_main.py       3      0   100%
------------------------------------------------------
TOTAL                        4      0   100%

[MattTheCuber]

@eleanorjboyd any update on this? Anything more I can provide to help debug?

[eleanorjboyd]

What is your root dir when you run it from the terminal? You can see what one is selected for running tests in vscode and so im trying to figure out what setting could be different between what you are putting in the terminal and the extension might be passing to run coverage in the UI.

[MattTheCuber]

My root directory is test_coverage.

I also ran the full command exactly as it shows in the terminal excluding the -p argument: pytest --cov=. --cov-report term-missing --rootdir=/home/mvine/Documents/test_coverage /home/mvine/Documents/test_coverage/app/tests/test_main.py::test and got the same results. My suspicions are that either the plugin is doing something or the --cov=. is not working because it's using a relative path. However, I tested running the previous command from /home/mvine/Documents/test_coverage/app/tests, /home/mvine/Documents/test_coverage/app, /home/mvine/Documents/test_coverage, and /home/mvine/Documents without replicating the problem.

[eleanorjboyd]

I wondering if it has to do with the plugins you have enabled because I am now seeing the error "unable to read testIds from temp fileNo...". Also I see you added --cov-report term-missing to your run in terminal, are you expecting this to be run in the extension because I don't see that in the logs.

[MattTheCuber]

I will create a new virtual environment trying to replicate VS Code's as closely as possible. But the test coverage is correct in the terminal, VS Code's is wrong. Do you know how I can get the vscode pytest plugin?

I think I can remove the term missing stuff from the command as well. That's just the command I normally run.

[MattTheCuber]

I created a new virtual environment, ran pip install monai pytest pytest-cov and the integrated test reported 100% coverage. I then ran pip install faker==22.2.0 pytest-timeout==2.3.1 anyio==4.4.0 pytest-cov==5.0.0 dash==2.16.1 pytest-xdist==3.5.0 pytest-drop-dup-tests==1.0.0 pytest-mock==3.14.0 typeguard==4.3.0 to install the identical plugins and it still reported 100% coverage. I am now going through every library in my massive virtual environment to find which is causing the problem 😩

[MattTheCuber]

It's opencv-python!

1. pip install opencv-python pytest pytest-cov
2. Have a file with the line import cv2 (change monai to cv2)
3. Test coverage breaks

pip freeze

``` coverage==7.6.4 iniconfig==2.0.0 numpy==2.1.2 opencv-python==4.10.0.84 packaging==24.1 pluggy==1.5.0 pytest==8.3.3 pytest-cov==6.0.0 ```

This now throws a couple warnings.

Pretty sure it can be ignored because it still works properly in the terminal.

[MattTheCuber]

Yeah, the warnings are fine: https://github.com/nedbat/coveragepy/issues/1653

[MattTheCuber]

Aha, problem solved! Adding the config file as described in this comment resolves the problem: https://github.com/nedbat/coveragepy/issues/1653#issuecomment-1653730387! (I opened an issue on the opencv-python repo as well)

This is probably still a problem though because my terminal doesn't match the integrated test coverage.