microsoft / vscode-python

Python extension for Visual Studio Code
https://aka.ms/pvsc-marketplace
MIT License
4.34k stars 1.2k forks source link

Test coverage wrong when importing large library #24261

Open MattTheCuber opened 1 month ago

MattTheCuber commented 1 month ago

Type: Bug

  1. Create a simple project that imports MONAI with a test
  2. Open the Testing tab and run tests with coverage
  3. Module-level lines (e.g. imports, class definitions, and function definitions) will show as missed in the coverage report

image

Extension version: 2024.16.0 VS Code version: Code 1.94.0 (d78a74bcdfad14d5d3b1b782f87255d802b57511, 2024-10-02T13:08:12.626Z) OS version: Windows_NT x64 10.0.19045 Modes: Remote OS version: Linux x64 4.18.0-348.23.1.el8_5.x86_64 Remote OS version: Linux x64 4.18.0-348.23.1.el8_5.x86_64 Remote OS version: Linux x64 4.18.0-348.23.1.el8_5.x86_64 Remote OS version: Linux x64 4.18.0-348.23.1.el8_5.x86_64

System Info |Item|Value| |---|---| |CPUs|AMD Ryzen 7 5825U with Radeon Graphics (16 x 1996)| |GPU Status|2d_canvas: enabled
canvas_oop_rasterization: enabled_on
direct_rendering_display_compositor: disabled_off_ok
gpu_compositing: enabled
multiple_raster_threads: enabled_on
opengl: enabled_on
rasterization: enabled
raw_draw: disabled_off_ok
skia_graphite: disabled_off
video_decode: enabled
video_encode: enabled
vulkan: disabled_off
webgl: enabled
webgl2: enabled
webgpu: enabled
webnn: disabled_off| |Load (avg)|undefined| |Memory (System)|38.83GB (21.36GB free)| |Process Argv|--file-uri file:///c%3A/Users/mvine/Documents/Programming/goliath-fragct.code-workspace| |Screen Reader|no| |VM|0%| |Item|Value| |---|---| |Remote|SSH: 10.10.10.137| |OS|Linux x64 4.18.0-348.23.1.el8_5.x86_64| |CPUs|AMD Ryzen 9 5950X 16-Core Processor (32 x 3650)| |Memory (System)|125.82GB (89.95GB free)| |VM|0%| |Item|Value| |---|---| |Remote|SSH: 10.10.10.137| |OS|Linux x64 4.18.0-348.23.1.el8_5.x86_64| |CPUs|AMD Ryzen 9 5950X 16-Core Processor (32 x 3650)| |Memory (System)|125.82GB (89.95GB free)| |VM|0%| |Item|Value| |---|---| |Remote|SSH: 10.10.10.137| |OS|Linux x64 4.18.0-348.23.1.el8_5.x86_64| |CPUs|AMD Ryzen 9 5950X 16-Core Processor (32 x 3650)| |Memory (System)|125.82GB (89.95GB free)| |VM|0%| |Item|Value| |---|---| |Remote|SSH: 10.10.10.137| |OS|Linux x64 4.18.0-348.23.1.el8_5.x86_64| |CPUs|AMD Ryzen 9 5950X 16-Core Processor (32 x 3589)| |Memory (System)|125.82GB (89.95GB free)| |VM|0%|
eleanorjboyd commented 1 month ago

thank you for the bug report! Could you include what is shown if you run coverage from the terminal as comparison?

MattTheCuber commented 1 month ago

thank you for the bug report! Could you include what is shown if you run coverage from the terminal as comparison?

Sure thing:

---------- coverage: platform linux, python 3.11.6-final-0 -----------
Name                     Stmts   Miss  Cover   Missing
------------------------------------------------------
app/__init__.py              0      0   100%
app/main.py                  1      0   100%
app/tests/__init__.py        0      0   100%
app/tests/test_main.py       3      0   100%
------------------------------------------------------
TOTAL                        4      0   100%
MattTheCuber commented 1 month ago

@eleanorjboyd any update on this? Anything more I can provide to help debug?

eleanorjboyd commented 1 month ago

What is your root dir when you run it from the terminal? You can see what one is selected for running tests in vscode and so im trying to figure out what setting could be different between what you are putting in the terminal and the extension might be passing to run coverage in the UI. Image

MattTheCuber commented 1 month ago

My root directory is test_coverage.

Image

I also ran the full command exactly as it shows in the terminal excluding the -p argument: pytest --cov=. --cov-report term-missing --rootdir=/home/mvine/Documents/test_coverage /home/mvine/Documents/test_coverage/app/tests/test_main.py::test and got the same results. My suspicions are that either the plugin is doing something or the --cov=. is not working because it's using a relative path. However, I tested running the previous command from /home/mvine/Documents/test_coverage/app/tests, /home/mvine/Documents/test_coverage/app, /home/mvine/Documents/test_coverage, and /home/mvine/Documents without replicating the problem.

eleanorjboyd commented 1 month ago

I wondering if it has to do with the plugins you have enabled because I am now seeing the error "unable to read testIds from temp fileNo...". Also I see you added --cov-report term-missing to your run in terminal, are you expecting this to be run in the extension because I don't see that in the logs.

MattTheCuber commented 1 month ago

I will create a new virtual environment trying to replicate VS Code's as closely as possible. But the test coverage is correct in the terminal, VS Code's is wrong. Do you know how I can get the vscode pytest plugin?

I think I can remove the term missing stuff from the command as well. That's just the command I normally run.

MattTheCuber commented 1 month ago

I created a new virtual environment, ran pip install monai pytest pytest-cov and the integrated test reported 100% coverage. I then ran pip install faker==22.2.0 pytest-timeout==2.3.1 anyio==4.4.0 pytest-cov==5.0.0 dash==2.16.1 pytest-xdist==3.5.0 pytest-drop-dup-tests==1.0.0 pytest-mock==3.14.0 typeguard==4.3.0 to install the identical plugins and it still reported 100% coverage. I am now going through every library in my massive virtual environment to find which is causing the problem 😩

MattTheCuber commented 1 month ago

It's opencv-python!

  1. pip install opencv-python pytest pytest-cov
  2. Have a file with the line import cv2 (change monai to cv2)
  3. Test coverage breaks
pip freeze ``` coverage==7.6.4 iniconfig==2.0.0 numpy==2.1.2 opencv-python==4.10.0.84 packaging==24.1 pluggy==1.5.0 pytest==8.3.3 pytest-cov==6.0.0 ```

This now throws a couple warnings.

Image

Pretty sure it can be ignored because it still works properly in the terminal.

Image

MattTheCuber commented 1 month ago

Yeah, the warnings are fine: https://github.com/nedbat/coveragepy/issues/1653

MattTheCuber commented 1 month ago

Aha, problem solved! Adding the config file as described in this comment resolves the problem: https://github.com/nedbat/coveragepy/issues/1653#issuecomment-1653730387! (I opened an issue on the opencv-python repo as well)

This is probably still a problem though because my terminal doesn't match the integrated test coverage.

eleanorjboyd commented 3 weeks ago

Hi @MattTheCuber, thanks for all your investigation help! I have just put in the a fix for the NoSource issue. Are you seeing the same issues with your terminal not matching like above or does it look different now? If it looks different and you could send an updated screenshot that would be great. One note is you might see rounding discrepancies in the coverage numbers between terminal and the sidebar so just an fyi there.

MattTheCuber commented 3 weeks ago

Which version of the extension should I test?