search

numaru / vscode-ceedling-test-adapter

Ceedling Test Adapter for the VS Code Test Explorer
MIT License
37 stars 14 forks source link

When a test fails, all subsequent tests will fail #41

Closed joel-felcana closed 4 years ago

joel-felcana commented 4 years ago

Hi,

I'm facing an issue. The test results returned by the plugin are not correct. I don't know if it's related, but I'm not getting any output in the "Output" window. I'm on latest windows 10, with latest VS Code (1.40.1) and latests plugin versions (1.4.1 and 2.14.6). 

$ ceedling version
  Ceedling:: 0.28.3
CException:: 1.3.1.18
     CMock:: 2.4.6.217
     Unity:: 2.4.3.122

$ ruby --version
ruby 2.4.4p296 (2018-03-28 revision 63013) [x64-mingw32]

$ gcc --version
gcc.exe (Rev2, Built by MSYS2 project) 9.2.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

1) Some tests are always returning error, even when they pass when run manually (in my project test_HW_test_flash passes if run on the command line, but not if run via the panel) 2) Some tests pass, so it's not that it always return an error (in my project, test_LED always passes) 3) Running all the tests will make all the tests after the first error appear as error (test_LED will fail if run via the "run all" button). 4) I'm not getting output in the Output windon. In a multifolder environment, I get the "failed to find project.yml" message on those folders that have no tests, but I don't get any output when running the tests.

This screenshot shows how test_HW_test_flash passes in the command line, but appears as failed when run via VS Code. Also, it shows how the output console shows an error message on a folder without a ceedling project (As expected) but that there is no other output, even both failed and passed tests haven been run.

image

joel-felcana commented 4 years ago

I did a bit more research. I disabled all the plugins. The behaviour is very erratic. If I reopen VS code and run another passing test (test_docking) and it shows as passing, and then I run test_LED (which you can see in the previous screenshot that passes) and returns error

image

I thought "maybe only the first test run returns its real result, and then the next ones will return always error", but it's not the case. I reopened VS Code. Running "test_HW_test_flash" (which should pass) fails, and then running "test_LED" (which should pass) also fails, but then running "test_docking" (wich shold pass) passes

image

numaru commented 4 years ago

Thank you for this detailed report. I'm trying to reproduce on a simple project with no success. Did it already worked correctly before? If you have another ceedling projet on your computer, does this happen also on this project?

joel-felcana commented 4 years ago

Hey,

Just tested with a completely new project and it works beautifully (both as a single folder project and as a multiroot workspace, both work). I tried to transplant my project.yml to see if it was that the reason it didn't work but it doesn't seem to be the case... how weird...

I'll do more research to see if I find the problem, but you can close this one as it seems to be an issue with my config, probably... If I find something I'll post it here

joel-felcana commented 4 years ago

I rebuilt my project with a "ceedling new" and then transplanted my code and config and it seems to be working now. I updated from 0.28.3 to 0.29 recently, maybe something old was left there and broke it?