search

kondratyev-nv / vscode-python-test-adapter

Python Test Adapter for the VS Code Test Explorer
https://marketplace.visualstudio.com/items?itemName=LittleFoxTeam.vscode-python-test-adapter
MIT License
117 stars 27 forks source link

[testplan] Extension throws error when tested application fails to start up #289

Open M6AI opened 1 year ago

M6AI commented 1 year ago

We’re using this extension to run a testplan test, for an application that can fail to start up. In this particular scenario, the output of the extension is as follows:

2022-09-14T14:09:16.667Z CRIT testplan at 'src': Execution of the test "unit_test" failed: TypeError: Cannot read properties of undefined (reading 'map'

If the application starts up successfully, the output and test states are available through the extension as expected.

Searching for “map” in the codebase suggests that the extension may always be looking for test results – could this be the case? Could the extension detect and report a startup failure?

https://github.com/kondratyev-nv/vscode-python-test-adapter/blob/324083feb3f8f358a040ca5742a3273dfcdce92b/src/testplan/testplanJunitTestStatesParser.ts#L78

kondratyev-nv commented 1 year ago

@M6AI do you have a reproducible example? I tried throwing an exception in setup 

def setup(self, env):
    raise Exception('unexpected')

and I don't see tests being labelled as failed (which is ofc an issue), but I don't see any critical errors in logs.

Also, I think you should be able to see stacktrace below the error line, can you please share it?

M6AI commented 1 year ago

@kondratyev-nv Thanks for picking this up. To be honest, I am in sort of under some administrative limitation as I am raising this issue on behalf of a colleague due to my organization membership and the issue is surfacing on proprietary code. For now I can say that the startup failure is coming from the C++ binary that underlies the Testplan driver. So basically, if there is a C++ binary that is being run as say an App driver and purposefully crashing on C++ layer, it should replicate the issue. Will try to put together something for you.