search

approvals / ApprovalTests.Python

ApprovalTests for python
Apache License 2.0
150 stars 52 forks source link

Approvaltests not able to launch meld when running with bazel test #93

Open sonulohani opened 3 years ago

sonulohani commented 3 years ago

Hi,

I have some approval tests which when I run using python3 command I am able to see the difference between the approved file and received file in meld. However when I am trying to do the same by running bazel test :my_approval_test I am not able to see the difference between received and approved file if the test fails as the process is getting killed before even started. I have tried using --spawn_strategy=local also however that also doesnt work.

sonulohani commented 3 years ago

I could use subprocess.run instead of subprocess.Popen, however the method run_command is static in GenericDiffReporter and I am not able to modify the behaviour.

obestwalter commented 3 years ago

Did you have the chance to look at this a bit more? If run works but Popen doesn't in your context (I don't know bazel and don't know what it does), it would be a good idea to first check why this happens. The only difference I am aware of between run and Popen is that run is blocking. But the non blocking launch does work in other context and that is also the idea. So what does bazel do differently?

sonulohani commented 3 years ago

Did you have the chance to look at this a bit more? If run works but Popen doesn't in your context (I don't know bazel and don't know what it does), it would be a good idea to first check why this happens. The only difference I am aware of between run and Popen is that run is blocking. But the non blocking launch does work in other context and that is also the idea. So what does bazel do differently?

The problem is that the bazel runs all its tasks in its own sandbox so whenever bazel is done with running the test cases it closes its server and due to that whatever the processes running inside bazel also get killed. The run works because it blocks the bazel workflow by stopping/blocking the execution context until the diffviewer gets closed or killed. I temporarily fixed it by making custom reporter class derived from GenericDiffReporter and overriding the report method and using run instead of Popen. It kind of works because the entire purpose is to see the diff of the failed test cases.

obestwalter commented 3 years ago

Interesting. So that means though that there definitely is a case to be made for setting the spawning of the differ to be blocking, maybe as a command line argument. I am quite new to the tool as a user but also have a vague feeling that I would prefer the blocking workflow even when running approvals normally. Opened #95 for that.

sonulohani commented 3 years ago

Hi @obestwalter You're right. Right now I am controlling the behaviour using "use_custom_reporter" as command line parameter because I want to run the same test in ci server as well and if I get test failure then I wouldn't need diff viewer to be opened in the server.