search

gleb-sevruk / pycrunch-engine

NCrunch inspired tool for continuous testing Python
https://pycrunch.com
Other
57 stars 8 forks source link

add some CI to run pycrunch tests #28

Closed yarikoptic closed 8 months ago

yarikoptic commented 3 years ago

Decided to take a shot at #27 but first wanted to check if I have broke anything... realized that if I submit a PR nobody would even know if I break something ;)

Since hosted on github, using github actions might be the best choice.

gleb-britecore commented 3 years ago

It's a bit hard to add meaningful tests, that will be possible to run via PyCrunch itself.

Maybe I was looking at this from the wrong direction, and full integration testing should be possible without loading all pycrunch-engine modules into memory, just via pytest command.

Anyway, I used to test both plugin and engine manually on Windows and macOS before making a new release

yarikoptic commented 3 years ago

I still have no clue of "how pycrunch works" so cannot really give any sensible advice, but I would expect it should be possible to test engines "alone", so to verify that they run the desired test and produce desired/expected output. E.g. I am still not sure how pytest run is triggered to be done with coverage reporting (so I could do the same for nose).

How do you test it "manually"? is it possible without bothering intellij/pycharm at all?

gleb-britecore commented 3 years ago

I made a quick and dirty prototype branch for nose and can run them with success. Hopefully, it will help you to finish your PR: https://github.com/gleb-sevruk/pycrunch-engine/pull/30

So coverage collection is done independently of pytest. Coverage is just line hits collected during arbitrary code execution, and started here: https://github.com/gleb-sevruk/pycrunch-engine/blob/master/pycrunch/child_runtime/test_runner.py#L41

As for testing, there is no easy answer, for example, I want to test pytest.main function using pytest. , for this, I need to recursively invoke pytest.

Pycrunch has 2 modules: [engine] -> [child_executor]

Child executor is run for one or more tests. task_id is assigned to every child executor. It fetches tests required to run via TCP, and then signals results over TCP back to the engine. There can be multiple child executor instances. For example, when you run 20 tests, by default they are split between 4 executors (on an 8-core CPU).

In Short, the link between engine and child processes is the most untestable piece of the system now.

As for manual testing, I usually use PyCharm or WebUI. Its URL is printed during engine startup like:

PyCrunch port will be 52879
PyCrunch Web-UI at http://0.0.0.0:52879/ui/

Though I do not recommend using WebUI, it is still an option.

yarikoptic commented 3 years ago

There is no some kind of a token minted and any user on a shared system could connect? (also with 0.0.0.0 it would listen even on public interface -- not safe)

yarikoptic commented 3 years ago

Thank you for #30 and thanks for all the details above! I only wish you have based https://github.com/gleb-sevruk/pycrunch-engine/pull/30 on top of #29 (you are most welcome even to push directly into my branch for that PR). Otherwise -- I will try to "merge" (to not loose any bit of history/attribution ;)) later today and give it a try.

gleb-sevruk commented 8 months ago

basic version is done. I will be working on more sophisticated integration test suite in meantime