search

pytest-dev / pytest-bdd

BDD library for the pytest runner
https://pytest-bdd.readthedocs.io/en/latest/
MIT License
1.31k stars 221 forks source link

Output should match the output of behave and lettuce #139

Closed thedrow closed 4 days ago

thedrow commented 9 years ago

Currently the output does not match BDD style test runners output. This is the only reason I chose why not to use this plugin.

py.test -vvvv
================================================================================================ test session starts ================================================================================================
platform linux2 -- Python 2.7.10 -- py-1.4.30 -- pytest-2.7.2 -- /home/omer/.virtualenvs/playground/bin/python2.7
rootdir: /home/omer/bddtest, inifile: 
plugins: bdd
collected 2 items 

test_outline.py::test_outlined[12-5.0-7] PASSED
test_outline.py::test_outlined[2-1.0-1] PASSED

============================================================================================= 2 passed in 0.01 seconds ==============================================================================================

This is taken from one your examples provided in the README. I'd expect the plugin to report the steps themselves and their status.

olegpidsadnyi commented 9 years ago

This is the standard output of the PyTest, however it supports pluggable reporting. We've implemented cucumber json format (BDD standard) that we use to integrate to Jenkins (See https://github.com/pytest-dev/pytest-bdd#reporting). What exactly are you using lettuce/behave console output for? Maybe it can be solved with the cucumber json format? You could also help writing lettuce-style reporting plugin if you like :)

thedrow commented 9 years ago

I'm using it to show our product team that the features they specify work as intended. The cucumber json format doesn't help us because we're using Travis and not Jenkins. I don't think I have the time to contribute a new reporting plugin right now.

olegpidsadnyi commented 9 years ago

I'll think about it. It shouldn't be too hard, because there are hooks that allow to log feature/scenario step execution and just dump it in the end of the pytest log. Do you have a link to some online example where the format is good for you?

thedrow commented 9 years ago

Just run either lettuce or behave with their examples. Their output mostly match.

bubenkoff commented 9 years ago

so please be kind to specify the output, as you're initiating a change

spinus commented 9 years ago

@bubenkoff I'm working on this. I already have something working but not yet all features covered. State for now selection_002

Do you prefer it as PR or I should create separate plugin?

btw, I plan to do the same for HTML report. PS. @thedrow I think the title of this issue could be like feature request rather then complaining about that others have this :-)

bubenkoff commented 9 years ago

@spinus that looks very promising! it certainly worth to be directly in pytest-bdd as we have to be user-friendly, as we see :) thanks for working on this, looking forward for PR!

thedrow commented 9 years ago

@spinus I did intend to ask for a new feature. The way to explain it is by showing other project's output. I apologize if I came off ranting.

Can you check what happens if pytest-sugar is installed with the output changes you've made?

spinus commented 9 years ago

@thedrow what sugar does (and my plugin as well, because I kind of fork it) is unregistering default terminal reporter and setting new one. So I assume this is race condition or some kind of "sorting" priority.

Shortly, nothing good probably.

olegpidsadnyi commented 9 years ago

The entire report of the execution is accumulated. You could dump it all together at certain point if it helps.

spinus commented 9 years ago

@thedrow - internal error appears when you have both (currently) as I'm not using some variables which are required by sugar. So it looks like my Terminal Reporter is used but then sugar hooks are working anyway, so, not good stuff is happening :-)

thedrow commented 9 years ago

@spinus We can either disallow it or try to fix it to work with the sugar plugin.

spinus commented 9 years ago

@thedrow how you want to do that? Two different "applications" writing on screen, each of them has its own picture of how the screen looks like at the moment.

spinus commented 9 years ago

I'm going to write tests for this feature but I'm not sure what's the proper way, using testdir should be ok to get terminal output or you suggest to use something different?

bubenkoff commented 9 years ago

yep testdir.runpytest, as it is for generation tests, for example

thedrow commented 9 years ago

@spinus Disallowing is easier. You can just check config.sugar and if it's installed stop the execution and report that the plugins are not compatible.

spinus commented 9 years ago

@bubenkoff OK, I'll try this. @thedrow good tip, I'll do.

thedrow commented 9 years ago

Oh it might be worthwhile to check compatibility with xdist as well. Check if config.slaveinput exists and if it does, do not try to use the reporter since it won't exist.

spinus commented 9 years ago

@thedrow Yup, I have it on my todo-list as well :-) But, applying this reporter to xdist is probably another task, I'll do reporter only for non-xdist installtions for now.

thedrow commented 9 years ago

:+1:

spinus commented 9 years ago

Hi guys, sorry for delay, I'm pretty busy recently. I wrote few scenarios and started implementing them. I pushed what I've done until now to https://github.com/pytest-dev/pytest-bdd/pull/147 just to have feedback from you. Please let me know if the direction is good, maybe I'm missing something or you would like to add/change/remove scenarios. Let me know. Over next weeks I'll try to implements tests one by one.

One question, do you know how I can emulate terminal with testdir? I'm not sure but maybe tests should check if colors are used on the terminal? What do you think?

jsa34 commented 4 days ago

I think we have now reached a point where the original request has been somewhat implemented: a BDD-style output is now implemented (in Cucumber JSON format and with the gherkin terminal reporter).

I suggest individual feature requests be raised for these two outputs as necessary.

We will hopefully also be able to move to messages once this is available from Cucumber opening other possibilities