search

mcdcorp / opentest

Open source test automation tool for web applications, mobile apps and APIs
https://getopentest.org
MIT License
452 stars 108 forks source link

BDD Integration #24

Open jbadeau opened 6 years ago

jbadeau commented 6 years ago

Hello,

I have looked into added gherkin support to opentest but i don't think it's a great fit due to missing built in features like:

It can be done but I think the effort is to high.

I think something closer to https://github.com/getgauge/gauge is a better fit.

I have forked opentest and am now working on an a DSL which better matches opentest

some of the features will include:

Regards,

SuperAlokB commented 6 years ago

Thank you for the update - was trying to fit with cucumber but it required a lot of efforts, and it is out os cope for me :(

jbadeau commented 6 years ago

I am currently working on the DSL grammar itself:

https://github.com/jbadeau/opentest/tree/BDD-Integration/bdd

I will be looking at differenty styles to try and find a good balance between flexibility and readability

adrianth commented 6 years ago

Hey Jose, thanks for looking into this. I think it would be best to add BDD (Gherkin) support to OpenTest due to its larger adoption, but, with what I learned so far, it would seem that the Gherkin syntax doesn't support all the features we need. We could probably work around the limitations that you described in the original post, however there are others that I'm not sure how we would handle. For example, how do I describe that I want a particular step to execute on a specific test actor? I'm thinking we might be able to use comments to describe things that we need that are not supported in Gherkin out of the box. Not sure how good that's going to work - I need some time to think through the downsides. In any case, let's define the problem correctly and investigate the best ways to solve it before you spend a lot of time working on integrating a DSL into OpenTest. I would hate for all your effort to be wasted.

jbadeau commented 6 years ago

Hi,

I am not implementing anything yet. I am looking into defining a grammar style that supports openetest features while staying as close to standards like gherkin and gauge.

The idea is that any BDD style language (gherkin, gauge, custom) is transpiled to opentest yaml files with sourcemaps for debugging

I have done similar things here https://github.com/Jose-Badeau/boomslang-core/wiki/User-Documentation

I have also started to document design concerns. For instance, why should a test writer manually define where a segment is executed? Should this not be be a concern of load balancing? Why cant the open-test server or test transpiler do this?

Maybe I should move the concerns to this issue?

jbadeau commented 6 years ago

Oops, accidentally closed the issue

jbadeau commented 6 years ago

I think I now understand what the segments are for. Correct me if I'm wrong but it appears the segments main usecase is not performance, but rather for test orchestration. For example:

  1. A book order is posted via a store rest api (segment-1 - actor1)
  2. The order is inserted in the stores database and should create an order message to the book supplier
  3. The message is checked in the book suppliers system via a rest api (segment2 -actor2)

Is this correct?

If so then maybe a separate dsl for test orchestration is needed.

adrianth commented 6 years ago

You are correct @jbadeau. Test segments execute sequentially form segment 1 to last segment. No work in segment 2 is being started until segment 1 finished execution. This is useful when doing distributed testing involving test actors running on two or more different machines.

jbadeau commented 6 years ago

Ok. I only now read through the docs, it was pretty clear actually 😬.

So I am now focusing on a test grammar and an orchestration grammar. To me the they are a separate concerns. I will of course keep the yaml format.

Give me a few weeks I come up with some proposals

dmcgill50 commented 5 years ago

Any update on this feature @jbadeau or @adrianth ? This is something that we would be interested in when it becomes available.

adrianth commented 5 years ago

Hey @dmcgill50, actually I did give this some thought. Initially, it seemed like it would be a very messy exercise to try and represent a test flow as a Gerkin file (as @jbadeau also realized), since Gerkin is missing the language to describe distributed tests, as well as a few other OpenTest features. For example, there's no syntax construct that we can use to include metadata such as what actor should execute a particular step. However I think I have some ideas how to implement it in such a way so that we don't have to muddy up the feature files with useless metadata, but also keep things clean and elegant in order to maximize maintainability of both the feature files and the feature steps. I still have a few more pieces missing in this puzzle, so I need to spend a bit more time on it. Will use this issue to keep you in the loop with where I'm at, but it's not going to be ready any time soon, because it's a pretty involved feature.

dmcgill50 commented 5 years ago

@adrianth anything that I can help with this on front?

adrianth commented 5 years ago

@dmcgill50 If you have some experience with BDD already, I might need some help understanding certain concepts. Have you used BDD before? What's your level of familiarity with it?

I"m trying to figure out things like what are all the possibilities to insert a data table in a feature file. I am trying to make sure I don't leave out any use cases when implementing BDD support.

dmcgill50 commented 5 years ago

I have. I'd be happy to help test your MVP and then see where the testing takes us. https://cucumber.io/docs/gherkin/reference/ would be a starting point for me in supporting the BDD Syntax.

jbadeau commented 5 years ago

Hi, I don't think cucumber is a good option as it doesn't support includable scenarios like gauge concepts which easily act as segments.

I have added a Gauge example here. It is a very quick and dirty way to demonstrate how integration could be done:

In general a bdd approach is nice because it:

When you decide u want a generator approach then I will add an API to make working with the test structure easier but this should provide enough to get started. Things to add:

dieperei commented 4 years ago

Hello guys, any update on this feature?

Thanks in advance

dmcgill50 commented 4 years ago

It looks like the POC has been moved as that link is broken.

Any thoughts Adrian?

On Wed, Jan 22, 2020 at 11:00 AM Diego Pereira notifications@github.com wrote:

Hello guys, any update on this feature?

Thanks in advance

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mcdcorp/opentest/issues/24?email_source=notifications&email_token=AABYHQETDWO2UFGDYVZUWFLQ7B3SPA5CNFSM4FQU3ZW2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJUKRDY#issuecomment-577284239, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABYHQCEXHOD5QRW63X3PRTQ7B3SPANCNFSM4FQU3ZWQ .

-- David McGill (405) 628-5297

adrianth commented 4 years ago

I believe the work stream started by @jbadeau has been abandoned.

OpenTest will eventually support the Gherkin syntax, eventually, but there are some other updates that must happen before we can properly implement that feature. One other thing to note is that the Gherkin syntax doesn't have the ability of expressing a distributed test that leverages multiple test actors, potentially running on different machines, so that will be a limitation when writing tests BDD-style.

dmcgill50 commented 4 years ago

I think that limitation is sufficient for the benefit. Please let me know if there are things that can be worked on to allow for this new BDD style to be supported.

adrianth commented 4 years ago

@dmcgill50 Realistically, BDD will not be available very soon, because it requires a non-trivial amount of work:

Thanks for the offer and I will involve you as soon as there's something that can be worked on collaboratively. Is there a big appetite for the BDD approach in your organization?