MetaTag Support

[wcdeich4]

Hi. Is there any support for putting @MetaTag featureName inside the .feature files so you can filter what tests should be run or not run based on the meta tags? I searched the code, but did not see anything like that.

.NET Core already has test filtering by attributes in the .cs files (https://docs.microsoft.com/en-us/dotnet/core/testing/selective-unit-tests) but support for meta tags inside the gherkin files is one of the key attributes of the BDD.

Don't get me wrong. Xunit.Gherkin.Quick is awesome!!!!!!!!! But, testers will not be happy with out support for meta tags inside the gherkin files

[ttutisani]

I don't yet know much about meta tags. If this is a standard and obvious thing within Gherkin, then answer will be Yes. But I have not yet looked into that part of Gherkin.

Goal of this framework is to support Gherkin language and naturally integrate it into Xunit tests.

[videege]

I think you could support translating Gherkin tags into XUnit traits. I can take a look at maybe doing a PR for this this weekend.

[ttutisani]

Sure. Please also provide a link where I can read about Gherkin tags and how they would translate into XUnit traits, if there is anything like that.

[wcdeich4]

https://github.com/cucumber/cucumber/wiki/Tags

[wcdeich4]

http://jbehave.org/reference/stable/meta-filtering.html

[wcdeich4]

In the Java world, there is a very slight syntactic shift between JBehave Gherkin & Cucumber Gherkin. Most noticable about how meta tag command line arguments work, but also, in JBehave you always say "Scenario" but in Cucumber you say "Scenario Outline" if you want to use an examples table. But then JBehave supports pointing to a totally separate file for the examples table, but Cucumber does not. (Actually I think Cucumber uses JBehave internally for some of its processing - so the relationship between Cucumber & JBehave is kind of like how Mac has a Linux micro kernel)

[ttutisani]

Ok, I understand the usage of tags now in Gherkin. Thank you! So, how should it map to Xunit traits? specifically, tag is just one value (e.g. @myTest), while traits in Xunit are key-value pairs. e.g. testType=UseCase. Do you suggest mapping all tags to the same trait key? e.g. mapping everything to Category trait would mean - @myTest tag translates into Category=myTest trait. At least that would make it deterministic. Any other thoughts and justifications for them?

[wcdeich4]

Hmm. The only way to use Xunit traits would be to dynamically add the trait at run time. There is a discussion w/ sample code on stack overflow, but I'm not sure if it will really work or not:

https://stackoverflow.com/questions/35936671/xunit-adding-trait-to-collectiondefinition

[ttutisani]

All right. Step back for a moment to clarify something for me.

Can anybody define what is the idea behind using tags in the Xunit.Gherkin.Quick framework? How do you envision using it?

I was thinking this is what you meant: first you apply tags to the feature, and then you pass them as Xunit traits when running the tests, and it would only run the features having that trait.

Is that the request here? or is there some other expectation from it?

[wcdeich4]

Yes, that is what I meant. But the tag would be inside the .feature file & "pass them as Xunit traits when running the tests" would be on the command line only. Normal Xunit tests rely on the trait being provided in an attribute tag above the unit test inside the .cs file. The Gherkin standard would have the meta tag in the .feature file & command line only so that a non-programmer can edit the feature files & pass arguments on the command line & control which tests are run w/out changing any C# code. That is what I meant when I said "at run time" - if a BA edits the .feature file, but not the .cs files & passed the same command line parameters as before .feature files were edited, the test behavior should dynamically change.

But how can we do this without putting the meta tag as a trait inside an attribute tag inside the C# "step" unit test file? Well, I can think of 2 ways.

(1) You have to apply the meta tags as a traits to the unit test method w/out actually writing the trait in an attribute tag above method in the unit test or "step" C# file. But how? Well, stack overflow has some sample code, but I'm not sure if it will work or not https://stackoverflow.com/questions/35936671/xunit-adding-trait-to-collectiondefinition

(2) Don't use traits meta tag filtering. When your code reads the .feature Gherkin file, make a list of the meta tags & compare it against the meta tags on the command line. If command line contains meta tags, but none of the meta tags from the command line were present in the file, then just skip ahead to the next test. (Note: if there are no meta tags on the command line, then run all tests. Also, in Cucumber Gherkin @meta_tag means to run that meta tag & ~@meta_tag means not to run that tag. But in JBehave Gherkin +meta_tag means to run that meta tag & -meta_tag mean not to run that tag --- so it is up to you if you want to support Cucumber flavor Gherkin, or JBehave flavor Gherkin, or be flexible enough to support both)

However. Option (2) may be harder than it seems. The Environment.CommandLine static variable only exists in .NET Framework, not .NET Core. I was just googling it this morning & Microsoft says the way Environment.CommandLine works is operating specific to Windows & so it would not on Unix / Linux / Mac. I'm not sure why this is so hard for Microsoft. I mean, Java is cross platform & they get command line parameters from unit tests. (The Gherkin standard is actually based on Java where it is easier to get the command line arguments in any operating system.)

[ttutisani]

My preference is that we should go with the standard, i.e. option 1 in your comment. I still don't know how to do that.

My previous question was about translating the gherkin tag (single word) into trait (2 words).

[wcdeich4]

Oh. Splitting the meta tag into two words for the trait should be easy. If the Gherkin Meta_Tag is "IntegrationSuite" then the trait would be [Trait("MetaTag","IntegrationSuite")]. The part that has me worried, is, how do you do that w/out actually writing "[Trait("MetaTag","IntegrationSuite")]" in the C# file? The code example on Stack Overflow may in fact to that; but, I think will take some fiddling around with to get it to work just right.

[ttutisani]

Yes, that will require some trial and error. But the link you provided indeed shows some hope.

MetaTag can be hard to guess. Maybe we should use "Category" as a keyword instead of MetaTag? After all, meta tags are really denoting categories of tests.

[wcdeich4]

Well, if the code works to add as many traits as we want dynamically, why not add both, so you could use either ...-trait "Category=Integration" or ...-trait "MetaTag=Integration" on the command line? I guess it does not really matter all that much

[ttutisani]

I see. That would be fine too.

[videege]

I just submitted PR #30 for this. Please take a look and let me know what you think. I was able to run just the scenarios with certain tags by using the built in filter commands, i.e., dotnet test --filter Category=@bigaddition

[ttutisani]

Ah, this answers one of my questions on that PR. Thanks! I just reviewed and provided couple of comments. Please address them upon your convenience. Good job!

[ttutisani]

Just one question about this command line that you provided now: I'm thinking that command line should like like this: dotnet test --filter Category=bigaddition (without @ sign). Don't you agree? @ sign is for Gherkin to find the tag I would think. Thoughts?

[wcdeich4]

dotnet test --filter Category=bigaddition and/or dotnet test --filter MetaTag=bigaddition Sound good. I think we have t accept a little variation from how thing are done in Java. Using @metatag or +metatag on the command line would require working around the Xunit builtin --filter command line flag, and probably need a lot more development.

[ttutisani]

I'm fine to proceed with Category at a minimum. But @ sign should not be needed in the command line. @videege please do what you can.

[videege]

I will update my PR with your suggestions, but I’m on travel currently so it won’t be before Saturday.

On Jun 11, 2018, at 10:50 AM, Tengiz Tutisani notifications@github.com wrote:

I'm fine to proceed with Category at a minimum. But @ sign should not be needed in the command line. @videege please do what you can.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[ttutisani]

Ok. Thank you!

[ttutisani]

@videege I don't know if you noticed, but your PR branch has conflicts with master branch. Because of that, I cannot merge. I left the same comment on the PR itself. Just FYI, not rush.

[ttutisani]

Thanks @videege for doing the PR! Now, can you also write up a small instruction page for it? Just like I did this one for DataTable: https://github.com/ttutisani/Xunit.Gherkin.Quick/blob/master/datatable-argument.md

[ttutisani]

All action items are complete. Good job! Closing this thread.