search

openfga / cli

A cross-platform CLI to interact with an OpenFGA server
https://openfga.dev
Apache License 2.0
50 stars 25 forks source link

Generate a tests report #293

Open le-yams opened 7 months ago

le-yams commented 7 months ago

Hello,

Having the possibility to generate a test report would be great (especially to publish them in CIs). No sure which format to use. JUnit is the most commonly used and a lot of tools supports it but ... xml 😢. The testing tools generating json reports seems to all use a custom format 😞. Looking into this I came across this initiative that is very new but could be interesting.

What do you think about it?

Regards, Yann

homorozeanu commented 7 months ago

Agree with @le-yams . We test our model using the openfga cli in azure devops pipeline and would love to be able to export it in a format that is understood by azure devops. Hence, +1.

aaguiarz commented 7 months ago

@homorozeanu @le-yams would using https://github.com/ctrf-io work for you?

homorozeanu commented 7 months ago

@aaguiarz Unfortunately from what I know neither Azure DevOps nor NUnit (the testing framework that we use for our dotnet tests) understand and can export/import test results in CTRF format.

yann-disanto commented 6 months ago

@aaguiarz it would be ok on our side (a github action to publish the test results is being written so it will fit our needs).

As I said, the downside of this choice is that, as it's pretty new, this format is not yet well supported by most of CI tools. But this downside would be also true if we use a "yet another home made test report fornat". This downside could be mitigated by the possibility for people to write ctrf to supported format transformer.

I would be interested in others openfga cli core-team developers opinion, @rhamzeh any thought about this?

ewanharris commented 6 months ago

As you might have seen @aaguiarz mention, last week we had a hackathon at Okta. I decided to spend some time working on integrating Cucumber (via godog) into the fga model test command to enable testing to live outside the store files. What's fun about this is that, given it's relatively easy to generate the Cucumber tests from a yaml file, we can use Cucumber as our test runner for those too and benefit from the reporters it offers, such as junit.

If you're curious, the work is at https://github.com/openfga/cli/tree/feat/support-gherkin-syntax (I'll be splitting this into multiple PRs eventually) and you can install it using go install github.com/openfga/cli/cmd/fga@feat/support-gherkin-syntax. Running your existing store based tests using fga model test --tests store.fga.yaml --reporter junit will print a junit report to the console with the test results. I've tested it against the store files in https://github.com/openfga/sample-stores but would be great to get any feedback from yourselves around the functionality.

There's also an example of the Gherkin syntax here that demonstrates the FolderBox store and ABAC store tests converted into the gherkin syntax

yann-disanto commented 6 months ago

Hello @ewanharris,

Is it planned to go further with the gherkin syntax support?

ewanharris commented 5 months ago

Hey @yann-disanto, sorry for the delay! Yes I think it's something that we'd like to continue with, although I don't really have a timeline for when.

We'd most likely release the gherkin syntax work initially and then at a later date switch to using it to as the "backend" for the yaml tests once we're confident in the syntax, my focus has been elsewhere the last few weeks but I'll see if I can set some time aside soon to do that.