search

fhoeben / allure-fitnesse-listener

jUnit Listener to incorporate FitNesse results in Allure
Apache License 2.0
1 stars 4 forks source link

Upgrade to Allure 2? #3

Open fhoeben opened 7 years ago

fhoeben commented 7 years ago

Allure1 is deprecated according to https://github.com/allure-framework/allure1

Should we upgrade to https://github.com/allure-framework/allure2. What would be impact and benefit?

fhoeben commented 6 years ago

I made a first stab at implementing Allure2 change in https://github.com/fhoeben/allure-fitnesse-listener/tree/allure2, and using these in https://github.com/fhoeben/sample-fitnesse-project/tree/allure2.

It seems to work, but the advantage is not clear to me yet :-)

tcnh commented 6 years ago

There should be some advantages in using allure 2.

  1. A separate listener is no longer needed for basic reporting. Allure 2 can serve a nice html aggregate report directly from the surefire-reports. (this would lack the fitnesse result pages, obviously)
  2. There's a Jenkins plugin to generate reports as a build step
  3. The new plug in framework should unlock the opportunity to convert the fitnesse results combined with the xunit/surefire output to an allure report containing all info (https://docs.qameta.io/allure/2.0/#_allure_plugins_system)
  4. Allure 2 supports reporting multiple runs (trend) in one dashboard. Also giving insight in f.e. flaky tests

Thinking out loud, I'd say that when we create a reader/aggregator plugin to do the report conversion from already existing output, The allure listener could become obsolete, making it easier to use. (as a post-build action) Would have to dig in to how allure 2 works.

cyborghound commented 6 years ago

Is this still something you're interested in developing further @fhoeben? I think what @tcnh said is something that i'm also very interested in. It would be a nice addition to FitNesse.

fhoeben commented 6 years ago

I'm not using Allure so I'm not going to take the initiative to create it. If someone else (e.g. @tcnh or you) builds it, I'm willing to support/merge...

tcnh commented 6 years ago

At this point I have no urgent need for Allure 2 in my projects. The current listener and Allure 1 are currently sufficient, so it's not on my (already too long) to-do list right now.

roydekleijn commented 4 years ago

I made a first stab at implementing Allure2 change in https://github.com/fhoeben/allure-fitnesse-listener/tree/allure2, and using these in https://github.com/fhoeben/sample-fitnesse-project/tree/allure2.

It seems to work, but the advantage is not clear to me yet :-)

will this be released? or should we fork it and release it ourselves?

tcnh commented 4 years ago

Visually I see no difference between the reports generated by the v1 and v2 listeners. (other than that the V1 has better info and structure (packages tab) because the v2 branch is pretty far behind) Using the v1 listener creates results that are interpreted fine by allure 2.x? What does the allure2 branch add in tems of functionality/compatibility?

roydekleijn commented 4 years ago

True there is not much of a difference, however:

tcnh commented 4 years ago

allure-fitnesse-listener 1.0.5 no longer works on my mac...

Could you describe what errors it's giving?

I can take a look at getting allure2 branch functionally equivalent to 1.0.5 listener, then I see no reason to stick with V1

roydekleijn commented 4 years ago

when adding -Pallure the tests are no longer executed.

it just says: [INFO] [INFO] ------------------------------------------------------- [INFO] T E S T S [INFO] ------------------------------------------------------- [INFO] [INFO] Results: [INFO] [INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0

When removing -Pallure it executes all my tests.

tcnh commented 4 years ago

Can you share the allure profile from your pom.xml?

roydekleijn commented 4 years ago

Sure,

<profile>
            <id>allure</id>
            <properties>
                <allure.report.directory>${project.build.directory}/allure-report</allure.report.directory>

<extraFailsafeListeners>,nl.hsac.fitnesse.junit.JUnitXMLPerPageListener,nl.hsac.fitnesse.junit.allure.JUnitAllureFrameworkListener</extraFailsafeListeners>
            </properties>
            <dependencies>
                <dependency>
                    <groupId>nl.hsac</groupId>
                    <artifactId>allure-fitnesse-listener</artifactId>
                    <version>${allure.fitnesse.listener.version}</version>
                    <scope>test</scope>
                </dependency>
            </dependencies>
            <build>
                <plugins>
                    <plugin>
                        <artifactId>maven-resources-plugin</artifactId>
                        <version>3.0.2</version>
                        <executions>
                            <execution>
                                <id>copy-resources</id>
                                <phase>site</phase>
                                <goals>
                                    <goal>copy-resources</goal>
                                </goals>
                                <configuration>
                                    <outputDirectory>${allure.report.directory}/data/fitnesseResults</outputDirectory>
                                    <resources>
                                        <resource>
                                            <directory>${project.build.directory}/fitnesse-results</directory>
                                            <filtering>true</filtering>
                                        </resource>
                                    </resources>
                                </configuration>
                            </execution>
                            <execution>
                                <id>copy-resources-allurehistory</id>
                                <phase>site</phase>
                                <goals>
                                    <goal>copy-resources</goal>
                                </goals>
                                <configuration>
                                    <outputDirectory>${project.build.directory}/allure-results/history</outputDirectory>
                                    <resources>
                                        <resource>
                                            <directory>${allure.report.directory}/history</directory>
                                            <filtering>true</filtering>
                                        </resource>
                                    </resources>
                                </configuration>
                            </execution>
                        </executions>
                        <dependencies>
                            <dependency>
                                <groupId>org.apache.maven.shared</groupId>
                                <artifactId>maven-filtering</artifactId>
                                <version>3.1.1</version>
                            </dependency>
                        </dependencies>
                    </plugin>
tcnh commented 4 years ago

Could you try:

<profile>
            <id>allure</id>
            <properties>
                <allure.report.directory>${project.build.directory}/allure-report</allure.report.directory>
                <extraFailsafeListeners>,nl.hsac.fitnesse.junit.JUnitXMLPerPageListener,nl.hsac.fitnesse.junit.allure.JUnitAllureFrameworkListener</extraFailsafeListeners>
            </properties>
            <dependencies>
                <dependency>
                    <groupId>nl.hsac</groupId>
                    <artifactId>allure-fitnesse-listener</artifactId>
                    <version>${allure.fitnesse.listener}</version>
                    <scope>test</scope>
                </dependency>

            </dependencies>
            <build>
                <plugins>
                    <plugin>
                        <artifactId>maven-resources-plugin</artifactId>
                        <version>3.0.2</version>
                        <executions>
                            <execution>
                                <id>copy-resources</id>
                                <phase>site</phase>
                                <goals>
                                    <goal>copy-resources</goal>
                                </goals>
                                <configuration>
                                    <outputDirectory>${allure.report.directory}/data/fitnesseResults</outputDirectory>
                                    <resources>
                                        <resource>
                                            <directory>${project.build.directory}/fitnesse-results</directory>
                                            <filtering>true</filtering>
                                        </resource>
                                    </resources>
                                </configuration>
                            </execution>
                            <execution>
                                <id>copy-resources-allurehistory</id>
                                <phase>site</phase>
                                <goals>
                                    <goal>copy-resources</goal>
                                </goals>
                                <configuration>
                                    <outputDirectory>${project.build.directory}/allure-results/history</outputDirectory>
                                    <resources>
                                        <resource>
                                            <directory>${allure.report.directory}/history</directory>
                                            <filtering>true</filtering>
                                        </resource>
                                    </resources>
                                </configuration>
                            </execution>
                        </executions>
                        <dependencies>
                            <dependency>
                                <groupId>org.apache.maven.shared</groupId>
                                <artifactId>maven-filtering</artifactId>
                                <version>3.1.1</version>
                            </dependency>
                        </dependencies>
                    </plugin>
                </plugins>
            </build>
            <reporting>
                <excludeDefaults>true</excludeDefaults>
                <plugins>
                <plugin>
                    <groupId>io.qameta.allure</groupId>
                    <artifactId>allure-maven</artifactId>
                    <version>${allure.maven.version}</version>
                    <configuration>
                        <resultsDirectory>allure-results</resultsDirectory>
                        <reportDirectory>${allure.report.directory}</reportDirectory>
                        <reportVersion>${allure.version}</reportVersion>
                    </configuration>
                </plugin>
                </plugins>
            </reporting>
        </profile>

Use the site target to run the reporting part

Using properties:

<allure.maven.version>2.10.0</allure.maven.version>
<allure.version>2.13.0</allure.version>
<allure.fitnesse.listener>1.0.5</allure.fitnesse.listener>

That should give you an allure2 report, including trend graphs, history, flakiness and new failed markers.

(be sure to exclude history when cleaning):

<fileset>
    <directory>target</directory>
    <includes>
        <include>**</include>
    </includes>
    <excludes>
        <exclude>allure-results/history/**</exclude>
    </excludes>
</fileset>
teunisv commented 4 years ago

Seems to run fine on openJDK 8, but with openJDK 13 (macos) the result remain empty until we used allure2 code

Using your example pom still result in no runnend test

tcnh commented 4 years ago

Ah, that could be a thing yes. I'm running everything on JDK8.. if you're able to debug a bit to see what can't be found inwhen using java 13... I expect that some JEE dependency is missing. The runner howerver fails silently in that kind of situation

tcnh commented 4 years ago

Just tested with openJDK13 on windows and test ran fine. So it seems to be osx specific.. I can't reproduce it as I have no Mac.

Could you run with -X to see if that produces any error worth investigating?