Add instructions for running the tests

[kinow]

Hi,

I will be presenting in a local conference in a few days about IIIF, and using Cantaloupe as Image server for experimentations and demo.

I have the latest version from the develop branch, but not all tests pass. I remember running the tests some time ago, and got a similar error. Are there any instructions on how to run the tests?

Normally, when I find a Java Maven project in GitHub, and mvn clean test does not work, I look for the Travis CI file in the project repository.

Maybe we could add instructions for building and running - if these instructions are not already somewhere - the project, and also include Travis-CI for the branches and pull requests?

I'm also getting very intriguing results when running the tests. This is the last commit I have in my working copy:

commit a4c874847894b2936aa81f0350349419890b7f5a
Author: Alex Dolski <alexd@n-ary.net>
Date:   Tue Jul 18 14:18:31 2017 -0500

Running mvn clean test, I got:

Tests run: 1341, Failures: 8, Errors: 156, Skipped: 21

After some minutes, running mvn clean test again, I got:

Tests run: 1341, Failures: 8, Errors: 135, Skipped: 21

The number of tests run, failures, and skipped is consistent. But the number of errors is not. I didn't investigate why, as I'm working to run Cantaloupe within Eclipse and also finishing slides. But if it's a known issue, I could try to dig deeper and perhaps help with this if necessary :-)

Ta Bruno

[adolski]

I can certainly add a testing section to the readme. The basic idea is:

1. Install dependencies: GraphicsMagick, ImageMagick, FFmpeg, Kakadu, OpenJPEG, Redis, and Firefox+geckodriver (needed by Selenium for ControlPanelTest). Whichever of these are missing will cause the corresponding tests to fail, so if Kakadu is missing, KakaduProcessorTest will fail, etc.

2. Copy test.properties.sample to test.properties and fill in whatever info you have -- again, anything missing will cause failures.

I don't run the tests as often as I should, so I'm not surprised that there are some failures. There should only a handful of consistent ones, though. I just ran and got 19 errors & 3 failures, and there should be less than that.

I can reproduce the inconsistent results across runs. Often I run individual package tests from IntelliJ, and they run fine. I don't know what the problem is -- my only guess would be state being carried over across the tests, but I don't know where it might be occurring. I'd love to get them all consistently passing.

[adolski]

The above commit moved some info into the readme that used to be buried in the user manual. I also fixed a couple of other failing tests, and ignored a few that have been failing for a long time for mysterious reasons. The rest now pass for me except ControlPanelTest, which breaks every now and then as new versions of Selenium, Firefox, and webdriver are released.

[kinow]

Thanks!! I already have some tests passing after running

apt-get install libav-tools graphicsmagick imagemagick libopenjp2-tools

I'm playing with the code in another branch https://github.com/kinow/cantaloupe/tree/fix-tests where I'm trying to fix the tests.

There are a few tests that fail at work PC, but not at home. These tests are related to fonts, and are not so hard to fix. Other tests that re failing involve cache and might take a bit longer for me to find a way to fix them.

I'm also exploring the idea of ignoring some tests by default (Azure, S3, Redis, ...) which most users won't have access to all these resources. Then creating a profile, say all, that could be used as mvn clean test -Pall.

That way we could add Travis CI in the near future too, instructing it to install the ffmpeg, openjpeg, etc, dependencies. Maybe even the Firefox and Redis tests could be enabled in Travis with some Shell + Docker-Fu.

Is that a good direction? Otherwise we can have all tests enabled by default and have a profile that limits what is executed. That's what we are doing at Apache Jena https://github.com/apache/jena/blob/68ee610984aea876780ce68c07334c0f0bd36e0f/pom.xml#L65

I'm happy either way, as long as I can get all tests passing :-)

Thanks again for all the instructions! Mind if I send another PR some other time, with instructions specifically for Ubuntu?

[adolski]

What about doing something like org.junit.Assume.assumeTrue(hasDependencies())? That could go in either a @Before method or a @Test to skip only what needs to be. It would take some work to implement all the hasDependencies() methods, but basically it would only be a matter of trying to do something that requires the dependency and returning false when catching the exception.

Absolutely send the Ubuntu instructions!

[kinow]

I never used this approach, but sounds useful for other similar cases. Would you have a link to some documentation or example for the hasDependencies() method?

[adolski]

It would vary from test to test. For AmazonS3ResolverTest & AmazonS3CacheTest it might look something like:

private boolean hasDependencies() {
    return (getAccessKeyId() != null && getBucket() != null
            && getSecretKey() != null);
}

Some of them would be easier than others, but I'm happy to write all of them myself if this is the route you want to take. I like the zero-configuration of this approach, but I don't like all the extra code, and if hasDependencies() breaks at some point, it might not get noticed right away.

I also like the limited-dependencies profile idea -- -Pnodeps or something. Actually I think I like it more than the above, having thought about it some more. 😄

I will even add a third idea, which is per-test-class toggle switches in test.properties which get checked in a @Before or @BeforeClass (again using Assume)

I don't really have a strong preference in any direction. Whatever you think would work best.

[kinow]

Oh, gotcha. For some reason I thought the hasDependencies() would deal with Maven dependencies :-)

Some of them would be easier than others, but I'm happy to write all of them myself if this is the route you want to take. I like the zero-configuration of this approach, but I don't like all the extra code, and if hasDependencies() breaks at some point, it might not get noticed right away.

I also like the limited-dependencies profile idea -- -Pnodeps or something. Actually I think I like it more than the above, having thought about it some more. :smile:

Haha. RIght now I'm excluding the dependencies with no profile, just so mvn clean test works. Then once it's done, I'll move the exclusions to a nodeps profile, and later we can investigate in another ticket the per-test-class toggle. I liked the idea too :-)

WDYT?

[adolski]

After the above change and fixing an issue in HttpResolver, I get:

Tests run: 1349, Failures: 0, Errors: 0, Skipped: 22

[kinow]

That's great! I tried that before, and got some failures, but that was before disabling Azure, AWS, Selenium tests, and creating a test.properties. I'm in a conference, without access to my Linux box. So testing on Windows today :-)

Here's what happens checking out the latest code, with no other change:

Tests run: 1343, Failures: 8, Errors: 107, Skipped: 22

Let me now disable the tests in pom.xml, and create the properties file for testing...

[kinow]

With

  <profiles>
    <profile>
      <id>nodeps</id>
      <activation>
        <activeByDefault>false</activeByDefault>
      </activation>
      <build>
        <plugins>
            <plugin>
              <groupId>org.apache.maven.plugins</groupId>
              <artifactId>maven-surefire-plugin</artifactId>
              <version>2.19.1</version>
              <configuration>
                <runOrder>random</runOrder>
                <reuseForks>false</reuseForks>
                <excludes>
                  <exclude>Azure*Test</exclude>
                  <exclude>Amazon*Test</exclude>
                  <exclude>Kakadu*Test</exclude>
                  <exclude>Redis*Test</exclude>
                </excludes>
              </configuration>
            </plugin>
        </plugins>
      </build>
    </profile>
  </profiles>

And

cp test.properties.sample test.properties

Running mvn clean test -Pnodeps I get the following:

Results :

Failed tests:
  StandaloneEntryTest.testMainWithDirectoryConfigFileArgumentPrintsUsage:194 expected:<...e\src\test\resources[]

]sage: java <VM op...> but was:<...e\src\test\resources[

Usage: java <VM op...>
  StandaloneEntryTest.testMainWithInvalidConfigFileArgumentPrintsUsage:174 expected:<Does not exist: [/bla/bla/bla]

]sage: java <VM op...> but was:<Does not exist: [C:\bla\bla\bla

Usage: java <VM op...>
  FilesystemCacheTest.testGetHashedStringBasedSubdirectory:83 expected:<[/]08\32\c1> but was:<[\]08\32\c1>
  BasicStringOverlayServiceTest.testGetOverlay:54 expected:<[Helvetica]> but was:<[SansSerif]>
  DelegateOverlayServiceTest.testGetOverlayReturningStringOverlay:73 expected:<[Helvetica]> but was:<[SansSerif]>
  StringOverlayTest.testToString:69 expected:<...e3bd8cb6803681_SE_5_[Helvetica]_12_2.0_0.1_#0000FFF...> but was:<...e3bd8cb6803681_SE_5_[SansSerif]_12_2.0_0.1_#0000FFF...>
  Java2DUtilTest.testApplyOverlayWithStringOverlay:194
  FilesystemResolverTest.testGetFile:79 Expected exception
Tests in error:
edu.illinois.library.cantaloupe.cache.FilesystemCacheTest.testNewDerivativeImageInputStreamWithOpListWithNonzeroTtl(edu.illinois.library.cantaloupe.cache.FilesystemCacheTest)
  Run 1: FilesystemCacheTest.testNewDerivativeImageInputStreamWithOpListWithNonzeroTtl:495
  Run 2: FilesystemCacheTest.tearDown:76 ▒ IO Unable to delete file: C:\Users\KINOSH~1\...

  FilesystemCacheTest.tearDown:76 ▒ IO Unable to delete file: C:\Users\KINOSH~1\...
  FilesystemCacheTest.tearDown:76 ▒ IO Unable to delete file: C:\Users\KINOSH~1\...
  FilesystemCacheTest.tearDown:76 ▒ IO Unable to delete file: C:\Users\KINOSH~1\...
  FilesystemCacheTest.tearDown:76 ▒ IO Unable to delete file: C:\Users\KINOSH~1\...
  FilesystemCacheTest.tearDown:76 ▒ IO Unable to delete file: C:\Users\KINOSH~1\...
  FfmpegProcessorTest>ProcessorTest.testProcessWithCropOperation:177->ProcessorTest.doProcessTest:340->ProcessorTest.doProcessTest:364 ▒ Processor
  FfmpegProcessorTest>ProcessorTest.testProcessWithFilterOperation:232->ProcessorTest.doProcessTest:340->ProcessorTest.doProcessTest:364 ▒ Processor
  FfmpegProcessorTest.testProcessWithFrameOption:128 ▒ Processor Cannot run prog...
  FfmpegProcessorTest.testProcessWithInvalidFrameOptionThrowsException:148 ▒ Processor
  FfmpegProcessorTest>ProcessorTest.testProcessWithRotateOperation:223->ProcessorTest.doProcessTest:340->ProcessorTest.doProcessTest:364 ▒ Processor
  FfmpegProcessorTest>ProcessorTest.testProcessWithScaleOperation:199->ProcessorTest.doProcessTest:340->ProcessorTest.doProcessTest:364 ▒ Processor
  FfmpegProcessorTest>ProcessorTest.testProcessWithSupportedOutputFormats:244->ProcessorTest.doProcessTest:340->ProcessorTest.doProcessTest:364 ▒ Processor
  FfmpegProcessorTest>ProcessorTest.testProcessWithSupportedSourceFormatsAndNoOpOperations:99->ProcessorTest.doProcessTest:340->ProcessorTest.doProcessTest:364 ▒ Processor
  FfmpegProcessorTest>ProcessorTest.testProcessWithSupportedSourceFormatsAndNoOperations:87->ProcessorTest.doProcessTest:340->ProcessorTest.doProcessTest:364 ▒ Processor
  FfmpegProcessorTest>ProcessorTest.testProcessWithTransposeOperation:212->ProcessorTest.doProcessTest:340->ProcessorTest.doProcessTest:364 ▒ Processor
  FfmpegProcessorTest.testReadImageInfo:168 ▒ Processor Cannot run program "ffpr...
  FfmpegProcessorTest.testValidate:175 ▒ Processor Cannot run program "ffprobe":...
  GraphicsMagickProcessorTest.testGetSupportedFeatures:29->MagickProcessorTest.testGetSupportedFeatures:58 ▒ NullPointer
  GraphicsMagickProcessorTest>ProcessorTest.testGetSupportedIiif11Qualities:255 ▒ NullPointer
  GraphicsMagickProcessorTest>ProcessorTest.testGetSupportedIiif20Qualities:277 ▒ NullPointer
  GraphicsMagickProcessorTest.testProcessPreservesMetadata:29->MagickProcessorTest.testProcessPreservesMetadata:81->MagickProcessorTest.assertXmpPresent:107 ▒ UnsupportedSourceFormat
  GraphicsMagickProcessorTest.testProcessStripsMetadata:29->MagickProcessorTest.testProcessStripsMetadata:91->MagickProcessorTest.assertXmpPresent:107 ▒ UnsupportedSourceFormat
  GraphicsMagickProcessorTest.testProcessWithRotationAndCustomBackgroundColorAndNonTransparentOutputFormat:29->MagickProcessorTest.testProcessWithRotationAndCustomBackgroundColorAndNonTransparentOutputFormat:190 ▒ UnsupportedSourceFormat
  GraphicsMagickProcessorTest.testProcessWithRotationAndCustomBackgroundColorAndTransparentOutputFormat:29->MagickProcessorTest.testProcessWithRotationAndCustomBackgroundColorAndTransparentOutputFormat:155 ▒ UnsupportedSourceFormat
  GraphicsMagickProcessorTest>ProcessorTest.testProcessWithSupportedOutputFormats:239 ▒ NullPointer
  GraphicsMagickProcessorTest>ProcessorTest.testReadImageInfo:68 ▒ UnsupportedSourceFormat
  ImageMagickProcessorTest.testGetOverlayTempFile:202 ▒ UnknownHost C
  ImageMagickProcessorTest.testGetSupportedFeatures:35->MagickProcessorTest.testGetSupportedFeatures:58 ▒ NullPointer
  ImageMagickProcessorTest>ProcessorTest.testGetSupportedIiif11Qualities:255 ▒ NullPointer
  ImageMagickProcessorTest>ProcessorTest.testGetSupportedIiif20Qualities:277 ▒ NullPointer
  ImageMagickProcessorTest.testProcessPreservesMetadata:35->MagickProcessorTest.testProcessPreservesMetadata:81->MagickProcessorTest.assertXmpPresent:107 ▒ UnsupportedSourceFormat
  ImageMagickProcessorTest.testProcessStripsMetadata:35->MagickProcessorTest.testProcessStripsMetadata:91->MagickProcessorTest.assertXmpPresent:107 ▒ UnsupportedSourceFormat
  ImageMagickProcessorTest.testProcessWithPageOption:211 ▒ UnsupportedSourceFormat
  ImageMagickProcessorTest.testProcessWithRotationAndCustomBackgroundColorAndNonTransparentOutputFormat:35->MagickProcessorTest.testProcessWithRotationAndCustomBackgroundColorAndNonTransparentOutputFormat:190 ▒ UnsupportedSourceFormat
  ImageMagickProcessorTest.testProcessWithRotationAndCustomBackgroundColorAndTransparentOutputFormat:35->MagickProcessorTest.testProcessWithRotationAndCustomBackgroundColorAndTransparentOutputFormat:155 ▒ UnsupportedSourceFormat
  ImageMagickProcessorTest>ProcessorTest.testProcessWithSupportedOutputFormats:239 ▒ NullPointer
  ImageMagickProcessorTest>ProcessorTest.testReadImageInfo:68 ▒ UnsupportedSourceFormat
  ImageMagickProcessorTest.testValidate:237 ▒ UnsupportedSourceFormat ImageMagic...
  OpenJpegProcessorTest>ProcessorTest.testProcessWithCropOperation:177->ProcessorTest.doProcessTest:340->ProcessorTest.doProcessTest:364 ▒ Processor
  OpenJpegProcessorTest>ProcessorTest.testProcessWithFilterOperation:232->ProcessorTest.doProcessTest:340->ProcessorTest.doProcessTest:364 ▒ Processor
  OpenJpegProcessorTest>ProcessorTest.testProcessWithRotateOperation:223->ProcessorTest.doProcessTest:340->ProcessorTest.doProcessTest:364 ▒ Processor
  OpenJpegProcessorTest>ProcessorTest.testProcessWithScaleOperation:199->ProcessorTest.doProcessTest:340->ProcessorTest.doProcessTest:364 ▒ Processor
  OpenJpegProcessorTest>ProcessorTest.testProcessWithSupportedOutputFormats:244->ProcessorTest.doProcessTest:340->ProcessorTest.doProcessTest:364 ▒ Processor
  OpenJpegProcessorTest>ProcessorTest.testProcessWithSupportedSourceFormatsAndNoOpOperations:99->ProcessorTest.doProcessTest:340->ProcessorTest.doProcessTest:364 ▒ Processor
  OpenJpegProcessorTest>ProcessorTest.testProcessWithSupportedSourceFormatsAndNoOperations:87->ProcessorTest.doProcessTest:340->ProcessorTest.doProcessTest:364 ▒ Processor
  OpenJpegProcessorTest>ProcessorTest.testProcessWithTransposeOperation:212->ProcessorTest.doProcessTest:340->ProcessorTest.doProcessTest:364 ▒ Processor
  ProcessorConnectorTest.testConnectWithFileResolverAndStreamProcessor:105 ▒ UnsupportedSourceFormat
  ProcessorConnectorTest.testConnectWithStreamResolverAndStreamProcessorWithCacheStrategy:226 ▒ UnsupportedSourceFormat
  ProcessorConnectorTest.testConnectWithStreamResolverAndStreamProcessorWithCacheStrategyAndSourceCacheDisabled:264 ▒ UnsupportedSourceFormat
  ProcessorConnectorTest.testConnectWithStreamResolverAndStreamProcessorWithStreamStrategy:190 ▒ UnsupportedSourceFormat
  ImageResourceTest.testPurgeFromCacheWhenSourceIsMissingAndOptionIsFalse:379->doPurgeFromCacheWhenSourceIsMissing:461 ▒ IO
  ImageResourceTest.testPurgeFromCacheWhenSourceIsMissingAndOptionIsTrue:385->doPurgeFromCacheWhenSourceIsMissing:461 ▒ IO
  InformationResourceTest.testPurgeFromCacheWhenSourceIsMissingAndOptionIsFalse:216->doPurgeFromCacheWhenSourceIsMissing:292 ▒ IO
  InformationResourceTest.testPurgeFromCacheWhenSourceIsMissingAndOptionIsTrue:222->doPurgeFromCacheWhenSourceIsMissing:292 ▒ IO
  ImageResourceTest.testPurgeFromCacheWhenSourceIsMissingAndOptionIsFalse:414->doPurgeFromCacheWhenSourceIsMissing:496 ▒ IO
  ImageResourceTest.testPurgeFromCacheWhenSourceIsMissingAndOptionIsTrue:420->doPurgeFromCacheWhenSourceIsMissing:496 ▒ IO
  InformationResourceTest.testPurgeFromCacheWhenSourceIsMissingAndOptionIsFalse:220->doPurgeFromCacheWhenSourceIsMissing:296 ▒ IO
  InformationResourceTest.testPurgeFromCacheWhenSourceIsMissingAndOptionIsTrue:226->doPurgeFromCacheWhenSourceIsMissing:296 ▒ IO

Tests run: 1295, Failures: 8, Errors: 59, Skipped: 22

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 05:24 min
[INFO] Finished at: 2017-08-02T11:33:10+12:00
[INFO] Final Memory: 34M/627M
[INFO] ------------------------------------------------------------------------

Just need to install the dependencies as I did in Ubuntu now I think....

[kinow]

Hmmm, installed imagemagic, ffmeg, jpeg2000, and graphicsmagick for Windows, added some directories to the PATH env var, confirmed they are accessible and no security warning when I run them... but running mvn clean test (closed terminal, opened again, checked PATH) still gives me the same numbers for failures & errors. Will try it again when I return from the conference :-( sorry

[kinow]

At home now. Tested with the following exclusions:

          <excludes>
            <exclude>Azure*Test</exclude>
            <exclude>Amazon*Test</exclude>
            <exclude>Kakadu*Test</exclude>
            <exclude>Redis*Test</exclude>
            <exclude>ControlPanelTest</exclude>
          </excludes>

mvn clean test gives me:

Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.333 sec - in edu.illinois.library.cantaloupe.script.ScriptEngineFactoryTest

Results :

Failed tests: 
  FilesystemCacheTest.testGetSourceImageFileWithIdentifierWithNonzeroTtl:422
  StringOverlayTest.testToString:69 expected:<...e3bd8cb6803681_SE_5_[Helvetica]_12_2.0_0.1_#0000FFF...> but was:<...e3bd8cb6803681_SE_5_[SansSerif]_12_2.0_0.1_#0000FFF...>
  Java2DUtilTest.testApplyOverlayWithStringOverlay:194

Tests run: 1294, Failures: 3, Errors: 0, Skipped: 21

[kinow]

The FilesystemCacheTest.testGetSourceImageFileWithIdentifierWithNonzeroTtl did not fail again. But Java2DUtilTest.testApplyOverlayWithStringOverlay doesn't work.

[adolski]

Not sure about the first one, maybe a timing issue? Never seen that fail before.

The overlay test tries to apply a string overlay and then samples the color of a couple of pixels. Maybe our JREs are producing different results? (Also it's using Helvetica)

[kinow]

I'm thinking about updating the other pull request and then preparing another PR for continuous integration Travis CI :-)

There we can test with a prepared environment, and compare our results. This test may not fail in travis too, and in that case I will just compare my env with travis' job env.

Will post the output of mvn -v once I'm in the office...

[kinow]

Apache Maven 3.3.9
Maven home: /usr/share/maven
Java version: 1.8.0_131, vendor: Oracle Corporation
Java home: /opt/jdk1.8.0_131/jre
Default locale: en_NZ, platform encoding: UTF-8
OS name: "linux", version: "4.10.0-27-generic", arch: "amd64", family: "unix"

[kinow]

I have one test failing, but we have instructions in place in README.md, so I'm closing this ticket to archive it and then will submit other issues for the test failing and Travis CI. Thanks!!