Use Guava's testlib for increased test coverage

[ben-manes]

It might be useful to run the object-based data structures using Guava's testlib. The below example might not be configured correctly, as I am unsure about the exact behavior expected for nulls in fastutil, but it does show minor mistakes like

junit.framework.ComparisonFailure: map.toString() incorrect expected:<{one=[January, two=February, three=]March}> but was:<{one=[>January, two=>February, three=>]March}>
    at junit.framework.Assert.assertEquals(Assert.java:100)
    at junit.framework.TestCase.assertEquals(TestCase.java:253)
    at com.google.common.collect.testing.testers.MapToStringTester.testToString_formatting(MapToStringTester.java:78)

import java.util.Map;
import java.util.function.Supplier;

import com.google.common.collect.testing.MapTestSuiteBuilder;
import com.google.common.collect.testing.TestStringMapGenerator;
import com.google.common.collect.testing.features.CollectionFeature;
import com.google.common.collect.testing.features.CollectionSize;
import com.google.common.collect.testing.features.MapFeature;

import it.unimi.dsi.fastutil.objects.Object2ObjectOpenHashMap;
import junit.framework.Test;
import junit.framework.TestCase;

public final class FastUtilMapTest extends TestCase {

  public static Test suite() {
    return suite("Object2ObjectOpenHashMap", Object2ObjectOpenHashMap::new);
  }

  public static Test suite(String name, Supplier<Map<String, String>> factory) {
    return MapTestSuiteBuilder
        .using(new TestStringMapGenerator() {
          @Override protected Map<String, String> create(Map.Entry<String, String>[] entries) {
            var map = factory.get();
            for (var entry : entries) {
              map.put(entry.getKey(), entry.getValue());
            }
            return map;
          }
        })
        .named(name)
        .withFeatures(
            CollectionSize.ANY,
            MapFeature.GENERAL_PURPOSE,
            MapFeature.ALLOWS_NULL_KEYS,
            MapFeature.ALLOWS_NULL_VALUES,
            MapFeature.ALLOWS_ANY_NULL_QUERIES,
            CollectionFeature.SUPPORTS_ITERATOR_REMOVE)
        .createTestSuite();
  }
}

[ben-manes]

ahh, toString is contention from prior discussions. Here are a few better failures

java.lang.ArrayIndexOutOfBoundsException: Index -4 out of bounds for length 33
    at it.unimi.dsi.fastutil.objects.Object2ObjectOpenHashMap.insert(Object2ObjectOpenHashMap.java:271)
    at it.unimi.dsi.fastutil.objects.Object2ObjectOpenHashMap.merge(Object2ObjectOpenHashMap.java:537)
    at com.google.common.collect.testing.testers.MapMergeTester.testMappedToNull(MapMergeTester.java:69)

junit.framework.AssertionFailedError
    at com.google.common.collect.testing.testers.MapComputeIfPresentTester.lambda$testComputeIfPresent_nullTreatedAsAbsent$3(MapComputeIfPresentTester.java:99)
    at it.unimi.dsi.fastutil.objects.Object2ObjectOpenHashMap.computeIfPresent(Object2ObjectOpenHashMap.java:495)

junit.framework.AssertionFailedError: Should not call merge function if value was null
    at com.google.common.collect.testing.testers.MapMergeTester.lambda$testMergeNullValue$6(MapMergeTester.java:158)
    at it.unimi.dsi.fastutil.objects.Object2ObjectOpenHashMap.merge(Object2ObjectOpenHashMap.java:540)
    at com.google.common.collect.testing.testers.MapMergeTester.testMergeNullValue(MapMergeTester.java:154)

[vigna]

These seem to be legit catches, albeit I cannot really understand how the first can happen. Are those generated by the code above?

[ben-manes]

Yep. I am not sure how you want to treat nulls, as it triggers different tests. There are suites for lists, sets, etc that you might want to leverage. It’s a very handy test kit.

[vigna]

Are those test builders in the main Guava jar? From which version?

[vigna]

Ah OK just found guava-testlib. Checking...

[vigna]

With Guava 31 the code does not compile for me—the methods need to return TestSuite, not Test. Is that correct?

[ben-manes]

I am using 31.0.1-jre. This is JUnit3-based where TestSuite implements Test, so it should be equivalent (JUnit 4.13.2 dependency). I ran it in Eclipse.

[ben-manes]

Attached is a minimal gradle project: fastutil-testlib.zip

./gradlew clean test --console plain

$ ./gradlew clean test --console plain
> Task :clean
> Task :compileJava NO-SOURCE
> Task :processResources NO-SOURCE
> Task :classes UP-TO-DATE
> Task :compileTestJava
> Task :processTestResources NO-SOURCE
> Task :testClasses

> Task :test FAILED

fastutil.testlib.FastUtilMapTest > com.google.common.collect.testing.testers.MapMergeTester.testMergeNullValue[Object2ObjectOpenHashMap [collection size: zero]] FAILED
    junit.framework.AssertionFailedError at MapMergeTester.java:160

fastutil.testlib.FastUtilMapTest > com.google.common.collect.testing.testers.CollectionToStringTester.testToString_size0[Object2ObjectOpenHashMap [collection size: zero] entrySet [collection size: zero]] FAILED
    junit.framework.ComparisonFailure at CollectionToStringTester.java:49

fastutil.testlib.FastUtilMapTest > com.google.common.collect.testing.testers.CollectionToStringTester.testToString_size0[Object2ObjectOpenHashMap [collection size: zero] keys [collection size: zero]] FAILED
    junit.framework.ComparisonFailure at CollectionToStringTester.java:49

fastutil.testlib.FastUtilMapTest > com.google.common.collect.testing.testers.CollectionToStringTester.testToString_size0[Object2ObjectOpenHashMap [collection size: zero] values [collection size: zero]] FAILED
    junit.framework.ComparisonFailure at CollectionToStringTester.java:49

fastutil.testlib.FastUtilMapTest > com.google.common.collect.testing.testers.MapComputeIfPresentTester.testComputeIfPresent_nullTreatedAsAbsent[Object2ObjectOpenHashMap [collection size: one]] FAILED
    junit.framework.AssertionFailedError at MapComputeIfPresentTester.java:99

fastutil.testlib.FastUtilMapTest > com.google.common.collect.testing.testers.MapMergeTester.testMappedToNull[Object2ObjectOpenHashMap [collection size: one]] FAILED
    java.lang.ArrayIndexOutOfBoundsException at MapMergeTester.java:69

fastutil.testlib.FastUtilMapTest > com.google.common.collect.testing.testers.MapMergeTester.testMergeNullValue[Object2ObjectOpenHashMap [collection size: one]] FAILED
    junit.framework.AssertionFailedError at MapMergeTester.java:158

fastutil.testlib.FastUtilMapTest > com.google.common.collect.testing.testers.MapToStringTester.testToString_size1[Object2ObjectOpenHashMap [collection size: one]] FAILED
    junit.framework.ComparisonFailure at MapToStringTester.java:57

fastutil.testlib.FastUtilMapTest > com.google.common.collect.testing.testers.MapToStringTester.testToStringWithNullKey[Object2ObjectOpenHashMap [collection size: one]] FAILED
    junit.framework.ComparisonFailure at MapToStringTester.java:78

fastutil.testlib.FastUtilMapTest > com.google.common.collect.testing.testers.MapToStringTester.testToString_formatting[Object2ObjectOpenHashMap [collection size: one]] FAILED
    junit.framework.ComparisonFailure at MapToStringTester.java:78

fastutil.testlib.FastUtilMapTest > com.google.common.collect.testing.testers.MapToStringTester.testToStringWithNullValue[Object2ObjectOpenHashMap [collection size: one]] FAILED
    junit.framework.ComparisonFailure at MapToStringTester.java:78

fastutil.testlib.FastUtilMapTest > com.google.common.collect.testing.testers.CollectionToStringTester.testToString_size1[Object2ObjectOpenHashMap [collection size: one] entrySet [collection size: one]] FAILED
    junit.framework.ComparisonFailure at CollectionToStringTester.java:55

fastutil.testlib.FastUtilMapTest > com.google.common.collect.testing.testers.CollectionToStringTester.testToString_size1[Object2ObjectOpenHashMap [collection size: one] keys [collection size: one]] FAILED
    junit.framework.ComparisonFailure at CollectionToStringTester.java:55

fastutil.testlib.FastUtilMapTest > com.google.common.collect.testing.testers.CollectionToStringTester.testToString_size1[Object2ObjectOpenHashMap [collection size: one] values [collection size: one]] FAILED
    junit.framework.ComparisonFailure at CollectionToStringTester.java:55

fastutil.testlib.FastUtilMapTest > com.google.common.collect.testing.testers.MapComputeIfPresentTester.testComputeIfPresent_nullTreatedAsAbsent[Object2ObjectOpenHashMap [collection size: several]] FAILED
    junit.framework.AssertionFailedError at MapComputeIfPresentTester.java:99

fastutil.testlib.FastUtilMapTest > com.google.common.collect.testing.testers.MapMergeTester.testMappedToNull[Object2ObjectOpenHashMap [collection size: several]] FAILED
    java.lang.ArrayIndexOutOfBoundsException at MapMergeTester.java:69

fastutil.testlib.FastUtilMapTest > com.google.common.collect.testing.testers.MapMergeTester.testMergeNullValue[Object2ObjectOpenHashMap [collection size: several]] FAILED
    junit.framework.AssertionFailedError at MapMergeTester.java:158

fastutil.testlib.FastUtilMapTest > com.google.common.collect.testing.testers.MapToStringTester.testToStringWithNullKey[Object2ObjectOpenHashMap [collection size: several]] FAILED
    junit.framework.ComparisonFailure at MapToStringTester.java:78

fastutil.testlib.FastUtilMapTest > com.google.common.collect.testing.testers.MapToStringTester.testToString_formatting[Object2ObjectOpenHashMap [collection size: several]] FAILED
    junit.framework.ComparisonFailure at MapToStringTester.java:78

fastutil.testlib.FastUtilMapTest > com.google.common.collect.testing.testers.MapToStringTester.testToStringWithNullValue[Object2ObjectOpenHashMap [collection size: several]] FAILED
    junit.framework.ComparisonFailure at MapToStringTester.java:78

955 tests completed, 20 failed

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':test'.
> There were failing tests. See the report at: file:///Users/ben/projects/fastutil-testlib/build/reports/tests/test/index.html

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 2s
3 actionable tasks: 3 executed

[vigna]

Great, the OutOfBoundException is a genuine algorithmic bug in fastutil—we haven't seen those for a while. I have to dig deeper into this! In this case, there is an || value[null] that will let a positive value of pos inside the if, which is not what you want... thanks for the suggestion.

[vigna]

I squeezed a lot of small bugs—great library, just horribly documented. But with some trial-and-error it worked. I have covered 5 classes, and slowly I'll add more. If you want add something yourself, please feel free...

[ben-manes]

That's awesome. I didn't even notice the suppressing option! definitely too many hidden gems.