search

jqwik-team / jqwik

Property-Based Testing on the JUnit Platform
http://jqwik.net
Eclipse Public License 2.0
578 stars 64 forks source link

Issue with multiple lists from suppliers #438

Closed jtraglia closed 1 year ago

jtraglia commented 1 year ago

Issue

Hello 👋 I've run into an issue with having multiple lists from suppliers. I've tried to create a simpler proof of concept, but for some reason it doesn't seem to trigger the bug. So I'll explain using the test I discovered this with. If this isn't enough information, let me know and I'll try to provide as much as I can. Also, just want to say thanks for this library. I've really enjoyed using it.

When I run this property test:

@Property
void verifyKZGCommitmentsAgainstTransactionsThrowsExpected(
    @ForAll final List<@From(supplier = TransactionSupplier.class) Transaction> transactions,
    @ForAll final List<@From(supplier = KZGCommitmentSupplier.class) KZGCommitment> commitments) {
  try {
    transactions.forEach(t -> System.out.println(t.getClass()));
    commitments.forEach(k -> System.out.println(k.getClass()));
    miscHelpers.verifyKZGCommitmentsAgainstTransactions(transactions, commitments);
  } catch (Exception e) {
    assertThat(e)
        .isInstanceOfAny(
            IllegalArgumentException.class,
            IndexOutOfBoundsException.class);
  }
}

It will fail with the following output. When printing getClass() for each item in both lists, it encounters a ClassCastException when iterating over the commitments list. It seems that the commitments argument is actually a list of Transaction objects.

timestamp = 2022-12-15T15:03:15.502988, MiscHelpersEip4844PropertyTest:verifyKZGCommitmentsAgainstTransactionsThrowsExpected = 
  java.lang.AssertionError:
    Expecting actual throwable to be an instance of any of the following types:
      [java.lang.IllegalArgumentException, java.lang.IndexOutOfBoundsException]
    but was:
      java.lang.ClassCastException: class tech.pegasys.teku.spec.datastructures.execution.Transaction cannot be cast to class tech.pegasys.teku.kzg.KZGCommitment (tech.pegasys.teku.spec.datastructures.execution.Transaction and tech.pegasys.teku.kzg.KZGCommitment are in unnamed module of loader 'app')
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
        at tech.pegasys.teku.spec.logic.versions.eip4844.helpers.MiscHelpersEip4844PropertyTest.verifyKZGCommitmentsAgainstTransactionsThrowsExpected(MiscHelpersEip4844PropertyTest.java:84)
        at jdk.internal.reflect.GeneratedMethodAccessor82.invoke(Unknown Source)
        ...(56 remaining lines not displayed - this can be changed with Assertions.setMaxStackTraceElementsDisplayed)

                              |-----------------------jqwik-----------------------
tries = 11                    | # of calls to property
checks = 11                   | # of not rejected calls
generation = RANDOMIZED       | parameters are randomly generated
after-failure = SAMPLE_FIRST  | try previously failed sample, then previous seed
when-fixed-seed = ALLOW       | fixing the random seed is allowed
edge-cases#mode = MIXIN       | edge cases are mixed in
edge-cases#total = 100        | # of all combined edge cases
edge-cases#tried = 0          | # of edge cases tried in current run
seed = 5833293952362977072    | random seed to reproduce generated values

Shrunk Sample (17 steps)
------------------------
  arg0: []
  arg1: [SszByteList{0x73d51abbd89cb8196f0efb6892f94d68fccc2c35f0b846}]

This property test method can be found here on GitHub:

The KZGCommitmentSupplier can be found here:

And the TransactionSupplier can be found here:

Also, if it's useful, here's a link to CircleCI where this issue happened (note I needed to scroll up just a bit to see the issue):

jlink commented 1 year ago

@jtraglia On which jqwik version is that happening?

jtraglia commented 1 year ago

We are using v1.7.1.

jlink commented 1 year ago

I slightly modified the property to check that the correct types of lists are being generated:

@Property(
    seed = "-7137675076485900209",
    shrinking = ShrinkingMode.OFF
)
void verifyKZGCommitmentsAgainstTransactionsThrowsExpected(
    @ForAll final List<@From(supplier = TransactionSupplier.class) Transaction> transactions,
    @ForAll final List<@From(supplier = KZGCommitmentSupplier.class) KZGCommitment> commitments
) {

    assertThat(transactions).allMatch(s -> s instanceof Transaction);
    assertThat(commitments).allMatch(i -> i instanceof KZGCommitment);

    try {
      miscHelpers.verifyKZGCommitmentsAgainstTransactions(transactions, commitments);
    } catch (Exception e) {
      assertThat(e)
          .isInstanceOfAny(IllegalArgumentException.class, IndexOutOfBoundsException.class);
    }
}

The result I got is:

MiscHelpersEip4844PropertyTest:verifyKZGCommitmentsAgainstTransactionsThrowsExpected = 
  java.lang.AssertionError:
    Expecting actual throwable to be an instance of any of the following types:
      [java.lang.IllegalArgumentException, java.lang.IndexOutOfBoundsException]
    but was:
      java.lang.ArithmeticException: BigInteger out of int range
        at java.base/java.math.BigInteger.intValueExact(BigInteger.java:4849)
        at org.apache.tuweni.units.bigints.UInt32Value.intValue(UInt32Value.java:298)
        at tech.pegasys.teku.spec.logic.versions.eip4844.helpers.MiscHelpersEip4844.txPeekBlobVersionedHashes(MiscHelpersEip4844.java:105)
        ...(69 remaining lines not displayed - this can be changed with Assertions.setMaxStackTraceElementsDisplayed)

                              |-----------------------jqwik-----------------------
tries = 1                     | # of calls to property
checks = 1                    | # of not rejected calls
generation = RANDOMIZED       | parameters are randomly generated
after-failure = SAMPLE_FIRST  | try previously failed sample, then previous seed
when-fixed-seed = ALLOW       | fixing the random seed is allowed
edge-cases#mode = MIXIN       | edge cases are mixed in
edge-cases#total = 100        | # of all combined edge cases
edge-cases#tried = 12         | # of edge cases tried in current run
seed = -7137675076485900209   | random seed to reproduce generated values

Sample
------
  arg0:
    [
      SszByteList{0x9b1167527b8c0771f38b}, SszByteList{0x38cac155ce1bfb0fbbcd}, 
      SszByteList{0x73d51abbd89cb8196f0efb6892f94d68fccc2c35f0b846}, 
      SszByteList{0x133038bbb8225cc1a5bff68f704de766ddbd315b61cd7a}, SszByteList{0xc68dd344636e66b5}, 
      SszByteList{0x7ea4c8ba98bfeeddd9b7a0c824756d9f54f71e31e9b281}, SszByteList{0xb8aeed3bd30cd5}, 
      SszByteList{0x66e8f04443f4095d}, SszByteList{0x1a0d6890f20e5887526695844efbd871a681}, 
      SszByteList{0x05a3d4d5848e2d9e049410276205310716aee53d4214a1565097}, 
      SszByteList{0xac8a99afecb069ae0520f0881e371337cb9ca47740}, SszByteList{0xc79d0b4e33adc18ef5}, 
      SszByteList{0x701a389ade3a86e41e206889056b542dfdbda3}, 
      SszByteList{0x54e572bb792ea3101122eedd2bf519649f9f3ba741f6e2}, 
      SszByteList{0x4d78de5d0fd8e7ecfbc411}, 
      SszByteList{0x96ff3aade0556eddfd4b04fcb555d511c1f2d63cde}, SszByteList{0xeaba2750b77f82ced42d}, 
      SszByteList{0x69b53f41dc8378ad}, SszByteList{0xba669c438016a1f7}, 
      SszByteList{0x0bdeebbad8f9b240192635c42f40f2d02ee524c5a3fe8c}, SszByteList{0x62b47f306211}, 
      SszByteList{0xb36cd944c3228770}, SszByteList{0x4c9326bb9805fa8f85882c12eae724cef0c62e118427f5}, 
      SszByteList{0xcddf1f4543970f36}, 
      SszByteList{0x8ebe66e9b6d98ea23d5c4b8be15665279171066706712285ef38d80f94}, 
      SszByteList{0xfe40af41fde6e590}, SszByteList{0x4d7bcfa3e337f28c8519bb6eb22779bd5aecd623}, 
      SszByteList{0x56d434eba031384f6944a5d0b9a18ffc42d6d377579e1c7a0ac95669c1}, 
      SszByteList{0x08618abbf9ff25fd3c165130ddd1c72f86933f5f68d33f}, 
      SszByteList{0x501088ba77ff86d3629810a63c564f6f99181477bf5242}, 
      SszByteList{0xa715658fcba939e04394cb8f36b1304b83}, SszByteList{0x73553561d81c01a26f8ed992}
    ]
  arg1:
    [
      0xb36cd944c3228770172024039eff4b8c695e7ea94cbbcd999ab39b6a0cc53766ca9b7fd3f638790e007558ce41eb4c4f,
      0xad47b0a2b5017d25566108fd89203c3b8943788b442a15d5690e73246cfd6e86a571179a6584a151e30e792a1dc26e2a,
      0xf6e15e5199d7478c0e51e770412c802615eb994764a6d18efc2fa24a4a331b8b763af0b0d062eab517ad3e676dd47cee,
      0x51010822744e7c34256ee7ac2ad98fc15034eca9a4c3ea3244c17a13e28d800e699e53689c157adb5be3b19be571c269,
      0x73d51abbd89cb8196f0efb6892f94d68fccc2c35f0b84609e5f12c55dd85aba8d5d9bef76808f3b572e5900112b81927,
      0x9f4bdf4423d7a72b21ddbcd7cb7637bf635b7f1795f2a4df6c3258a2326480db337cb974fcd79820b95c15227a23e968,
      0x53c7f644a3a82a184dd11f2a7c53e58a4a4f83cfbccf01f7b52d4a81c8e0a2b0d4fda0f9125417699cfb08715d045acf,
      0x6c758a4df22cf27908ff43c14f6ee5ee29e4f48457956cffd774abeebf61bb085bd62e8430a649e38e8303c75d279439,
      0x8c2ae544838bc8e62c9a55acf7ed22f25c588085df297c253eb114da5803c9509b5cf3150177b8327244d275b25b8582,
      0x9f4bdf4423d7a72b21ddbcd7cb7637bf635b7f1795f2a4df6c3258a2326480db337cb974fcd79820b95c15227a23e968,
      0xb1e68e37099785bf3b1a75bcaa2127d31f249c7b29277566a6e1f4df71a6ce804b18048d1dc04436c242862e49e9de0a,
      0x13f8f95be4224bbb787cae1f12c5304eda42a6144043e0318978c9344e2e96d165b74b5147f78f59c05207305614e428,
      0xf80d520b5e3a80f0d382e940c785627da874ca295805cccf8395779d36ab354b65ce7b3911eb5e6011ae6b27013fd9dc,
      0xf72e242738f08fb099dfb9af1b0b6c73fd245c896225c144a1c0ee02313314f4167bcfe7aa67da39bc1eddfa9155e870,
      0x922d1c48c1213a8e5618dbf093d95c6d1c68c8192725d511be115666eb5446caeaa5a15bcf9f5e64497a810b45694f73,
      0x4f4a9545c4ae9dd46fc13b962ae5bae9a1fd9d6981a4b454e18d2b62c0a94b0fd7a8bbfaa8196d52196cf5455bf2d382,
      0x42ab70224f7f5fd7fa7d6a71939cddacfe90917267df69574832fd6010c2ec087bc776cc371088b6c58f97ec1350cfee,
      0x54e572bb792ea3101122eedd2bf519649f9f3ba741f6e2203760389912d9ec87f3ff226abb59cdc54579a2e9660341a7,
      0x04bcdc6e0df2919226f52c6826d558f1b1b990e06662f1651044874747aaeab5ec86ad99ba38c3c04cb7320ed3413f95,
      0x97f1f63fda0e51be21d60e20b3d7612e9ce2ab2203bbbc7daf2ec60abf5acbd0ede457ae57e929fa43a82b22fc235200,
      0x39722cbbf8b91a4b9045c5e6175f1001eac32f7fcd5eccda5c6e62fc4e6385080f7b6cdb79e551ec9c9cc7fcbd60ee73,
      0x43855ff1be58d0a3a65164fc937aea5e0b6d0eabeb3ba2544e79d82749b5fd62be628dee9964c704d0a408c9abb5c86b,
      0xeeafc2995d23f845465b79ed8e1bef327afa8f6b98b4ea5cdaa394c42f9e3386dc480641d8d659c6fbc9a4666465d2b1,
      0x58e9a61b43f08e7986e9cfa045d19bcc38106d413df29196348f0aef90bef46cc44e8b61ca31f8c11fa22b1a624e2e52
    ]

java.lang.AssertionError: 
Expecting actual throwable to be an instance of any of the following types:
  [java.lang.IllegalArgumentException, java.lang.IndexOutOfBoundsException]
but was:
  java.lang.ArithmeticException: BigInteger out of int range
    at java.base/java.math.BigInteger.intValueExact(BigInteger.java:4849)
    at org.apache.tuweni.units.bigints.UInt32Value.intValue(UInt32Value.java:298)
    at tech.pegasys.teku.spec.logic.versions.eip4844.helpers.MiscHelpersEip4844.txPeekBlobVersionedHashes(MiscHelpersEip4844.java:105)
    ...(69 remaining lines not displayed - this can be changed with Assertions.setMaxStackTraceElementsDisplayed)

    at tech.pegasys.teku.spec.logic.versions.eip4844.helpers.MiscHelpersEip4844PropertyTest.verifyKZGCommitmentsAgainstTransactionsThrowsExpected(MiscHelpersEip4844PropertyTest.java:91)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
    at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727)
    at org.junit.platform.commons.support.ReflectionSupport.invokeMethod(ReflectionSupport.java:198)
    at net.jqwik.engine.execution.CheckedPropertyFactory.lambda$createRawFunction$1(CheckedPropertyFactory.java:84)
    at net.jqwik.engine.execution.CheckedPropertyFactory.lambda$createRawFunction$2(CheckedPropertyFactory.java:91)
    at net.jqwik.engine.properties.CheckedFunction.execute(CheckedFunction.java:17)
    at net.jqwik.api.lifecycle.AroundTryHook.lambda$static$0(AroundTryHook.java:57)
    at net.jqwik.engine.execution.lifecycle.HookSupport.lambda$wrap$2(HookSupport.java:48)
    at net.jqwik.engine.hooks.lifecycle.TryLifecycleMethodsHook.aroundTry(TryLifecycleMethodsHook.java:57)
    at net.jqwik.engine.execution.lifecycle.HookSupport.lambda$wrap$3(HookSupport.java:53)
    at net.jqwik.engine.execution.CheckedPropertyFactory.lambda$createTryExecutor$0(CheckedPropertyFactory.java:60)
    at net.jqwik.engine.execution.lifecycle.AroundTryLifecycle.execute(AroundTryLifecycle.java:23)
    at net.jqwik.engine.properties.GenericProperty.testPredicate(GenericProperty.java:166)
    at net.jqwik.engine.properties.GenericProperty.check(GenericProperty.java:68)
    at net.jqwik.engine.execution.CheckedProperty.check(CheckedProperty.java:67)
    at net.jqwik.engine.execution.PropertyMethodExecutor.executeProperty(PropertyMethodExecutor.java:90)
    at net.jqwik.engine.execution.PropertyMethodExecutor.executeMethod(PropertyMethodExecutor.java:69)
    at net.jqwik.engine.execution.PropertyMethodExecutor.lambda$execute$0(PropertyMethodExecutor.java:49)
    at net.jqwik.api.lifecycle.AroundPropertyHook.lambda$static$0(AroundPropertyHook.java:46)
    at net.jqwik.engine.execution.lifecycle.HookSupport.lambda$wrap$0(HookSupport.java:26)
    at net.jqwik.engine.hooks.lifecycle.PropertyLifecycleMethodsHook.aroundProperty(PropertyLifecycleMethodsHook.java:57)
    at net.jqwik.engine.execution.lifecycle.HookSupport.lambda$wrap$1(HookSupport.java:31)
    at net.jqwik.engine.execution.lifecycle.HookSupport.lambda$wrap$0(HookSupport.java:26)
    at net.jqwik.engine.hooks.statistics.StatisticsHook.aroundProperty(StatisticsHook.java:37)
    at net.jqwik.engine.execution.lifecycle.HookSupport.lambda$wrap$1(HookSupport.java:31)
    at net.jqwik.engine.execution.lifecycle.HookSupport.lambda$wrap$0(HookSupport.java:26)
    at net.jqwik.engine.hooks.lifecycle.AutoCloseableHook.aroundProperty(AutoCloseableHook.java:13)
    at net.jqwik.engine.execution.lifecycle.HookSupport.lambda$wrap$1(HookSupport.java:31)
    at net.jqwik.engine.execution.PropertyMethodExecutor.execute(PropertyMethodExecutor.java:47)
    at net.jqwik.engine.execution.PropertyTaskCreator.executeTestMethod(PropertyTaskCreator.java:166)
    at net.jqwik.engine.execution.PropertyTaskCreator.lambda$createTask$1(PropertyTaskCreator.java:51)
    at net.jqwik.engine.execution.lifecycle.CurrentDomainContext.runWithContext(CurrentDomainContext.java:28)
    at net.jqwik.engine.execution.PropertyTaskCreator.lambda$createTask$2(PropertyTaskCreator.java:50)
    at net.jqwik.engine.execution.pipeline.ExecutionTask$1.lambda$execute$0(ExecutionTask.java:31)
    at net.jqwik.engine.execution.lifecycle.CurrentTestDescriptor.runWithDescriptor(CurrentTestDescriptor.java:17)
    at net.jqwik.engine.execution.pipeline.ExecutionTask$1.execute(ExecutionTask.java:31)
    at net.jqwik.engine.execution.pipeline.ExecutionPipeline.runToTermination(ExecutionPipeline.java:82)
    at net.jqwik.engine.execution.JqwikExecutor.execute(JqwikExecutor.java:46)
    at net.jqwik.engine.JqwikTestEngine.executeTests(JqwikTestEngine.java:70)
    at net.jqwik.engine.JqwikTestEngine.execute(JqwikTestEngine.java:53)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:147)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:127)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:90)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:55)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:102)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:54)
    at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
    at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
    at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
    at org.junit.platform.launcher.core.SessionPerRequestLauncher.execute(SessionPerRequestLauncher.java:53)
    at com.intellij.junit5.JUnit5IdeaTestRunner.startRunnerWithArgs(JUnit5IdeaTestRunner.java:57)
    at com.intellij.rt.junit.IdeaTestRunner$Repeater$1.execute(IdeaTestRunner.java:38)
    at com.intellij.rt.execution.junit.TestsRepeater.repeat(TestsRepeater.java:11)
    at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:35)
    at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:235)
    at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:54)

This suggests to me that the stacktrace you got may be a result of (erroneous) shrinking and the original failure is due to a BigInteger out of int range.

I'll try again, maybe there is more than one problem hiding here.

jlink commented 1 year ago

I ran it 10 times with different random seeds. Each time the original exception was an ArithmeticException. The supplier implementations are too involved for me to really understand how they work by just looking at them. They might have some hidden or global state underneath which confuses the shrinker. Your CI log suggests the same thing since the original exception there is also java.lang.ArithmeticException: BigInteger out of int range

So, if there is an underlying jqwik bug - which is absolutely possible - it's probably a bug in shrinking. And you have to get rid of the bug leading to "BigInteger out of int range" ;-)

jlink commented 1 year ago

Looks like I found the underlying shrinking problem. It's really esoteric, so I won't bother you with the details. It only shows up in your case, because you catch all exceptions and then sort out some specific ones. If you change the code to:

void verifyKZGCommitmentsAgainstTransactionsThrowsExpected(
    @ForAll final List<@From(supplier = TransactionSupplier.class) Transaction> transactions,
    @ForAll final List<@From(supplier = KZGCommitmentSupplier.class) KZGCommitment> commitments
) {
    try {
      miscHelpers.verifyKZGCommitmentsAgainstTransactions(transactions, commitments);
    } catch (IllegalArgumentException| IndexOutOfBoundsException e) {
        // expected
    }
}

the original exception should be kept in order during shrinking as well.

And many thanks for taking the time to report the finding!

jlink commented 1 year ago

I cover the shrinking bug in this new issue: https://github.com/jlink/jqwik/issues/439

I'll probably not fix the bug in the near future, because it requires more effort than I can currently afford.

You could also just rethrow the exception if it's not in the list of expected ones. That would also solve the problem for you.

jtraglia commented 1 year ago

Of course! Thanks for helping me figure this out 💯