jqwik-team / jqwik

Property-Based Testing on the JUnit Platform
http://jqwik.net
Eclipse Public License 2.0
578 stars 64 forks source link

Add @CheckReturnValue to methods that return modified values (e.g. methods on Arbitrary) #422

Closed vlsi closed 1 year ago

vlsi commented 1 year ago

Testing Problem

See https://youtrack.jetbrains.com/issue/IDEA-265263/Generalise-Stream-missing-terminal-operation-inspections-to-warn-about-missing-calls-to-terminal-operations-of-third-party#focus=Comments-27-4804303.0-0

Frankly speaking, I would expect that methods named like addAction would mutate the object, however, it turns out addAction does not really mutate the chain.

fun <S> actionChains(
    init: () -> S,
    vararg actions: Pair<Int, Action<S>>
): ActionChainArbitrary<S> =
    ActionChain.startWith(init).apply {
        actions.forEach { (weight, action) ->
            addAction(weight, action)
        }
    }

It turns out addAction adds nothing, and the generated chain fails quite surprisingly in the runtime.

net.jqwik.api.JqwikException: [] does not contain any positive frequencies.
    at app//net.jqwik.engine.support.ChooseRandomlyByFrequency.<init>(ChooseRandomlyByFrequency.java:17)
    at app//net.jqwik.engine.properties.state.DefaultChainArbitrary.generator(DefaultChainArbitrary.java:26)
    at app//net.jqwik.api.Arbitrary.generatorWithEmbeddedEdgeCases(Arbitrary.java:113)
    at app//net.jqwik.engine.properties.arbitraries.ArbitraryMap.generatorWithEmbeddedEdgeCases(ArbitraryMap.java:25)
    at app//net.jqwik.api.arbitraries.ArbitraryDecorator.generatorWithEmbeddedEdgeCases(ArbitraryDecorator.java:47)
    at app//net.jqwik.engine.facades.ArbitraryFacadeImpl.generator(ArbitraryFacadeImpl.java:191)
    at app//net.jqwik.engine.facades.ArbitraryFacadeImpl.lambda$memoizedGenerator$0(ArbitraryFacadeImpl.java:185)
    at app//net.jqwik.engine.facades.Memoize.memoizedGenerator(Memoize.java:25)
    at app//net.jqwik.engine.facades.ArbitraryFacadeImpl.memoizedGenerator(ArbitraryFacadeImpl.java:185)
    at app//net.jqwik.api.Arbitrary.generator(Arbitrary.java:96)
    at app//net.jqwik.engine.properties.RandomizedParameterGenerator.getGenerator(RandomizedParameterGenerator.java:40)
    at app//net.jqwik.engine.properties.RandomizedParameterGenerator.selectGenerator(RandomizedParameterGenerator.java:36)
    at app//net.jqwik.engine.properties.RandomizedParameterGenerator.next(RandomizedParameterGenerator.java:24)
    at app//net.jqwik.engine.properties.PurelyRandomShrinkablesGenerator.lambda$generateNext$0(PurelyRandomShrinkablesGenerator.java:21)
    at java.base@11.0.13/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
    at java.base@11.0.13/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)
    at java.base@11.0.13/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
    at java.base@11.0.13/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
    at java.base@11.0.13/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
    at java.base@11.0.13/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
    at java.base@11.0.13/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
    at app//net.jqwik.engine.properties.PurelyRandomShrinkablesGenerator.generateNext(PurelyRandomShrinkablesGenerator.java:22)
    at app//net.jqwik.engine.properties.RandomizedShrinkablesGenerator.next(RandomizedShrinkablesGenerator.java:214)
    at app//net.jqwik.engine.properties.RandomizedShrinkablesGenerator.next(RandomizedShrinkablesGenerator.java:17)
    at app//net.jqwik.engine.execution.ResolvingParametersGenerator.next(ResolvingParametersGenerator.java:36)
    at app//net.jqwik.engine.properties.GenericProperty.check(GenericProperty.java:57)
    at app//net.jqwik.engine.execution.CheckedProperty.check(CheckedProperty.java:67)
    at app//net.jqwik.engine.execution.PropertyMethodExecutor.executeProperty(PropertyMethodExecutor.java:90)
    at app//net.jqwik.engine.execution.PropertyMethodExecutor.executeMethod(PropertyMethodExecutor.java:69)
    at app//net.jqwik.engine.execution.PropertyMethodExecutor.lambda$execute$0(PropertyMethodExecutor.java:49)
    at app//net.jqwik.api.lifecycle.AroundPropertyHook.lambda$static$0(AroundPropertyHook.java:46)
    at app//net.jqwik.engine.execution.lifecycle.HookSupport.lambda$wrap$0(HookSupport.java:26)
    at app//com.netcracker.platform.rdbl.test.fuzz.jqwik.PropertyTimeoutHook.aroundProperty(PropertyTimeoutHook.kt:36)
    at app//net.jqwik.engine.execution.lifecycle.HookSupport.lambda$wrap$1(HookSupport.java:31)
    at app//net.jqwik.engine.execution.lifecycle.HookSupport.lambda$wrap$0(HookSupport.java:26)
    at app//net.jqwik.api.PropertyDefaults$PropertyDefaultsHook.aroundProperty(PropertyDefaults.java:91)
    at app//net.jqwik.engine.execution.lifecycle.HookSupport.lambda$wrap$1(HookSupport.java:31)
    at app//net.jqwik.engine.execution.lifecycle.HookSupport.lambda$wrap$0(HookSupport.java:26)
    at app//net.jqwik.engine.hooks.lifecycle.PropertyLifecycleMethodsHook.aroundProperty(PropertyLifecycleMethodsHook.java:57)
    at app//net.jqwik.engine.execution.lifecycle.HookSupport.lambda$wrap$1(HookSupport.java:31)
    at app//net.jqwik.engine.execution.lifecycle.HookSupport.lambda$wrap$0(HookSupport.java:26)
    at app//net.jqwik.engine.hooks.statistics.StatisticsHook.aroundProperty(StatisticsHook.java:37)
    at app//net.jqwik.engine.execution.lifecycle.HookSupport.lambda$wrap$1(HookSupport.java:31)
    at app//net.jqwik.engine.execution.lifecycle.HookSupport.lambda$wrap$0(HookSupport.java:26)
    at app//net.jqwik.engine.hooks.lifecycle.AutoCloseableHook.aroundProperty(AutoCloseableHook.java:13)
    at app//net.jqwik.engine.execution.lifecycle.HookSupport.lambda$wrap$1(HookSupport.java:31)
    at app//net.jqwik.engine.execution.PropertyMethodExecutor.execute(PropertyMethodExecutor.java:47)
    at app//net.jqwik.engine.execution.PropertyTaskCreator.executeTestMethod(PropertyTaskCreator.java:166)
    at app//net.jqwik.engine.execution.PropertyTaskCreator.lambda$createTask$1(PropertyTaskCreator.java:51)
    at app//net.jqwik.engine.execution.lifecycle.CurrentDomainContext.runWithContext(CurrentDomainContext.java:28)
    at app//net.jqwik.engine.execution.PropertyTaskCreator.lambda$createTask$2(PropertyTaskCreator.java:50)
    at app//net.jqwik.engine.execution.pipeline.ExecutionTask$1.lambda$execute$0(ExecutionTask.java:31)
    at app//net.jqwik.engine.execution.lifecycle.CurrentTestDescriptor.runWithDescriptor(CurrentTestDescriptor.java:17)
    at app//net.jqwik.engine.execution.pipeline.ExecutionTask$1.execute(ExecutionTask.java:31)
    at app//net.jqwik.engine.execution.pipeline.ExecutionPipeline.runToTermination(ExecutionPipeline.java:82)
    at app//net.jqwik.engine.execution.JqwikExecutor.execute(JqwikExecutor.java:46)
    at app//net.jqwik.engine.JqwikTestEngine.executeTests(JqwikTestEngine.java:70)
    at app//net.jqwik.engine.JqwikTestEngine.execute(JqwikTestEngine.java:53)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:108)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:88)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:54)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:67)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:52)
    at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:96)
    at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:75)
    at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.processAllTestClasses(JUnitPlatformTestClassProcessor.java:99)
    at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.access$000(JUnitPlatformTestClassProcessor.java:79)
    at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor.stop(JUnitPlatformTestClassProcessor.java:75)
    at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.stop(SuiteTestClassProcessor.java:61)
    at java.base@11.0.13/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base@11.0.13/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base@11.0.13/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base@11.0.13/java.lang.reflect.Method.invoke(Method.java:566)
    at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
    at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
    at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
    at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
    at com.sun.proxy.$Proxy2.stop(Unknown Source)
    at org.gradle.api.internal.tasks.testing.worker.TestWorker$3.run(TestWorker.java:193)
    at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129)
    at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100)
    at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60)
    at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
    at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:133)
    at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:71)
    at app//worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
    at app//worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)
vlsi commented 1 year ago

It looks like

  1. ActionChainBuilder would help to make the API easier to use (e.g. the builder might throw exception if no action was added)
  2. @CheckReturnValue might help to mark methods in jqwik that return important values that should not be discarded: https://youtrack.jetbrains.com/issue/KTIJ-7061
jlink commented 1 year ago

All „mutating“ methods of Arbitrary subtypes in jqwik return new instances, so the behaviour is consistent with the rest of jqwik. Would a different method name (eg withAction) be more revealing?

I didn’t know about CheckReturnValue and will have a look at it.

vlsi commented 1 year ago

Well, there's withMaxTransformations, so naming alone is probably not enough.

I guess the issue boils down to that I thought ActionChain.startWith was ChainBuilder while in practice it returns a free-floating arbitrary.

vlsi commented 1 year ago

I just checked, and adding @javax.annotation.CheckReturnValue on top of addAction does help in Java code:

unused addAction produces warning

For Kotlin, the issue is still unresolved. AFAIK the idea is to implement something like that at the language level.

jlink commented 1 year ago

Well, there's withMaxTransformations, so naming alone is probably not enough.

I guess the issue boils down to that I thought ActionChain.startWith was ChainBuilder while in practice it returns a free-floating arbitrary.

A builder usually requires a final build() call, which makes the code more involved. That’s why I try to steer clear of builders where possible.

vlsi commented 1 year ago

A builder usually requires a final build() call, which makes the code more involved

Even though build() is verbose (well, Java is verbose anyway), explicit build() method provides a clear boundary when you consider the object to be fully configured.

In the case of chain arbitrary, it would be better to have validation during build(), so the errors like "please add at least one action, and remember to use the result of .addAction call" closer to the user code. Currently, the exception is thrown in the runtime from the deep internals of jqwik.

vlsi commented 1 year ago

Would a different method name (eg withAction) be more revealing?

It might indeed be a better idea to use withAction for mutation method. For instance, with Arbitrary.filter, it is crystal clear that you expect that the original arbitrary is not modified, and the resulting is filtered. At the same time Arbitrary.addFilter would look like adding a filter to already existing arbitrary.

chain.withAction(..) would indeed suggest that the method produces new chain rather than mutates the old in-place.

jlink commented 1 year ago

Released in 1.7.2-SNAPSHOT

vlsi commented 1 year ago

@jlink , I suggest that type annotations should be in api scope, so they become transitive dependencies.

Even though Java specification says everyone should be ignoring the annotations that are not on the classpath, there are known issues when various compilers (e.g. Scala compiler) observe a classfile with a public method that is annotated with "unknown annotation".

That is basically the reason Guava exposes nullability annotations as transitive dependencies.

See https://github.com/pgjdbc/pgjdbc/issues/2053#issuecomment-775915780

jlink commented 1 year ago

@jlink , I suggest that type annotations should be in api scope, so they become transitive dependencies.

Will consult what others think and write about that. 1.8. at the earliest.

Current plan is to include annotations dependency in samples and in userguide recommendation.