Closed alisevych closed 1 year ago
I am happy to discuss about clusters for fuzzing tests bazed on the utExecution.result.
At this moment all unsuccessful fuzzing execution are grouped to the one cluster with the name "EXPLICITLY_THROWN_UNCHECKED_EXCEPTIONS", probably the common name is incorrect
Also, I could create a separate clusters for all types of utExecution.result
If it collects exceptions of any kind, I'd prefer name like "EXCEPTIONS_THROWN" without any details about checked/unchecked.
If it collects exceptions of any kind, I'd prefer name like "EXCEPTIONS_THROWN" without any details about checked/unchecked.
We should at least differentiate Error suite from regression suite
As a suggestion: EXPECTED_EXCEPTIONS UNEXPECTED_EXCEPTIONS
@alisevych what do you mean under EXPECTED_EXCEPTIONS and UNEXPECTED_EXCEPTIONS from the following list? Is it the same as the proposed by @korifey the splitting of exceptions and errors (failures)?
Am I right?
@korifey is it correct to say that all result types could be grouped in Errors and Exceptions?
Exceptions: UtExplicitlyThrownException UtImplicitlyThrownException UtTimeoutException
Failures: UtOverflowFailure UtSandboxFailure UtConcreteExecutionFailure
@korifey @Markoutte @alisevych
Also as a varinat we could unify it with the clusters produced by the Symbolic Engine. It has the following logic based on type of exceptions (checked/unchecked)
is UtExecutionSuccess -> ExecutionGroup.SUCCESSFUL_EXECUTIONS
is UtImplicitlyThrownException -> if (this.exception.isCheckedException) ExecutionGroup.CHECKED_EXCEPTIONS else ExecutionGroup.ERROR_SUITE
is UtExplicitlyThrownException -> if (this.exception.isCheckedException) ExecutionGroup.CHECKED_EXCEPTIONS else ExecutionGroup.EXPLICITLY_THROWN_UNCHECKED_EXCEPTIONS
is UtOverflowFailure -> ExecutionGroup.OVERFLOWS
is UtTimeoutException -> ExecutionGroup.TIMEOUTS
is UtConcreteExecutionFailure -> ExecutionGroup.CRASH_SUITE
is UtSandboxFailure -> ExecutionGroup.SECURITY
@amandelpie @Markoutte @korifey Please correct me if needed:
As for Overflow exceptions - they should be treated as errors or ignored - based on the setting: Also , what is meant by "ignored": just skipped?
From my point of view implicitly thrown (by other classes/packages) and explicitly thrown (by user code) - should be in different suites. Implicitly - possible errors. Should be added to error suite (?) Explicitly - expected ones. Should be treated as regression suite - add to successful executions (?)
Sandbox failures should be shown separately - with distinct explanation how to add permissions and regenerate the tests if needed.
Timeout exceptions should be added to error suite.
Concrete execution failures - are UnitTestBot issues. Should be shown in error suite or separately.
@rudolf101 @zishkaz Currently generated test cases for Java are divided into the following regions:
<Source name> : <Suite name> for method <Method signature>
You can find some details in PR - #1250.
Also there is Errors report
region to collect internal errors from concrete execution (only count, full details should be in logs).
Let's consider and discuss an appropriate solution for JavaScript test cases division into suites and their naming.
Description
Clarify scope of generated suite with EXPLICITLY_THROWN_UNCHECKED_EXCEPTIONS.
Expected behavior
Is it correct that only explicitly thrown exceptions without try-catch should be located in this suite?
Environment
Fuzzing is on.
Context
Example:
Test with String.repeat(-1) throws [java.lang.IllegalArgumentException: count is negative: -1]
Expected result
The test should be put into FUZZING: IMPLICITLY_THROWN_UNCHECKED_EXCEPTIONS region.
Actual result
The test is currently put into FUZZING: EXPLICITLY_THROWN_UNCHECKED_EXCEPTIONS region.
Originally posted by @alisevych in https://github.com/UnitTestBot/UTBotJava/issues/1102#issuecomment-1269877351