[BUG] Flaky tests high-level tracking - Githubissues

opensearch-project / data-prepper

OpenSearch Data Prepper is a component of the OpenSearch project that accepts, filters, transforms, enriches, and routes data at scale.

https://opensearch.org/docs/latest/clients/data-prepper/index/

Apache License 2.0

265 stars 203 forks source link

[BUG] Flaky tests high-level tracking #3481

Open graytaylor0 opened 1 year ago

graytaylor0 commented 1 year ago

Describe the bug Running the data-prepper build and github actions frequently has failing flaky tests, and often requires retrying the tests multiple times before it succeeds. This issue is meant to allow easy tracking of these flaky tests, by adding an item to the checklist when a flaky test is seen.

Flaky tests

[ ] > Task :data-prepper-core:integrationTest PipelinesWithAcksIT > simple_pipeline_with_multiple_records() FAILED java.lang.NullPointerException at PipelinesWithAcksIT.java:85
[ ] ```testcase name="three_pipelines_with_route_and_multiple_records()" classname="org.opensearch.dataprepper.integration.PipelinesWithAcksIT" time="11.003"> java.lang.NullPointerException at org.opensearch.dataprepper.integration.PipelinesWithAcksIT.three_pipelines_with_route_and_multiple_records(PipelinesWithAcksIT.java:130)```
[x] #3470
[ ] #1374
[ ] Router_ThreeRoutesIT https://github.com/opensearch-project/data-prepper/issues/3481#issuecomment-1793501480
[ ] AwsCloudMapPeerListProviderTest.java:345 - https://github.com/opensearch-project/data-prepper/issues/3481#issuecomment-1795279643
[ ] OTelTraceRawProcessorTest.java:270 - https://github.com/opensearch-project/data-prepper/issues/3481#issuecomment-1796339462
[ ] OTelTraceRawProcessorTest.java:193 - https://github.com/opensearch-project/data-prepper/issues/3481#issuecomment-1806598375
[x] #4298
[ ] HistogramAggregateActionTests.java:370 - https://github.com/opensearch-project/data-prepper/issues/3481#issuecomment-2207244965
[x] LambdaSinkServiceTest.java:316 - https://github.com/opensearch-project/data-prepper/issues/3481#issuecomment-2221378060; fixed by #4723
[ ] RDS ExportSchedulerTest - https://github.com/opensearch-project/data-prepper/issues/3481#issuecomment-2393877779
[ ] RDS StreamScheduler - https://github.com/opensearch-project/data-prepper/issues/3481#issuecomment-2394058183. (added sleep time in #5016, will check again see if this helps)
[ ] Truncate processor - https://github.com/opensearch-project/data-prepper/issues/3481#issuecomment-2405626969

dlvenable commented 1 year ago

> Task :data-prepper-core:integrationTest

Router_ThreeRoutesIT > sending_alpha_and_beta_events_sends_to_sinks_that_take_both(String) > [3] ConditionalRoutingIT_alpha_beta_gamma FAILED
    java.lang.AssertionError at Router_ThreeRoutesIT.java:89

Router_ThreeRoutesIT > sending_alpha_and_beta_events_sends_to_both_sinks() FAILED
    java.lang.AssertionError at Router_ThreeRoutesIT.java:112

dlvenable commented 1 year ago

AwsCloudMapPeerListProviderTest > WithSeveralFailedAttempts > listener_gets_list_after_several_failed_attempts() FAILED
    java.lang.AssertionError at AwsCloudMapPeerListProviderTest.java:345

dlvenable commented 1 year ago

PipelinesWithAcksIT > two_pipelines_with_multiple_records() FAILED
    java.lang.NullPointerException at PipelinesWithAcksIT.java:105

dlvenable commented 1 year ago

OTelTraceRawProcessorTest > traceGroupCacheMaxSize_provides_an_upper_bound(long, int) > [1] 1, 4 FAILED
    java.lang.AssertionError at OTelTraceRawProcessorTest.java:270

dlvenable commented 1 year ago

OTelTraceRawProcessorTest > testExportRequestFlushByMixedMultiThread() FAILED
    org.opentest4j.AssertionFailedError at OTelTraceRawProcessorTest.java:193

dlvenable commented 1 year ago

I saw this error again in a recent build:

Router_ThreeRoutesIT > sending_alpha_and_beta_events_sends_to_sinks_that_take_both(String) > [1] ConditionalRoutingIT_all FAILED
    java.lang.AssertionError at Router_ThreeRoutesIT.java:89

Router_ThreeRoutesIT > sending_alpha_and_beta_events_sends_to_sinks_that_take_both(String) > [3] ConditionalRoutingIT_alpha_beta_gamma FAILED
    java.lang.AssertionError at Router_ThreeRoutesIT.java:89

dlvenable commented 10 months ago

I'm seeing lots of failures in the kafka-plugins project in the Java 21 build.


> Task :data-prepper-plugins:kafka-plugins:test

KafkaBufferTest > test_kafkaBuffer_producerThrows() FAILED
    java.lang.IllegalAccessError at KafkaBufferTest.java:155

KafkaBufferTest > test_kafkaBuffer_isEmpty_BufferNotEmpty() FAILED
    java.lang.NullPointerException at KafkaBufferTest.java:155

KafkaBufferTest > testShutdown_Successful() FAILED
    java.lang.NullPointerException at KafkaBufferTest.java:155

KafkaBufferTest > test_kafkaBuffer_isEmpty_TopicNotEmpty() FAILED
    java.lang.NullPointerException at KafkaBufferTest.java:155

KafkaBufferTest > test_kafkaBuffer_getDrainTimeout() FAILED
    java.lang.NullPointerException at KafkaBufferTest.java:155

KafkaBufferTest > test_kafkaBuffer_doCheckpoint() FAILED
    java.lang.NullPointerException at KafkaBufferTest.java:155

KafkaBufferTest > isWrittenOffHeapOnly_returns_true() FAILED
    java.lang.NullPointerException at KafkaBufferTest.java:155

KafkaBufferTest > test_kafkaBuffer_isEmpty_True() FAILED
    java.lang.NullPointerException at KafkaBufferTest.java:155

KafkaBufferTest > test_kafkaBuffer_basicFunctionality() FAILED
    java.lang.NullPointerException at KafkaBufferTest.java:155

KafkaBufferTest > test_kafkaBuffer_postProcess() FAILED
    java.lang.NullPointerException at KafkaBufferTest.java:155

KafkaBufferTest > test_kafkaBuffer_doWriteAll() FAILED
    java.lang.NullPointerException at KafkaBufferTest.java:155

KafkaBufferTest > testShutdown_Timeout() FAILED
    java.lang.NullPointerException at KafkaBufferTest.java:155

KafkaBufferTest > testShutdown_InterruptedException() FAILED
    java.lang.NullPointerException at KafkaBufferTest.java:155

KafkaBufferTest > test_kafkaBuffer_isEmpty_MultipleTopics_AllNotEmpty() FAILED
    java.lang.NullPointerException at KafkaBufferTest.java:155

KeyFactoryTest > getKeySupplier_returns_using_first_InnerKeyFactory_that_supports_the_TopicConfig() FAILED
    java.lang.NullPointerException at KeyFactoryTest.java:54


> Task :data-prepper-plugins:kafka-plugins:test

EncryptionContextTest > fromEncryptionKey_includes_correct_Key() FAILED
    java.lang.AssertionError at EncryptionContextTest.java:55

EncryptionContextTest > encryption_and_decryption_are_symmetric() FAILED
    java.lang.NullPointerException at EncryptionContextTest.java:36

MessageFormatSerializationFactoryTest > getSerializer_returns_expected_deserializer(MessageFormat, Class) > [1] BYTES, class org.apache.kafka.common.serialization.ByteArraySerializer FAILED
    java.lang.NullPointerException at MessageFormatSerializationFactoryTest.java:72

MessageFormatSerializationFactoryTest > getSerializer_returns_expected_deserializer(MessageFormat, Class) > [2] PLAINTEXT, class org.apache.kafka.common.serialization.StringSerializer FAILED
    java.lang.NullPointerException at MessageFormatSerializationFactoryTest.java:72

MessageFormatSerializationFactoryTest > getSerializer_returns_expected_deserializer(MessageFormat, Class) > [3] JSON, class org.apache.kafka.connect.json.JsonSerializer FAILED
    java.lang.NullPointerException at MessageFormatSerializationFactoryTest.java:72

MessageFormatSerializationFactoryTest > getSerializer_returns_expected_deserializer(MessageFormat, Class) > [4] AVRO, class io.confluent.kafka.serializers.KafkaAvroSerializer FAILED
    java.lang.NullPointerException at MessageFormatSerializationFactoryTest.java:72

MessageFormatSerializationFactoryTest > getDeserializer_returns_same_instance_for_multiple_calls(MessageFormat) > [1] PLAINTEXT FAILED
    java.lang.NullPointerException at MessageFormatSerializationFactoryTest.java:59

MessageFormatSerializationFactoryTest > getDeserializer_returns_same_instance_for_multiple_calls(MessageFormat) > [2] JSON FAILED
    java.lang.NullPointerException at MessageFormatSerializationFactoryTest.java:59

MessageFormatSerializationFactoryTest > getDeserializer_returns_same_instance_for_multiple_calls(MessageFormat) > [3] AVRO FAILED
    java.lang.NullPointerException at MessageFormatSerializationFactoryTest.java:59

MessageFormatSerializationFactoryTest > getDeserializer_returns_same_instance_for_multiple_calls(MessageFormat) > [4] BYTES FAILED
    java.lang.NullPointerException at MessageFormatSerializationFactoryTest.java:59

MessageFormatSerializationFactoryTest > getSerializer_returns_same_instance_for_multiple_calls(MessageFormat) > [1] PLAINTEXT FAILED
    java.lang.NullPointerException at MessageFormatSerializationFactoryTest.java:87

MessageFormatSerializationFactoryTest > getSerializer_returns_same_instance_for_multiple_calls(MessageFormat) > [2] JSON FAILED
    java.lang.NullPointerException at MessageFormatSerializationFactoryTest.java:87

MessageFormatSerializationFactoryTest > getSerializer_returns_same_instance_for_multiple_calls(MessageFormat) > [3] AVRO FAILED
    java.lang.NullPointerException at MessageFormatSerializationFactoryTest.java:87

MessageFormatSerializationFactoryTest > getSerializer_returns_same_instance_for_multiple_calls(MessageFormat) > [4] BYTES FAILED
    java.lang.NullPointerException at MessageFormatSerializationFactoryTest.java:87

MessageFormatSerializationFactoryTest > getDeserializer_returns_deserializer_instance(MessageFormat) > [1] PLAINTEXT FAILED
    java.lang.NullPointerException at MessageFormatSerializationFactoryTest.java:51

MessageFormatSerializationFactoryTest > getDeserializer_returns_deserializer_instance(MessageFormat) > [2] JSON FAILED
    java.lang.NullPointerException at MessageFormatSerializationFactoryTest.java:51

MessageFormatSerializationFactoryTest > getDeserializer_returns_deserializer_instance(MessageFormat) > [3] AVRO FAILED
    java.lang.NullPointerException at MessageFormatSerializationFactoryTest.java:51

MessageFormatSerializationFactoryTest > getDeserializer_returns_deserializer_instance(MessageFormat) > [4] BYTES FAILED
    java.lang.NullPointerException at MessageFormatSerializationFactoryTest.java:51

MessageFormatSerializationFactoryTest > getDeserializer_returns_expected_deserializer(MessageFormat, Class) > [1] BYTES, class org.apache.kafka.common.serialization.ByteArrayDeserializer FAILED
    java.lang.NullPointerException at MessageFormatSerializationFactoryTest.java:44

MessageFormatSerializationFactoryTest > getDeserializer_returns_expected_deserializer(MessageFormat, Class) > [2] PLAINTEXT, class org.apache.kafka.common.serialization.StringDeserializer FAILED
    java.lang.NullPointerException at MessageFormatSerializationFactoryTest.java:44

MessageFormatSerializationFactoryTest > getDeserializer_returns_expected_deserializer(MessageFormat, Class) > [3] JSON, class org.apache.kafka.connect.json.JsonDeserializer FAILED
    java.lang.NullPointerException at MessageFormatSerializationFactoryTest.java:44

MessageFormatSerializationFactoryTest > getDeserializer_returns_expected_deserializer(MessageFormat, Class) > [4] AVRO, class io.confluent.kafka.serializers.KafkaAvroDeserializer FAILED
    java.lang.NullPointerException at MessageFormatSerializationFactoryTest.java:44

MessageFormatSerializationFactoryTest > getSerializer_returns_deserializer_instance(MessageFormat) > [1] PLAINTEXT FAILED
    java.lang.NullPointerException at MessageFormatSerializationFactoryTest.java:79

MessageFormatSerializationFactoryTest > getSerializer_returns_deserializer_instance(MessageFormat) > [2] JSON FAILED
    java.lang.NullPointerException at MessageFormatSerializationFactoryTest.java:79

MessageFormatSerializationFactoryTest > getSerializer_returns_deserializer_instance(MessageFormat) > [3] AVRO FAILED
    java.lang.NullPointerException at MessageFormatSerializationFactoryTest.java:79

MessageFormatSerializationFactoryTest > getSerializer_returns_deserializer_instance(MessageFormat) > [4] BYTES FAILED
    java.lang.NullPointerException at MessageFormatSerializationFactoryTest.java:79

AuthConfigTest > testSaslAuthConfigWithPlainText() FAILED
    java.lang.ClassCastException at AuthConfigTest.java:56

AuthConfigTest > testSaslAuthConfigWithMskIam() FAILED
    java.lang.ClassCastException at AuthConfigTest.java:56

AuthConfigTest > testSaslAuthConfigWithOAuth() FAILED
    java.lang.ClassCastException at AuthConfigTest.java:56

AuthConfigTest > testPlainTextAuthConfig() FAILED
    java.lang.ClassCastException at AuthConfigTest.java:56

CommonTopicConfigTest > testConfigValues_from_yaml() FAILED
    java.lang.ClassCastException at CommonTopicConfigTest.java:41

CommonTopicConfigTest > test_topicsConfig_not_null() FAILED
    java.lang.ClassCastException at CommonTopicConfigTest.java:41

CommonTopicConfigTest > testConfigValues_default() FAILED
    java.lang.ClassCastException at CommonTopicConfigTest.java:41

CommonTopicConfigTest > testConfigValues_from_yaml_not_null() FAILED
    java.lang.ClassCastException at CommonTopicConfigTest.java:41

KmsConfigTest > toCredentialsOptions_returns_options_with_correct_values() FAILED
    java.lang.IllegalArgumentException at KmsConfigTest.java:33
        Caused by: com.fasterxml.jackson.databind.exc.MismatchedInputException at KmsConfigTest.java:33

MskBrokerConnectionTypeTest > fromTypeValue(MskBrokerConnectionType) > [1] null FAILED
    java.lang.NullPointerException at MskBrokerConnectionTypeTest.java:19

MskBrokerConnectionTypeTest > fromTypeValue(MskBrokerConnectionType) > [2] null FAILED
    java.lang.NullPointerException at MskBrokerConnectionTypeTest.java:19

MskBrokerConnectionTypeTest > fromTypeValue(MskBrokerConnectionType) > [3] null FAILED
    java.lang.NullPointerException at MskBrokerConnectionTypeTest.java:19

OAuthConfigTest > assertNotNullForConfigs() FAILED
    java.lang.ClassCastException at OAuthConfigTest.java:49

OAuthConfigTest > assertConfigValues() FAILED
    java.lang.ClassCastException at OAuthConfigTest.java:49

OAuthConfigTest > testConfig() FAILED
    java.lang.ClassCastException at OAuthConfigTest.java:49

PlainTextAuthConfigTest > testConfigValues() FAILED
    java.lang.ClassCastException at PlainTextAuthConfigTest.java:35

SchemaConfigTest > test_schemaConfig_not_null() FAILED
    java.lang.ClassCastException at SchemaConfigTest.java:37

SchemaConfigTest > test_schemaConfig_values() FAILED
    java.lang.ClassCastException at SchemaConfigTest.java:37

KafkaCustomConsumerTest > testJsonDeserializationErrorWithAcknowledgements() FAILED
    java.lang.IllegalAccessError at KafkaCustomConsumerTest.java:162

KafkaCustomConsumerTest > testAwsGlueErrorWithAcknowledgements() FAILED
    java.lang.IllegalAccessError at KafkaCustomConsumerTest.java:162

KafkaCustomConsumerTest > testPlainTextConsumeRecordsWithAcknowledgements() FAILED
    java.lang.IllegalAccessError at KafkaCustomConsumerTest.java:162

KafkaCustomConsumerTest > testJsonConsumeRecords() FAILED
    java.lang.IllegalAccessError at KafkaCustomConsumerTest.java:162

KafkaCustomConsumerTest > testPlainTextConsumeRecordsWithNegativeAcknowledgements() FAILED
    java.lang.IllegalAccessError at KafkaCustomConsumerTest.java:162

KafkaCustomConsumerTest > testPlainTextConsumeRecords() FAILED
    java.lang.IllegalAccessError at KafkaCustomConsumerTest.java:162

TopicPartitionCommitTrackerTest > test(List) > [1] [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] FAILED
    java.lang.AssertionError at TopicPartitionCommitTrackerTest.java:51

TopicPartitionCommitTrackerTest > test(List) > [2] [9, 8, 7, 6, 5, 4, 3, 2, 1, 0] FAILED
    java.lang.AssertionError at TopicPartitionCommitTrackerTest.java:51

TopicPartitionCommitTrackerTest > test(List) > [3] [1, 0, 2, 3, 5, 9, 4, 8, 6, 7] FAILED
    java.lang.AssertionError at TopicPartitionCommitTrackerTest.java:51

TopicPartitionCommitTrackerTest > test(List) > [4] [5, 7, 4, 0, 2, 1, 3, 9, 8, 6] FAILED
    java.lang.AssertionError at TopicPartitionCommitTrackerTest.java:51

TopicPartitionCommitTrackerTest > test(List) > [5] [1, 6, 2, 5, 7, 4, 0, 9, 8, 3] FAILED
    java.lang.AssertionError at TopicPartitionCommitTrackerTest.java:51

TopicPartitionCommitTrackerTest > test(List) > [6] [1, 2, 6, 5, 3, 9, 4, 7, 0, 8] FAILED
    java.lang.AssertionError at TopicPartitionCommitTrackerTest.java:51

TopicPartitionCommitTrackerTest > test(List) > [7] [8, 4, 0, 6, 3, 1, 7, 2, 5, 9] FAILED
    java.lang.AssertionError at TopicPartitionCommitTrackerTest.java:51

TopicPartitionCommitTrackerTest > test(List) > [8] [0, 2, 5, 7, 1, 8, 6, 3, 4, 9] FAILED
    java.lang.AssertionError at TopicPartitionCommitTrackerTest.java:51

TopicPartitionCommitTrackerTest > test(List) > [9] [3, 8, 2, 6, 7, 1, 5, 0, 9, 4] FAILED
    java.lang.AssertionError at TopicPartitionCommitTrackerTest.java:51

TopicPartitionCommitTrackerTest > test(List) > [10] [4, 9, 2, 3, 7, 8, 0, 1, 5, 6] FAILED
    java.lang.AssertionError at TopicPartitionCommitTrackerTest.java:51

TopicPartitionCommitTrackerTest > test(List) > [11] [0, 6, 8, 1, 5, 9, 4, 7, 2, 3] FAILED
    java.lang.AssertionError at TopicPartitionCommitTrackerTest.java:51

TopicPartitionCommitTrackerTest > test(List) > [12] [9, 7, 0, 1, 2, 8, 5, 6, 3, 4] FAILED
    java.lang.AssertionError at TopicPartitionCommitTrackerTest.java:51

TopicPartitionCommitTrackerTest > testNonZeroOffsets(List) > [1] [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] FAILED
    java.lang.AssertionError at TopicPartitionCommitTrackerTest.java:75

TopicPartitionCommitTrackerTest > testNonZeroOffsets(List) > [2] [9, 8, 7, 6, 5, 4, 3, 2, 1, 0] FAILED
    java.lang.AssertionError at TopicPartitionCommitTrackerTest.java:75

TopicPartitionCommitTrackerTest > testNonZeroOffsets(List) > [3] [1, 4, 7, 5, 0, 6, 8, 2, 9, 3] FAILED
    java.lang.AssertionError at TopicPartitionCommitTrackerTest.java:75

TopicPartitionCommitTrackerTest > testNonZeroOffsets(List) > [4] [9, 6, 3, 4, 2, 8, 0, 5, 1, 7] FAILED
    java.lang.AssertionError at TopicPartitionCommitTrackerTest.java:75

TopicPartitionCommitTrackerTest > testNonZeroOffsets(List) > [5] [1, 6, 2, 0, 8, 5, 7, 4, 9, 3] FAILED
    java.lang.AssertionError at TopicPartitionCommitTrackerTest.java:75

TopicPartitionCommitTrackerTest > testNonZeroOffsets(List) > [6] [3, 2, 4, 0, 5, 1, 7, 8, 9, 6] FAILED
    java.lang.AssertionError at TopicPartitionCommitTrackerTest.java:75

TopicPartitionCommitTrackerTest > testNonZeroOffsets(List) > [7] [8, 2, 6, 4, 3, 5, 7, 1, 9, 0] FAILED
    java.lang.AssertionError at TopicPartitionCommitTrackerTest.java:75

TopicPartitionCommitTrackerTest > testNonZeroOffsets(List) > [8] [8, 6, 9, 7, 1, 0, 4, 2, 5, 3] FAILED
    java.lang.AssertionError at TopicPartitionCommitTrackerTest.java:75

TopicPartitionCommitTrackerTest > testNonZeroOffsets(List) > [9] [3, 1, 8, 2, 6, 5, 4, 9, 7, 0] FAILED
    java.lang.AssertionError at TopicPartitionCommitTrackerTest.java:75

TopicPartitionCommitTrackerTest > testNonZeroOffsets(List) > [10] [6, 1, 2, 8, 9, 5, 4, 0, 7, 3] FAILED
    java.lang.AssertionError at TopicPartitionCommitTrackerTest.java:75

TopicPartitionCommitTrackerTest > testNonZeroOffsets(List) > [11] [6, 9, 1, 5, 7, 4, 3, 2, 8, 0] FAILED
    java.lang.AssertionError at TopicPartitionCommitTrackerTest.java:75

TopicPartitionCommitTrackerTest > testNonZeroOffsets(List) > [12] [9, 8, 0, 6, 7, 4, 5, 2, 3, 1] FAILED
    java.lang.AssertionError at TopicPartitionCommitTrackerTest.java:75

KafkaClusterConfigTest > testConfigWithTestExtension() FAILED
    java.lang.annotation.AnnotationFormatError at KafkaClusterConfigTest.java:40

KafkaCustomProducerTest > testGetGenericRecord() FAILED
    java.lang.ExceptionInInitializerError at KafkaCustomProducerTest.java:249
        Caused by: java.lang.NullPointerException at KafkaCustomProducerTest.java:249

KafkaCustomProducerTest > producePlainTextRecords() FAILED
    java.lang.NullPointerException at KafkaCustomProducer.java:112

KafkaCustomProducerTest > producePlainTextRecords_callbackException() FAILED
    java.lang.NullPointerException at KafkaCustomProducer.java:112

KafkaCustomProducerTest > validateSchema() FAILED
    com.github.fge.jsonschema.core.exceptions.ProcessingException at KafkaCustomProducerTest.java:261

KafkaCustomProducerTest > produceRawDataTest() FAILED
    java.lang.NullPointerException at KafkaCustomProducerTest.java:105

KafkaCustomProducerTest > producePlainTextRecords_sendError() FAILED
    java.lang.NullPointerException at KafkaCustomProducer.java:112

KafkaCustomProducerTest > produceJsonRecordsTest() FAILED
    java.lang.NullPointerException at KafkaCustomProducer.java:112

KafkaCustomProducerTest > produceAvroRecordsTest() FAILED
    java.lang.NoClassDefFoundError at KafkaCustomProducerTest.java:227

KafkaCustomProducerTest > produceRawData_sendError() FAILED
    org.mockito.exceptions.misusing.UnfinishedStubbingException at KafkaCustomProducerTest.java:87

DLQSinkTest > testPerform() FAILED
    java.lang.ClassCastException at DLQSinkTest.java:68

KafkaSinkConfigTest > testDLQConfiguration() FAILED
    java.util.NoSuchElementException at KafkaSinkConfigTest.java:97

KafkaSinkTest > doOutputExceptionTest() FAILED
    java.lang.ClassCastException at KafkaSinkTest.java:105

KafkaSinkTest > doOutputTest() FAILED
    java.lang.ClassCastException at KafkaSinkTest.java:105

KafkaSinkTest > doInitializeNullPointerExceptionTest() FAILED
    java.lang.ClassCastException at KafkaSinkTest.java:105

KafkaSinkTest > isReadyTest() FAILED
    java.lang.ClassCastException at KafkaSinkTest.java:105

KafkaSinkTest > shutdownTest() FAILED
    java.lang.ClassCastException at KafkaSinkTest.java:105

KafkaSinkTest > shutdownExceptionTest() FAILED
    java.lang.ClassCastException at KafkaSinkTest.java:105

KafkaSinkTest > doOutputTestForAutoTopicCreate() FAILED
    java.lang.ClassCastException at KafkaSinkTest.java:105

KafkaSinkTest > doInitializeTest() FAILED
    java.lang.ClassCastException at KafkaSinkTest.java:105

KafkaSinkTest > doOutputEmptyRecordsTest() FAILED
    java.lang.ClassCastException at KafkaSinkTest.java:105

KafkaSourceConfigTest > test_topics_not_null() FAILED
    java.lang.ClassCastException at KafkaSourceConfigTest.java:47

KafkaSourceConfigTest > test_kafka_config_not_null() FAILED
    java.lang.ClassCastException at KafkaSourceConfigTest.java:47

KafkaSourceConfigTest > test_setters() FAILED
    java.lang.ClassCastException at KafkaSourceConfigTest.java:47

KafkaSourceConfigTest > test_bootStrapServers_not_null() FAILED
    java.lang.ClassCastException at KafkaSourceConfigTest.java:47

KafkaSourceTest > test_kafkaSource_basicFunctionality() FAILED
    java.lang.RuntimeException at KafkaSourceTest.java:150
        Caused by: java.lang.NullPointerException at KafkaSourceTest.java:150

KafkaSourceTest > test_kafkaSource_retry_consumer_create() FAILED
    java.lang.RuntimeException at KafkaSource.java:155
        Caused by: java.lang.NullPointerException at KafkaSourceTest.java:165

AuthenticationPropertyConfigurerTest > testSetSaslPlainTextProperties() FAILED
    java.lang.ClassCastException at AuthenticationPropertyConfigurerTest.java:34

AuthenticationPropertyConfigurerTest > testSetSaslSslProperties() FAILED
    java.lang.ClassCastException at AuthenticationPropertyConfigurerTest.java:34

AuthenticationPropertyConfigurerTest > testSetSaslOauthProperties() FAILED
    java.lang.ClassCastException at AuthenticationPropertyConfigurerTest.java:34

AuthenticationTypeTest > getAuthTypeByName(AuthenticationType) > [1] plaintext FAILED
    java.lang.NullPointerException at AuthenticationTypeTest.java:17

AuthenticationTypeTest > getAuthTypeByName(AuthenticationType) > [2] ssl FAILED
    java.lang.NullPointerException at AuthenticationTypeTest.java:17

AuthenticationTypeTest > getAuthTypeByName(AuthenticationType) > [3] oauth FAILED
    java.lang.NullPointerException at AuthenticationTypeTest.java:17

KafkaTopicConsumerMetricsTests > KafkaTopicMetricTest_checkMetricUpdates(int) > [1] 1 FAILED
    java.lang.NullPointerException at KafkaTopicConsumerMetricsTests.java:211

KafkaTopicConsumerMetricsTests > KafkaTopicMetricTest_checkMetricUpdates(int) > [2] 5 FAILED
    java.lang.NullPointerException at KafkaTopicConsumerMetricsTests.java:211

KafkaTopicConsumerMetricsTests > KafkaTopicMetricTest_checkMetricUpdates(int) > [3] 10 FAILED
    java.lang.NullPointerException at KafkaTopicConsumerMetricsTests.java:211

KafkaTopicProducerMetricsTests > KafkaTopicMetricTest_checkMetricUpdates(int) > [1] 1 FAILED
    java.lang.NullPointerException at KafkaTopicProducerMetricsTests.java:169

KafkaTopicProducerMetricsTests > KafkaTopicMetricTest_checkMetricUpdates(int) > [2] 5 FAILED
    java.lang.NullPointerException at KafkaTopicProducerMetricsTests.java:169

KafkaTopicProducerMetricsTests > KafkaTopicMetricTest_checkMetricUpdates(int) > [3] 10 FAILED
    java.lang.NullPointerException at KafkaTopicProducerMetricsTests.java:169

SinkPropertyConfigurerTest > testGetProducerPropertiesForPlainText() FAILED
    java.lang.ClassCastException at SinkPropertyConfigurerTest.java:43

SinkPropertyConfigurerTest > testGetProducerPropertiesForNoSerde() FAILED
    java.lang.ClassCastException at SinkPropertyConfigurerTest.java:43

SinkPropertyConfigurerTest > testGetProducerPropertiesForAvro() FAILED
    java.lang.ClassCastException at SinkPropertyConfigurerTest.java:43

SinkPropertyConfigurerTest > testGetProducerPropertiesForJson() FAILED
    java.lang.ClassCastException at SinkPropertyConfigurerTest.java:43

258 tests completed, 132 failed

https://github.com/opensearch-project/data-prepper/actions/runs/7479248734/job/20362415558?pr=3937

dlvenable commented 8 months ago

Seen today:

ExportPartitionWorkerTest > testProcessPartitionSuccess(String) > [1] test.collection|0|1|java.lang.Integer FAILED
    org.mockito.exceptions.verification.WantedButNotInvoked at ExportPartitionWorkerTest.java:120

dlvenable commented 4 months ago

Seen today:

HistogramAggregateActionTests > testHistogramAggregateOTelFormatWithStartAndEndTimesInTheEvent(int) > [1] 10 FAILED
    java.lang.AssertionError at HistogramAggregateActionTests.java:370

dlvenable commented 4 months ago

Some investigation on AggregateProcessorIT.

[2] 15.0

java.lang.AssertionError: 
Expected: a numeric value within <1.0> of <3000.0>
     but: <2994.0> differed by <5.0> more than delta <1.0>
    at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
    at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6)
    at org.opensearch.dataprepper.plugins.processor.aggregate.AggregateProcessorIT.aggregateWithPercentSamplerAction(AggregateProcessorIT.java:292)

This assertion is failing:

assertThat((double)allowedEventsCount.get(), closeTo(NUM_THREADS * NUM_EVENTS_PER_BATCH * testPercent/100, 1.0));

dlvenable commented 4 months ago

A new failure:

LambdaSinkServiceTest > lambda_sink_test_batch_enabled() FAILED
    org.mockito.exceptions.verification.WantedButNotInvoked at LambdaSinkServiceTest.java:316

dlvenable commented 4 months ago

New failure: > Task :data-prepper-plugins:log-generator-source:test

LogGeneratorSourceTest > GIVEN_logGeneratorSourceAndBlockingBuffer_WHEN_reachedLimit_THEN_stopsWritingToBuffer() FAILED org.mockito.exceptions.verification.WantedButNotInvoked at LogGeneratorSourceTest.java:115

dlvenable commented 4 months ago

Another AggregateProcessorIT:

AggregateProcessorIT > aggregateWithCountAggregateActionWithCondition() > repetition 2 of 2 FAILED
    java.lang.AssertionError at AggregateProcessorIT.java:439

dlvenable commented 1 month ago

RDS now has a flaky test:

> Task :data-prepper-plugins:rds-source:test

ExportSchedulerTest > test_shutDown() FAILED
    org.mockito.exceptions.verification.NoInteractionsWanted at ExportSchedulerTest.java:271

https://github.com/opensearch-project/data-prepper/actions/runs/11172396206/job/31058770072?pr=5021

@oeyh , Can you look at this one?

dlvenable commented 1 month ago

> Task :data-prepper-plugins:rds-source:test

StreamSchedulerTest > test_shutdown() FAILED
    org.mockito.exceptions.verification.NoInteractionsWanted at StreamSchedulerTest.java:118

@oeyh , Here is another failure in RDS.

oeyh commented 1 month ago

Truncate processor test. https://github.com/opensearch-project/data-prepper/actions/runs/11278262054/job/31366340484?pr=5043

> Task :data-prepper-plugins:truncate-processor:test

TruncateProcessorConfigTests > testInvalidConfiguration_StartAt_Length_Negative() FAILED
    org.opentest4j.AssertionFailedError: expected: <false> but was: <true>