search

masesgroup / KEFCore

Entity Framework Core provider for Apache Kafka™
https://kefcore.masesgroup.com/
Apache License 2.0
6 stars 1 forks source link

Add test execution at the end of build process #243

Closed masesdevelopers closed 3 weeks ago

masesdevelopers commented 1 month ago

Is your feature request related to a problem? Please describe. Currently no tests are executed.

Describe the solution you'd like The build workflow can execute some unit tests to verify execution. Meanwhile the test projects can be enhanced to use the Microsoft.NET.Test.Sdk and/or Microsoft.TestPlatform.TestHost

Describe alternatives you've considered N/A

Additional context N/A

masesdevelopers commented 3 weeks ago

For MacOS and Windows try to start local ZooKeeper and Apache Kafka broker then execute the tests

masesdevelopers commented 3 weeks ago

The test execution reports:

00::44::23:8223 - System.InvalidOperationException: BridgeInstance is null.
   at MASES.JCOBridge.C2JBridge.JVMBridgeException.IExecute[TReturn](String methodName, Object[] args)
   at Java.Lang.Throwable.get_Message()
   at MASES.EntityFrameworkCore.KNet.Storage.Internal.KafkaCluster.CreateTable(IEntityType entityType, Int32 cycle) in /_/src/net/KEFCore/Storage/Internal/KafkaCluster.cs:line 230
masesdevelopers commented 3 weeks ago

The test execution reports:

00::44::23:8223 - System.InvalidOperationException: BridgeInstance is null.
   at MASES.JCOBridge.C2JBridge.JVMBridgeException.IExecute[TReturn](String methodName, Object[] args)
   at Java.Lang.Throwable.get_Message()
   at MASES.EntityFrameworkCore.KNet.Storage.Internal.KafkaCluster.CreateTable(IEntityType entityType, Int32 cycle) in /_/src/net/KEFCore/Storage/Internal/KafkaCluster.cs:line 230

Add a check in https://github.com/masesgroup/KEFCore/blob/9312fea886f58adfe8868e9719b1b1f29474d852/src/net/KEFCore/Storage/Internal/KafkaCluster.cs#L230 to verify if ex.BridgeInstance is not null, otherwise cast to JVMBridgeException and retrieve Message. The problem comes from https://github.com/masesgroup/JNet/issues/429 and how Java.Lang.Throwable manages the information.

masesdevelopers commented 3 weeks ago

Within https://github.com/masesgroup/KEFCore/actions/runs/9655461268 many tests fails. Here the summary:

The errors occurs always in net8.0, and sparsely for many JDK 21, when the array of bytes of key related to the Kafka record is retrieved from the JVM. However the problem is raised in many way within KNet code, so the issue shall be solved there.

masesdevelopers commented 3 weeks ago

Within https://github.com/masesgroup/KEFCore/actions/runs/9655461268 many tests fails. Here the summary:

* .NET: net8.0

* OS

  * Windows

    * JDK: Corretto 21
  * Linux

    * Oracle 21
    * Zulu 21
    * Temurin 21
    * Microsoft 21
  * MacOS 13 and MacOS 14 (macos-latest)

    * Corretto 21
    * Microsoft 21
    * Temurin 21
    * Oracle 21
    * Zulu 21

The errors occurs always in net8.0, and sparsely for many JDK 21, when the array of bytes of key related to the Kafka record is retrieved from the JVM. However the problem is raised in many way within KNet code, so the issue shall be solved there.

Executing other tests, some exceptions are raised within KEFCore itself:

Unhandled exception. System.Text.Json.JsonException: The input does not contain any JSON tokens. Expected the input to start with a valid JSON token, when isFinalBlock is true. Path: $ | LineNumber: 0 | BytePositionInLine: 0.
 ---> System.Text.Json.JsonReaderException: The input does not contain any JSON tokens. Expected the input to start with a valid JSON token, when isFinalBlock is true. LineNumber: 0 | BytePositionInLine: 0.
   at System.Text.Json.ThrowHelper.ThrowJsonReaderException(Utf8JsonReader& json, ExceptionResource resource, Byte nextByte, ReadOnlySpan`1 bytes)
   at System.Text.Json.Utf8JsonReader.Read()
   at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)
   --- End of inner exception stack trace ---
   at System.Text.Json.ThrowHelper.ReThrowWithPath(ReadStack& state, JsonReaderException ex)
   at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)
   at System.Text.Json.JsonSerializer.ReadFromSpan[TValue](ReadOnlySpan`1 utf8Json, JsonTypeInfo`1 jsonTypeInfo, Nullable`1 actualByteCount)
   at System.Text.Json.JsonSerializer.Deserialize[TValue](ReadOnlySpan`1 utf8Json, JsonSerializerOptions options)
   at MASES.EntityFrameworkCore.KNet.Serialization.Json.DefaultKEFCoreSerDes.ValueContainer`1.JsonRaw`1.DeserializeWithHeaders(String topic, Headers headers, Byte[] data) in /_/src/net/KEFCore.SerDes/DefaultKEFCoreSerDes.cs:line 318
   at MASES.EntityFrameworkCore.KNet.Serialization.Json.DefaultKEFCoreSerDes.ValueContainer`1.JsonRaw`1.Deserialize(String topic, Byte[] data) in /_/src/net/KEFCore.SerDes/DefaultKEFCoreSerDes.cs:line 313
   at MASES.KNet.Streams.KeyValue`4..ctor(IGenericSerDesFactory factory, KeyValue`2 value, ISerDes`2 keySerDes, ISerDes`2 valueSerDes, Boolean fromPrefetched)
   at MASES.KNet.Streams.State.KeyValueIterator`4.PrefetchableLocalEnumerator.ConvertObject(Object input)
   at MASES.JCOBridge.C2JBridge.JVMBridgeBasePrefetchableEnumerator.Advance()
   at MASES.JCOBridge.C2JBridge.JVMBridgeBasePrefetchableEnumerator.Execute(Object o)
   at System.Threading.Thread.StartHelper.Callback(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)

and the process fails due to unhandled exception in the thread used to prefetch the information. The specific exception is System.Text.Json.JsonReaderException which reports LineNumber: 0 | BytePositionInLine: 0: seems the data received does not contains information, i.e. the array of bytes is empty.

The previous condition pairs with the following:

Unhandled exception. System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection. (Parameter 'startIndex')
   at System.BitConverter.ToInt32(Byte[] value, Int32 startIndex)
   at MASES.KNet.Serialization.KNetSerialization.DeserializeInt(Boolean fallbackToKafka, String topic, Byte[] data)
   at MASES.KNet.Serialization.SerDes`2.Deserialize(String topic, TJVMT data)
   at MASES.KNet.Streams.KeyValue`4..ctor(IGenericSerDesFactory factory, KeyValue`2 value, ISerDes`2 keySerDes, ISerDes`2 valueSerDes, Boolean fromPrefetched)
   at MASES.KNet.Streams.State.KeyValueIterator`4.PrefetchableLocalEnumerator.ConvertObject(Object input)
   at MASES.JCOBridge.C2JBridge.JVMBridgeBasePrefetchableEnumerator.Advance()
   at MASES.JCOBridge.C2JBridge.JVMBridgeBasePrefetchableEnumerator.Execute(Object o)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)

where System.ArgumentOutOfRangeException most probably reports that the value input of System.BitConverter.ToInt32 has a length of zero and MASES.KNet.Serialization.KNetSerialization.DeserializeInt try to start from zero: see explanation in https://learn.microsoft.com/it-it/dotnet/api/system.bitconverter.toint32?view=net-8.0#system-bitconverter-toint32(system-byte()-system-int32)

The first kind of exception shall be managed in KEFCore, while the second one is related to KNet.

masesdevelopers commented 3 weeks ago

Windows is not a supported OS to run Apache Kafka and many tests fails on Windows because the broker, or ZooKeeper, closes.

Try use Docker, only on Windows, since it seems to be installed in Windows 2022 image, the command shall be something like:

docker run -e KNET_DOCKER_RUNNING_MODE=server -p 9092:9092 ghcr.io/masesgroup/knet:latest

maybe the access to ghcr.io will be forbidden, so it is needed an authentication.

masesdevelopers commented 3 weeks ago

Windows is not a supported OS to run Apache Kafka and many tests fails on Windows because the broker, or ZooKeeper, closes.

Try use Docker, only on Windows, since it seems to be installed in Windows 2022 image, the command shall be something like:

docker run -e KNET_DOCKER_RUNNING_MODE=server -p 9092:9092 ghcr.io/masesgroup/knet:latest

maybe the access to ghcr.io will be forbidden, so it is needed an authentication.

The docker version available in Windows runner cannot activate linux images due to virtualization limitation, so there is no chance to use this method.

To avoid problems on OSes not supporting Apache Kafka maybe it is mandatory to activate an external container/VM, but the network can be a bottleneck.

masesdevelopers commented 2 weeks ago

Some tests fails for other reason than the previous: specifically under windows and with JDK 17. Looking at the available dumps, sometime there is an access violation during preparation of the KNetSerialization class. However, trying to reproduce the issue locally, the exception is not raised; between GitHub and local test the differences are:

It is possible to highlight that KNetReplicator never raises such exception and the exception is not raised from the same test even if the order of execution is always the same.