The input is not a valid Base64 string of encoded floats.

[aminshahnazari]

Context / Scenario

I am trying to use the KernelMemory embedding scenario with the basic configuration of the kernelmemory/service Docker container.

Here is the configuration summery in appsettings.Production.json:

...

"DocumentStorageType": "SimpleFileStorage",
"DocumentStorageType": "SimpleFileStorage",
"TextGeneratorType": "OpenAI",
"EmbeddingGeneratorTypes": ["OpenAI"],
"Retrieval": {
      "MemoryDbType": "SimpleVectorDb",
      "EmbeddingGeneratorType": "OpenAI"
      ...
},
"DataIngestion": {
      "OrchestrationType": "Distributed",
      "DistributedOrchestration": {
        "QueueType": "SimpleQueues"
      },
      "EmbeddingGenerationEnabled": true,
      "EmbeddingGeneratorTypes": [
        "OpenAI"
      ],
      "MemoryDbTypes": [
        "SimpleVectorDb"
      ]
      ...
},
"OpenAI": {
        "TextModel": "gpt-4o-mini",
        "TextModelMaxTokenTotal": 16384,
        "TextGenerationType": "Auto",
        "EmbeddingModel": "text-embedding-3-small",
        "EmbeddingModelMaxTokenTotal": 8191,
        "APIKey": "xxx",
        "OrgId": "",
        "Endpoint": "xxx",
        "MaxRetries": 10,
        "EmbeddingDimensions": null,
        "MaxEmbeddingBatchSize": 100
      }
...

What happened?

When I attempt to embed something or use the ask API, I encounter the following error in the container console: System.FormatException: The input is not a valid Base64 string of encoded floats. at OpenAI.Embeddings.Embedding.<ConvertToVectorOfFloats>

Importance

I cannot use Kernel Memory

Platform, Language, Versions

Docker Image: kernelmemory/service:sha-d6af98f

Relevant log output

fail: Microsoft.AspNetCore.Server.Kestrel[13] Connection id "0HN6JNDT5NDUO", Request id "0HN6JNDT5NDUO:0000000E": 
An unhandled exception was thrown by the application. System.FormatException: The input is not a valid Base64 string of encoded floats.    
at OpenAI.Embeddings.Embedding.<ConvertToVectorOfFloats>g__ThrowInvalidData|11_0()    
at OpenAI.Embeddings.Embedding.ConvertToVectorOfFloats(BinaryData binaryData)    
at OpenAI.Embeddings.Embedding..ctor(Int32 index, BinaryData embeddingProperty, InternalEmbeddingObject object, IDictionary`2 serializedAdditionalRawData)    
at OpenAI.Embeddings.Embedding.DeserializeEmbedding(JsonElement element, ModelReaderWriterOptions options)    
at OpenAI.Embeddings.EmbeddingCollection.DeserializeEmbeddingCollection(JsonElement element, ModelReaderWriterOptions options)    
at OpenAI.Embeddings.EmbeddingCollection.FromResponse(PipelineResponse response)   
at OpenAI.Embeddings.EmbeddingClient.GenerateEmbeddingsAsync(IEnumerable`1 inputs, EmbeddingGenerationOptions options, CancellationToken cancellationToken)    
at Microsoft.SemanticKernel.Connectors.OpenAI.ClientCore.RunRequestAsync[T](Func`1 request)    
at Microsoft.SemanticKernel.Connectors.OpenAI.ClientCore.GetEmbeddingsAsync(String targetModel, IList`1 data, Kernel kernel, Nullable`1 dimensions, CancellationToken cancellationToken)    
at Microsoft.SemanticKernel.AI.Embeddings.TextEmbeddingGenerationExtensions.GenerateEmbeddingAsync(ITextEmbeddingGenerationService generator, String text, CancellationToken cancellationToken) in /src/service/Abstractions/SemanticKernel/TextEmbeddingGenerationExtensions.cs:line 28    
at Microsoft.KernelMemory.MemoryStorage.DevTools.SimpleVectorDb.GetSimilarListAsync(String index, String text, ICollection`1 filters, Double minRelevance, Int32 limit, Boolean withEmbeddings, CancellationToken cancellationToken)+MoveNext() in /src/service/Core/MemoryStorage/DevTools/SimpleVectorDb.cs:line 119   
at Microsoft.KernelMemory.MemoryStorage.DevTools.SimpleVectorDb.GetSimilarListAsync(String index, String text, ICollection`1 filters, Double minRelevance, Int32 limit, Boolean withEmbeddings, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()    
at Microsoft.KernelMemory.Search.SearchClient.AskAsync(String index, String question, ICollection`1 filters, Double minRelevance, IContext context, CancellationToken cancellationToken) in /src/service/Core/Search/SearchClient.cs:line 237    
at Microsoft.KernelMemory.Search.SearchClient.AskAsync(String index, String question, ICollection`1 filters, Double minRelevance, IContext context, CancellationToken cancellationToken) in /src/service/Core/Search/SearchClient.cs:line 336    
at Microsoft.KernelMemory.Service.AspNetCore.WebAPIEndpoints.<>c.<<AddAskEndpoint>b__5_0>d.MoveNext() in /src/service/Service.AspNetCore/WebAPIEndpoints.cs:line 221 --- End of stack trace from previous location ---    
at Microsoft.AspNetCore.Http.RequestDelegateFactory.<TaskOfTToValueTaskOfObject>g__ExecuteAwaited|92_0[T](Task`1 task)    
at Microsoft.KernelMemory.Service.HttpAuthEndpointFilter.InvokeAsync(EndpointFilterInvocationContext context, EndpointFilterDelegate next) in /src/service/Service/Auth/HttpAuthHandler.cs:line 38    
at Microsoft.AspNetCore.Http.RequestDelegateFactory.<ExecuteValueTaskOfObject>g__ExecuteAwaited|129_0(ValueTask`1 valueTask, HttpContext httpContext, JsonTypeInfo`1 jsonTypeInfo)    
at Microsoft.AspNetCore.Http.RequestDelegateFactory.<>c__DisplayClass102_2.<<HandleRequestBodyAndCompileRequestDelegateForJson>b__2>d.MoveNext() --- End of stack trace from previous location ---    
at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)    
at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)    
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)

[alex521]

+1

[aminshahnazari]

I used the previous version of the Kernel Memory service, and everything worked smoothly:

kernelmemory/service:sha-07a1d0d

@alex521

[dluc]

"Endpoint": "xxx",

are you using a custom endpoint?

[wmioch]

I have the same exception using a custom endpoint to LM Studio

[L261173157]

i have the same bug when using a custom endpoint

[dluc]

As you can see from the stack trace, the error is thrown by OpenAI .NET library, which is an internal dependency of Semantic Kernel.

OpenAI.Embeddings.Embedding.g__ThrowInvalidData|11_0()

LM Studio and similar projects attempt to mimic OpenAI service output, however, sometimes the output has minor differences that can lead to exceptions, for example if a type doesn't match, if a special token is missing, etc.

The previous version of KM used an older version of Semantic Kernel, which used different code/libraries/versions.

I would report the issue to LM Studio asking to check for differences and compatibility with OpenAI .NET library.