MS Examples are crashing on MemoryStorage

[securigy]

I took the example source code - Example38_Pinecone, and pasted it into my simple WinForms application. The made some changes. Specifically I delete the all the existing indexes and collection before I do any operations, like creating index, etc. The entire code is below:

`
public static async Task RunAsync() { using (Log.VerboseCall()) { string apiKey = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"; string pineconeEnvironment = "us-west1-gcp-free";

            string openAiKey = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx";

            PineconeMemoryStore memoryStore = new(pineconeEnvironment, apiKey);
            using var pinecone = new PineconeClient(pineconeEnvironment, apiKey);

            // Delete all existing indexes
            List<string> indexNames = new List<string>();
            IAsyncEnumerable<string> IAsyncEnum = pinecone.ListIndexesAsync();
            await foreach (var item in IAsyncEnum)
            {
                await pinecone.DeleteIndexAsync(item);
            }

            try
            {
                int i = 0;
                IAsyncEnumerable<string> collections = memoryStore.GetCollectionsAsync();
                await foreach (string collection in collections)
                {
                    collection.Remove(i);
                    Console.WriteLine(collection);
                    i++;
                }
            }
            catch(Exception ex)
            {
                Console.WriteLine(ex.Message);
            }

            try
            {
                // Create new index
                await pinecone.CreateIndexAsync(new("pinecone-index")
                {
                    Dimension = 1536,
                    PodType = PodType.S1X1,
                    Metric = IndexMetric.Cosine
                });
            }
            catch(Exception ex)
            {
                Console.WriteLine(ex.Message);
            }

            Task<PineconeIndex> idx = pinecone.DescribeIndexAsync("pinecone-index");
            idx.Wait();
            Console.WriteLine("idx.Status: {0}", idx.Status);

            IKernel kernel = Kernel.Builder
                .WithOpenAITextCompletionService("text-davinci-003", openAiKey)
                .WithOpenAITextEmbeddingGenerationService("text-embedding-ada-002", openAiKey)
                .WithMemoryStorage(memoryStore)
                .Build();

            Console.WriteLine("== Printing Collections in DB ==");

            IAsyncEnumerable<string> collections2 = memoryStore.GetCollectionsAsync();
            await foreach (string collection in collections2)
            {
                Console.WriteLine(collection);
            }

            Console.WriteLine("== Adding Memories ==");

            Dictionary<string, object> metadata = new()
            {
                { "type", "text" },
                { "tags", new List<string>() { "memory", "cats" } }
            };

            string additionalMetadata = System.Text.Json.JsonSerializer.Serialize(metadata);

            Log.Verbose("Calling CreateIndexAsync...");

            try
            {
               //  !!! THIS LINE THROWS EXCEPTION, ALWAYS, NO MATTER WHAT...!!!
                string key1 = await kernel.Memory.SaveInformationAsync(MemoryCollectionName, "british short hair", "cat1", null, additionalMetadata);
                string key2 = await kernel.Memory.SaveInformationAsync(MemoryCollectionName, "orange tabby", "cat2", null, additionalMetadata);
                string key3 = await kernel.Memory.SaveInformationAsync(MemoryCollectionName, "norwegian forest cat", "cat3", null, additionalMetadata);

                Console.WriteLine("== Retrieving Memories Through the Kernel ==");
                MemoryQueryResult lookup = await kernel.Memory.GetAsync(MemoryCollectionName, "cat1");
                Console.WriteLine(lookup != null ? lookup.Metadata.Text : "ERROR: memory not found");

                Console.WriteLine("== Retrieving Memories Directly From the Store ==");
                var memory1 = await memoryStore.GetAsync(MemoryCollectionName, key1);
                var memory2 = await memoryStore.GetAsync(MemoryCollectionName, key2);
                var memory3 = await memoryStore.GetAsync(MemoryCollectionName, key3);

                Console.WriteLine(memory1 != null ? memory1.Metadata.Text : "ERROR: memory not found");
                Console.WriteLine(memory2 != null ? memory2.Metadata.Text : "ERROR: memory not found");
                Console.WriteLine(memory3 != null ? memory3.Metadata.Text : "ERROR: memory not found");

                Console.WriteLine("== Similarity Searching Memories: My favorite color is orange ==");
                IAsyncEnumerable<MemoryQueryResult> searchResults = kernel.Memory.SearchAsync(MemoryCollectionName, "My favorite color is orange", 1, 0.8);

                await foreach (MemoryQueryResult item in searchResults)
                {
                    Console.WriteLine(item.Metadata.Text + " : " + item.Relevance);
                }
            }
            catch (Exception ex)
            {
                Log.Verbose(ex);
            }
        }
    }`

The indicated line with the comment above throws exception, no matter what... and this behavior is observed not only in this example but in QDRANT example as well: Example19_Qdrant. The same line throw exception. So, at this point it looks like a general problem... Am I the only one trying these examples?

Just to note that I did deployed the correct resource in Azure with correct model "text-embedding-ada-002". Any my Pinecone key and environment that I got from Pinecone are correct, as well as ApiKey of OpenAI. Anything else maybe I need to do?

The exception is: Microsoft.SemanticKernel.Connectors.Memory.Pinecone.PineconeMemoryException: 'Index creation is not supported within memory store. It should be created manually or using CreateIndexAsync. Ensure index state is Ready.'

But, I do create index in the code as shown below and it ran to completion. (idx.Status = RanToCompletion)

Screenshots Stack Trace: at Microsoft.SemanticKernel.Connectors.Memory.Pinecone.PineconeMemoryStore.d2.MoveNext() in /home/vsts/work/1/s/semantic-kernel/dotnet/src/Connectors/Connectors.Memory.Pinecone/PineconeMemoryStore.cs:line 68 at Microsoft.SemanticKernel.Memory.SemanticTextMemory.d3.MoveNext() at GptModelTrainer.MainForm.d__21.MoveNext() in C:\Projects\GptModelTrainer\GptModelTrainer\MainForm.cs:line 376

Desktop (please complete the following information):

• Windows 11
• IDE: VS2022
• NuGet Package Version: 0.15.230531.5-preview

[flatluna]

I see also errors in some of these Memory samples.

[securigy]

It has been 2 weeks since I posted my bug report. Is there any update on the status? Had it been reproduced? resolved? being worked on?

I managed to successfully produce a tool that can answer questions about private data - but I had to do it using OpenAI Embeddings API in C# + Pinecone C# bindings + OpenAI Completions. However, my goal is to be able to achieve it with SK. I think Microsoft need to hurry with SK as the open-source development with LangChain gains momentum (unfortunately in Python)

[shawncal]

@dmytrostruk can you take a look please?

[securigy]

If you guys need me to share my screen and demo to you - I can do it.

[dmytrostruk]

The exception is: Microsoft.SemanticKernel.Connectors.Memory.Pinecone.PineconeMemoryException: 'Index creation is not supported within memory store. It should be created manually or using CreateIndexAsync. Ensure index state is Ready.'

@securigy The reason why you receive this exception on SaveInformationAsync method is because SemanticTextMemory logic checks whether collection exists and if not, it tries to create it. This flow works fine for other memory connectors, but not for Pinecone. That's because collection creation in Pinecone is asynchronous, meaning that when you receive HTTP response from Pinecone API after collection creation request, the creation process has been started on Pinecone side, but collection (i.e. index) is not provisioned yet, provisioning is in progress. As mentioned in exception message, index state should be Ready before usage.

There are two ways how to resolve this problem:

• You can use CreateIndexAsync in your code, but you have to add logic on your side to perform HTTP polling with your preferred strategy to get to the point, when index state on Pinecone side is Ready, so then you will be able to proceed with other operations.
• You can pre-create your indexes manually on Pinecone portal and re-use it in the code when the state of your index creation will be Ready.

But, I do create index in the code as shown below and it ran to completion. (idx.Status = RanToCompletion)

Pinecone index state can have values: Initializing, ScalingUp, ScalingDown, Terminating, Ready. In your case, status RanToCompletion it's not a state of Pinecone index creation, it's a status of your Task (documentation: https://learn.microsoft.com/en-us/dotnet/api/system.threading.tasks.taskstatus?view=net-7.0#fields).

You output status of Task, because in your example there is no await when calling DescribeIndexAsync method, and you are working with Task. You can either add await or get index state details by using idx.Result.Status.State.

Task<PineconeIndex> idx = pinecone.DescribeIndexAsync("pinecone-index");
idx.Wait();
Console.WriteLine("idx.Status: {0}", idx.Status);

and this behavior is observed not only in this example but in QDRANT example as well: Example19_Qdrant

It would be really helpful if you can provide more details about exception you are receiving in Qdrant example, so we can investigate the problem.

Thanks a lot!

[securigy]

Dmytro, your post is really detailed and seems helpful - I will try it as soon as I can. THANK YOU!

======== I wonder if you can provide a general guidance (not sure where else to ask):

1. I see now that the training of the models is possible now with SK and in conjunction with many other vector DBs. However, I also see that:
2. MS came out now with Azure Cognitive Service Vector Search: https://github.com/Azure/cognitive-search-vector-pr and it is not clear what vector DB is used there...
3. And then there is ADX, which is also quite new (Azure Data Explorer) that I also have problem to run its samples.

The bottom line is: my head is spinning from multitude of option for model training through Embeddings (due to vector DB search options). Where can I find a top-level guidance or comparison between all options? and which can be used locally on my machine with or without complication of Docker...

[dmytrostruk]

@securigy In general, there is no specific recommendation what vector DB to use, because it all depends on what scenario you want to cover and what result you want to achieve. All options are different in terms of cost, performance and set of available features.

My recommendation would be to try each available option manually and see which one works best for your scenario. In repository we have an integration with multiple vector DBs including examples. Some of them can be used locally on your machine using Docker containers (e.g. Redis, Postgres, Qdrant, Chroma): https://github.com/microsoft/semantic-kernel/tree/main/dotnet/src/Connectors

I would also suggest joining our Discord so you will be able to discuss this question with wider community: https://aka.ms/SKDiscord

If you don't mind, I will close this issue, because your use-case has expected behavior due to limitations on Pinecone side. If you encounter any problems with Pinecone or any other functionality - feel free to open another issue, this will be very helpful.

Thank you!