search

microsoft / kernel-memory

RAG architecture: index and query any data using LLM and natural language, track sources, show citations, asynchronous memory patterns.
https://microsoft.github.io/kernel-memory
MIT License
1.34k stars 252 forks source link

[Bug] Summarization failed, the content is getting longer: 811 tokens => 811 tokens #670

Closed pyliakm closed 2 weeks ago

pyliakm commented 2 weeks ago

Context / Scenario

I am using PostgreSQL + pgvector, and I got an exception when I enabled Summarizing and saved the result to the database.

var memoryConnector = new KernelMemoryBuilder()
    .WithPostgresMemoryDb(new PostgresConfig() { ConnectionString = config.GetValue<string>("PostgresSql:ConnectionString")! })
    .WithOpenAIDefaults(apiKey)
    .Build<MemoryServerless>();

await memoryConnector.ImportDocumentAsync("PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf", documentId: Guid.NewGuid().ToString(), index: "default", steps: Constants.PipelineWithSummary);

Table: 

create table
  public.km - default (
    id text not null,
    embedding public.vector null,
    tags text[] not null default '{}'::text[],
    content text not null default ''::text,
    payload jsonb not null default '{}'::jsonb,
    constraint km - default_pkey primary key (id)
  ) tablespace pg_default;

create index if not exists idx_tags on public."km-default" using gin (tags) tablespace pg_default;

create index if not exists idx_embedding_cosine on public."km-default" using hnsw (embedding vector_cosine_ops) tablespace pg_default;

What happened?

I expect that Summarizing should work for any PDF documents.

Importance

I cannot use Kernel Memory

Platform, Language, Versions

Microsoft.KernelMemory.MemoryDb.Postgres v0.62.240605.1

Relevant log output

info: Microsoft.KernelMemory.Handlers.TextExtractionHandler[0]
      Handler 'extract' ready
info: Microsoft.KernelMemory.Handlers.TextPartitioningHandler[0]
      Handler 'partition' ready
info: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Handler 'summarize' ready
info: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Handler 'gen_embeddings' ready, 1 embedding generators
info: Microsoft.KernelMemory.Handlers.SaveRecordsHandler[0]
      Handler save_records ready, 1 vector storages
info: Microsoft.KernelMemory.Handlers.DeleteDocumentHandler[0]
      Handler 'private_delete_document' ready
info: Microsoft.KernelMemory.Handlers.DeleteIndexHandler[0]
      Handler 'private_delete_index' ready
info: Microsoft.KernelMemory.Handlers.DeleteGeneratedFilesHandler[0]
      Handler 'delete_generated_files' ready
info: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsParallelHandler[0]
      Handler 'gen_embeddings_parallel' ready, 1 embedding generators
info: Microsoft.KernelMemory.Handlers.SummarizationParallelHandler[0]
      Handler 'summarize_parallel' ready
info: Microsoft.KernelMemory.Pipeline.InProcessPipelineOrchestrator[0]
      Queueing upload of 1 files for further processing [request 343e6358-3738-4f90-bfc7-70d77f2ec9e0]
dbug: Microsoft.KernelMemory.Pipeline.InProcessPipelineOrchestrator[0]
      Uploading 1 files, pipeline 'default/343e6358-3738-4f90-bfc7-70d77f2ec9e0'
dbug: Microsoft.KernelMemory.Pipeline.InProcessPipelineOrchestrator[0]
      Uploading file 'PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf', size 389705 bytes
info: Microsoft.KernelMemory.Pipeline.InProcessPipelineOrchestrator[0]
      File uploaded: PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf, 389705 bytes
dbug: Microsoft.KernelMemory.Pipeline.InProcessPipelineOrchestrator[0]
      Saving pipeline status to 'default/343e6358-3738-4f90-bfc7-70d77f2ec9e0/__pipeline_status.json'
dbug: Microsoft.KernelMemory.Pipeline.InProcessPipelineOrchestrator[0]
      Saving pipeline status to 'default/343e6358-3738-4f90-bfc7-70d77f2ec9e0/__pipeline_status.json'
dbug: Microsoft.KernelMemory.Handlers.TextExtractionHandler[0]
      Extracting text, pipeline 'default/343e6358-3738-4f90-bfc7-70d77f2ec9e0'
dbug: Microsoft.KernelMemory.Handlers.TextExtractionHandler[0]
      Extracting text from file 'PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf' mime type 'application/pdf' using extractor 'Microsoft.KernelMemory.DataFormats.Pdf.PdfDecoder'
dbug: Microsoft.KernelMemory.DataFormats.Pdf.PdfDecoder[0]
      Extracting text from PDF file
dbug: Microsoft.KernelMemory.Handlers.TextExtractionHandler[0]
      Saving extracted text file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.extract.txt
dbug: Microsoft.KernelMemory.Handlers.TextExtractionHandler[0]
      Saving extracted content PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.extract.json
info: Microsoft.KernelMemory.Pipeline.InProcessPipelineOrchestrator[0]
      Handler 'extract' processed pipeline 'default/343e6358-3738-4f90-bfc7-70d77f2ec9e0' successfully
dbug: Microsoft.KernelMemory.Pipeline.InProcessPipelineOrchestrator[0]
      Saving pipeline status to 'default/343e6358-3738-4f90-bfc7-70d77f2ec9e0/__pipeline_status.json'
dbug: Microsoft.KernelMemory.Handlers.TextPartitioningHandler[0]
      Partitioning text, pipeline 'default/343e6358-3738-4f90-bfc7-70d77f2ec9e0'
dbug: Microsoft.KernelMemory.Handlers.TextPartitioningHandler[0]
      Partitioning text file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.extract.txt
dbug: Microsoft.KernelMemory.Handlers.TextPartitioningHandler[0]
      Saving 28 file partitions
dbug: Microsoft.KernelMemory.Handlers.TextPartitioningHandler[0]
      Partition size: 830 tokens
dbug: Microsoft.KernelMemory.Handlers.TextPartitioningHandler[0]
      Partition size: 808 tokens
dbug: Microsoft.KernelMemory.Handlers.TextPartitioningHandler[0]
      Partition size: 818 tokens
dbug: Microsoft.KernelMemory.Handlers.TextPartitioningHandler[0]
      Partition size: 783 tokens
dbug: Microsoft.KernelMemory.Handlers.TextPartitioningHandler[0]
      Partition size: 971 tokens
dbug: Microsoft.KernelMemory.Handlers.TextPartitioningHandler[0]
      Partition size: 809 tokens
dbug: Microsoft.KernelMemory.Handlers.TextPartitioningHandler[0]
      Partition size: 972 tokens
dbug: Microsoft.KernelMemory.Handlers.TextPartitioningHandler[0]
      Partition size: 833 tokens
dbug: Microsoft.KernelMemory.Handlers.TextPartitioningHandler[0]
      Partition size: 788 tokens
dbug: Microsoft.KernelMemory.Handlers.TextPartitioningHandler[0]
      Partition size: 933 tokens
dbug: Microsoft.KernelMemory.Handlers.TextPartitioningHandler[0]
      Partition size: 966 tokens
dbug: Microsoft.KernelMemory.Handlers.TextPartitioningHandler[0]
      Partition size: 940 tokens
dbug: Microsoft.KernelMemory.Handlers.TextPartitioningHandler[0]
      Partition size: 826 tokens
dbug: Microsoft.KernelMemory.Handlers.TextPartitioningHandler[0]
      Partition size: 792 tokens
dbug: Microsoft.KernelMemory.Handlers.TextPartitioningHandler[0]
      Partition size: 953 tokens
dbug: Microsoft.KernelMemory.Handlers.TextPartitioningHandler[0]
      Partition size: 903 tokens
dbug: Microsoft.KernelMemory.Handlers.TextPartitioningHandler[0]
      Partition size: 929 tokens
dbug: Microsoft.KernelMemory.Handlers.TextPartitioningHandler[0]
      Partition size: 979 tokens
dbug: Microsoft.KernelMemory.Handlers.TextPartitioningHandler[0]
      Partition size: 957 tokens
dbug: Microsoft.KernelMemory.Handlers.TextPartitioningHandler[0]
      Partition size: 942 tokens
dbug: Microsoft.KernelMemory.Handlers.TextPartitioningHandler[0]
      Partition size: 930 tokens
dbug: Microsoft.KernelMemory.Handlers.TextPartitioningHandler[0]
      Partition size: 956 tokens
dbug: Microsoft.KernelMemory.Handlers.TextPartitioningHandler[0]
      Partition size: 856 tokens
dbug: Microsoft.KernelMemory.Handlers.TextPartitioningHandler[0]
      Partition size: 994 tokens
dbug: Microsoft.KernelMemory.Handlers.TextPartitioningHandler[0]
      Partition size: 984 tokens
dbug: Microsoft.KernelMemory.Handlers.TextPartitioningHandler[0]
      Partition size: 980 tokens
dbug: Microsoft.KernelMemory.Handlers.TextPartitioningHandler[0]
      Partition size: 982 tokens
dbug: Microsoft.KernelMemory.Handlers.TextPartitioningHandler[0]
      Partition size: 566 tokens
trce: Microsoft.KernelMemory.Handlers.TextPartitioningHandler[0]
      Skipping file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.extract.json (not original text)
info: Microsoft.KernelMemory.Pipeline.InProcessPipelineOrchestrator[0]
      Handler 'partition' processed pipeline 'default/343e6358-3738-4f90-bfc7-70d77f2ec9e0' successfully
dbug: Microsoft.KernelMemory.Pipeline.InProcessPipelineOrchestrator[0]
      Saving pipeline status to 'default/343e6358-3738-4f90-bfc7-70d77f2ec9e0/__pipeline_status.json'
dbug: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Generating embeddings, pipeline 'default/343e6358-3738-4f90-bfc7-70d77f2ec9e0'
trce: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Skipping file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.extract.txt (not a partition, not syntheticdata)
trce: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Skipping file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.extract.json (not a partition, not synthetic data)
trce: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Processing file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.0.txt
trce: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Generating embeddings using AI.OpenAI.OpenAITextEmbeddingGenerator, file: PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.0.txt
dbug: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Saving embedding file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.0.txt.AI.OpenAI.OpenAITextEmbeddingGenerator.TODO.text_embedding
trce: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Processing file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.1.txt
trce: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Generating embeddings using AI.OpenAI.OpenAITextEmbeddingGenerator, file: PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.1.txt
dbug: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Saving embedding file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.1.txt.AI.OpenAI.OpenAITextEmbeddingGenerator.TODO.text_embedding
trce: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Processing file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.2.txt
trce: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Generating embeddings using AI.OpenAI.OpenAITextEmbeddingGenerator, file: PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.2.txt
dbug: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Saving embedding file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.2.txt.AI.OpenAI.OpenAITextEmbeddingGenerator.TODO.text_embedding
trce: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Processing file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.3.txt
trce: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Generating embeddings using AI.OpenAI.OpenAITextEmbeddingGenerator, file: PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.3.txt
dbug: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Saving embedding file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.3.txt.AI.OpenAI.OpenAITextEmbeddingGenerator.TODO.text_embedding
trce: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Processing file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.4.txt
trce: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Generating embeddings using AI.OpenAI.OpenAITextEmbeddingGenerator, file: PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.4.txt
dbug: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Saving embedding file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.4.txt.AI.OpenAI.OpenAITextEmbeddingGenerator.TODO.text_embedding
trce: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Processing file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.5.txt
trce: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Generating embeddings using AI.OpenAI.OpenAITextEmbeddingGenerator, file: PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.5.txt
dbug: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Saving embedding file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.5.txt.AI.OpenAI.OpenAITextEmbeddingGenerator.TODO.text_embedding
trce: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Processing file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.6.txt
trce: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Generating embeddings using AI.OpenAI.OpenAITextEmbeddingGenerator, file: PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.6.txt
dbug: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Saving embedding file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.6.txt.AI.OpenAI.OpenAITextEmbeddingGenerator.TODO.text_embedding
trce: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Processing file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.7.txt
trce: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Generating embeddings using AI.OpenAI.OpenAITextEmbeddingGenerator, file: PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.7.txt
dbug: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Saving embedding file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.7.txt.AI.OpenAI.OpenAITextEmbeddingGenerator.TODO.text_embedding
trce: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Processing file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.8.txt
trce: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Generating embeddings using AI.OpenAI.OpenAITextEmbeddingGenerator, file: PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.8.txt
dbug: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Saving embedding file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.8.txt.AI.OpenAI.OpenAITextEmbeddingGenerator.TODO.text_embedding
trce: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Processing file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.9.txt
trce: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Generating embeddings using AI.OpenAI.OpenAITextEmbeddingGenerator, file: PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.9.txt
dbug: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Saving embedding file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.9.txt.AI.OpenAI.OpenAITextEmbeddingGenerator.TODO.text_embedding
trce: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Processing file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.10.txt
trce: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Generating embeddings using AI.OpenAI.OpenAITextEmbeddingGenerator, file: PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.10.txt
dbug: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Saving embedding file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.10.txt.AI.OpenAI.OpenAITextEmbeddingGenerator.TODO.text_embedding
trce: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Processing file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.11.txt
trce: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Generating embeddings using AI.OpenAI.OpenAITextEmbeddingGenerator, file: PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.11.txt
dbug: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Saving embedding file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.11.txt.AI.OpenAI.OpenAITextEmbeddingGenerator.TODO.text_embedding
trce: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Processing file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.12.txt
trce: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Generating embeddings using AI.OpenAI.OpenAITextEmbeddingGenerator, file: PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.12.txt
dbug: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Saving embedding file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.12.txt.AI.OpenAI.OpenAITextEmbeddingGenerator.TODO.text_embedding
trce: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Processing file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.13.txt
trce: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Generating embeddings using AI.OpenAI.OpenAITextEmbeddingGenerator, file: PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.13.txt
dbug: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Saving embedding file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.13.txt.AI.OpenAI.OpenAITextEmbeddingGenerator.TODO.text_embedding
trce: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Processing file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.14.txt
trce: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Generating embeddings using AI.OpenAI.OpenAITextEmbeddingGenerator, file: PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.14.txt
dbug: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Saving embedding file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.14.txt.AI.OpenAI.OpenAITextEmbeddingGenerator.TODO.text_embedding
trce: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Processing file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.15.txt
trce: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Generating embeddings using AI.OpenAI.OpenAITextEmbeddingGenerator, file: PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.15.txt
dbug: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Saving embedding file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.15.txt.AI.OpenAI.OpenAITextEmbeddingGenerator.TODO.text_embedding
trce: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Processing file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.16.txt
trce: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Generating embeddings using AI.OpenAI.OpenAITextEmbeddingGenerator, file: PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.16.txt
dbug: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Saving embedding file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.16.txt.AI.OpenAI.OpenAITextEmbeddingGenerator.TODO.text_embedding
trce: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Processing file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.17.txt
trce: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Generating embeddings using AI.OpenAI.OpenAITextEmbeddingGenerator, file: PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.17.txt
dbug: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Saving embedding file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.17.txt.AI.OpenAI.OpenAITextEmbeddingGenerator.TODO.text_embedding
trce: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Processing file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.18.txt
trce: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Generating embeddings using AI.OpenAI.OpenAITextEmbeddingGenerator, file: PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.18.txt
dbug: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Saving embedding file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.18.txt.AI.OpenAI.OpenAITextEmbeddingGenerator.TODO.text_embedding
trce: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Processing file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.19.txt
trce: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Generating embeddings using AI.OpenAI.OpenAITextEmbeddingGenerator, file: PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.19.txt
dbug: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Saving embedding file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.19.txt.AI.OpenAI.OpenAITextEmbeddingGenerator.TODO.text_embedding
trce: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Processing file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.20.txt
trce: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Generating embeddings using AI.OpenAI.OpenAITextEmbeddingGenerator, file: PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.20.txt
dbug: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Saving embedding file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.20.txt.AI.OpenAI.OpenAITextEmbeddingGenerator.TODO.text_embedding
trce: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Processing file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.21.txt
trce: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Generating embeddings using AI.OpenAI.OpenAITextEmbeddingGenerator, file: PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.21.txt
dbug: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Saving embedding file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.21.txt.AI.OpenAI.OpenAITextEmbeddingGenerator.TODO.text_embedding
trce: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Processing file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.22.txt
trce: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Generating embeddings using AI.OpenAI.OpenAITextEmbeddingGenerator, file: PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.22.txt
dbug: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Saving embedding file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.22.txt.AI.OpenAI.OpenAITextEmbeddingGenerator.TODO.text_embedding
trce: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Processing file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.23.txt
trce: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Generating embeddings using AI.OpenAI.OpenAITextEmbeddingGenerator, file: PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.23.txt
dbug: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Saving embedding file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.23.txt.AI.OpenAI.OpenAITextEmbeddingGenerator.TODO.text_embedding
trce: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Processing file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.24.txt
trce: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Generating embeddings using AI.OpenAI.OpenAITextEmbeddingGenerator, file: PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.24.txt
dbug: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Saving embedding file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.24.txt.AI.OpenAI.OpenAITextEmbeddingGenerator.TODO.text_embedding
trce: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Processing file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.25.txt
trce: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Generating embeddings using AI.OpenAI.OpenAITextEmbeddingGenerator, file: PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.25.txt
dbug: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Saving embedding file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.25.txt.AI.OpenAI.OpenAITextEmbeddingGenerator.TODO.text_embedding
trce: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Processing file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.26.txt
trce: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Generating embeddings using AI.OpenAI.OpenAITextEmbeddingGenerator, file: PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.26.txt
dbug: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Saving embedding file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.26.txt.AI.OpenAI.OpenAITextEmbeddingGenerator.TODO.text_embedding
trce: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Processing file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.27.txt
trce: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Generating embeddings using AI.OpenAI.OpenAITextEmbeddingGenerator, file: PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.27.txt
dbug: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
      Saving embedding file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.27.txt.AI.OpenAI.OpenAITextEmbeddingGenerator.TODO.text_embedding
info: Microsoft.KernelMemory.Pipeline.InProcessPipelineOrchestrator[0]
      Handler 'gen_embeddings' processed pipeline 'default/343e6358-3738-4f90-bfc7-70d77f2ec9e0' successfully
dbug: Microsoft.KernelMemory.Pipeline.InProcessPipelineOrchestrator[0]
      Saving pipeline status to 'default/343e6358-3738-4f90-bfc7-70d77f2ec9e0/__pipeline_status.json'
dbug: Microsoft.KernelMemory.Handlers.SaveRecordsHandler[0]
      Saving memory records, pipeline 'default/343e6358-3738-4f90-bfc7-70d77f2ec9e0'
trce: Microsoft.KernelMemory.Handlers.SaveRecordsHandler[0]
      Creating index 'default'
trce: Microsoft.KernelMemory.Postgres.PostgresMemory[0]
      Checking if table km-default exists
trce: Microsoft.KernelMemory.Postgres.PostgresMemory[0]
      Schema: public, Table: km-default, SQL:
                SELECT table_name
                FROM information_schema.tables
                    WHERE table_schema = @schema
                        AND table_name = @table
                        AND table_type = 'BASE TABLE'
                LIMIT 1

trce: Microsoft.KernelMemory.Handlers.SaveRecordsHandler[0]
      Saving record d=343e6358-3738-4f90-bfc7-70d77f2ec9e0//p=1c70f974af004ed683638964c8e876e0 in index 'default'
trce: Microsoft.KernelMemory.Postgres.PostgresMemory[0]
      Upserting record 'd=343e6358-3738-4f90-bfc7-70d77f2ec9e0//p=1c70f974af004ed683638964c8e876e0' in table 'public."km-default"'
trce: Microsoft.KernelMemory.Handlers.SaveRecordsHandler[0]
      Saving record d=343e6358-3738-4f90-bfc7-70d77f2ec9e0//p=c1ba6e90963c4cb4bee7a2ec199f3edd in index 'default'
trce: Microsoft.KernelMemory.Postgres.PostgresMemory[0]
      Upserting record 'd=343e6358-3738-4f90-bfc7-70d77f2ec9e0//p=c1ba6e90963c4cb4bee7a2ec199f3edd' in table 'public."km-default"'
trce: Microsoft.KernelMemory.Handlers.SaveRecordsHandler[0]
      Saving record d=343e6358-3738-4f90-bfc7-70d77f2ec9e0//p=93e6727d1ac44cbb9698ed518133910e in index 'default'
trce: Microsoft.KernelMemory.Postgres.PostgresMemory[0]
      Upserting record 'd=343e6358-3738-4f90-bfc7-70d77f2ec9e0//p=93e6727d1ac44cbb9698ed518133910e' in table 'public."km-default"'
trce: Microsoft.KernelMemory.Handlers.SaveRecordsHandler[0]
      Saving record d=343e6358-3738-4f90-bfc7-70d77f2ec9e0//p=8fa3d7e30c174acd8616bb8625b7fdf3 in index 'default'
trce: Microsoft.KernelMemory.Postgres.PostgresMemory[0]
      Upserting record 'd=343e6358-3738-4f90-bfc7-70d77f2ec9e0//p=8fa3d7e30c174acd8616bb8625b7fdf3' in table 'public."km-default"'
trce: Microsoft.KernelMemory.Handlers.SaveRecordsHandler[0]
      Saving record d=343e6358-3738-4f90-bfc7-70d77f2ec9e0//p=72c551f31873402e8bd0194099285590 in index 'default'
trce: Microsoft.KernelMemory.Postgres.PostgresMemory[0]
      Upserting record 'd=343e6358-3738-4f90-bfc7-70d77f2ec9e0//p=72c551f31873402e8bd0194099285590' in table 'public."km-default"'
trce: Microsoft.KernelMemory.Handlers.SaveRecordsHandler[0]
      Saving record d=343e6358-3738-4f90-bfc7-70d77f2ec9e0//p=10d1722153db4c5a8100648b82e1d5cd in index 'default'
trce: Microsoft.KernelMemory.Postgres.PostgresMemory[0]
      Upserting record 'd=343e6358-3738-4f90-bfc7-70d77f2ec9e0//p=10d1722153db4c5a8100648b82e1d5cd' in table 'public."km-default"'
trce: Microsoft.KernelMemory.Handlers.SaveRecordsHandler[0]
      Saving record d=343e6358-3738-4f90-bfc7-70d77f2ec9e0//p=fed4d37ad89541dcbfd2b1c2d271bb50 in index 'default'
trce: Microsoft.KernelMemory.Postgres.PostgresMemory[0]
      Upserting record 'd=343e6358-3738-4f90-bfc7-70d77f2ec9e0//p=fed4d37ad89541dcbfd2b1c2d271bb50' in table 'public."km-default"'
trce: Microsoft.KernelMemory.Postgres.PostgresMemory[0]
      Upserting record 'd=343e6358-3738-4f90-bfc7-70d77f2ec9e0//p=1721c0f228cf4aa39f55a9d2344ed017' in table 'public."km-default"'
trce: Microsoft.KernelMemory.Handlers.SaveRecordsHandler[0]
      Saving record d=343e6358-3738-4f90-bfc7-70d77f2ec9e0//p=1721c0f228cf4aa39f55a9d2344ed017 in index 'default'
trce: Microsoft.KernelMemory.Handlers.SaveRecordsHandler[0]
      Saving record d=343e6358-3738-4f90-bfc7-70d77f2ec9e0//p=65555403cc1746f295da8988da3ca922 in index 'default'
trce: Microsoft.KernelMemory.Postgres.PostgresMemory[0]
      Upserting record 'd=343e6358-3738-4f90-bfc7-70d77f2ec9e0//p=65555403cc1746f295da8988da3ca922' in table 'public."km-default"'
trce: Microsoft.KernelMemory.Handlers.SaveRecordsHandler[0]
      Saving record d=343e6358-3738-4f90-bfc7-70d77f2ec9e0//p=1cfc4d42d62948db8c107d54ffc80acf in index 'default'
trce: Microsoft.KernelMemory.Postgres.PostgresMemory[0]
      Upserting record 'd=343e6358-3738-4f90-bfc7-70d77f2ec9e0//p=1cfc4d42d62948db8c107d54ffc80acf' in table 'public."km-default"'
trce: Microsoft.KernelMemory.Handlers.SaveRecordsHandler[0]
      Saving record d=343e6358-3738-4f90-bfc7-70d77f2ec9e0//p=3038352ef5f74d1888436980267295e8 in index 'default'
trce: Microsoft.KernelMemory.Postgres.PostgresMemory[0]
      Upserting record 'd=343e6358-3738-4f90-bfc7-70d77f2ec9e0//p=3038352ef5f74d1888436980267295e8' in table 'public."km-default"'
trce: Microsoft.KernelMemory.Handlers.SaveRecordsHandler[0]
      Saving record d=343e6358-3738-4f90-bfc7-70d77f2ec9e0//p=0aaa04876c364a17a69e55290c5b1b64 in index 'default'
trce: Microsoft.KernelMemory.Postgres.PostgresMemory[0]
      Upserting record 'd=343e6358-3738-4f90-bfc7-70d77f2ec9e0//p=0aaa04876c364a17a69e55290c5b1b64' in table 'public."km-default"'
trce: Microsoft.KernelMemory.Handlers.SaveRecordsHandler[0]
      Saving record d=343e6358-3738-4f90-bfc7-70d77f2ec9e0//p=eb7d24e41e6a4811a8c01a00b803b849 in index 'default'
trce: Microsoft.KernelMemory.Postgres.PostgresMemory[0]
      Upserting record 'd=343e6358-3738-4f90-bfc7-70d77f2ec9e0//p=eb7d24e41e6a4811a8c01a00b803b849' in table 'public."km-default"'
trce: Microsoft.KernelMemory.Handlers.SaveRecordsHandler[0]
      Saving record d=343e6358-3738-4f90-bfc7-70d77f2ec9e0//p=f22e8ae6b8564d4d9df55fb9b88eb067 in index 'default'
trce: Microsoft.KernelMemory.Postgres.PostgresMemory[0]
      Upserting record 'd=343e6358-3738-4f90-bfc7-70d77f2ec9e0//p=f22e8ae6b8564d4d9df55fb9b88eb067' in table 'public."km-default"'
trce: Microsoft.KernelMemory.Handlers.SaveRecordsHandler[0]
      Saving record d=343e6358-3738-4f90-bfc7-70d77f2ec9e0//p=ef1fa334a1d24947a5ce49e31bf74186 in index 'default'
trce: Microsoft.KernelMemory.Postgres.PostgresMemory[0]
      Upserting record 'd=343e6358-3738-4f90-bfc7-70d77f2ec9e0//p=ef1fa334a1d24947a5ce49e31bf74186' in table 'public."km-default"'
trce: Microsoft.KernelMemory.Handlers.SaveRecordsHandler[0]
      Saving record d=343e6358-3738-4f90-bfc7-70d77f2ec9e0//p=bbb221d2324d400d9828ffadd41ebe11 in index 'default'
trce: Microsoft.KernelMemory.Postgres.PostgresMemory[0]
      Upserting record 'd=343e6358-3738-4f90-bfc7-70d77f2ec9e0//p=bbb221d2324d400d9828ffadd41ebe11' in table 'public."km-default"'
trce: Microsoft.KernelMemory.Handlers.SaveRecordsHandler[0]
      Saving record d=343e6358-3738-4f90-bfc7-70d77f2ec9e0//p=8830f6d43bf04587bcefcb3c40c731a0 in index 'default'
trce: Microsoft.KernelMemory.Postgres.PostgresMemory[0]
      Upserting record 'd=343e6358-3738-4f90-bfc7-70d77f2ec9e0//p=8830f6d43bf04587bcefcb3c40c731a0' in table 'public."km-default"'
trce: Microsoft.KernelMemory.Handlers.SaveRecordsHandler[0]
      Saving record d=343e6358-3738-4f90-bfc7-70d77f2ec9e0//p=c3fe5e906c77478a966c48465da42b08 in index 'default'
trce: Microsoft.KernelMemory.Postgres.PostgresMemory[0]
      Upserting record 'd=343e6358-3738-4f90-bfc7-70d77f2ec9e0//p=c3fe5e906c77478a966c48465da42b08' in table 'public."km-default"'
trce: Microsoft.KernelMemory.Handlers.SaveRecordsHandler[0]
      Saving record d=343e6358-3738-4f90-bfc7-70d77f2ec9e0//p=3494065cd5284d8db486715d619782d7 in index 'default'
trce: Microsoft.KernelMemory.Postgres.PostgresMemory[0]
      Upserting record 'd=343e6358-3738-4f90-bfc7-70d77f2ec9e0//p=3494065cd5284d8db486715d619782d7' in table 'public."km-default"'
trce: Microsoft.KernelMemory.Handlers.SaveRecordsHandler[0]
      Saving record d=343e6358-3738-4f90-bfc7-70d77f2ec9e0//p=8081c7893eba448cba08b0fc5a839bad in index 'default'
trce: Microsoft.KernelMemory.Postgres.PostgresMemory[0]
      Upserting record 'd=343e6358-3738-4f90-bfc7-70d77f2ec9e0//p=8081c7893eba448cba08b0fc5a839bad' in table 'public."km-default"'
trce: Microsoft.KernelMemory.Handlers.SaveRecordsHandler[0]
      Saving record d=343e6358-3738-4f90-bfc7-70d77f2ec9e0//p=b96f4076cf58460484cc8f253fb06c72 in index 'default'
trce: Microsoft.KernelMemory.Postgres.PostgresMemory[0]
      Upserting record 'd=343e6358-3738-4f90-bfc7-70d77f2ec9e0//p=b96f4076cf58460484cc8f253fb06c72' in table 'public."km-default"'
trce: Microsoft.KernelMemory.Handlers.SaveRecordsHandler[0]
      Saving record d=343e6358-3738-4f90-bfc7-70d77f2ec9e0//p=a6a030c191834b54a8c03177aaf3e4dd in index 'default'
trce: Microsoft.KernelMemory.Postgres.PostgresMemory[0]
      Upserting record 'd=343e6358-3738-4f90-bfc7-70d77f2ec9e0//p=a6a030c191834b54a8c03177aaf3e4dd' in table 'public."km-default"'
trce: Microsoft.KernelMemory.Handlers.SaveRecordsHandler[0]
      Saving record d=343e6358-3738-4f90-bfc7-70d77f2ec9e0//p=1d59a3d0ddd946f0a2f4f88d204f7c27 in index 'default'
trce: Microsoft.KernelMemory.Postgres.PostgresMemory[0]
      Upserting record 'd=343e6358-3738-4f90-bfc7-70d77f2ec9e0//p=1d59a3d0ddd946f0a2f4f88d204f7c27' in table 'public."km-default"'
trce: Microsoft.KernelMemory.Handlers.SaveRecordsHandler[0]
      Saving record d=343e6358-3738-4f90-bfc7-70d77f2ec9e0//p=ac0e7015201f4bd1a33c3747988f0a84 in index 'default'
trce: Microsoft.KernelMemory.Postgres.PostgresMemory[0]
      Upserting record 'd=343e6358-3738-4f90-bfc7-70d77f2ec9e0//p=ac0e7015201f4bd1a33c3747988f0a84' in table 'public."km-default"'
trce: Microsoft.KernelMemory.Handlers.SaveRecordsHandler[0]
      Saving record d=343e6358-3738-4f90-bfc7-70d77f2ec9e0//p=28c6c9dff20f43d3920a307667e0edd1 in index 'default'
trce: Microsoft.KernelMemory.Postgres.PostgresMemory[0]
      Upserting record 'd=343e6358-3738-4f90-bfc7-70d77f2ec9e0//p=28c6c9dff20f43d3920a307667e0edd1' in table 'public."km-default"'
trce: Microsoft.KernelMemory.Handlers.SaveRecordsHandler[0]
      Saving record d=343e6358-3738-4f90-bfc7-70d77f2ec9e0//p=b6aa21a676cd4058a1eef4a9d9662a57 in index 'default'
trce: Microsoft.KernelMemory.Postgres.PostgresMemory[0]
      Upserting record 'd=343e6358-3738-4f90-bfc7-70d77f2ec9e0//p=b6aa21a676cd4058a1eef4a9d9662a57' in table 'public."km-default"'
trce: Microsoft.KernelMemory.Handlers.SaveRecordsHandler[0]
      Saving record d=343e6358-3738-4f90-bfc7-70d77f2ec9e0//p=767cf8bcb61b449caca477e4a3775e27 in index 'default'
trce: Microsoft.KernelMemory.Postgres.PostgresMemory[0]
      Upserting record 'd=343e6358-3738-4f90-bfc7-70d77f2ec9e0//p=767cf8bcb61b449caca477e4a3775e27' in table 'public."km-default"'
trce: Microsoft.KernelMemory.Handlers.SaveRecordsHandler[0]
      Saving record d=343e6358-3738-4f90-bfc7-70d77f2ec9e0//p=f633a1105b6e4188b2ca375b229deb6a in index 'default'
trce: Microsoft.KernelMemory.Postgres.PostgresMemory[0]
      Upserting record 'd=343e6358-3738-4f90-bfc7-70d77f2ec9e0//p=f633a1105b6e4188b2ca375b229deb6a' in table 'public."km-default"'
info: Microsoft.KernelMemory.Pipeline.InProcessPipelineOrchestrator[0]
      Handler 'save_records' processed pipeline 'default/343e6358-3738-4f90-bfc7-70d77f2ec9e0' successfully
dbug: Microsoft.KernelMemory.Pipeline.InProcessPipelineOrchestrator[0]
      Saving pipeline status to 'default/343e6358-3738-4f90-bfc7-70d77f2ec9e0/__pipeline_status.json'
dbug: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Generating summary, pipeline 'default/343e6358-3738-4f90-bfc7-70d77f2ec9e0'
dbug: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Summarizing text file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.extract.txt
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Paragraphs to summarize: 7
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Summarizing paragraph 0
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Summarizing paragraph 1
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Summarizing paragraph 2
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Summarizing paragraph 3
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Summarizing paragraph 4
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Summarizing paragraph 5
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Summarizing paragraph 6
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Summary length: 23634 => 811
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Paragraphs to summarize: 1
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Summarizing paragraph 0
fail: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Summarization failed, the content is getting longer: 811 tokens => 811 tokens
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Skipping file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.extract.json
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Skipping file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.0.txt
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Skipping file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.1.txt
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Skipping file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.2.txt
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Skipping file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.3.txt
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Skipping file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.4.txt
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Skipping file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.5.txt
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Skipping file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.6.txt
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Skipping file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.7.txt
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Skipping file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.8.txt
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Skipping file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.9.txt
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Skipping file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.10.txt
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Skipping file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.11.txt
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Skipping file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.12.txt
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Skipping file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.13.txt
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Skipping file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.14.txt
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Skipping file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.15.txt
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Skipping file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.16.txt
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Skipping file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.17.txt
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Skipping file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.18.txt
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Skipping file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.19.txt
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Skipping file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.20.txt
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Skipping file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.21.txt
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Skipping file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.22.txt
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Skipping file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.23.txt
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Skipping file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.24.txt
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Skipping file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.25.txt
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Skipping file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.26.txt
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Skipping file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.27.txt
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Skipping file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.0.txt.AI.OpenAI.OpenAITextEmbeddingGenerator.TODO.text_embedding
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Skipping file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.1.txt.AI.OpenAI.OpenAITextEmbeddingGenerator.TODO.text_embedding
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Skipping file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.2.txt.AI.OpenAI.OpenAITextEmbeddingGenerator.TODO.text_embedding
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Skipping file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.3.txt.AI.OpenAI.OpenAITextEmbeddingGenerator.TODO.text_embedding
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Skipping file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.4.txt.AI.OpenAI.OpenAITextEmbeddingGenerator.TODO.text_embedding
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Skipping file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.5.txt.AI.OpenAI.OpenAITextEmbeddingGenerator.TODO.text_embedding
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Skipping file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.6.txt.AI.OpenAI.OpenAITextEmbeddingGenerator.TODO.text_embedding
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Skipping file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.7.txt.AI.OpenAI.OpenAITextEmbeddingGenerator.TODO.text_embedding
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Skipping file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.8.txt.AI.OpenAI.OpenAITextEmbeddingGenerator.TODO.text_embedding
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Skipping file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.9.txt.AI.OpenAI.OpenAITextEmbeddingGenerator.TODO.text_embedding
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Skipping file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.10.txt.AI.OpenAI.OpenAITextEmbeddingGenerator.TODO.text_embedding
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Skipping file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.11.txt.AI.OpenAI.OpenAITextEmbeddingGenerator.TODO.text_embedding
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Skipping file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.12.txt.AI.OpenAI.OpenAITextEmbeddingGenerator.TODO.text_embedding
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Skipping file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.13.txt.AI.OpenAI.OpenAITextEmbeddingGenerator.TODO.text_embedding
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Skipping file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.14.txt.AI.OpenAI.OpenAITextEmbeddingGenerator.TODO.text_embedding
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Skipping file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.15.txt.AI.OpenAI.OpenAITextEmbeddingGenerator.TODO.text_embedding
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Skipping file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.16.txt.AI.OpenAI.OpenAITextEmbeddingGenerator.TODO.text_embedding
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Skipping file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.17.txt.AI.OpenAI.OpenAITextEmbeddingGenerator.TODO.text_embedding
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Skipping file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.18.txt.AI.OpenAI.OpenAITextEmbeddingGenerator.TODO.text_embedding
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Skipping file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.19.txt.AI.OpenAI.OpenAITextEmbeddingGenerator.TODO.text_embedding
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Skipping file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.20.txt.AI.OpenAI.OpenAITextEmbeddingGenerator.TODO.text_embedding
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Skipping file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.21.txt.AI.OpenAI.OpenAITextEmbeddingGenerator.TODO.text_embedding
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Skipping file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.22.txt.AI.OpenAI.OpenAITextEmbeddingGenerator.TODO.text_embedding
trce: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
      Skipping file PurposePilot - Vervolg Stappen Bespreken - June 7th.pdf.partition.23.txt.AI.OpenAI.OpenAITextEmbeddingGenerator.TODO.text_embedding
dluc commented 2 weeks ago

Looks like a problem with the given file. For some reason the content of the file cannot be summarized, e.g. the code can shrink it down to 811 token but not more. A few things you can try:

dluc commented 2 weeks ago

The code has changed to avoid blocking in similar situations. If the summarization cannot compress further, it will log an error but keep the summary created. Note: you can also use Request Context to override some of the summarization settings, see PR #673