Closed oxaroky02 closed 4 months ago
The goal is to iterate over chunks and embed them.
Chonker#each
TheIngestor#ingest
Changes to parsers to actually incrementally yield chunks per parser will come afterwards.
The goal is to iterate over chunks and embed them.
Chonker#eachcan take a block to yield chunks (or returns iterator when called without a block)TheIngestor#ingestmodified to iterate through chunks and embed them.Changes to parsers to actually incrementally yield chunks per parser will come afterwards.