@Ansh5461 @ngupta10 to go through code and create a complete mind map of how things are interacting with each other.
Look at high level how workflow starts with ingestors tasks and engine tasks
Understand how collected bytes from files are streamed into ingestor manager
How ingestor manager runs multiple collectors as separate tasks and gather them
each task above collects bytes for files and ingest one whole file once all bytes are collected
Ingestors then based on file type runs the processing and send out IngestedTokens, understand each field in IngestedTokens type and know why it is there and what are the use cases of them
Look at llama and gpt engine codes and make sure ingested tokens are correctly processing and events are generated
Ideally a nice diagram of this whole process would be ideal to close this issue if not its fine
ngupta10
commented
5 months ago
@Ansh5461 @ngupta10 to go through code and create a complete mind map of how things are interacting with each other.
Ideally a nice diagram of this whole process would be ideal to close this issue if not its fine