[Enhancement] Support for Multiple Data Sources

[shreyas-damle]

Pebblo (as of 0.1.12 version) supports single data source. Having support multiple data sources within single RAG application would be a good feature.

Description: When I have multiple data sources to be used in my app, I should be able to see all those data sources and their details in the pebblo report.

As part of this feature, following changes would need to be done in the report:

1. Report Summary: Aggregate details about all data data sources.
2. Top Files With Most Findings: Add new column to show to which data source the file belongs.
3. Data Source: It would show snippets about all data sources.

[rahul-trip]

Analysis from the plugin side:

Problem at hand: What should be the trigger to generate the report? Earlier (In case of single loader) we used to send a flag, loading_end: bool. Now with multiple document loaders we have the following scenarios:

1. loader1 = PebbloSafeLoader(ABCDocumentLoader(abcpath), abcname, abcdescription, abcowner) #discocery loader2 = PebbloSafeLoader(XYZDocumentLoader(xyzpath), xyzname, xyzdescription, xyzowner) loader1.load() #classification loader2.load() 2. loader1 = PebbloSafeLoader(ABCDocumentLoader(abcpath), abcname, abcdescription, abcowner) #discocery loader1.load() #classification loader2 = PebbloSafeLoader(XYZDocumentLoader(xyzpath), xyzname, xyzdescription, xyzowner) loader2.load()

Note: PebbloSafeLoader instantiation --> app:loader discovery, and it's .load() --> classifies the loaded data.

In 1st case we will get to know the first loader and then the 2nd loader, and then both loaders will be loaded (and their data classified) one by one (generating reports twice, on every loading_end = true from both the loaders). But in 2nd case when the first loader is discovered and loaded just after its discovery and just before the discovery of 2nd loader, server will get loading_end = True (first loader) and report generation will get triggered, We need to make some adjustments here.

As of today, the pebblo server will generate the report with every new loading_end = true. the 2ne case mentioned above is a kind of race condition where we won't be able to get a trigger point to make sure the loading is actually over in an RAG app. Let's wait for a few seconds before generating the report, a new loader discovery might just kick in. Also, we generate(overwrite) a new report with every new loading_end=True

[shreyas-damle]

In order to support multiple data sources, we need to update out folder structure within scratch space as follows:

.pebblo/
       /<app-name>/
                  /metadata/metadata.json             => app name and run_id to load_id mappings.
                                                         This file will be protected by file lock.
                  /<run-id>/
                           /metadata/metadata.json    => All the details needed to generate report.json.
                                                         This will include details for all successful loads.
                           /report.json               => This is input to reports module.
                                                         This will include details for all successful loads.
                           /pebblo_report.pdf         => Report pdf including details for all successful loads.
                  /<load-id>/
                           /metadata/metadata.json.   => All the details needed to generate report.json.
                                                         This will include details for all successful runs.
                                                         This is only for backward compatibility
                           /report.json               => This is input to reports module.
                                                         This will include details for all successful runs.
                                                         This is only for backward compatibility
                           /pebblo_report.pdf         => Report pdf including details for all successful runs.
                                                         This is only for backward compatibility