2.2 Conduct E2E Testing using other protocols from Boardroom

raid-guild / gaianet-rag-api-pipeline

Supercharge your Gaianet node by generating a vector knowledge base from any API. Demo slides: https://hackmd.io/@santteegt/ByoykY4nC#/ Link to Docs below

https://raid-guild.github.io/gaianet-rag-api-pipeline/

MIT License

1 stars 0 forks source link

2.2 Conduct E2E Testing using other protocols from Boardroom #28

Closed earth2travis closed 1 month ago

earth2travis commented 2 months ago

Conduct end-to-end (E2E) testing with four additional DAO protocols to ensure the robustness and scalability of the pipeline. The process includes:

Identify Four DAO Protocols: Select four DAO protocols from the Boardroom Governance API for testing. Ensure that these protocols represent a diverse set of data and use cases to thoroughly validate the system's performance.

Run End-to-End Tests: Execute the end-to-end testing process with the selected DAO protocols.

This includes:

Extracting data from the Boardroom Governance API for each DAO.
Parsing, normalizing, and transforming the data into a structured format.
Generating vector embeddings and storing them in the Qdrant vector store.
Validating the accuracy, completeness, and performance of the data processing and retrieval.

Comprehensive E2E testing with these additional DAO protocols is crucial for ensuring that the pipeline can handle various data contexts and deliver reliable and accurate outputs.

santteegt commented 2 months ago

Criteria:

dao framework diversity
discourse content

wtfsayo commented 2 months ago

Suggestions

NounsDAO (NFT + Gov Project) - nounsdao
VitaDAO (Science related DAO) - votevitadaoeth
SeedClub (Web3 Startup accelerator) - clubeth

santteegt commented 2 months ago

A 4th testing dao should be "moloch-based":

molochdao (doesn't use discourse so it's good for testing edge cases)
metacartel

santteegt commented 1 month ago

Preliminary Results with base chunk parameters

Protocol	discourseTopics	discourseCategories	discourseTopicPosts	Proposals	Total Records	Chunked	run-all	chunk-only	embed-only
Aave	204	7	15126	1267	16604 + 1	152619	--	--	--
NounsDAO	505	5	2095	632	3237 + 1	32190	--	--	--
Optimism	119	17	19853	188	20177 + 1	120409	--	291.28s	--
VitaDAO	0	0	0	127	127 + 1	2128	94.87s	21.56s	62.97s
SeedClub	0	0	0	21	21 + 1	277	28.55s	18.56s	8.56s
Metacartel	45	5	1811	624	2485 + 1	13547	376.95s	59.97s	304.41s
Molochdao	0	0	0	49	49 +1	153	26.42s	27.54s	6.20s

wtfsayo commented 1 month ago

I think pipeline needs to be tweaked bit for number of pending/blocking requests to not overwhelm ollama or qdrant endpoint.
we should pass cname command line as an option if possible from environment

@santteegt

wtfsayo commented 1 month ago

Protocol	discourseTopics	discourseCategories	discourseTopicPosts	Proposals	Total Records	Chunked	run-all	chunk-only	embed-only
Aave	204	7	15126	1267	16604 + 1	152619	3578.19s	274.59s	2709.52s
NounsDAO	505	5	2095	632	3237 + 1	32190	847.02s	90.83s	759.23s
Optimism	119	17	19853	188	20177 + 1	120409	3784.94s	279.40s	2414.94s
VitaDAO	0	0	0	127	127 + 1	2128	92.66s	24.85s	52.58s
SeedClub	0	0	0	21	21 + 1	277	29.86s	16.63s	9.85s
Metacartel	45	5	1811	624	2485 + 1	13547	455.06s	58.10s	369.77s
Molochdao	0	0	0	49	49 +1	153	24.42s	19.05s	5.90s