posts/2024-06-25-evaluation-finetuning-manual-dataset

[utterances-bot]

Alex Strick van Linschoten - How to think about creating a dataset for LLM finetuning evaluation

I summarise the kinds of evaluations that are needed for a structured data generation task.

https://mlops.systems/posts/2024-06-25-evaluation-finetuning-manual-dataset.html

[SoMaCoSF]

This is great. Whats really amazing is that so many people are popping up with similiarish ideas on how they want to use models to pull meaningful structured data out of the vast firehose of information we see.

For me this is exactly what I want to do, as I am sure Quiver Quantative would as well, where I want to take a similar press release regarding a law/act that passes and the resultant government contacts, congressional stock moves, international trade deals, aid dispersal, company bonuses, etc can be tied to such. In the same way this creates a DB for conflict meta data that can provide accountability - telemetry on conflict casualties etc -- applying exactly this to legal/financial/political accountability and transparency.

Apply this to treading the panama paper trail and an amazing lattice may reveal.

https://www.quiverquant.com/congresstrading/

[SoMaCoSF]

Follow up - I realize that what I am working on is very similar with respect to correlating the similar notions between contexts - where I am calling my approach "Contextual Archetype Framework"

Contextually-Validated Tags for Enhanced Organizational AI Use

In an organization, leveraging large language models (LLMs) and AI can be significantly optimized by using contextually-validated tags, similar to hashtags. These tags would act as weighted indicators of how well a piece of information is validated within the organization's knowledge graph. When correlated phrases are triggered within a prompt, these validated contextual notions link the new query to existing solutions, creating a highly efficient and context-aware AI system.

So when youre calling our the phrases which need ccategorizing/classification - what I am proposing in my project is that you would evaluate the phrases such as "a male" a "person" etc - and assign the "Validated Archetype" to that phrase or concept - meaning that by doing so the model wll already have a libray of pre-valided context vectors that define that part of the prompt's output - ideally as your library of validated notional archetypes grows, you have abstracted the prompt's compute footprint...

[SoMaCoSF]

Not to spam - but as I cannot edit my previous - in reading more about your goal, I highly recommend taking a peak at the post history of this particular Reddit user: https://old.reddit.com/user/backcountrydrifter/comments/?sort=top&t=month <--- They have a ridiculous amount of well linked posts about events people and data that you're documenting with this project and including this users attempt to share a similar lens of information, it would be great for YSK.