Test out BERTopic for query dataset issue#291 - Githubissues

Samagra-Development / ai-tools

AI Tooling to bootstrap applications fast

41 stars 109 forks source link

Test out BERTopic for query dataset issue#291 #309

Closed pmukesh31 closed 2 months ago

pmukesh31 commented 2 months ago

Aim: Get an accurate list of topics (around 20 topics max) for an agri dataset of queries (has around 20k unique queries) using BERTTopic.

Description:

Implemented BERTopic model to accurately segment the agriculture dataset into 20 distinct topics.
Utilized the 'questioninEnglish' column containing approximately 20,000 unique queries for topic analysis.
Successfully generated 20 topics using BERTopic, leveraging contextual embeddings from BERT for clustering.
Open to suggestions for improving topic cluster evaluation and enhancing the clustering process.