Clustering of projects using embeddings

[andrew]

To help automate discovery of new projects, I'd like to experiment with https://github.com/pgvector/pgvector and embeddings from a large language model to cluster projects together.

My plan is:

• [x] generate embeddings from the readme of each reviewed project
• [x] add the pgvector extension to postgresql
• [x] query the database for other projects closest to the embedding of the project
• [x] compare the "nearest" projects to their categories and topics
• [x] produce average vectors for each category of projects
• [x] produce an average of the vectors for all the reviewed projects
• [ ] provide a interface (private for now due to API costs) for, a newly proposed project, to find out:
  • [ ] the closest existing projects
  • [x] the closest categories
  • [x] distance from each category average
  • [ ] distance from the total average
• [ ] experiment with a selection of open source repositories (both climate related and totally unrelated) to find good distances to use as cut-off thresholds
• [x] experiment with including repo name, topics, description and other metadata when generating embeddings

[andrew]

Working on this branch: https://github.com/ecosyste-ms/ost/tree/pgvector

[andrew]

Experimental output for project recommendations based on the similarity of readmes: https://gist.github.com/andrew/096b4dc209cbcb8701d6acbf812a4244

[andrew]

Also working on a related item in https://github.com/ecosyste-ms/awesome/issues/3 which should help with classification of projects, in theory you'll be able to see which lists a project is most similar to.