Sign Up (1k chunks free) | HN Search and RAG |Documentation | Meeting Link | Discord | Matrix

Trieve HN Discovery is Home to All Code for Hacker News Search and RAG

About

This repository contains the code for an engine which provides sub-100ms vector search (SPLADE, dense vector, cross-encoder re-rank'ed hybrid), recommendations, RAG, and analytics for Hacker News.

Quick Links

• Launching Trieve HN Discovery - A Discovery Focused Search Engine for Hacker News

Contributing

Issues and contributions are welcome! There are three main folders for this project and they contain the code for their respective components of the application.

• actix-frontend for the nojs version at hnnojs.trieve.ai built using Rust actix-web and Minijinja
• solidjs-spa for the power tool JS version at hn.trieve.ai
• ingest for the various scripts relying on Redis queue's for pulling all of the data from the API and indexing it into a Trieve instance for search and RAG

Why Make This?

1. Dense vector semantic search, re-rankers, SPLADE, and other techniques have gotten a lot of hype recently, but it's hard to figure out where each technique is best applied. We're hackers and built hacker'y features into Trieve to make it easier to test and experiment with this new tech, and needed a dataset us and our friends would have knowledge of to mess with. We're hoping to build lots of cool shareable projects on top of this demo to learn and collect feedback with.

2. Make HN search more explorable with recommendations plus RAG and more open with public analytics. We added a feedback button under the search bar and are collecting CTR data such that we can continously fine-tune our models to improve the search experience. If you have some time, try out our blind comparison at hn-comparison.trieve.ai to help us collect higher quality training data.

3. Open up a more fully-featured discovery API for others who want to build on top of HN data. The firebase HN API is great, but storing all the data yourself is expensive and time-consuming. We are hoping to lower that barrier in the old Octopart HNSearch spirit.

4. Show what's possible with Trieve, work out scaling/stability bugs, and pressure test.

5. HN is a cool place where we like to spend time and showing what we are working on to this community we feel apart is something we're excited about!