Bayesian beagle blog 🐶

Welcome to the Bayesian beagle blog! This project is a unique intersection of machine learning and scientific communication, providing a platform where readers can quickly get insights from the latest research papers hosted on ArXiv. Utilizing state-of-the-art Large Language Models (LLMs), our system generates concise, comprehensible summaries of complex research articles, covering a wide array of disciplines.

Our blog is built using Quarto, an open-source scientific and technical publishing system designed for creating beautiful, data-driven content. It is then published with Netlify.

graph LR
    A["Download weekly Arxiv articles"] --> B["Predict and Filter LLM topic"]
    B --> C["Summarize short docs"]
    B --> D["Summarize by Map-Reduce long docs"]
    C --> E["Update website with summaries weekly"]
    D --> E

Features

Curated ArXiv Articles: A handpicked selection of the most intriguing and high-impact research papers from various fields on ArXiv.
Automated Summaries: Each article is accompanied by a summary automatically generated by a sophisticated Large Language Model tailored for scientific content, utilizing Arxiv's new HTML (beta) formatting.
Regular Updates: Our collection is updated regularly via GitHub actions to include new research findings and innovations.
LLM-research: Coverage focuses on LLM-related research.

How It Works

Article Selection: We curate a list of ArXiv articles based on recency, impact, and relevance to a diverse audience.
Summary Generation: LLMs are employed to read and understand the selected articles and provide a human-readable summary.
Blog Publication: These summaries are formatted and published as blog posts on our Quarto-powered platform.

Usage

The blog is live at https://bayesian-beagle.netlify.app/

Navigate to the blog using the provided link and enjoy the latest research summaries. If you're interested in how the blog is generated or want to suggest improvements, feel free to check the repository or open an issue.

Installation and Setup

To clone and run this project locally, you'll need Git, Quarto, and the necessary Python packages installed on your computer. From your command line:

# Clone this repository
git clone https://github.com/wesslen/bayesian-beagle.git

# Go into the repository
cd bayesian-beagle

# Create venv
python3.9 -m venv venv
source venv/bin/activate

# Install dependencies for summary
pip install -r requirements-summarizer.txt

# Install dependencies for build
pip install -r requirements-build.txt

# Install dependencies for langchain
pip install -r requirements-langchain.txt

# Curate arxiv ids in data/input.jsonl, ensure they have HTML renderings

# Generate summaries
python scripts/summarizer.py data/input.jsonl

# Create quarto posts of summaries
python scripts/generate_qmd.py data/output.jsonl posts

# Build the Quarto blog
quarto render

License

Distributed under the MIT License. See LICENSE for more information.

Acknowledgments

ArXiv for making scientific articles openly accessible to all.
Vincent Warmerdam for his Arxiv-Frontpage project, which I extended for custom LLM labels and models
Posit for their outstanding publishing tool, Quarto.
Simon Willison's helpful strip-tags library

wesslen / bayesian-beagle

readme