search

openstreetmap / openstreetmap-website

The Rails application that powers OpenStreetMap
https://www.openstreetmap.org/
GNU General Public License v2.0
2.22k stars 918 forks source link

Guidance on Enhancing OSM Profile Pages with Contribution Data #5356

Open kcne opened 14 hours ago

kcne commented 14 hours ago

Problem

We want to enhance the OSM profile pages with more engaging and informative features, similar to HDYC. These features aim to display user contribution summaries, activity patterns, and insights into changesets. Our goal is to integrate this functionality directly into the openstreetmap-website while ensuring efficiency and scalability.

Key Challenges:

  1. Data Access:

    • The required data already exists in the OSM database, so we won’t rely on planet dumps for this integration.
    • For development purposes, we want to import a subset of data locally to test queries and ensure they work well with larger datasets.

  2. Performance:

    • Querying user contribution data efficiently at scale is critical. Local testing will help us benchmark and optimize queries for production.

  3. Integration:

    • The solution should align with OSM's infrastructure, avoiding external systems or unnecessary overhead.

Description

To achieve this, we’re seeking guidance on:

Questions

  1. Has anyone implemented similar features for OSM profile pages? If so, what approach did you take?
  2. Are there recommended tools or workflows (e.g., Osmium, osm-history-renderer) for importing and testing user contribution data efficiently?
  3. How can we test and benchmark queries locally without committing to a full-scale data import?
  4. Can these queries be handled within OSM's infrastructure, or would a separate system be necessary for performance?

We welcome any advice, insights, or references to similar projects to guide us forward. Tagging: @gravitystorm @tomhughes @AntonKhorev @pa5cal

pnorman commented 10 hours ago
  • Changesets in the planet OSM files contain user data, but tools like Osmosis and Osmium cannot directly link users to contributions in a database.
  • Full history dumps provide the necessary data but are massive (~223 GB compressed, ~3.7 TB uncompressed) and complex to process.

If you're looking in the context of openstreetmap-website, this data is already present in the database and you'd never look at the dumps. You'd only use the planet files if you were building something external.

kcne commented 9 hours ago

Thank you for the quick response! Yes, we’re focused on the openstreetmap-website and plan to integrate this directly. Testing queries and performance locally with larger datasets ensures reliability and simplifies development. That’s why we aim to import a subset of data for development, enabling us to test and refine features effectively before integration.