Service: Establish Dataverse Project Metrics Service

cmbz commented 5 months ago

Purpose

Design, implement & operationalize a metrics service for the Dataverse Project and Harvard Dataverse Repository.

Background

A number of metrics about Dataverse Project and Harvard Dataverse are collected at different time intervals and published to different locations. For instance, some metrics are available from the Dataverse.org home page: https://dataverse.org/metrics, the IQSS metrics webpage: https://www.iq.harvard.edu/metrics. Metrics also appear on the Harvard Dataverse Repository homepage: https://dataverse.harvard.edu/. Use metrics for individual datasets are also shown on Harvard Dataverse Dataset pages. Other metrics related to Harvard Dataverse Repository (e.g., number of user searches or most popular users searches) are currently produced and tracked by the software but not always publicized.
Within IQSS there is a need for ready access to these, and additional new metrics about the Dataverse Project and Harvard Dataverse Repository for monitoring and marketing purposes and to quantify and demonstrate growth of the software and repository over time.
The project will:
- Centralize all Dataverse and Harvard Dataverse-related metrics
- Identify the existing and new elements needed to collect and present Dataverse Project and Harvard Dataverse Repository metrics
- Improve existing or implement new processes (technical and administrative) to collect desired data and produce metrics in a repeatable, highly automated fashion.
- Establish an implementation timeline, including specific short, medium, and long term tasks needed to produce all required metrics
- Define a repeatable, highly automated process for aggregating and delivering existing metrics and for supporting new metrics as needed.
Eventually, hub.dataverse.org will become the only authoritative source of information of all the original three metrics pages.

Harvard Dataverse Repository Metrics

Service will provide the following:

Regular automated monitoring and sharing of metrics about HDV and related services, including storage usage, downloads, and curation
Static and interactive visualizations of data about HDV for visual exploration and use in Web pages and reports
Provide input for IQSS and Harvard Library strategic planning efforts and HDV research projects
Support user dataset creation and usage report requests for NIH and other grants

Tasks

[ ] Develop a services definition and plan outlining different program components, design, and timeline
[ ] https://github.com/IQSS/dataverse.harvard.edu/issues/280

Monitoring Issues

Resources

Dataverse Hub
Dataverse Hub repo: https://github.com/jp-tosca/dataverse-hub
Service: Establish Dataverse Project and Harvard Dataverse Repository Metrics Service (Deprecated)

Metrics Resources

cmbz commented 5 months ago

Status: June 2024

Proposal outline created by @siacus here: https://iqss.slack.com/archives/C03R1E7T4KA/p1719237600176579

cmbz commented 5 months ago

Status: July 2024

Met with @jp-tosca and @siacus to discuss plan and itemize tasks.
New channel created: https://iqss.slack.com/archives/C07CFTZ0YG6
Began tracking work: https://github.com/IQSS/dataverse-pm/issues/303
Additional tasks:
- Coordinate with @scolapasta on Market place work.
- Then prioritize and schedule tasks.
Related: https://github.com/IQSS/dataverse-pm/issues/278

jp-tosca commented 2 months ago

Status: August 2024

Completed Analysis of the data source from https://www.iq.harvard.edu/metrics and reported several issues related to this website.
Completed analysis of the data source from https://dataverse.harvard.edu/
Proposal of design philosophy and features.
An initial definition of data is required for metrics generation.

qqmyers commented 1 month ago

FWIW: #10865 fixes a bug that has affected metrics coming from Dataverse, particularly for those like Harvard that have many harvested datasets.

jp-tosca commented 1 month ago

Status: September 2024

We presented to de Dataverse team a document on this link it includes the results of the initial analysis, an initial proposal for the data schema and a list of features that will make an MVP.
We deprecated the original proposal in favor of the new Dataverse Hub document. Future updates to design, plans, and timeline will be made to this new document.

jp-tosca commented 2 weeks ago

Status: October 2024

We started the development of the prototype for the Dataverse Hub, some of the tasks completed and features that have been implemented are:

Endpoints:

Endpoint to retrieve all the registered Dataverse installations
Endpoint to retrieve all the current status of the registered Dataverse installations
Endpoint to add a new Dataverse Installation to the database

Scheduled Jobs

Scheduled job to retrieve the current list of installations, this takes our file from this link
Scheduled job to retrieve the status and basic information from each Dataverse installation.

Features

Basic non-encrypted token based authentication to protect endpoints like the one that adds a Dataverse installation.
100% Compliant OpenAPI documentation document which we hope will guide us as a team in the future to improve the current implementation on Dataverse that is currently at 10%.
Database setup and connection with Postgres and versioning using Flyway.

Frontend

We have developed a Shiny app to test the endpoints creating a simple map and a list of the installations and current status.
Poetry for dependency management.
Python for Shiny development
Shiny Express for shiny.
Polars over panda due performance concerns.

jp-tosca commented 2 days ago

Status: November 2024

We continued the development of the prototype for the Dataverse Hub maintaining 100% compliance with all the OpenAPI specification rules and adding new features and endpoints:

Features:

-Added Jacoco plugin to measure our test code coverage and added minimal unit tests for the current codebase. -Introduced basic server logging to the scheduled jobs. -Refactored the documentation annotations for OpenAPI with custom tags to keep the documentation tags separate from the functional code.

Endpoints:

Development metrics

/api/dev Endpoint to retrieve metrics from the Dataverse repository.
/api/dev/releases Endpoint to retrieve all the Dataverse releases metrics.

Dataverse metrics.

/api/installation/metrics Endpoint to retrieve the latest metrics from all the healthy Dataverse Installations
/api/installation/country Endpoint to retrieve the number of installations by country

Scheduled Jobs

Added scheduled job to retrieve the current metrics of the Dataverse repository.
Added scheduled job to retrieve the current metrics of each Dataverse Installation that is healthy.

IQSS / dataverse-pm