search

opensearch-project / opensearch-metrics

OpenSearch Metrics
https://metrics.opensearch.org
Apache License 2.0
3 stars 5 forks source link

[FEATURE] Client Download Stats #58

Open dblock opened 3 months ago

dblock commented 3 months ago

Is your feature request related to a problem?

We'd like to understand usage of OpenSearch clients over time.

What solution would you like?

A dashboard that groups clients together (as in projects under https://github.com/opensearch-project/OpenSearch-clients) and shows their downloads and such.

As an example, we collected some spot data with @nhtruong and @Xtansia:

prudhvigodithi commented 3 months ago

[Triage] Thanks @dblock , its good to have some dashboards on client download and usage numbers, we can observe the trends and patterers for each version and derive some action items. Adding @getsaurabh02 @bshien

prudhvigodithi commented 1 month ago

Using the following APIs and for the list of clients from this link, we can collect daily total download numbers, index them as new documents each day, and leverage OpenSearch to analyze trends and growth metrics. To begin with we can collect the total download numbers combining for all versions.

Example:

{
  "id": "opensearch-py",
  "total_downloads": 590593208,
}

We can use this number and index daily and using OpenSearch It should be possible to see the incremental growth

Nuget:

https://api.nuget.org/v3/index.json

Npm:

https://api.npmjs.org/downloads/point

Ruby:

https://rubygems.org/api/v1/gems

Python:

https://api.pepy.tech/api/v2/projects

Rust:

https://crates.io/api/v1/crates/opensearch

PHP:

https://packagist.org/packages/opensearch-project/opensearch-php/stats.json

Maven:

Maven does not provide direct download numbers (https://search.maven.org/solrsearch/select?q=g:%22org.opensearch.client%22+AND+a:%22opensearch-java%22&rows=20&wt=json), from GitHub side we collect the repo stats (but wont save the history, the documents will be updated daily with new numbers. Possible to re-index the same data with new _id for the trends)

  "_source": {
    "id": 388880322,
    "description": "Java Client for OpenSearch",
    "forks_count": 182,
    "stargazers_count": 119,
    "watchers_count": 119,
    "size": 27949,
    "open_issues_count": 134,
    "network_count": 182,
    "subscribers_count": 27,
    "transformed_at": "2024-10-11T15:45:15.482189423Z",
    "repository": "opensearch-java"
  },

We can also ingest the repo traffic https://docs.github.com/en/rest/metrics/traffic?apiVersion=2022-11-28#about-repository-traffic.

@dblock @getsaurabh02 WDYT?

prudhvigodithi commented 1 month ago

Go lang

https://pkg.go.dev/github.com/opensearch-project/opensearch-go/v2 Coming from here at present go lang does not provide any metrics/stats that shows the total downloads https://github.com/golang/go/issues/41805#issuecomment-1319196676.

Terraform

https://registry.terraform.io/v1/providers/opensearch-project/opensearch

prudhvigodithi commented 1 month ago

Once the data flows and once we have right visualizations created, similar to https://opensearch.org/release-dashboard, we can have https://opensearch.org/client-stats. :)