Automate update of metrics

clyne commented 2 years ago

Metrics for the landing page are currently generated manually. They should be automated where possible. In particular, # of publications and # of downloads should be automated.

clyne commented 2 years ago

see also #12

jukent commented 2 years ago

Hi @clyne Can I have higher permissions to this repository so I can add a PAT to the settings for the GitHub actions necessary to do this? -- or you could add a PAT?

clyne commented 2 years ago

done

jukent commented 2 years ago

I have this working for publications now!!

I reached out to Virginia to get the SIParCS info (#30) which I can then update similarly.

However, I'm not sure how to do the downloads metrics. @clyne Could you post some relevant links to where that info might be? Do we want the total downloads of every VAST package?

clyne commented 2 years ago

I have this working for publications now!!

Awesome!!

I reached out to Virginia to get the SIParCS info (#30) which I can then update similarly.

However, I'm not sure how to do the downloads metrics. @clyne Could you post some relevant links to where that info might be? Do we want the total downloads of every VAST package?

Hmm. That is a question @erogluorhan. Orhan is it possible do script the collection of conda download metrics here?

erogluorhan commented 2 years ago

Hmm. That is a question @erogluorhan. Orhan is it possible do script the collection of conda download metrics here?

Hi @jukent , we have this https://github.com/NCAR/geocat_conda_metrics/blob/main/scrape_metrics.py for our own scraping purposes for several GeoCAT tools distributed in various channels at Conda. I believe a similar scraper code would work to read metrics for VAST tools. Please have a look at it and let me know if you have any questions.

jukent commented 2 years ago

Thanks this helps a lot! I'll look more closely at it tomorrow

jukent commented 2 years ago

@NihanthCW could you give me a list of any relevant VAPOR packages to include?

jukent commented 2 years ago

I created a yaml file of packages indata/packages.yml that just has the package name, team, and repository url. Do we want to turn this into a viewable page as well with a brief summary of each package, current release number, maybe a logo, and documentation link? If so I will need information from the project leads.

NihanthCW commented 2 years ago

@NihanthCW could you give me a list of any relevant VAPOR packages to include?

I don't think we have any additional packages. The upcoming python api could be one candidate to include once it is released. I'll keep you posted on that.

erogluorhan commented 2 years ago

Hi @jukent , I think the automation of scraping metrics is still a work in progress, but'd like to give a few suggestions FWIW:

In the GeoCAT case, we needed to record the daily scraped metrics into a DB for future cases where we'd want to run analysis for date intervals. In the VAST webpage case however, I think you'd not need to record things into a DB. Instead, if you only have a daily-running workflow, that would scrape up-to-date metrics from Conda for each package and then update the metrics numbers of the website, that would suffice. If this makes sense, you can get rid of the DB-related lines from scraper python file or others.
That being said, I think you'd not need an on-demand scraping workflow as well (since a daily automated workflow would work sufficient for the website)

jukent commented 2 years ago

Thank @erogluorhan I believe I've done these steps. I'm going to work on the environment failure and will reach out if I hit a roadblock.

jukent commented 2 years ago

I was having issues with binstar-client. Looks like that moved to now be within anaconda-client. Trying that now. Posting documentation for reference: https://github.com/Anaconda-Platform/anaconda-client/tree/master/binstar_client/utils

NCAR / vast

Automate update of metrics #42