search

jenkins-infra / helpdesk

Open your Infrastructure related issues here for the Jenkins project
https://github.com/jenkins-infra/helpdesk/issues/new/choose
17 stars 10 forks source link

[INFRA-3143] Create an Azure blob storage container and creds for Jenkins usage stats raw data #2688

Open jenkins-infra-bot opened 2 years ago

jenkins-infra-bot commented 2 years ago

Please create an Azure blob storage container/bucket/whatever the term is for Jenkins usage stats. The contents of /srv/bigger-usage/usage-stats on usage.jenkins.io will be synced to this container. That currently amounts to ~105GB of data, but obviously will be constantly increasing. I'm planning to use the cool access tier for this data, since any individual gz file will theoretically only be read once (when the new jenkins-usage-stats tool checks the container to see if there are any new gzip files it hasn't yet imported into its database and then downloads said new gzip files) but it probably doesn't really matter - the monthly storage costs for the current data would be $1.89 for hot tier and $1.05 for cool tier, so...eh. Anyway.

This container should be pretty well locked down, because why not. We'll need two credentials for this - one for use on usage.jenkins.io to upload new files (so this cred will need write access) and one for use when running the new jenkins-usage-stats tool's fetch command to download not-yet-imported files as part of a daily job running on an ephemeral pod in the Jenkins project infrastructure (so this cred will just need read access). I don't know enough about Azure's credentials model to be sure, but I think that SAS tokens would do the trick. If y'all know better, I'm happy to use whatever approach you want so long as we can use it with the azcopy CLI tool for uploading and azure-sdk-for-go for downloading. =)

Thanks!

Originally reported by abayer, imported from: Create an Azure blob storage container and creds for Jenkins usage stats raw data
  • assignee: hlemeur
  • status: Open
  • priority: Minor
  • resolution: Unresolved
  • imported: 2022/01/10
jenkins-infra-bot commented 2 years ago

hlemeur:

I'll look into it this month or in the beginning of next year if it's OK for you.

Part of https://github.com/jenkins-infra/jenkins-usage-stats/issues/2