Replacing existing stats.jenkins.io code with https://github.com/jenkins-infra/stats.jenkins.io

krisstern commented 2 months ago

Service(s)

Helpdesk, stats.jenkins.io

Summary

As the Implementing UI for Jenkins Infra Statistics project has officially completed, we would like to request for the existing code for the website https://stats.jenkins.io/ with that in https://github.com/jenkins-infra/stats.jenkins.io

c.c. @shlomomdahan @gounthar @Vandit1604

Reproduction steps

No response

dduportal commented 2 months ago

That would be the (almost) last mile of https://github.com/jenkins-infra/helpdesk/issues/4132 🥳

krisstern commented 2 months ago

Good to know @dduportal 👍🏼

hervelemeur commented 2 months ago

I'll prepare a plan for that today hopefully.

lemeurherveCB commented 2 months ago

From https://github.com/jenkins-infra/helpdesk/issues/4132#issuecomment-2312568426:

It means planning the effective switch to the new webservice.

We would also need to define what we do with the existing service:

How much time do we keep it ?

Archiving ?

etc.

As the existing website is running from Github Pages, I propose to just change its subdomain from stats.jenkins.io to old.stats.jenkins.io and keep it as there is no related infra cost, WDYT?

If we don't want to keep it, we can remove these lines: https://github.com/jenkins-infra/azure-net/blob/a51c2d937433c51eb289b319123beeb8e9477364/dns-records.tf#L287-L290

To switch to the new service we just have to exchange the DNS records.

Note: there is nothing to archive per se as https://github.com/jenkins-infra/infra-statistics is still used by the new website.

lemeurherveCB commented 2 months ago

To switch to the new service we just have to exchange the DNS records.

Opened https://github.com/jenkins-infra/azure-net/pull/291

dduportal commented 2 months ago

From #4132 (comment):

It means planning the effective switch to the new webservice. We would also need to define what we do with the existing service:

How much time do we keep it ?

Archiving ?

etc.

As the existing website is running from Github Pages, I propose to just change its subdomain from stats.jenkins.io to old.stats.jenkins.io and keep it as there is no related infra cost, WDYT?

If we don't want to keep it, we can remove these lines: https://github.com/jenkins-infra/azure-net/blob/a51c2d937433c51eb289b319123beeb8e9477364/dns-records.tf#L287-L290

To switch to the new service we just have to exchange the DNS records.

Note: there is nothing to archive per se as https://github.com/jenkins-infra/infra-statistics is still used by the new website.

What about the "process" which updates the branch gh-pages?

lemeurherveCB commented 2 months ago

@dduportal the pipeline https://github.com/jenkins-infra/infra-statistics/blob/main/Jenkinsfile must be kept running.

It generates and pushes data to the gh-pages:

The data generated is then used by the new stats website, which is only a frontend retrieving the data by downloading them from the content of gh-pages branch: https://github.com/jenkins-infra/stats.jenkins.io/blob/46f61c7264d72ae0b6d79551652ee4036c50a4ee/retrieve-infra-statistics-data.sh

lemeurherveCB commented 2 months ago

We may want to add some explanations on https://github.com/jenkins-infra/infra-statistics README to clarify this.

dduportal commented 2 months ago

DNS change: LGTM. Let's plan it for 11 September if no objection (to be confirmed during team meeting)
Old site on GH Page: it must be decommissioned. We don't want to have a resource published (it uses common resources: domain names, etc.) and it can be abused (like a DNS not renewed).
- How much time should we keep it "just in case" once the DNS is switched? 1, 2 weeks? More ? (will discuss it during team meeting and I'll report here)
- WDYT about changing the branch to data (instead of gh-pages) at this moment? It will allow us to disable the GH Page website once done

lemeurherveCB commented 2 months ago

data looks fine to me 🙂

lemeurherveCB commented 2 months ago

It will allow us to disable the GH Page website once done

Note that it can also be disabled by setting "none" as branch in GitHub Pages settings of the repository:

lemeurherveCB commented 2 months ago

As the job run once per month there will be plenty of time switching the branch name from gh-pages to data in both repositories without interrupting the service: https://github.com/jenkins-infra/infra-statistics/blob/8209b583ee5be409d2dc8cdf780bd4358579b8de/Jenkinsfile#L18-L19

// Make sure we run this job once a month (at 3am on the 2nd of the month) when the raw stats are available.
properties([pipelineTriggers([cron('0 3 2 * *')])])

lemeurherveCB commented 2 months ago

@MarkEWaite @shlomomdahan I'd also suggest that when https://github.com/jenkins-infra/azure-net/pull/291 is merged and released, one of you could announce the new website on https://groups.google.com/g/jenkinsci-users and https://groups.google.com/g/jenkinsci-dev, WDYT?

Maybe a blog post too?

krisstern commented 2 months ago

Hi @lemeurherveCB Thanks! Or I could make an announcement for the revamped project too.

dduportal commented 2 months ago

@krisstern @lemeurherveCB We propose to switch the production to the new site tomorrow, Wednesday 11 September at 08:00am UTC (10h00am Paris time).

The rest can wait (by default 1 week).

No objection?

hervelemeur commented 2 months ago

@dduportal looks fine to me, but my opinion is not that important, @krisstern's one is.

krisstern commented 2 months ago

No objection from me

dduportal commented 2 months ago

We did change the DNS:

The PR in Azure failed (TTL not up to date and non atomic change). We fixed it manually
The stats website is now broken as the production in AKS does not have the required ingress setup. We are operating it right now to hotfix but the website is in outaghe until it's done

dduportal commented 2 months ago

Update: the new site is live (https://stats.jenkins.io/) and the old site is now available again (https://old.stats.jenkins.io/)

Needed to fix the ingress on the AKS production to handle requests with the host stats.jenkins.io
- See https://github.com/jenkins-infra/kubernetes-management/pull/5675
Needed to change the expected CNAME on the GH Page setup
- See https://github.com/jenkins-infra/infra-statistics/pull/54

krisstern commented 2 months ago

I will make an announcement about the new revamped stats.jenkins.io site within the next couple of hours

krisstern commented 2 months ago

My apologies, I have been waiting for https://github.com/jenkins-infra/plugin-site/pull/1890 to be merged first before making an announcement...

dduportal commented 2 months ago

My apologies, I have been waiting for jenkins-infra/plugin-site#1890 to be merged first before making an announcement...

many thanks @krisstern !

dduportal commented 2 months ago

I'm closing this issue as the scope "put the GSoC new website online as default" is ok. See https://github.com/jenkins-infra/helpdesk/issues/4132#issuecomment-2346204602 for the next steps.

lemeurherveCB commented 2 months ago

EDIT: copied at https://github.com/jenkins-infra/helpdesk/issues/4132#issuecomment-2347009570 to continue the discussion there (still open)

As noted in https://github.com/badges/shields/pull/10522#issuecomment-2346864474, there are far more consumers than we (I) initially though: https://github.com/search?q=stats.jenkins.io&type=code 😱 Links to (old.)stats.jenkins.io were pointing to files (still) stored on GitHub (currently on the `gh-pages`, generated from the pipeline of https://github.com/jenkins-infra/infra-statistics), while the new frontend doesn't serve them. To avoid breaking existing consumers and without serving these files from our service, I propose to put in place redirections from the three folders of https://github.com/jenkins-infra/infra-statistics/tree/gh-pages (then from the `data` branch) to `https://raw.githubusercontent.com/jenkins-infra/infra-statistics/gh-pages/` with the help of https://github.com/jenkins-infra/helm-charts/pull/1332, WDYT? Ex: https://stats.jenkins.io/plugin-installation-trend/view-job-filters.stats.json would redirect to https://raw.githubusercontent.com/jenkins-infra/infra-statistics/gh-pages/plugin-installation-trend/view-job-filters.stats.json

jenkins-infra / helpdesk