search

jenkins-infra / helpdesk

Open your Infrastructure related issues here for the Jenkins project
https://github.com/jenkins-infra/helpdesk/issues/new/choose
17 stars 10 forks source link

To host stats.jenkins.io GSoC 2024 project in jenkins-infra #4132

Closed krisstern closed 1 month ago

krisstern commented 5 months ago

Service(s)

Helpdesk, stats.jenkins.io

Summary

We are in the process of redeveloping the frontend for the data presentation of the Jenkins Infra Statitics project, and will need help migrating the GSoC contributor's repo (https://github.com/shlomomdahan/stats.jenkins.io/) from his personal GitHub account to one in the jenkins-infra org.

@lemeurherve @gounthar @shlomomdahan

Reproduction steps

N/A

lemeurherve commented 5 months ago

@shlomomdahan could you initiate the transfer of your repository to https://githubcom/jenkins-infra please?

See https://docs.github.com/en/repositories/creating-and-managing-repositories/transferring-a-repository#transferring-a-repository-owned-by-your-personal-account

lemeurherve commented 5 months ago

Action plan

Similar to:

lemeurherve commented 5 months ago

@krisstern transfered the repository to jenkins-infra org (thanks!).

I've set @krisstern as "maintainer", and https://github.com/orgs/jenkins-infra/teams/core as "admin".

krisstern commented 5 months ago

Thanks @lemeurherve!

dduportal commented 5 months ago

Question about Fastly: what is the rationale behind using Fastly in this case (compared to serving the website ourselves)?

lemeurherve commented 5 months ago

Using it as CDN, same logic as for other websites like jenkins.io, stories.jenkins.io, contributors.jenkins.io, docs.jenkins.io.

The website would be served from a nginx on publick8s at (new.)stats.origin.jenkins.io like the others.

Am I missing something?

dduportal commented 5 months ago

Using it as CDN, same logic as for other websites like jenkins.io, stories.jenkins.io, contributors.jenkins.io, docs.jenkins.io.

The website would be served from a nginx on publick8s at (new.)stats.origin.jenkins.io like the others.

Am I missing something?

For jenkins.io, the rationale has always been because of the amount of requests costs too much in outbound bandwidth. But it was measured.

do we have current stats for stats.jenkins.io ?

the question is related to: do we need a CDN ? If no then what is the point.

the same question I have for stories and contributors but i never had the answer or at least i do not see anything around numbers and facts for it.

lemeurherve commented 5 months ago

As stats.jenkins.io is currently hosted on GitHub Pages, I'm not sure we can access traffic stats from it. I found https://github.blog/2014-01-07-introducing-github-traffic-analytics/ but it doesn't seem active anymore.

Then I don't think there's a lot of traffic on these pages, we can start serving it without Fastly then see what are the actual numbers in term of visits.

Removing the Fastly step points and reworking https://github.com/jenkins-infra/azure-net/pull/257

lemeurherve commented 5 months ago

FTR, I invited @shlomomdahan as writer on stats.jenkins.io.

Could be revised when a dedicated team is created.

lemeurherve commented 5 months ago

Opened https://github.com/jenkins-infra/stats.jenkins.io/pull/3 to add a pipeline for building the website on ci.jenkins.io and infra.ci.jenkins.io (pending https://github.com/jenkins-infra/kubernetes-management/pull/5310).

krisstern commented 5 months ago

Hi @lemeurherve, would it be possible to add @gounthar as a co-maintainer?

lemeurherve commented 5 months ago

https://github.com/orgs/jenkins-infra/teams/stats-jenkins-io team including you and @gounthar created and set as "maintainer" on https://github.com/jenkins-infra/stats.jenkins.io/.

lemeurherve commented 5 months ago

I've set again ci.jenkins.io as required status check on the main branch protection.

Please don't change again this setting and keep it that way, it's here to ensure the main build isn't broken.

lemeurherve commented 5 months ago

CI part completed 🙂

shlomomdahan commented 5 months ago

@lemeurherve Great!! Thank you

lemeurherve commented 5 months ago

FTR, I invited @shlomomdahan as writer on stats.jenkins.io.

Could be revised when a dedicated team is created.

Changed @shlomomdahan as "reader" like the other GSoC students.

cc @krisstern

lemeurherve commented 5 months ago

Renovate configured on the repo, dashboard pinned: https://github.com/jenkins-infra/stats.jenkins.io/issues/22

lemeurherve commented 5 months ago

Added an item to the plan: watch the cost associated to the standard File Share cf https://github.com/jenkins-infra/azure/pull/729#discussion_r1643941847

shlomomdahan commented 5 months ago

FTR, I invited @shlomomdahan as writer on stats.jenkins.io. Could be revised when a dedicated team is created.

Changed @shlomomdahan as "reader" like the other GSoC students.

cc @krisstern

@lemeurherve @krisstern I lost my ability to add reviewers to my PRs. Do you know If it's possible to maintain that feature? Thank you

timja commented 5 months ago

FTR, I invited @shlomomdahan as writer on stats.jenkins.io. Could be revised when a dedicated team is created.

Changed @shlomomdahan as "reader" like the other GSoC students.

This doesn't sound right, at least in a new repository. In an existing production service sure.

But this just adds friction whilst unneeded imo.

krisstern commented 5 months ago

GSoC contributors are not supposed to have the ability to add their own reviewers either with triage or write rights, nor are they supposed to have the ability to review or approve merge requests in a Jenkins repo.

timja commented 5 months ago

GSoC contributors are not supposed to have the ability to add their own reviewers either with triage or write rights, nor are they supposed to have the ability to review or approve merge requests in a Jenkins repo.

Why? Historically they have been able to.

CODEOWNERS can be setup to add reviewers automatically.

krisstern commented 5 months ago

Why? Historically they have been able to.

I don't think so, not since I have been an org admin with Jenkins or from my own personal experiences working as a GSoC contributor. Frankly, I myself would not ask for it before the GSoC project has completed and I myself have gained the community's trust to do so. From what I observe previously most projects are first set up outside of Jenkins before migrating to the Jenkins org eventually upon completion. We do not normally set up repos under the Jenkins org first for GSoC during the coding period. This year is a first for me.

If the repos are under the contributor's own account, that is another story.

krisstern commented 5 months ago

Why do the GSoC contributors need such rights anyway may I ask?

timja commented 5 months ago

Why do the GSoC contributors need such rights anyway may I ask?

because its 'their' project. They are maintaining and creating it. The work is done async and mentors should be there to guide and not necessarily check the whole work.

this is only for new projects that they have created

historically we generally encouraged development in the jenkinsci org and there was a mix of people doing it on personal accounts and in the org.

krisstern commented 5 months ago

It is the GSoC contributors working for us on a project, but these are not their projects. They are sponsored by Google to work for us, the code belongs to us not to them. Also, they are not necessarily the ones maintaining, it is the mentors job to do so.

krisstern commented 5 months ago

@MarkEWaite @alyssat @jmMeessen @gounthar I feel like in this year there has been too many bad precedences in terms of GSoC project setup, and I feel like I cannot perform my role as a GSoC Org Admin. What are your opinions on this matter?

lemeurherve commented 5 months ago

Production part completed: https://new.stats.jenkins.io/

krisstern commented 5 months ago

Also, I would also like to seek clarification on the role of the Infra team in any GSoC projects, not just this one. It seems like in this thread members of the Infra team has taken on mentorship role on the project lead mentor and is interfering with the actual mentorship of the GSoC mentoring team, and perhaps not necessarily in a good way. Some decisions of the project should fall on the lead mentor, but these does not seem be respected.

lemeurherve commented 5 months ago

Sorry if I overstepped on your role, I just wanted to help.

I'll restrict myself to pure infra related tasks from now on.

gounthar commented 5 months ago

I concur with @krisstern's perspective regarding write and triage permissions. Based on my limited experience as a lead mentor, admin, and co-mentor, I have not witnessed contributors being granted such permissions to existing or new repositories within the Jenkins organization during the GSoC period. Even if this has been the case previously, it does not align with my personal views. When contributors have their own repository that will be transferred to the Jenkins organization later, they undoubtedly have every right to manage it. However, when it comes to existing or new repositories within the Jenkins organization, I believe granting such permissions is premature at this stage. This is not a matter of doubting their trustworthiness or skills; it is simply a prudent approach. Nor does it imply that they are considered "second-class Jenkins citizens" – not at all. I hold certain privileged permissions in some Jenkins repositories, and I occasionally make mistakes or push work that should be promptly reverted or forgotten. Conversely, I also contribute code that I take pride in to Jenkins repositories where I will never hold such permissions, and I am perfectly content with that arrangement. This issue is more of an ethical consideration. The contributors are here to learn, and granting extensive permissions is not aligned with our standard practices outside of GSoC projects. Across most repositories, rights and permissions are typically granted after an extended "probationary" period (although this process can be expedited for abandoned plugins).

dduportal commented 5 months ago

Thanks @krisstern for speaking up, this is important and healthy. I believe it would be useful to define precisely how the infra-team should be able to help but also the limits for other GSoC projects and next years.

krisstern commented 5 months ago

Sorry if I overstepped on your role, I just wanted to help.

I'll restrict myself to pure infra related tasks from now on.

No worries @lemeurherve, I understand

krisstern commented 5 months ago

Thanks @krisstern for speaking up, this is important and healthy. I believe it would be useful to define precisely how the infra-team should be able to help but also the limits for other GSoC projects and next years.

  • Is there any urgent/short time things we (Jenkins Infra team) should stop doing (or start doing ;)) for this year?
  • I propose we (Jenkins Infra) will plan a kind of postmortem exchange to produce a written runbook. I might contact you to interview you so we can improve.

Sure, I will get in touch with you privately

lemeurherve commented 4 months ago

I had to push an hotfix with the correct STATS_SERVICE_PRINCIPAL_WRITER_CLIENT_ID value to fix production deployment error: https://github.com/jenkins-infra/charts-secrets/commit/4dca10bd26da859393aedff61ec672df0f297464 (private repo)

The CD part is now (almost) ready, https://new.stats.jenkins.io content is updated continuously on every new push on the main branch of the repository.

lemeurherve commented 4 months ago

After discussing with @krisstern about https://github.com/jenkins-infra/stats.jenkins.io/issues/2#issuecomment-2180111448

this project is a part of the ongoing GSoC 2024 and is not in production yet. There will be many additions to its documentation as the project progress. Currently it is still in its development phase.

I propose to temporarily restrict repository interactions to "prior contributors" (include @jenkins-infra members) on https://github.com/jenkins-infra/stats.jenkins.io:

![image](https://github.com/jenkins-infra/helpdesk/assets/91831478/fe26d419-ac6d-42bd-baa0-97fd5302b5dd)

And to (optionally) open a pinned issue in https://github.com/jenkins-infra/stats.jenkins.io to have a visible public status about this restriction.

WDYT?

MarkEWaite commented 4 months ago

I propose to temporarily restrict repository interactions to "prior contributors"

That sounds fine to me.

lemeurherve commented 4 months ago

A temporary interaction limit to prior contributors has been added to https://github.com/jenkins-infra/stats.jenkins.io/ for a month:

> image

And an event in Jenkins Infra calendar pointing to this comment to renew the limit or not in a month.

krisstern commented 4 months ago

@lemeurherve Please add @Vandit1604 to the team with "triage" rights.

lemeurherve commented 4 months ago

@krisstern added as "triage":

image

@Vandit1604 you should have received an invitation to join @jenkins-infra and accept this role, please note that it will expire in a few days.

Vandit1604 commented 4 months ago

Thanks, @lemeurherve I have accepted it.

krisstern commented 4 months ago

Hi @lemeurherve, is site preview for the PR's not an option at the moment? I am currently seeing the following:

Screenshot 2024-06-23 at 4 16 36 PM
lemeurherve commented 4 months ago

@krisstern: as @shlomomdahan doesn't have write permissions on the repo for now, builds on infra.ci.jenkins.io aren't triggered. (Security measure)

We discussed the case with @dduportal and came to the conclusion that we should create a distinct infra.ci.jenkins.io pipeline dedicated to previews that could be triggered without this requirement, while keeping the "production deployment" one restricted.

WIP, incoming soon.

lemeurherve commented 4 months ago

we should create a distinct infra.ci.jenkins.io pipeline dedicated to previews that could be triggered without this requirement, while keeping the "production deployment" one restricted.

WIP, incoming soon.

Should be addressed by:

lemeurherve commented 4 months ago

Note that in order to implement https://github.com/jenkins-infra/stats.jenkins.io/pull/53 I made the compromise of duplicating "Install dependencies" and "Build" stages between this new pipeline dedicated to previews and the main one.

Currently reworking https://github.com/jenkins-infra/kubernetes-management/pull/5345 after discussing it with @dduportal and @smerle33:

lemeurherve commented 4 months ago

Previews are now working on all stats.jenkins.io pull requests, even those from contributors without write permission on the repository:

image
krisstern commented 4 months ago

Thanks @lemeurherve!

krisstern commented 4 months ago

@lemeurherve Could you please let us know what will be the arrangement for the data to be fetched for the project. Currently this appears to be a major blocker. Will data be fetched internally or externally?

lemeurherve commented 4 months ago

An helpdesk issue should be open to discuss how to put in place a public storage of the processed data from https://github.com/jenkins-infra/infra-statistics so they could be fetched when building https://github.com/jenkins-infra/stats.jenkins.io.

dduportal commented 4 months ago

As discussed during the Jenkins Infra SIG 09 July 2024: we keep this issue open until the GSoC project is finished and we have to deploy to the production URL + if infra work is needed.

In the latter case, please comment here and we'll evaluate during the next team SIG meeting.