search

eclipse-platform / .github

Common contribution content for eclipse-platform repositories
https://www.eclipse.org/eclipse/
5 stars 10 forks source link

Merge certain repositories #12

Closed laeubi closed 2 years ago

laeubi commented 2 years ago

Recent changes in resources also would trigger changes in team as well what reminds me that this splitting across different repositories does really hinder us to create changes that adjust code crossing these repositories in a consistent way.

I therefore like to suggest to start with merge the following:

jukzi commented 2 years ago

if the repoitories get merged also the CI builds would need to get merged? -> the build time increases drastically?!

laeubi commented 2 years ago

Why should it be faster to setup a machine four times, download dependencies four times and resolve target dependencies four times than doing it once?

jukzi commented 2 years ago

i mean when i propose a PR the CI needs to run tests of all projects instead of only one project -> wait 1 hour instead of 15min

laeubi commented 2 years ago

We better improve the build times and run tests in parallel whenever possible and lots of time is currently spent for all the download and resolving stuff, so you can't just add the individual build times.

Beside that, if we get a better repository structure I better wait some more time and be confident that code do not cause issues than getting a scary mail next day that my change in resources breaks something on teams or where else...

akurtakov commented 2 years ago

Current maven build time of master branches in Jenkins is 7(platform) + 9(resources)+ 7(team)+ 10(runtime) =33mins aka worst case. That time have to be reduced by time spent in downloads and etc. so we are probably looking for under 30 mins response time which is good trade off for reduced time lost chasing changes between repos.

vogella commented 2 years ago

ALso I think we already decided this in PMC (or going to after the successful merge of PDE build, into PDE).

vogella commented 2 years ago

Just to be sure, I bring this up in today PMC meeting and will update. In general the PMC decided already that reducing the number of repos is a good goal. as it reduces the complexity of setup, issue reporting, cross dependencies, etc.

jukzi commented 2 years ago

iIt is 4 times the test time times a power of 4 in the chance of getting a random error. We now have like 50% failing rate we will end up in 85% chance of random failure and have to rebuild almost every times instead of every second time. we can't be proud of those random failing. but they are real. Please take that in consideration. If there would be a technical solution to only verify the part of the project that got changed i would have no concerns. but i doubt there will be. I foresee big disadvantage with the current plan.

jukzi commented 2 years ago

see https://github.com/mrguamos/jenkins-pipeline-maven-multi-module that it would be possible to rebuild only affected modules.

laeubi commented 2 years ago

Maybe there are different definitions of "drastically" then. If we have flapping tests, the solution is not to split code into smaller repositories but to either fix those or delete them, a test that fails in 50% of the time for no reason is just a pita and don't help neither proves anything.

SarikaSinha commented 2 years ago

I propose to merge https://github.com/eclipse-platform/eclipse.platform.resources https://github.com/eclipse-platform/eclipse.platform.runtime

As these 2 are relatively active and have common contributors. I see 2 issues in merging all the 4 repos -

  1. Tests randomness and taking more time
  2. Contributors listening to the new issues will have no way to filter the relevant issues.

Cloning the repo is one time activity and is less frustrating than not able to listen to the relevant issues.

laeubi commented 2 years ago

@SarikaSinha for sure once could simply ignore certain code, but at least from the contributor guidelines it is (was??) recommended to register for the bugzilla-channels so actually one should already listen to all those repositories as a platform-committer anyways. all of those don't justify splitting code that is closely related anyways, if we need more fine grained filtering we should find better ways for that.

akurtakov commented 2 years ago

I've just managed to get Codeowners feature working for swt https://github.com/eclipse-platform/eclipse.platform.swt/issues/66 . So this might be interesting feature for @SarikaSinha

jukzi commented 2 years ago

f we have flapping tests, the solution is not to split code into smaller repositories but to either fix those

please do so before merging. Everyday it's another test.

SarikaSinha commented 2 years ago

@SarikaSinha for sure once could simply ignore certain code, but at least from the contributor guidelines it is (was??) recommended to register for the bugzilla-channels so actually one should already listen to all those repositories as a platform-committer anyways. all of those don't justify splitting code that is closely related anyways, if we need more fine grained filtering we should find better ways for that.

There is a difference between listening and the responsibility to respond to each new issue or PR. If we have some better ways to do that, I am very much open for that but I don't want to miss to respond to any issue or PR in Debug or Ant because of these merges.

vogella commented 2 years ago

PMC decided that we should go ahead with the merge.

@laeubi do you want to provide a PR for merging the first repo? team looks like it is a good candicate to start, as no PR is currently open for it.

jarthana commented 2 years ago

Current maven build time of master branches in Jenkins is 7(platform) + 9(resources)+ 7(team)+ 10(runtime) =33mins aka worst case. That time have to be reduced by time spent in downloads and etc. so we are probably looking for under 30 mins response time which is good trade off for reduced time lost chasing changes between repos.

I have no stakes here, but if I was contributing to only one of the four repos and if my jenkins jobs take four times longer, I will be miffed. In that I agree with @jukzi. May be we should look for some middle ground here.

tjwatson commented 2 years ago

Right now we have serious pains in our releng process and very little additional help to make that better. If we don't make that easier then we may have to take drastic steps to allow us to keep releasing the project (and don't mistake my definition of drastic here, because it will be drastic). Merging repositories where we can will make releng measurably better. Aside from releng I believe fewer repositories makes it easier to develop and report issues for the project as a whole.

If PR test time or intermittent test failures becomes the loudest squeaky wheel then I am sure we will have more developers interested in progressing the project to come and fix the tests (they need to be fixed anyway).

As for traffic on notifications, too much contribution traffic that leads to "too much noise" is not necessarily a bad thing and I hope we can manage the notifications.

If not clear, I am for merging repositories.

SarikaSinha commented 2 years ago

As for traffic on notifications, too much contribution traffic that leads to "too much noise" is not necessarily a bad thing and I hope we can manage the notifications.

How do we manage that? Any suggestions?

Merging active and inactive repos will mean letting the inactive repo to die.

tjwatson commented 2 years ago

As for traffic on notifications, too much contribution traffic that leads to "too much noise" is not necessarily a bad thing and I hope we can manage the notifications.

How do we manage that? Any suggestions?

Not sure. What sorts of numbers are we talking about? I would think we all would be somewhat interested in at least getting notified of each issue or PR that is opened against the specific repositories we are working on. I know in the past my bugzilla traffic was very heavy when I monitored many inboxes and we had a large volume of bugs. It never seemed unmanageable to me. Similarly, I work on several github projects and the notifications don't seem any worse than bugzilla triage did from the past.

Merging active and inactive repos will mean letting the inactive repo to die.

I do not understand this point. Inactive repos should die either way.

SarikaSinha commented 2 years ago

As for traffic on notifications, too much contribution traffic that leads to "too much noise" is not necessarily a bad thing and I hope we can manage the notifications.

How do we manage that? Any suggestions?

Not sure. What sorts of numbers are we talking about? I would think we all would be somewhat interested in at least getting notified of each issue or PR that is opened against the specific repositories we are working on. I know in the past my bugzilla traffic was very heavy when I monitored many inboxes and we had a large volume of bugs. It never seemed unmanageable to me. Similarly, I work on several github projects and the notifications don't seem any worse than bugzilla triage did from the past.

Well after the migration, there are hardly any issues created.

Merging active and inactive repos will mean letting the inactive repo to die.

I do not understand this point. Inactive repos should die either way.

I don't watch all the new issues in team, resource and runtime but I watch for all the Platform Ant issues which is part of platform repo. Ant receives only very few critical issues and major part is to support new Ant versions. After the merge, for me to listen to new Ant issues, I will have to listen to team, resources and runtime issues as well. Which might be difficult for me and there are hardly any other contributors for Ant. I don't want to give up to support Ant because I can not segregate issues caused by merge. HTH

vogella commented 2 years ago

runtime has been merged into eclipse.platform via https://github.com/eclipse-platform/eclipse.platform/pull/27

https://github.com/eclipse-platform/eclipse.platform.resources https://github.com/eclipse-platform/eclipse.platform.team

are not yet merged.

vogella commented 2 years ago

I suggest to merge as next repo the team repo. @akurtakov is this week good for this?

@laeubi I saw some notifications in my vacations that you suggested to merge even more repos. I suggest to finish first the already planned merges before starting to merge more.

akurtakov commented 2 years ago

M2 was last week so should be good. Unfortunately, I won't be able to help much as I'm too pressed with other things.

laeubi commented 2 years ago

I suggest to finish first the already planned merges before starting to merge more.

Any progress forward is appreciated ... I'll try to propose a PR for eclipse.platform.team

iloveeclipse commented 2 years ago

I'll try to propose a PR for eclipse.platform.team

@laeubi: Since the #64 is merged now, please send a mail to platform dev / platform releng list with explanation which repo is merged into which one and which one is obsoleted now.

laeubi commented 2 years ago

@iloveeclipse I think this requires adjusting the setup (@merks ?) and aggregators (@akurtakov ?). Currently only the code is migrated to the other repository. Not sure if it maybe even requires a new I-Build....

merks commented 2 years ago

Yes the setup will need attention ASAP, but I'm not sure if I'll have time today.

akurtakov commented 2 years ago

I'll look into it between calls today.

iloveeclipse commented 2 years ago

I think this requires adjusting the aggregators

You haven't tried SDK build locally? So I guess that would fail then.

laeubi commented 2 years ago

@iloveeclipse I'm don't know much about the aggregator build, the official documentation only mentions how to do this for a master build, thus I only did a build of the repository itself, I assume one needs to remove the team submodule(?) and then everything should be fine, but as @akurtakov has more insights here I think it is better done by someone more familiar with these steps (I already broke the SDK :-) )

iloveeclipse commented 2 years ago

@laeubi : Please look at https://github.com/eclipse-platform/eclipse.platform.releng.aggregator/commits/master?after=adcf2db9803014df55d37bc3d396969700fd5b24+104&branch=master&qualified_name=refs%2Fheads%2Fmaster

iloveeclipse commented 2 years ago

Regarding platform build as whole: please see https://wiki.eclipse.org/Platform-releng/Platform_Build

TL;DR:

git clone -b master --recursive https://github.com/eclipse-platform/eclipse.platform.releng.aggregator.git
cd eclipse.platform.releng.aggregator
git pull --recurse-submodules
git submodule update
mvn clean verify -DskipTests=true

On a modern workstation this usually takes ~30 minutes / 1 hour depending on Internet.

vogella commented 2 years ago

See https://github.com/eclipse-platform/eclipse.platform.common/pull/58 and https://github.com/eclipse-platform/eclipse.platform.releng.aggregator/pull/434

merks commented 2 years ago

FYI, I've updated the setups:

https://github.com/eclipse-platform/eclipse.platform.releng.aggregator/pull/435

vogella commented 2 years ago

Thanks @laeubi for working on the merge of teams.

We still have resources left, which would be also merged into the eclipse.platform. Maybe @laeubi wants to take this one too?

laeubi commented 2 years ago

I'll see if I find some time the next days...

iloveeclipse commented 2 years ago

I'll try to propose a PR for eclipse.platform.team

@laeubi: Since the #64 is merged now, please send a mail to platform dev / platform releng list with explanation which repo is merged into which one and which one is obsoleted now.

Ping.

akurtakov commented 2 years ago

Merges planned here are done thus closing. Further merges of repositories should be handled in new issues.