search

eclipse-platform / eclipse.platform

https://eclipse.dev/eclipse/
Eclipse Public License 2.0
82 stars 113 forks source link

Combine other repositories with platform #7

Closed vogella closed 2 years ago

vogella commented 2 years ago

I suggest we start merging repos and team seems to be an easy candidate to start with.

vogella commented 2 years ago

@akurtakov @tjwatson WDYT?

iloveeclipse commented 2 years ago

I would consider to move not teams but runtime, releng & resources to this repo.

I would group it like:

Or even simpler

laeubi commented 2 years ago

Can someone enable github discussions for this repo? This would be more suitable for discussions than using issues / mailing list I think...

merks commented 2 years ago

@laeubi

Until we have discussions (which is a good idea because we will need a place for people to ask questions as opposed to using issues for that purpose), we "discuss" and plan here.

@iloveeclipse

Thanks for generalizing the theme and starting the discussion about where we would like to end up after the merges are completed.

rolfth commented 2 years ago

Just an additional consideration w.r.t. SWT binaries, never used it myself, but how about Git LFS (Large File Extensions) for that repository? It would decrease complexity from the consistency of the two repositories, at the expense of added LFS complexity.

iloveeclipse commented 2 years ago

Regarding SWT binaries & LFS: yes, that's would be interesting option => https://github.com/eclipse-platform/eclipse.platform.swt.binaries/issues/2

mickaelistria commented 2 years ago

I like the proposals made by @iloveeclipse in a previous comment.

Merging text, team, ua, debug into existing eclipse.platform.ui seems relatively consensual. I would be glad to create individual issue to individual repos to discuss the move individually if you think it helps. Similarly, merging runtime and resources seems to be consistent as well.

About SWT, it depends on whether there are contributors that have focus on SWT only without interest in other parts of Platform. If those contributors are numerous enough, we may want to keep the SWT repo separated to help them keeping focus. I personally don't know the answer to such question For releng, I've opened https://github.com/eclipse-platform/eclipse.platform.releng/issues/20 . I think it's better to not merge releng with eg UI at the moment, as they way it's maintained and the people who are involved in it is really different from eclipse.platform.ui or eclipse.platform.text for instance. Maybe later we can consider merging it. I'll also open another ticket regarding docs, to rename the repository into something more human friendly.

mickaelistria commented 2 years ago

Also, I think this particular "eclipse.platform" repo would better be incorporated directly in eclipse.platform.releng.aggregator.

merks commented 2 years ago

Do you folks think we could we spell out a complete plan in just one place for all of us to review?

To me that seems far better than distributed set of issues across multiple repositories...

SarikaSinha commented 2 years ago

We should also see the impact on cloning, Platform UI cloning itself times out many time, If we club that with debug etc it will be even more tougher for new comers.

vogella commented 2 years ago

I bring the merge of the repos into the next PMC meeting. Also it would be nice if we can stop storing build artifacts (binaries) in our repo and publish them instead.

niraj-modi commented 2 years ago

We should also see the impact on cloning, Platform UI cloning itself times out many time, If we club that with debug etc it will be even more tougher for new comers.

Adding on above point: In general and specifically existing Eclipse/SWT wiki articles for setup/configuration are written with SWT(Source & Binaries) repo wise and it may also need to be fixed.

Another important point to consider is back-porting of patches which will become challenging if we merge repos:

Currently both of SWT-Sources & SWT-Binaries repo itself is combination of code from below 3 different platforms:

Sorry it's a NoGo for SWT(Sources & Binaries) repo merging.

vogella commented 2 years ago

We should also see the impact on cloning, Platform UI cloning itself times out many time, If we club that with debug etc it will be even more tougher for new comers.

Clone via Github seems much faster and more stable, see @merks comment https://github.com/eclipse-platform/eclipse.platform.swt.binaries/issues/2#issuecomment-1084351862

akurtakov commented 2 years ago

We should also see the impact on cloning, Platform UI cloning itself times out many time, If we club that with debug etc it will be even more tougher for new comers.

Adding on above point: In general and specifically existing Eclipse/SWT wiki articles for setup/configuration are written with SWT(Source & Binaries) repo wise and it may also need to be fixed.

Another important point to consider is back-porting of patches which will become challenging if we merge repos:

* Also all old Eclipse build setups which we manage to work with might break(be it cherry-pick or SWT native builds)

* And it's no brainier to rework any of the old Eclipse build setups(build scripts, libraries etc...)

Currently both of SWT-Sources & SWT-Binaries repo itself is combination of code from below 3 different platforms:

* Windows

* MAC

* Linux

Sorry it's a NoGo for SWT(Sources & Binaries) repo merging.

@niraj-modi I ask again for everyone to stop putting his own opinion as the one. Discussions are held so majority can make a decision and drive - this is simply how democracy works and luckily EDP too. From https://www.eclipse.org/projects/dev_process/#4_1_Committers - "All Project Committers have equal rights and responsibilities within the Project.". The attitude that things can not change because they have been certain way is not constructive and led us to the current state where we have these discussions on daily basis. I have disagreed with fair major set of changes/policies/etc. so we can all work together towards a common goal/goods and this is expected by everyone.

The only other option to majority decides what/if/how changes I can see is "Authoritarianism" - PMC/PLs decides without any care about what committers think. Is this really where we want to go?

akurtakov commented 2 years ago

On request by @merks some part of an email copied here: So this is not a whim but rather a long going effort started years ago:

So let me explain what my vision is. There are certain questions which drive it:

  1. Is the current state of XYZ creating extra work which we can reduce? E.g. few more commits for each new stream, separately maintained Jenkinsfiles, build scripts update, non functional version checks and etc. that could all be reduced
  2. Will such a change make it better or harder for committers and potential contributors in the long run? E.g. repositories to clone, ease to setup new development machine, finding your way in the code.
  3. Will such a change make it better or harder for the releng team in the long run? E.g. build automation, using more standard tools, giving better reports/feedback to developers for their daily work
  4. Do we have a deadline? E.g. service will be terminated, lower level dependency will no longer be part of the next version of OS and etc.
  5. Do we have any limitations that prevent us from doing that change now? E.g. too slow build times destroying CI feedback

So can I say that there is a strategy based on ^^ - Yes and no. So the current strategy is to look up for cases which will have "Yes" as an answer to 1-3 and after that analyze 4. and 5. If there is a deadline act fast so we are ahead of the deadline (I still remember the time when we had no builds for certain period as all scripts and etc. were run by committer account with cron jobs on machines not being accessible and the stress from that) so I'm 100% convinced that at the first point a change is "inevitable" it should be acted upon. Last comes 5. - quite often it requires some reorganization and even further changes to make it work. Merging all platform repos would not work right now as it will make CI response time totally unbearable but this doesn't prevent doing steps in that direction. E.g. merging platform.team into platform:

So is all of the above some grand plan? No. Having a plan requires being able to set dates, plan work, etc. which we don't have the luxury anymore. The surrounding ecosystem changes so fast (Java 19, Win 11, Gtk 4, ..... all out of our control and we being at the mercy of what JDK devs put in Java 19 or GTK devs put in GTK 4 or Microsoft puts in Win11) that we are forced to act reactively quite often. Of course, I wish we were able to react to all these changes and still execute "our" plan. Unfortunately, this is not the case and quite often these "reactive" changes even eat the time for planning. That's why I stopped even looking or trying to do grand plans. "Pruning" as done in orchards seems to be quite good explanation of the workflow - the roots are strong enough for X number of fruits, prune some branches that have "defects" or even just growing in the wrong direction and preventing others to see enough sunshine/air, create comfort for "master" (ecj, swt...) branches as without them the whole tree would be compromised, look for ways to "fertilize" ( direct maven usage) and so on. The most important thing there is realizing that what looked promising in the spring may happen to be broken by summer storms and requires significant changes in the way you plan the tree's growth.

Please share your thoughts on all of the above as I fully agree with Ed - we have to decide where we want to go? do we want to go to the same place? and can we do it?

merks commented 2 years ago

@akurtakov Thanks!

With respect the @niraj-modi comment, I think it's fair to state a personal opinion on a specific topic. Stating an opinion, is not the same as making a decision for the group. I think it's important that the group understands the opinions of those closely involved in the maintenance and support of any of these specific repositories. In addition, sometimes opinion evolve...

akurtakov commented 2 years ago

With respect the @niraj-modi comment, I think it's fair to state a personal opinion on a specific topic. Stating an opinion, is not the same as making a decision for the group. I think it's important that the group understands the opinions of those closely involved in the maintenance and support of any of these specific repositories. In addition, sometimes opinion evolve...

My goal is to keep discussions from escalating. Many people seem to read things quite differently when one says "I disagree/dislike X for Y" and "X is a NoGo for Y". The later leads to escalation which should be the first thing we try to prevent.

niraj-modi commented 2 years ago

@akurtakov Request you to take it easy please, sounding rude doesn't help.

Below are two discussion points from my side:

  1. Possible breakage of various SWT native build compilation setups: SWT Source is already a merge of code from 3 different platforms and same applies for SWT Binaries. Since, SWT Native source build compilation happen on various platforms and is a complex job and any extra step here may break our working setups(for non-supported platforms and older Eclipse versions) and same applies to all corporate customers as well.

  2. There would be performance implication of merging SWT Sources and Binaries, so 2 SWT repos make life easier for us. I echo Sravan's comments: https://github.com/eclipse-platform/eclipse.platform.swt.binaries/issues/2#issuecomment-1084367245 Any alternate location for SWT binaries with quicker performance is welcomed. Thanks!

iloveeclipse commented 2 years ago

There would be performance implication of merging SWT Sources and Binaries

@niraj-modi : could you elaborate what kind of performance issue do you mean?

If we merge native repo into main SWT repo, for building older releases I assume we can leave the "old" binaries repository there, read only. Sure the old could be broken by the binary artifacts now appearing in SWT directory tree, but I believe that shouldn't be actually an issue, because old build scripts will probably just happily use old binary repo anyway.

akurtakov commented 2 years ago

Let's try to keep the swt binaries discussion in its own ticket so we have all the data in one place.

iloveeclipse commented 2 years ago

Sorry, previous comment was meant to be posted on another issue.

Regarding general strategy. A good point was raised regarding increasing test times for PR validation.

1) Surely everyone wants to run only related tests for the touched code (to get fast feedback), not all test suites in all bundles in the merged repo. I don't know if that can be achieved by fine tuning of the Jenkins file? Anyone has experience doing that? 2) However, sometimes even the opposite would be great - if we do some significant change in lower layers, it would be great to run all available tests to see if someone will be affected by the change. I don't know if that would be possible, but would be cool. May be by temporarily touching files in other bundles.

So before merging anything we need to know how we would deal with that above.

jonahgraham commented 2 years ago

@iloveeclipse - I invested some time in CDT to do similar things. So there are now 5 different verify jobs for CDT and it uses modified paths to know which subset to trigger. That was for gerrit, but I would be surprised if a similar thing wasn't available for Jenkins+GitHub.

EPP does something similar too - rather than different jobs it analyses the changed paths (e.g. since master) and runs only profiles for those paths. The EPP way is not gerrit dependent, it just runs a script before running maven.

iloveeclipse commented 2 years ago

@jonahgraham : OK, so the Jenkinsfile can do probably same as the shell script you've mentioned, may be there is even some clever jenkins commands to detect changed code ("when" seem to match paths), but I assume shell & git should be also possible in Jenkins.

Means: we can merge without thinking about build times if we update Jenkinsfile to trigger different builds based on changed files, so the build times should be exact same as before.

Back to the original topic: what should we merge / not merge?

Updated proposal:

Platform Core (Resources, runtime, jobs, releng, test infra, etc)

SWT

Platform UI (platform text, team, debug, ui & ua)

All other repos remain unchanged (they all are very special)

sravanlakkimsetti commented 2 years ago

https://github.com/eclipse-platform/eclipse.platform.releng.buildtools

is not used in aggregator build. So better keep it out of repo merge.

jarthana commented 2 years ago

I am not a committer in any of the projects being discussed here, but I tend to lean on not merging too many projects unless there's a compelling reason. I would like to extent the reasoning of @mickaelistria for releng for other platform projects as well if it will make life easier for the committers. In other words, this will give more weight to point (2) from @akurtakov as this what will make a difference in our daily life. Then there is the point of keeping the committers unworried about the incoming changes not related to them. Of course, this all depends on how many of these projects share committers and vice versa.

akurtakov commented 2 years ago

For the record - these are not separate projects but repositories of the same project (quite often even with circular dependencies between the repositories) so we speak about exactly the same group of committers. @jarthana would you please describe the difference in daily life that would be negative?

jarthana commented 2 years ago

@jarthana would you please describe the difference in daily life that would be negative?

When time permits when doing a pull, I (I am sure others do, too), go over all the incoming commits to get some idea of what changed since the last pull. Now, larger the group of bundles (not projects), the more time-consuming this activity becomes (i.e. assuming I am not interested in the other sub-projects or bundles or however we put it.

akurtakov commented 2 years ago

@jarthana would you please describe the difference in daily life that would be negative?

When time permits when doing a pull, I (I am sure others do, too), go over all the incoming commits to get some idea of what changed since the last pull. Now, larger the group of bundles (not projects), the more time-consuming this activity becomes (i.e. assuming I am not interested in the other sub-projects or bundles or however we put it.

Is any Platform committer doing that? Please say so if you do. I can see how this makes sense for JDT core but wonder whether anyone does it here as that would be quite an adventure.

laeubi commented 2 years ago

But is it easier if you have to pull/review 12 repositories and guess which of these changes are dependent and which are related?

Additional, if it comes to branching/backporting its just anightmare to tag/branch/checkout multiple repositories to get to the state where it is all even (I really not envy the ecliplse platform releng staff for all this stupid repetitive work on each release even though scripts might help a bit here...)

vogella commented 2 years ago

I think most of the current committer work on multiple repos anyway so having the separate makes their life more difficult. Especially now that I have to maintain and update my own fork this become very time consuming.

@jarthana EGit allows to filter commits by selected projects so you could setup a working set and configure egit to show only commits for your projects.

SarikaSinha commented 2 years ago

@jarthana would you please describe the difference in daily life that would be negative?

When time permits when doing a pull, I (I am sure others do, too), go over all the incoming commits to get some idea of what changed since the last pull. Now, larger the group of bundles (not projects), the more time-consuming this activity becomes (i.e. assuming I am not interested in the other sub-projects or bundles or however we put it.

Is any Platform committer doing that? Please say so if you do. I can see how this makes sense for JDT core but wonder whether anyone does it here as that would be quite an adventure.

I work on Platform Ant and Debug and I am generally not involved in all the commits in Platform UI, resource, runtime, text etc. And this will definitely impact me. Also for triaging issues we will have to go though a much bigger list and chances of issues being missed out is higher. I am still struggling to properly filter out the email notifications from eclipse.platform. Is there a simple way out doing these things even after merge?

Bananeweizen commented 2 years ago

I work on Platform Ant and Debug and I am generally not involved in all the commits in Platform UI, resource, runtime, text etc. And this will definitely impact me. Also for triaging issues we will have to go though a much bigger list and chances of issues being missed out is higher. I am still struggling to properly filter out the email notifications from eclipse.platform. Is there a simple way out doing these things even after merge?

Working on issues could be as simple as before the merge, if labels on the issues are used to describe the component. The default labels are not really well suited (e.g. bug, documentation, issue, helpwanted etc. as you can see in the dropdown of an issues list). People have created better labeling schemes, and I totally like those where all labels are "SomeDimension : SomeValueOfTheDimension", e.g. "Priority: Low", "Priority: High", "Component: Foo", "Component: Bar". grafik That scales, is readable, and can be used for quite good filtering. See https://medium.com/@dave_lunny/sane-github-labels-c5d2e6004b63 for a deeper explanation. I have seen similar schemes in many projects, I'm not sure if there is some tooling/automation to create such a set of labels.

If you are able to filter by components, then you would also be able to create the same custom filters for your github inbox, to separate important notifications from unimportant ones. Of course, that is not mail, but maybe it would be a sufficient replacement.

vogella commented 2 years ago

I suggest we start with part of the proposal of @iloveeclipse and try to combine some of the core repos.

Platform Core (Resources, runtime, jobs, releng, test infra, etc)

https://github.com/eclipse-platform/eclipse.platform https://github.com/eclipse-platform/eclipse.platform.releng https://github.com/eclipse-platform/eclipse.platform.resources https://github.com/eclipse-platform/eclipse.platform.runtime

UPDATE: https://github.com/eclipse-platform/eclipse.platform.releng.buildtools was removed from this list of merges

The learning we make here can be used to decide if we want to continue this or not.

I will bring this plan to the next PMC meeting. Also I will suggest to merge the two PDE repos and two of the Equinox repos.

akurtakov commented 2 years ago

Please don't merge releng.buildtools - it contains only things that don't end up in deliverables so they are better out of sight in general as they are now.

vogella commented 2 years ago

PMC decided to start with PDE and Equinox for the merges.

SarikaSinha commented 2 years ago

PMC decided to start with PDE and Equinox for the merges.

Sorry, I could not find any communication in mail or minutes.

What is the alternative solution suggested? I will request to merge after we have some alternative way to segregate issues from the common pool.

vogella commented 2 years ago

See https://github.com/eclipse-platform/eclipse.platform/issues/7#issuecomment-1104045170 for the PMC decision at this point in time. PDE merge is already done and @tjwatson is still planning to merge Equinox but we did not yet start that.

For the next item the PMC decided already on https://github.com/eclipse-platform/.github/issues/12

It is a bit annoying that we have to many repos and issues in these so it harder to document things especially with the Eclipse wiki planned to be deleted by the foundation. See https://github.com/eclipse-platform/.github/issues/4 for the wiki discssion which has to my knowledge not concluded with a decision.

vogella commented 2 years ago

Closing this, current approach is to do this in small steps via individual issues.