eclipse-packaging / packages

Eclipse IDE product definitions.
Eclipse Public License 2.0
4 stars 11 forks source link

Refactor EPP build to be more parallel and in stages #120

Closed jonahgraham closed 6 months ago

jonahgraham commented 6 months ago

The EPP build takes a very long time and recently that has been causing issues - see https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/issues/4336 for an example.

This issue is to track my work to refactor the build to have numerous small jobs instead of a single big job.

General idea is to do build in stages, and place the intermediary parts on https://download.eclipse.org/technology/epp/staging/

Step 1: Build the EPP p2 site Step 2: Build all the products in parallel from the EPP p2 site + Simrel p2 site.

merks commented 6 months ago

This sounds like a good approach!

jonahgraham commented 6 months ago

AFAIU I cannot create the full p2 site without doing the whole build. That is because the p2 site contains the information about the product, which come as a side effect of making those products, such as units with id toolingepp.*, epp.package.*.executable.* and probably others. So my idea of making the p2 site first and then products second is a non-starter.

Splitting up the build differently, e.g. build each product in its own job, and then combining the p2 sites may work though and that is the next thing I will try.

jonahgraham commented 6 months ago

Actually I can get a p2 site faster than a full build, by skipping the materialize-products and archive-products phases of tycho-p2-director-plugin. Question is can I build the product from that p2 site (sort of like Oomph does I guess).

merks commented 6 months ago

It definitely should be possible in principle to build products purely from the update site. How to actually do that with tycho is the question.

jonahgraham commented 6 months ago

https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/issues/4358 and https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/issues/4359 are needed to complete the transition, but neither are blocking.

merks commented 6 months ago

FYI, I often use the folder name archive under which to store content that I don't want mirrored because I know mirroring ignores everything under a folder with that name. E.g., that's why I generated the Oomph report into this subfolder of the repository:

https://download.eclipse.org/releases/2024-03/202402231000/buildInfo/archive/

This report shows that mirrors don't mirror that information:

https://download.eclipse.org/oomph/archive/eclipse/releases/2024-03/202402231000/buildInfo/index.html

jonahgraham commented 6 months ago

Thanks @merks for the advice. In the meantime EF IT made "building" excluded from mirrors in https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/issues/4358

https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/issues/4359 is likely to take longer to resolve and I have split it out into #122