DependencyComputer is computing several times reexported bundles dependencies

[pdulth]

For Eclipse Capella project, the migration from Tycho 2.0 to 2.3 and 2.4 lead to a longer dependency computation . From ~30min of compile to 1h30 with 2.3.0 and ~45min with 2.4.0

From 2.3 to 2.4, there had been a fix around DependencyComputer leading to this improvement. https://github.com/eclipse/tycho/commit/d8d4a0c694b0a6785e7c779c91ff8ae0bbedabd8 (Cache ModuleWiring wires to speed-up dependency computation )

Likewise, in DependencyComputer.populateFromWiring and its sub methods, for a moduleWiring, we compute several times getRequiredBundlePackages of the same "providerBundleWires"

[pdulth]

I'm no expert of Tycho, WDYT of the patch?

[pdulth]

On Capella without the patch, dependency computation takes around ~11min with the patch, ~3min

[mickaelistria]

See recent change https://github.com/eclipse/tycho/pull/161 (and more importantly, the explanation). It could be that just by updating to newer Equinox/p2 fixes the issue? About #160, please add more explanation in the commit message, as it's not trivial code, more explanation would make review and further analysis easier.

[pdulth]

I will try with the latest snapshot, but i had also upgraded Capella to use Eclipse 2021-06 at same time, the osgi release seems to be the expected one in my previous tests.

[pdulth]

Hi, On Eclipse Capella based on Eclipse 2021-06, (with mentioned org.eclipse.osgi-3.16.300) Tycho 2.1.0 => 2min resolve dependencies Tycho 2.3.0 => 30min resolve dependencies Tycho 2.4.0-snapshot a8d7bb7 => 16min resolve dependencies Tycho 2.4.0 with patch #160 => 3min resolve dependencies

[pdulth]

The issue:

getRequiredBundlePackages will be called twice for A package.

when you have a large collection of plugins (around 1200 plugins in Eclipse Capella) (based on Eclipse Sirius, Mylyn, Amalgam, Kitalpha, EGF, GMF, CDO, EGIT, etc), there is many reexport of same plugins in the required tree of each plugins, so we will compute in such packaging several times the dependencies of the "A" plugin. By removing the recursive approach, we compute once the getRequiredBundlePackages for each plugins.

[pdulth]

I have done some statistic tests on capella, listing for each module how many times getRequiredBundlePackages is called for each required modules.

see results, (and related code)

@@computeDependencies(plugin)
requiredModule: <count of calls of getRequiredBundlePackages(requiredModule)> 

For instance , in one of the plugins below, we see that getRequiredBundlePackages(org.eclipse.core.runtime) is called 15k times.

I think there is of course an issue in capella or some of its dependencies, reexporting probably too many times the same bundles, but required bundle haven't changed that much since ~7years and we didn't notice slow performance at run-time. and i think its highly probable that modeling tools based on same Eclipse technological stack will raise same issues at build-time.

@@org.polarsys.capella.core.ui.properties.richtext
 com.google.guava:17
 com.ibm.icu:2
 com.sun.jna:1
 com.sun.jna.platform:1
 javax.annotation:1
 javax.inject:2
 javax.xml:2
 org.apache.batik.bridge:3
 org.apache.batik.constants:2
 org.apache.batik.css:4
 org.apache.batik.dom:5
 org.apache.batik.dom.svg:4
 org.apache.batik.ext.awt:5
 org.apache.batik.parser:2
 org.apache.batik.pdf:1
 org.apache.batik.svggen:3
 org.apache.batik.transcoder:5
 org.apache.batik.util:12
 org.apache.batik.util.gui:3
 org.apache.batik.xml:5
 org.apache.commons.jxpath:1
 org.apache.commons.lang:7
 org.apache.log4j:7241
 org.apache.xerces:1
 org.apache.xml.resolver:2
 org.apache.xml.serializer:1
 org.eclipse.ant.core:3
 org.eclipse.compare:2
 org.eclipse.compare.core:5
 org.eclipse.core.commands:3988
 org.eclipse.core.contenttype:15302
 org.eclipse.core.databinding:13
 org.eclipse.core.databinding.beans:1
 org.eclipse.core.databinding.observable:18
 org.eclipse.core.databinding.property:6
 org.eclipse.core.expressions:42
 org.eclipse.core.filebuffers:12
 org.eclipse.core.filesystem:16
 org.eclipse.core.jobs:15306
 org.eclipse.core.resources:15601
 org.eclipse.core.runtime:15301
 org.eclipse.core.variables:8
 org.eclipse.debug.core:7
 org.eclipse.debug.ui:1
 org.eclipse.draw2d:51
 org.eclipse.e4.core.commands:5
 org.eclipse.e4.core.contexts:14
 org.eclipse.e4.core.di:14
 org.eclipse.e4.core.di.annotations:15
 org.eclipse.e4.core.di.extensions:5
 org.eclipse.e4.core.services:10
 org.eclipse.e4.emf.xpath:2
 org.eclipse.e4.ui.bindings:3
 org.eclipse.e4.ui.css.core:8
 org.eclipse.e4.ui.css.swt:7
 org.eclipse.e4.ui.css.swt.theme:3
 org.eclipse.e4.ui.di:6
 org.eclipse.e4.ui.dialogs:1
 org.eclipse.e4.ui.ide:19
 org.eclipse.e4.ui.model.workbench:6
 org.eclipse.e4.ui.services:7
 org.eclipse.e4.ui.widgets:2
 org.eclipse.e4.ui.workbench:6
 org.eclipse.e4.ui.workbench.addons.swt:1
 org.eclipse.e4.ui.workbench.renderers.swt:1
 org.eclipse.e4.ui.workbench.swt:2
 org.eclipse.e4.ui.workbench3:458
 org.eclipse.egit.core:1
 org.eclipse.emf:2
 org.eclipse.emf.codegen:2
 org.eclipse.emf.codegen.ecore:1
 org.eclipse.emf.common:47755
 org.eclipse.emf.common.ui:89
 org.eclipse.emf.databinding:1
 org.eclipse.emf.diffmerge:9
 org.eclipse.emf.diffmerge.generic:10
 org.eclipse.emf.diffmerge.structures:11
 org.eclipse.emf.ecore:39155
 org.eclipse.emf.ecore.change:10313
 org.eclipse.emf.ecore.edit:25
 org.eclipse.emf.ecore.xmi:6870
 org.eclipse.emf.edit:6925
 org.eclipse.emf.edit.ui:85
 org.eclipse.emf.transaction:3384
 org.eclipse.emf.transaction.ui:28
 org.eclipse.emf.validation:5421
 org.eclipse.emf.workspace:3344
 org.eclipse.equinox.app:15302
 org.eclipse.equinox.bidi:2
 org.eclipse.equinox.common:15354
 org.eclipse.equinox.frameworkadmin:1
 org.eclipse.equinox.p2.core:2
 org.eclipse.equinox.p2.director:1
 org.eclipse.equinox.p2.director.app:1
 org.eclipse.equinox.p2.engine:3
 org.eclipse.equinox.p2.garbagecollector:1
 org.eclipse.equinox.p2.jarprocessor:1
 org.eclipse.equinox.p2.metadata:4
 org.eclipse.equinox.p2.publisher:1
 org.eclipse.equinox.p2.publisher.eclipse:1
 org.eclipse.equinox.p2.repository:1
 org.eclipse.equinox.p2.repository.tools:2
 org.eclipse.equinox.p2.touchpoint.eclipse:1
 org.eclipse.equinox.preferences:15308
 org.eclipse.equinox.registry:15318
 org.eclipse.equinox.security:2
 org.eclipse.equinox.simpleconfigurator.manipulator:1
 org.eclipse.gef:24
 org.eclipse.gmf.runtime.common.core:253
 org.eclipse.gmf.runtime.common.ui:72
 org.eclipse.gmf.runtime.common.ui.action:50
 org.eclipse.gmf.runtime.common.ui.action.ide:2
 org.eclipse.gmf.runtime.common.ui.printing:2
 org.eclipse.gmf.runtime.common.ui.services:28
 org.eclipse.gmf.runtime.common.ui.services.action:53
 org.eclipse.gmf.runtime.common.ui.services.properties:9
 org.eclipse.gmf.runtime.diagram.core:22
 org.eclipse.gmf.runtime.diagram.ui:19
 org.eclipse.gmf.runtime.diagram.ui.actions:2
 org.eclipse.gmf.runtime.diagram.ui.printing:3
 org.eclipse.gmf.runtime.diagram.ui.printing.render:2
 org.eclipse.gmf.runtime.diagram.ui.properties:3
 org.eclipse.gmf.runtime.diagram.ui.providers:7
 org.eclipse.gmf.runtime.diagram.ui.providers.ide:1
 org.eclipse.gmf.runtime.diagram.ui.render:1
 org.eclipse.gmf.runtime.diagram.ui.resources.editor:4
 org.eclipse.gmf.runtime.diagram.ui.resources.editor.ide:1
 org.eclipse.gmf.runtime.draw2d.ui:23
 org.eclipse.gmf.runtime.draw2d.ui.render:6
 org.eclipse.gmf.runtime.draw2d.ui.render.awt:3
 org.eclipse.gmf.runtime.emf.clipboard.core:84
 org.eclipse.gmf.runtime.emf.commands.core:57
 org.eclipse.gmf.runtime.emf.core:83
 org.eclipse.gmf.runtime.emf.type.core:49
 org.eclipse.gmf.runtime.emf.ui:25
 org.eclipse.gmf.runtime.emf.ui.properties:4
 org.eclipse.gmf.runtime.gef.ui:20
 org.eclipse.gmf.runtime.notation:50
 org.eclipse.gmf.runtime.notation.edit:1
 org.eclipse.help:7
 org.eclipse.jdt.core:6439
 org.eclipse.jdt.debug:1
 org.eclipse.jdt.launching:3
 org.eclipse.jface:382
 org.eclipse.jface.databinding:6
 org.eclipse.jface.text:86
 org.eclipse.jgit:1
 org.eclipse.ltk.core.refactoring:2
 org.eclipse.ltk.ui.refactoring:1
 org.eclipse.nebula.cwt:2
 org.eclipse.nebula.widgets.cdatetime:1
 org.eclipse.nebula.widgets.richtext:5
 org.eclipse.osgi:15315
 org.eclipse.osgi.services:3
 org.eclipse.pde.build:1
 org.eclipse.pde.core:6436
 org.eclipse.sirius:2465
 org.eclipse.sirius.common:2415
 org.eclipse.sirius.common.ui:5
 org.eclipse.sirius.diagram:20
 org.eclipse.sirius.diagram.formatdata:1
 org.eclipse.sirius.diagram.sequence:1
 org.eclipse.sirius.diagram.ui:7
 org.eclipse.sirius.diagram.ui.ext:1
 org.eclipse.sirius.ecore.extender:2414
 org.eclipse.sirius.ext.draw2d:9
 org.eclipse.sirius.ext.emf.edit:1
 org.eclipse.sirius.ext.gmf.runtime:8
 org.eclipse.sirius.table:3
 org.eclipse.sirius.ui:12
 org.eclipse.swt:820
 org.eclipse.team.core:78
 org.eclipse.team.ui:2
 org.eclipse.text:94
 org.eclipse.ui:363
 org.eclipse.ui.console:74
 org.eclipse.ui.editors:7
 org.eclipse.ui.forms:63
 org.eclipse.ui.ide:18
 org.eclipse.ui.navigator:35
 org.eclipse.ui.navigator.resources:1
 org.eclipse.ui.views:195
 org.eclipse.ui.views.properties.tabbed:31
 org.eclipse.ui.workbench:456
 org.eclipse.ui.workbench.texteditor:79
 org.eclipse.urischeme:1
 org.polarsys.capella.common:1648
 org.polarsys.capella.common.data.activity.gen:147
 org.polarsys.capella.common.data.activity.gen.edit:19
 org.polarsys.capella.common.data.behavior.gen:335
 org.polarsys.capella.common.data.behavior.gen.edit:39
 org.polarsys.capella.common.data.core.gen:785
 org.polarsys.capella.common.data.core.gen.edit:80
 org.polarsys.capella.common.data.helpers:18
 org.polarsys.capella.common.ef:3193
 org.polarsys.capella.common.flexibility.properties:3
 org.polarsys.capella.common.flexibility.wizards:1
 org.polarsys.capella.common.helpers:1618
 org.polarsys.capella.common.libraries:8
 org.polarsys.capella.common.libraries.gen:10
 org.polarsys.capella.common.linkedtext.ui:2
 org.polarsys.capella.common.mdsofa.common:6433
 org.polarsys.capella.common.menu.dynamic:14
 org.polarsys.capella.common.model:800
 org.polarsys.capella.common.model.helpers:787
 org.polarsys.capella.common.platform.sirius.ted:2405
 org.polarsys.capella.common.queries:5
 org.polarsys.capella.common.re.gen:1
 org.polarsys.capella.common.tools.report:806
 org.polarsys.capella.common.tools.report.appenders.reportlogview:2
 org.polarsys.capella.common.tools.report.appenders.usage:1
 org.polarsys.capella.common.ui:27
 org.polarsys.capella.common.ui.resources:5
 org.polarsys.capella.common.ui.services:14
 org.polarsys.capella.common.ui.toolkit:7
 org.polarsys.capella.common.ui.toolkit.browser:1
 org.polarsys.capella.core.af.integration:1
 org.polarsys.capella.core.data.business.queries:5
 org.polarsys.capella.core.data.gen:88
 org.polarsys.capella.core.data.gen.edit:18
 org.polarsys.capella.core.data.gen.edit.decorators:1
 org.polarsys.capella.core.data.helpers:16
 org.polarsys.capella.core.diagram.helpers:1
 org.polarsys.capella.core.libraries:2
 org.polarsys.capella.core.linkedtext.ui:1
 org.polarsys.capella.core.model.handler:35
 org.polarsys.capella.core.model.helpers:13
 org.polarsys.capella.core.model.preferences:16
 org.polarsys.capella.core.platform.sirius.ui.actions:1
 org.polarsys.capella.core.platform.sirius.ui.menu:1
 org.polarsys.capella.core.preferences:8
 org.polarsys.capella.core.ui.properties:1
 org.polarsys.capella.core.ui.resources:5
 org.polarsys.capella.core.ui.toolkit:4
 org.polarsys.capella.core.validation:1
 org.polarsys.capella.shared.id.handler:15
 org.polarsys.kitalpha.ad.common:8
 org.polarsys.kitalpha.ad.metadata:4
 org.polarsys.kitalpha.ad.metadata.model:4
 org.polarsys.kitalpha.ad.metadata.model.edit:1
 org.polarsys.kitalpha.ad.services:5
 org.polarsys.kitalpha.ad.viewpoint:2
 org.polarsys.kitalpha.ad.viewpoint.coredomain.model:6
 org.polarsys.kitalpha.ad.viewpoint.predicate:6
 org.polarsys.kitalpha.emde:2411
 org.polarsys.kitalpha.emde.model:1447
 org.polarsys.kitalpha.emde.model.edit:82
 org.polarsys.kitalpha.emde.ui:1
 org.polarsys.kitalpha.emde.xmi:2253
 org.polarsys.kitalpha.report:2421
 org.polarsys.kitalpha.report.model:2423
 org.polarsys.kitalpha.report.model.edit:2422
 org.polarsys.kitalpha.resourcereuse:15
 org.polarsys.kitalpha.resourcereuse.emfscheme:1
 org.polarsys.kitalpha.richtext.common:5
 org.polarsys.kitalpha.richtext.nebula.widget:4
 org.polarsys.kitalpha.richtext.widget:2
 org.polarsys.kitalpha.richtext.widget.ext:1
 org.polarsys.kitalpha.richtext.widget.tools:3
 org.sat4j.core:2
 org.sat4j.pb:1
 org.tukaani.xz:3
 org.w3c.css.sac:6
 org.w3c.dom.smil:2
 org.w3c.dom.svg:7

[mickaelistria]

OK, got it, thanks. I agree with the analysis, that re-exported bundles should be treated like directly required bundles to avoid crawling them multiple times.

[mickaelistria]

Done with #160