MRO performance on large application

[clemensutschig]

Today the MRO still produces quite some overhead (even with allowPartialRebuild turned on).

8 repos(3 parallel groups), one integration test - 11mins (with the pr below)

one part is opendevstack/ods-quickstarters#294 see also #339

High level trace below:

2020-05-19 15:46:53Z OpenShift project 'cd' exists

Main pod starttime: 699ms !---------------------------------------------------------------------------------------------- STARTING orchestration stage Init (71 secs)

[10] > [Pipeline] sh

• git config --global user.email undefined
• git config --global user.name 'ODS Jenkins Shared Library System User'

2020-05-19 15:47:03Z Release Manager Build Parameters

[6] > ?

2020-05-19 15:47:09Z Run Project#load

[32] > ?

2020-05-19 15:47:41Z Checkout repositories into the workspace

[9] > repo clones

2020-05-19 15:47:56Z Load configs from each repo's release-manager.yml

[5] > Docs

2020-05-19 15:48:02Z Document CSD-5 uploaded

ENDED orchestration stage Init (time: 71399ms) !---------------------------------------------------------------------------------------------- STARTING orchestration stage Build (33 secs)

docs ENDED orchestration stage Build (time: 33505ms) !---------------------------------------------------------------------------------------------- STARTING orchestration stage Deploy (53 secs) docs ENDED orchestration stage Deploy (time: 52457ms) !---------------------------------------------------------------------------------------------- STARTING orchestration stage Test (373 secs) 2020-05-19 15:49:31Z Creating document of type 'CFTP' for project 'pltfmdev' in phase 'Test' and stage 'POST_START'

[30] > Docs

2020-05-19 15:50:01Z Document TCP uploaded

[205] ODS Pipeline Build pod 'pod-56fd3bdb-4fa1-425e-9784-f1b55594ba09' start time:55607ms

**** ENDED stage 'SonarQube Analysis' for component 'demo-app-test' branch 'master' (time: 30411ms) ****

gradle: BUILD SUCCESSFUL in 1m 27s

ODS Component Pipeline 'demo-app-test-13' took 205526ms

2020-05-19 15:53:49Z Collecting JUnit XML Reports for demo-app-test

[50] > JIRA updates (iv tests)

2020-05-19 15:54:40Z Repo 'demo-app-orders' is of type ODS Code Component. Nothing to do in phase 'Test' for target environment 'D'.

[30] > Docs - project docs

2020-05-19 15:55:11Z Creating document of type 'SSDS' for project

[14] > Docs - resurrect SCRR

2020-05-19 15:55:24Z Document SSDS-5 uploaded

[37] > Docs

2020-05-19 15:55:41Z Document TRC uploaded ENDED orchestration stage Test (time: 372493ms) !---------------------------------------------------------------------------------------------- STARTING orchestration stage Release (4 secs)

[] > Docs

ENDED orchestration stage Release (time: 3427ms) !---------------------------------------------------------------------------------------------- STARTING orchestration stage Finalize (227 secs) 2020-05-19 15:55:48Z OpenShift project 'pltfmdev-dev' exists

[60] stash/unstash

[113] Orchestration pipeline pod 'mro-jenkins-agent-13' starttime: 113112ms

2020-05-19 15:58:35Z Repo 'demo-app-test' is of type ODS Test Component. Nothing to do in phase 'Finalize' for target environment'D'.

[46] > OC export // git tag ...

2020-05-19 15:59:21Z Creating document of type 'OVERALL_TIR' for project 'pltfmdev' in phase 'Finalize' and stage 'PRE_END'

[7] > Docs

2020-05-19 15:59:28Z Document Overall-TIR-Cover

ENDED orchestration stage Finalize (time: 227576ms) !----------------------------------------------------------------------------------------------

[clemensutschig]

one thought is on stage finalize - here we are NOT doing any functional component calls - so we could run all repos in parallel, rather than the groups .. this would save us ~10 secs per repo

[clemensutschig]

I gave this some more thought.. - what if we check all repos.. and in case all are resurrected we skip tests as well? And run fully parallel?

[clemensutschig]

More thoughts..

1. we could start the mro pipeline slave at the beginning of the mro run and keep it alive for the finalize stage...
2. Store the Change id somehow next to the jenkins build id (instead of the url) so the optimization works during assemble as well - this includes an early Check if the change Nexus folder exists
3. Make the Check for resurrect more robust with a git diff (deployments.json vs current) to see if anything more was touched aside the files mro owns