Improve Ballerina Standard Library Daily Build

[shafreenAnfar]

Background

At the moment each standard library module has it is own repository and the health of the each module is tested by running daily builds against the master. This daily build get triggered on a preconfigured cron expression.

Further, since there are standard libraries which depends on other standard libraries, each module must be built in certain order. Therefore, in order to decide When to build What, we have come up with a level system. Each module belongs to certain level starting from 0 to N.

Problem

Even though this allows us to check the status of the each module, this has some drawbacks such as,

• In case something goes wrong, we don't really get updated instantly.
• Build order of standard library modules is ignored as all the modules get triggered at the same time. Of course, this can be fixed by changing the trigger time of each module but that is not the ideal solution.

The short term solution

As the short term, it was decided to move from cron based daily builds to pub-sub style of builds. Idea is quite simple,

1. Each standard library module allows interested parties to get notified every time a build is completed on master.
2. Each module that depends on a certain module, need to subscribe to this build event.

What this means in terms of github actions is each module needs to have a final step which triggers a build on interested parties. Each interested party needs to have configured repository_dispatch event in their daily build workflow.

Note: Since all the modules depends the ballerina-lang, all the modules are required to subscribe for build events on ballerina-lang

Issues with the short term solution

Even though this solves the aforementioned problems, it does have it is own problems such as,

• Each module is responsible for subscribing in every module it depends on.
• This could trigger redundant builds. For instance, some of the language changes require standard libraries to be built level by level. But in the propose solution ignores the level each module belongs to.

The long term solution

Centralized build pipeline is needed to solve this problem properly. Something similar to the release pipeline. Following is the high level idea,

• Each module needs to updated an entry in the centralized build pipeline with all the dependencies.
• Each time a module is built, that particular module notifies the centralized build pipeline. Then the centralized build pipeline triggers a list of builds based on dependencies.

Note: This centralized build pipeline can be made smarter and smarter. For instance, it can be improved to automatically update the dependency entry of each modules by walking each repo.

This solution is complex compared to the the short term solution but has no drawbacks.

[shafreenAnfar]

@ThisaruGuruge @Maninda @BuddhiWathsala @niveathika do let me know if I have missed anything.

[ThisaruGuruge]

Note: Since all the modules depends the ballerina-lang, all the modules are required to subscribe for build events on ballerina-lang

We don't need to subscribe all the modules to the ballerina-lang, but the level 0 and the level 1 modules. Then the subsequent builds will trigger automatically right? Or am I missing something here?

[shafreenAnfar]

Yeah, you are right. Just wanted to keep it simple at the expense of redundant builds due to two reasons.

1) Each module owner do not need to worry about the level their module belongs to. 2) Centralize the responsibility to each module owner. In case their module didn't get triggered, it is simply because (s)he didn't subscribe to it. Other module owners has nothing to do with it.

The long term solution can address all these things in a better way.

[ThisaruGuruge]

With the PR ballerina-platform/ballerina-lang#26083 we are now at the following state:

• ballerina-lang is now publishing the SNAPSHOT artifacts to the Github on an event of pushing a commit to the master branch. (This is done via Publish Ballerina Snapshot workflow in ballerina-lang).
• When the ballerina-lang SNAPSHOT is published to the Github, another workflow will be triggered to publish all the standard library module SNAPSHOT artifacts to the Github. (This is done via Publish Stdlid Snapshots workflow in ballerina-lang).
• This standard library module publishing will be done sequentially, from the basic modules which aren't depending on any other standard library module, to the modules depending on them, and so on.
• Additionally, on an event of a commit to the master branch of a stdlib module, the module will be built, and the SNAPSHOT artifact will be published to the respective repository.
• After a stdlib module is built and the artifacts published, it will trigger a workflow to build and publish all the dependent modules of that particular module to the Github. (This is also done via Publish Stdlid Snapshots workflow in ballerina-lang).

[niveathika]

@shafreenAnfar Shall we close this, as it no longer valid?