search

loopbackio / loopback-next

LoopBack makes it easy to build modern API applications that require complex integrations.
https://loopback.io
Other
4.96k stars 1.07k forks source link

Migrate connector repos into monorepo #7368

Open achrinza opened 3 years ago

achrinza commented 3 years ago

This issue is to track the feasibility of migration the connector repos into this monorepo.

Problem statement

The connectors are in separate repos, which results in...

  1. Inconsistent automated maintenance and CI workflows (i.e. dependency/vuln. management, code/commit linting, CLA/DCO)
  2. Disparate issue trackers (harder to keep track of connector-specific issues)
  3. Non-integrated CI testing (i.e. integration tests are not executed on changes within the connector repo)

Potential concerns

  1. Slower CI build times - This concern was raised when this repo was utilising Travis CI, which had slow a VM spin-up duration, and rather limited co-current actions. With GitHub Actions, this concern is potentially mitigated.
  2. Maxing out CI minutes - This was an issue with Travis CI where there was a quota on build times. This is not an issue with GitHub Actions which has unlimited minutes for public repos.
  3. Loss of commit history - lerna import --preserve-commit will preserve the relative commit history, and retain the original commit authors and committers.
  4. Linux-only testing - GitHub Actions only supports containers/services with ubuntu-* images. Currently, the CI pipeline for most connectors are hosted under cis-jenkins.swg-devops.com, which isn't currently publicly-accessible AFAIK.

Connectors

These are the connectors that are officially maintained:

achrinza commented 3 years ago

Potential blocker—GitHub Actions only supports containers/services with ubuntu-* images. Currently, the CI pipeline for most connectors are hosted under cis-jenkins.swg-devops.com, which isn't currently publicly-accessible AFAIK.

dhmlau commented 3 years ago

If we move our CI to GitHub Actions, @bajtos mentioned before that juggle won't be able to have downstream testing. One of the possible solutions could be using https://verdaccio.org. @achrinza, do you think moving the repos into monorepo help us eliminating this downstream test issue?

achrinza commented 3 years ago

@dhmlau Thanks for raising the issue; From my understanding, downstream testing is running the connector tests against the to-be-merged codebase. Please correct me if I'm wrong :P

With the current monorepo setup, Lerna bootstraps local packages with each other locally, which would replicate the same behavior.

The other concern that can't be solved with GitHub Actions is the inability to spin up containers in non-Linux machines; This means that the connectors can't be tested against Windows environments. Though I'm not sure if the connectors are currently tested against Windows or Linux-only, and if this is a non-issue.

pktippa commented 3 years ago

@achrinza @dhmlau I would like to contribute for loopback-connector-mysql, please guide me on getting started.

achrinza commented 3 years ago

@pktippa Apologies for missing your comment; Currently, it's a bit difficult as there's quite a bit of internal CI pipelines that aren't visible. Hence the requirements of the new CI pipeline isn't really clear at the moment.

pktippa commented 3 years ago

@achrinza Okay, let me know once the task is available to take up again.