search

greenkeeperio / greenkeeper-lockfile

:lock: Your lockfile, up to date, all the time
https://greenkeeper.io
183 stars 73 forks source link

Monorepos #140

Closed janl closed 6 years ago

janl commented 6 years ago

This is mostly done, but do not merge yet, before we’ve had some more eyes on this 👀

💥 Boom, monorepo support, complete with tests.

This is in anticipation of monorepo support landing in Greenkeeper soonish, but we won’t merge this before then.

Firstly, this commits adds an integration test suite that allows us to test different types of repositories.

Secondly. instead of operating on the top level ./package.json and assorted lock files, operations are now split into three distinct steps:

  1. validate we are running in the right CI branch.
  2. for each package.json file in ./ or sub directories:
    • check if we have to run the lockfile update code
  3. if any changes have been made, commit all changes in a single commit

This is a big change, please review thoroughly.

This also will be a semver-major update, just to make sure we don’t trip anyone up with a subtle change.

gr2m commented 6 years ago

you can install https://github.com/apps/wip on this repo to have pull requests blocked automatically while you have "WIP" or "DO NOT MERGE" in the title :)

janl commented 6 years ago

TODO:

janl commented 6 years ago

Previously on this branch.

  1. How gk-lockfile works today.
  2. How does it have to change to work with GK’s monorepo support?
  3. How did I try to get it going?
  4. What are the next steps in getting the implementation there?
  5. What’s the release plan for this?
  6. How do we ensure continued broad CI service support with the new version?

1. How gk-lockfile works today

gk-lockfile is designed to run as a companion to Greenkeeper proper. The idea is that the lockfile update runs as part of the CI test for a Greenkeeper branch, where the CI services pushes a lockfile commit back to GitHub onto the Greenkeeper branch,

In order to avoid an infinite loop where a lockfile commit on a branch creates a new CI build where the lockfile update runs again, gk-lockfile only runs on the first push to a branch. To support multiple CI services, where detecting whether we are on the first push would be different, gk-lockfile abstracts a CI’s environment away from the updating lockfile logic.

2. How does it have to change to work with GK’s monorepo support?

Greenkeeper support for Monorepos can create multiple commits on a single branch (albeit in a single operation), so detecting if gk-lockfile should to run needs to change.

Instead of detecting whether we are the first push, we now try toking through all commits on a branch to see if it already has a lockfile commit. The problem here is that CI services (and we are only dealing with Travis on this PR for now, until we figured out the right approach), do a shallow clone of the branch-to-be-tested, so a git log between the current branch and master fails.

That’s why we introduced updating the git clone, to get to the commit messages: https://github.com/greenkeeperio/greenkeeper-lockfile/pull/140/files#diff-7ad4cafc41ad8cab90eba75a264db196R26

This has far-reaching implications on further operation, so it needs to be considered carefully.

Secondly, the previous update-lockfile functionality has been split up in two operations: update-lockfile and commit-lockfile, so we can iterate over our package.jsons and produce the required diffs and then commit them in one go: https://github.com/greenkeeperio/greenkeeper-lockfile/pull/140/files#diff-45d5ed8022b09fcdb199707473e2fe33R19

Thirdly, all this requires a way to get all package.json files in the repo (or the ones that were updated in this branch). Currently the branch tries to do a glob while applying .gitignore, but that fails for certain monorepo configurations, because after npm i we might have node_module directories in the CI checkout that we shouldn’t be checking.

3. How did I try to get it going?

4. What are the next steps in getting the implementation there?

@Realtin suggests we look into some git magic that gives us just the changed files in the branch or, a little broader, only the package.json files that are tracked by git.

After that we need to check for the other CI services, if the same works, or if we have to make this part of the CI-specific implementation.

5. What’s the release plan for this?

I suggest we make this a semver-major update (2.0.0) and initially release it as a beta, and non-latest dist-tag until we have worked all the kinks out on the new implementation and all CI services.

That way monorepo-users can opt into the new feature and help bake it, while existing users don’t get into a breaking situation.

6. How do we ensure continued broad CI service support with the new version?

I suggest we contact the original implementors mentioned in the README and work this though with them.

Realtin commented 6 years ago

failed to update lockfiles on test repository.

bildschirmfoto 2018-05-30 um 10 23 49

exited here: https://github.com/greenkeeperio/greenkeeper-lockfile/blob/master/update.js#L40-L42 info.branchName should have been greenkeeper/default/neighbourhoodie-test-package-25.0.1

mAAdhaTTah commented 6 years ago

I'd be interested in helping to test. Is this testable yet?

Realtin commented 6 years ago

Edit: updated in the README.md

❗️ Important setup steps: make sure that the CI install step doesn't use npm ci to install, because:

If dependencies in the package lock do not match those in package.json, npm ci will exit with an error, instead of updating the package lock.

which will cause the build to fail for greenkeeper updates!

example travis.yml

language: node_js
cache:
  directories:
  - node_modules
notifications:
  email: false
node_js:
  - '8'
before_install:
  - npm install -g npm
  - npm install -g greenkeeperio/greenkeeper-lockfile#feat/monorepos
install:
  - npm install
before_script: greenkeeper-lockfile-update
after_script: greenkeeper-lockfile-upload
Realtin commented 6 years ago

Edit: fixed now!

🐛 Updating dependencies from monorepos in monorepos. The regex in https://github.com/greenkeeperio/greenkeeper-lockfile/blob/ec44210dff2cd60eeab60fbdae8bf4415be622e8/lib/extract-dependency.js#L24 won't match because the branch name uses the monorepo-definition group name of the dependency i.e. babel instead of babel-plugin-transform-react-jsx-compat or all the others that are updated in that branch

bildschirmfoto 2018-05-31 um 14 43 04
greenkeeperio-bot commented 6 years ago

:tada: This PR is included in version 2.0.0 :tada:

The release is available on:

Your semantic-release bot :package::rocket:

espy commented 6 years ago

Hello everyone (and especially @mAAdhaTTah , who asked): this is now released as 2.0.0 under the next tag and available for testing, more details here: https://github.com/greenkeeperio/greenkeeper-lockfile/issues/164

Thanks!