feat: git interactor for remote instance registration

lvlcn-t commented 3 months ago

Motivation

We currently only support gitlab as remote registration backend through its API. This should change to support all git based repositories. For more information on that please refer to the issue.

Closes #66

Changes

I've introduced a new remote interactor type, that is able to perform our defined interactor interfaces' actions on any git repository. Currently the git interactor is only able to communicate via HTTP, but that could be changed in a future PR if a user needs ssh communication.

As you've already seen with my last two PRs regarding the issue, I've tried to do it as simple as possible so I hope the code is easy to understand. 😅 I'm also sorry that it's still ~1,5k lines changed but I think that was inevitable.

For additional information look at the commits.

Tests done

I've provided unit tests for the interactor. To do that I've done several things like hiding the remote interactions (Clone, Pull, Push) behind an interface so we can mock it and setup a repomock package that sets up a new local repository where the local git interactions are performed. I've consciously decided not to hide the whole go-git library behind an interface because it'd not bring us that much more benefit, but if you think we should still be able to test error cases of local git actions then we'd need to create an interface for it.

[x] Unit tests succeeded
[ ] E2E tests succeeded - we have none for the target manager

Manual e2e tests

Config

targetManager:
  type: git
  checkInterval: 1m
  registrationInterval: 1m
  updateInterval: 1m
  unhealthyThreshold: 3m
  git:
    repoUrl: https://github.com/lvlcn-t/sparrow-registration.git

Logs

❯ go run ./main.go run --config .tmp/start-config.yaml
Using config file: .tmp/start-config.yaml
time=20:29:11 level=INFO source=/home/tom/dev/sparrow/cmd/run.go:89 msg="Running sparrow"
time=20:29:11 level=INFO source=/home/tom/dev/sparrow/pkg/sparrow/targets/manager.go:82 msg="Starting target manager reconciliation"
time=20:29:11 level=INFO source=/home/tom/dev/sparrow/pkg/api/api.go:76 msg="Serving Api" addr=:8080
time=20:29:11 level=INFO source=/home/tom/dev/sparrow/pkg/config/file.go:71 msg="File Loader disabled"
time=20:30:11 level=INFO source=/home/tom/dev/sparrow/pkg/sparrow/targets/remote/git/git.go:82 msg="Successfully fetched all target files" files=0
time=20:30:11 level=DEBUG source=/home/tom/dev/sparrow/pkg/sparrow/targets/manager.go:251 msg="Updated global targets" targets=0
time=20:30:11 level=DEBUG source=/home/tom/dev/sparrow/pkg/sparrow/targets/manager.go:196 msg="Not registered as global target, no update done."
time=20:30:11 level=DEBUG source=/home/tom/dev/sparrow/pkg/sparrow/targets/manager.go:177 msg="Registering as global target"
time=20:30:11 level=DEBUG source=/home/tom/dev/sparrow/pkg/sparrow/targets/remote/git/git.go:97 msg="Creating new file in repository" file=dev-tom.local.sparrow.com.json
time=20:30:12 level=INFO source=/home/tom/dev/sparrow/pkg/sparrow/targets/remote/git/git.go:121 msg="File committed and pushed" file=dev-tom.local.sparrow.com.json
time=20:30:12 level=DEBUG source=/home/tom/dev/sparrow/pkg/sparrow/targets/manager.go:183 msg="Successfully registered"
time=20:31:12 level=INFO source=/home/tom/dev/sparrow/pkg/sparrow/targets/remote/git/git.go:82 msg="Successfully fetched all target files" files=1
time=20:31:12 level=DEBUG source=/home/tom/dev/sparrow/pkg/sparrow/targets/manager.go:251 msg="Updated global targets" targets=1
time=20:31:12 level=DEBUG source=/home/tom/dev/sparrow/pkg/sparrow/targets/manager.go:209 msg="Updating instance registration"
time=20:31:12 level=DEBUG source=/home/tom/dev/sparrow/pkg/sparrow/targets/remote/git/git.go:89 msg="Updating file in repository" file="{Branch:main AuthorEmail:dev-tom.local.sparrow.com@sparrow AuthorName:dev-tom.local.sparrow.com CommitMessage:Updated registration Content:{Url:https://dev-tom.local.sparrow.com LastSeen:2024-03-14 19:31:12.212252862 +0000 UTC} Name:dev-tom.local.sparrow.com.json}"
time=20:31:13 level=INFO source=/home/tom/dev/sparrow/pkg/sparrow/targets/remote/git/git.go:121 msg="File committed and pushed" file=dev-tom.local.sparrow.com.json
time=20:31:13 level=DEBUG source=/home/tom/dev/sparrow/pkg/sparrow/targets/manager.go:215 msg="Successfully updated registration"
time=20:31:13 level=DEBUG source=/home/tom/dev/sparrow/pkg/sparrow/targets/manager.go:164 msg="Already registered as global target"
^Ctime=20:31:20 level=INFO source=/home/tom/dev/sparrow/cmd/run.go:98 msg="Signal received, shutting down"
time=20:31:20 level=ERROR source=/home/tom/dev/sparrow/pkg/sparrow/targets/manager.go:86 msg="Error while reconciling targets" err="context canceled"
time=20:31:20 level=INFO source=/home/tom/dev/sparrow/pkg/sparrow/run.go:168 msg="Shutting down sparrow gracefully"
time=20:31:20 level=DEBUG source=/home/tom/dev/sparrow/pkg/sparrow/targets/manager.go:127 msg="Shut down signal received"
time=20:31:21 level=INFO source=/home/tom/dev/sparrow/pkg/sparrow/targets/remote/git/git.go:149 msg="File deleted, committed and pushed successfully" file="{Branch:main AuthorEmail:dev-tom.local.sparrow.com@sparrow AuthorName:dev-tom.local.sparrow.com CommitMessage:Unregistering global target Content:{Url: LastSeen:0001-01-01 00:00:00 +0000 UTC} Name:dev-tom.local.sparrow.com.json}" file=dev-tom.local.sparrow.com.json
time=20:31:21 level=DEBUG source=/home/tom/dev/sparrow/pkg/sparrow/targets/manager.go:149 msg="Stopping gitlab reconciliation routine"
time=20:31:21 level=ERROR source=/home/tom/dev/sparrow/pkg/api/api.go:78 msg="Failed to serve api" error="http: Server closed"
time=20:31:21 level=DEBUG source=/home/tom/dev/sparrow/pkg/config/file.go:135 msg="Sending signal to shut down file loader"
time=20:31:21 level=INFO source=/home/tom/dev/sparrow/pkg/sparrow/controller.go:83 msg="Shutting down checks controller"
time=20:31:21 level=ERROR source=/home/tom/dev/sparrow/pkg/sparrow/run.go:123 msg="Non-recoverable error in sparrow component" error="context canceled"
time=20:31:21 level=ERROR source=/home/tom/dev/sparrow/pkg/sparrow/run.go:123 msg="Non-recoverable error in sparrow component" error="context canceled"
time=20:31:21 level=ERROR source=/home/tom/dev/sparrow/pkg/sparrow/run.go:123 msg="Non-recoverable error in sparrow component" error="failed serving API: context canceled"

Remote Git Repository

You can find the commit history of the repository here.

TODO

[x] I've assigned this PR to myself
[x] I've labeled this PR correctly

niklastreml commented 3 months ago

I discussed whether we should merge this now with @lvlcn-t. We decided that it's best to leave this out-of-scope for now, as it is not required currently and to make future changes easier. We will keep the code around in case we need to implement the git integration later on

lvlcn-t commented 3 months ago

I'll still address your reviews. After the PR would be ready to merge I'll close it, and move the branch into my fork so it won't be accidentally deleted. Additional to that I'll close the issue as not planned for now and label it with out-of-scope.

caas-team / sparrow