[Spike] Use /wei/pull APP instead of FriendsOfGalaxy synchronizer

UncleGoogle commented 2 years ago

Context:

I've started to rewriting FoG's https://github.com/GOG-Nebula/galaxy-integrations-synchronizer/blob/master/init.py (check also "scripts.py") hoping for quick adjustment to just work with a new organization, but then I realized the stuff is quite tightly coupled with the infra setup which is bare 2 accounts used: FriendsOfGaalxy and FriendsOfGalaxyBot. Bot personal token was used to synchronize repos periodically with github actions and assign reviews to FriendsOfGalaxy. Then I found https://github.com/wei/pull which seems more scalable and secure for us: synchronizer done using github bot API. Pros: properly done bot, configurable, mature project. But need to spent more time for experiments

ToDo:

[x] fork one of our plugins and try to setup end to end using this APP
[x] install APP in org.
[x] update the upstream and check if the app sync's it
[x] explore the app configuration file
[x] evaluate and compare with galaxy-integrations-synchronizer
[ ] review security concerns of using this solution

Acceptance criteria

decision made about if we should use this way of synchronization it or not and if we should still use some parts of galaxy-integrations-synchronizer or not

UncleGoogle commented 2 years ago

Fun fact: they implemented manual trigger for synchronization using GET method so I was synced when I enter this address https://pull.git.ci/process/GOG-Nebula/galaxy-integration-humblebundle

Despite the fact of showing "success" I see no PR created on the fork. Could you guys check what I made wrong? This should trigger PR... https://github.com/wei/pull/blob/f5e10f6d2d82f3e999e5a6d73392e12ce356502b/lib/pull.js#L44

urwrstkn8mare commented 2 years ago

Ok I'm not sure what's happening but I can't see it either. And I've tried to get it working but somethings wrong.

urwrstkn8mare commented 2 years ago

@UncleGoogle could your try making another commit to ur repo? It may be that its watching for changes instead of comparing when installed

urwrstkn8mare commented 2 years ago

Also won't this create a PR every time you commit? Couldn't that become annyoing?

AndrewDWhite commented 2 years ago

I tested a few things but I don't know what is going on without spending more time with it.

Validated yaml (note on page about it taking a day to update)
- https://pull.git.ci/check/GOG-Nebula/galaxy-integration-humblebundle
Manually merged the two branches to verify that they were clean and not causing the issue when using a recursive strategy
- https://pull.git.ci/process/GOG-Nebula/galaxy-integration-humblebundle
- Got several not founds then it ran manually
  - Successful runs go here:
  - https://github.com/issues?q=author%3Aapp%2Fpull

urwrstkn8mare commented 2 years ago

Well whatever you did made it work now

AndrewDWhite commented 2 years ago

No idea if it is successful automation or just a successful manual run.

urwrstkn8mare commented 2 years ago

Oh yeah says "every few hours" but it does say you can do it manually. But yeah it could just take a little time.

urwrstkn8mare commented 2 years ago

It also says if you star the repo you get priority. So @UncleGoogle you should probably star it.

UncleGoogle commented 2 years ago

It also says if you star the repo you get priority. So @UncleGoogle you should probably star it.

yea, I'm afraid it is a part of bussines model for this app :D Ii would be better to not to rely on my personal account "star mark".

I saw some issues in that project where the maintainter said that the server is overloaded with many jobs in the queue, and if asking for financial support.

There is also a way to self-host the pulling server, but I have not much exp in setting it up. Maybe @Mixaill can estimate the cost?

Also won't this create a PR every time you commit? Couldn't that become annyoing?

Agree. The advantage of FriendsOfGalaxy hacky script is that it creates PR only when manifest.json has bumped version.

I'll check if the PR will be updated on another commit. That is quite important to not block new changes by old PR

Mixaill commented 2 years ago

You can actually just set up a github action to sync periodically. https://stackoverflow.com/a/61574295

AndrewDWhite commented 2 years ago

It could also have started working from me clicking on the manage button under the install button on the github project page to see what that said but I don’t know.

AndrewDWhite commented 2 years ago

I think that the old FoG repo used a cron workflow methodology and someone mentioned that it gets turned off after some time.

urwrstkn8mare commented 2 years ago

There is also a way to self-host the pulling server, but I have not much exp in setting it up.

It can be self-hosted simply with docker. I believe I can find some free cloud compute resources or use my own server at home if it comes to it.

You can actually just set up a github action to sync periodically.

This could be a good alternative especially as we'll be able to do it based on version bump. I've made an initial attempt at something. See: https://github.com/GOG-Nebula/syncer/pull/1

I think that the old FoG repo used a cron workflow methodology and someone mentioned that it gets turned off after some time.

@AndrewDWhite For GitHub actions?

AndrewDWhite commented 2 years ago

I am probably misremembering it. I don't remember if this was it. https://github.com/FriendsOfGalaxy/update-test-integration/blob/master/.github/workflows/sync.yml

UncleGoogle commented 2 years ago

I think that the old FoG repo used a cron workflow methodology and someone mentioned that it gets turned off after some time.

scheduled workflows are automatically disabled when no repository activity has occurred in 60 days.

https://docs.github.com/en/actions/managing-workflow-runs/disabling-and-enabling-a-workflow

Then you have to click "enable" in UI or enable it pragmatically. But probably it is not possible to enable them by running another scheduled workflow :D

This could be a good alternative especially as we'll be able to do it based on version bump. I've made an initial attempt at something. See: https://github.com/GOG-Nebula/syncer/pull/1

IMO your solution is very good:

it can compare versions so create PR only when necessary
it is versionable, but we can also pin this github action to master so to never have to update manually
more readable than FriendsOfGalaxy scripts. And don't depend on external services (except github crons of cource).
it does not require special user-bot account (most probably)

Problems:

the problem with automatically disabled workflows
having it written as a composition of other github actions has limited ability to be extended with new features. Eg. I was looking for action that simply returns list of org members to use them as asigneers in athoer step. But well - we can always have a fully-flagged js action.

UncleGoogle commented 2 years ago

So -- what do you think -- is the wei/pull still an option to be explored or we should focus on github sheduled workflow (either our own action or re-using FoG scripts in the same way)?

urwrstkn8mare commented 2 years ago

having it written as a composition of other github actions has limited ability to be extended with new features.

oh that's what's great about composite actions, you don't only have to use other actions, you can use shell scripts as well. See ions/creating-actions/creating-a-composite-action

the problem with automatically disabled workflows

Honestly, if this org has been made so that there are people to maintain the plugins I think the same people can deal with this if it ever occurs. Also it only happens if no activity occurs. So while this is annoying I don't think its a major issue.

it does not require special user-bot account (most probably)

yeah just a token with PR write permissions (fingers crossed)

it can compare versions so create PR only when necessary

another thing it should be able to do but obviously untested, is if the PR is unmerged and the upstream repo gets another version bump, it will just update the current PR to do that version instead of creating a new PR. (this is mainly a functionality of the create-pull-request action)

So -- what do you think -- is the wei/pull still an option to be explored or we should focus on github sheduled workflow (either our own action or re-using FoG scripts in the same way)?

imo wei/pull is a nice app just doesn't fit our specific use case as well as it could.

UncleGoogle commented 2 years ago

oh that's what's great about composite actions, you don't only have to use other actions, you can use shell scripts as well. See ions/creating-actions/creating-a-composite-action

yes, then it looks very flexible

the problem with automatically disabled workflows

Honestly, if this org has been made so that there are people to maintain the plugins I think the same people can deal with this if it ever occurs. Also it only happens if no activity occurs. So while this is annoying I don't think its a major issue.

true

it does not require special user-bot account (most probably)

yeah just a token with PR write permissions (fingers crossed)

so far so good. secrets.GITHUB_TOKEN is much more secure, much more handy.

it can compare versions so create PR only when necessary

another thing it should be able to do but obviously untested, is if the PR is unmerged and the upstream repo gets another version bump, it will just update the current PR to do that version instead of creating a new PR. (this is mainly a functionality of the create-pull-request action)

life quality - we'll improve in time.

imo wei/pull is a nice app just doesn't fit our specific use case as well as it could.

That was my thought too - we won't save much time and add some other problems of 3rd party dependency.

So I'm closing the issue and we'll switch to your github action.

GOG-Nebula / .github