Automatic sync v2

[rndquu]

This PR introduced a workflow for syncing https://github.com/ubiquity/ts-template repository template with all of our other repositories.

There are a couple of issues:

1. It doesn't work as expected. Check this PR for example which: a) Clears the main page b) Removes required dependencies
2. For "auto sync" to work we need to keep the sync-template.yml in all of our repositories

"Auto sync" may be useful for small changes like CI updates, ts config edits, etc... Such PRs introduce too many changes which are hard to debug and fix.

What should be done:

1. Remove sync-template.yml from all of our repositories and keep it only in a single place like https://github.com/ubiquity/.github
2. Refactor sync-template.yml to sync only whitelisted files (like deploy.yml or tsconfig.json) with the following strategy:
   • ts-template: file added, target-repo: file missing => propose adding a file
   • ts-template: file modified, target-repo: file exists => propose change
   • ts-template: file deleted, target-repo: file exists => propose delete

P.S. For authentication you may use github PAT of this user https://github.com/ubiquity-devpool

[0x4007]

I think we should give it a chance to figure out how to include the commit in the merge. Once the commit is included I suspect that the same changes won't be pulled next month due to the commit timestamp. This was true in my initial research/testing before funding the idea.

[rndquu]

include the commit in the merge

You mean that sync-template.yml should propose changes only to the files that have been added/updated/removed in the https://github.com/ubiquity/ts-template repo during the last month?

[0x4007]

After the initial run it should only sync new changes. That's the point of this tool.

But we need to merge the commit for git to resolve future pulls in this manner.

[rndquu]

After the initial run it should only sync new changes. That's the point of this tool.

But we need to merge the commit for git to resolve future pulls in this manner.

I still don't understand how it's supposed to work.

Check these index.html pages:

• https://github.com/ubiquity/ts-template/blob/858ab74ead68d8954a8e32074c9b0eb805c5b6ca/static/index.html
• https://github.com/ubiquity/work.ubq.fi/blob/8a8a831fb3222eddabc145ec0b5ac668eb083829/static/index.html

When we open a PR from ts-template to sync the index.html file in the work.ubq.fi the index.html is rewritten with the content from the ts-template which is expected. So I don't fully understand why ts-template should rewrite ones files and don't rewrite others.

[0x4007]

Once the merge commit is included, Git should know not to request the same changes again. Here’s the precise behavior:

1. Git Merge Tracking: When you merge changes from the template repo into the project repo, Git records the merge commit. This means Git is aware of the common history between the two repos, and it won’t attempt to apply the same changes again. As long as the commits from the template repo have already been merged, future merges or pull requests should only contain new changes from the template.
2. Redundant Change Requests: Redundant changes will only happen if: • The merge process resets, ignoring previous merges (e.g., if your cron job force-resets the project repo to an earlier state). • The cron job does not correctly track what has been merged, such as if it creates a pull request without checking the project repo’s current state.

If you merge the pull request created by the cron job and do not reset or rewrite the history in the project repo, Git will not request changes that have already been merged.

In summary:

• As long as the cron job pulls the latest changes and the project repo has the merge commit from the previous pull request, Git will know what has been merged and won’t suggest the same changes again.

I think for initial setup I'll need to merge the pull, and then cherry pick or "revert" the changes on top of that commit.

Once that commit is in, I don't anticipate redundant changes to be requested again. Instead, only new changes to the template will be requested to be merged in, which is precisely the behavior that we want.

[0x4007]

I tried for a few hours tonight testing at pay.ubq.fi but couldn't figure it out. Claude suggested an interesting approach that involved cherry picking commits which I think is more on the right track.

https://github.com/ubiquity/pay.ubq.fi/blob/dbe772fe36d7fffbe4c6ca75a3862336092c3e2d/.github/workflows/sync-template.yml#L42-L54

2. Refactor sync-template.yml to sync only whitelisted files (like deploy.yml or tsconfig.json) with the following strategy:

• ts-template: file added, target-repo: file missing => propose adding a file
• ts-template: file modified, target-repo: file exists => propose change
• ts-template: file deleted, target-repo: file exists => propose delete

P.S. For authentication you may use github PAT of this user https://github.com/ubiquity-devpool

Perhaps a full whitelist should be defined here. I think it makes sense to sync every file in the root, .github/* and .husky/*

[ariesgun]

/wallet 0xE80FC2700ec6faF5f0347a2E7E7798FAf548e1c3

[ubiquity-os[bot]]

+ Successfully registered wallet address

[ariesgun]

/start

[ubiquity-os[bot]]

Deadline	Sun, Oct 13, 7:50 AM UTC
Beneficiary	0xE80FC2700ec6faF5f0347a2E7E7798FAf548e1c3

[!TIP]

• Use /wallet 0x0000...0000 if you want to update your registered payment wallet address.
• Be sure to open a draft pull request as soon as possible to communicate updates on your progress.
• Be sure to provide timely updates to us when requested, or you will be automatically unassigned from the task.

[rndquu]

@ariesgun Regarding failing https://github.com/ariesgun/work.ubq.fi/actions/runs/11304622140/job/31443217530

You may either create a new github app, install to your forked instance of https://github.com/ariesgun/work.ubq.fi (with necessary permissions) and setup APP_ID and APP_PRIVATE_KEY in github secrets so that they could be fetched here

Either pass your personal github PAT here, but keep in mind that in the end sync-template workflow will be run using a github app's token

[0x4007]

If we continue with this idea: I think it might actually make more sense to run a single cron that authenticates with our GitHub app from a dedicated repo.

This script can dynamically find all the active repositories across all three of our organizations and open pulls against them.

Then we don't need to worry about managing every instance of this script across all of our orgs.

[ariesgun]

@ariesgun Regarding failing https://github.com/ariesgun/work.ubq.fi/actions/runs/11304622140/job/31443217530

You may either create a new github app, install to your forked instance of https://github.com/ariesgun/work.ubq.fi (with necessary permissions) and setup APP_ID and APP_PRIVATE_KEY in github secrets so that they could be fetched here

Either pass your personal github PAT here, but keep in mind that in the end sync-template workflow will be run using a github app's token

I updated the script based on https://github.com/ubiquity/pay.ubq.fi/blob/dbe772fe36d7fffbe4c6ca75a3862336092c3e2d/.github/workflows/sync-template.yml#L42-L54.

Here's the results:

First time sync is always painful since there will be lots of conflicts. These must be solve manually as well. Check this PR: https://github.com/ariesgun/work.ubq.fi/pull/6.

Subsequent syncs might and might not have conflicts. https://github.com/ariesgun/work.ubq.fi/pull/8. In this PR, some files can be merged just fine and some are not.

Do you guys think this mechanism is acceptable?

[0x4007]

It's hard to say but in my experience the run didn't go smoothly. We might need to settle with dialing back the scope of the script's responsibility. Perhaps just sync the root level files until we figure a reliable technique?

[ariesgun]

The best strategy at this moment is to use whitelisting strategy then. Don't use git merge or cherry pick at all (because it might trigger merge conflicts) and just simply copy paste whitelisted files from ts-template to the target repos.

Even if we only sync root files, it does not guarantee that it wont replace changes made in the target repo. (e.g. package.json)

[0x4007]

We could make more advanced custom logic that uses AI to figure out what to sync.

For example:

• we run it about once a month
• we save the last run commit hash inside ts-template
• we look at a diff between the last run commit and branch HEAD
• if any differences then we clone the to-be-synced repo and ask Claude to resolve the change and open a pull.

Seems a bit overkill but I think it could solve the problem. I would make this a low priority project that would take about a week.

[ariesgun]

Do you guys have an example on using Claude AI?

Using an AI to solve merge conflicts sounds dangerous

[0x4007]

Do you guys have an example on using Claude AI?

Using an AI to solve merge conflicts sounds dangerous

Via pull request the results are approved by humans

[ariesgun]

/stop

[ariesgun]

/start

[ubiquity-os[bot]]

! ariesgun you were previously unassigned from this task. You cannot be reassigned.

[ariesgun]

@0x4007, can you assign this to me again? I would like to try to use Claude AI to handle the merge conflicts.

[ubiquity-os[bot]]

@ariesgun the deadline is at Mon, Oct 28, 10:45 PM UTC

[ariesgun]

Is https://github.com/ubiquity/.github still the best place to put the script?

[ubiquity-os[bot]]

Passed the deadline and no activity is detected, removing assignees: @ariesgun.

[ariesgun]

/start

[zugdev]

Yes, the sync workflow should only propose changes automatically on shared files between repos, I agree those include some of the workflows and ts settings. What I would propose is to enable manual workflow runs with input files too. This way if you want every repo that uses the template to receive a specific new file or to create an overwriting, but not too big PR, you can do it. I believe the AI analysis even though might help, would still need a thorough review and would not reduce any time overhead. I am not sure centralizing the sync workflow is a good idea, if so we would need another way to identify which repos implement the template, to only open PRs in those.

[zugdev]

/start

[ubiquity-os[bot]]

! You have reached your max task limit. Please close out some tasks before assigning new ones.

[ubiquity-os[bot]]

@zugdev the deadline is at Sat, Nov 9, 2:50 PM UTC

[ubiquity-os[bot]]

Passed the deadline and no activity is detected, removing assignees: @zugdev.

[0x4007]

GitHub iOS UI is broken and I can't access the menu to assign you but there's a change that should be made before merging.

[ubiquity-os[bot]]

@zugdev the deadline is at Tue, Nov 19, 4:11 PM UTC