Ignore metarepo events within each group

sushain97 commented 6 years ago

This is a bit tricky since apertium-all needs to act on metarepo events. Maybe we need a ignore key in the REPOS config?

shardulc commented 6 years ago

Just to clarify:

event on level-2 repo (e.g. apertium-apy) causes sync for level-1 meta-repos (e.g. apertium-tools) but not for level-0 meta-repo apertium-all
event on level-1 meta-repo doesn't sync level-1 meta-repos but syncs level-0 meta-repo
event on level-0 meta-repo doesn't sync anything

sushain97 commented 6 years ago

I don't think we should make it particularly intelligent (yet). We should just make it possible for a sync of a particular key in REPOS to not happen if the event in question's repo matches some criterion.

You do bring up a good related point though. Right now, the git update commands are recursive which could cause severe problems in repos with submodules and apertium-all.

sushain97 commented 6 years ago

One idea I have to fix this and other related problems are limiting the frequency of syncs, effectively debouncing them. If someone makes a build config change over 40 submodules and pushes them, we'll get a lot of unnecessary syncing since there will be 40 different events fired. We could avoid this problem by syncing at most one time every 5 seconds. Thoughts?

shardulc commented 6 years ago

What if we have two instances of the server started in the script on two different ports? Level-1 events go to port A, level-2 events go to port B; the server on port A updates level-0, the server on port B updates level-1.

shardulc commented 6 years ago

What other problems does the frequency limit solve? If it solves more problems than those just related to this, it is probably a better idea.

sushain97 commented 6 years ago

What if we have two instances of the server started in the script on two different ports? Level-1 events go to port A, level-2 events go to port B; the server on port A updates level-0, the server on port B updates level-1.

I'm already confused... need more time to parse this.

What other problems does the frequency limit solve? If it solves more problems than those just related to this, it is probably a better idea.

I mentioned two problems it solves: 1) the reason this ticket exists, 2) "someone makes a build config change over 40 submodules and pushes them, we'll get a lot of unnecessary syncing since there will be 40 different events fired"

shardulc commented 6 years ago

OK, I misunderstood that, we'll go with the frequency limit. My proposal doesn't solve the other problem.

shardulc commented 6 years ago

Just to clarify my two-server proposal:

sync.py instantiates two servers on two different ports: port A and port B
all level-2 repos (e.g. apertium-eng) are configured to point to port A
the server on port A only updates level-1 repos (e.g. apertium-languages) when it receives an event
all level-1 repos are configured to point to port B
the server on port B only updates the level-0 repo (i.e. apertium-all) when it receives an event

This solves the recursion problem but not the 'too many updates' problem.

sushain97 commented 6 years ago

The solution you describe still seems overly complicated to me. Ignore lists are a pretty common motif in software and I think it would be the simpler solution (in terms of code and understanding). It also doesn't easily scale. What if we have level-3 repos? The configuration and setup will be annoying and there are more points of failure.

If it would seem less like a hack, we can give each repo a function that takes an event and says True if that event should make us update and False otherwise?

shardulc commented 6 years ago

The latter idea seems reasonable. With #11 maybe it isn't even needed, we can just deal with the extra event.

sushain97 commented 6 years ago

I agree. I'll test #11 and then merge. We can always take another look at this if necessary.

sushain97 commented 6 years ago

OK, I just realized something. Consider the following set of events:

apertium-eng gets updated
sync occurs; push to apertium-languages
sync occurs; push to apertium-all
sync occurs; no pushes

This can be fixed IF we sync the repos in a particular order. If we made REPOS look like:

REPOS = [
    {
        'apertium-incubator': {'apertium-incubator'},
        'apertium-languages': {'apertium-languages'},
        'apertium-nursery': {'apertium-nursery'},
        'apertium-staging': {'apertium-staging'},
        'apertium-tools': {'apertium-tools'},
        'apertium-trunk': {'apertium-trunk'},
    },
    {
        'apertium-all': {'apertium-core', 'apertium-all'},
    },
]

then we could sync the groups of repos in order and the sequence would become:

apertium-eng gets updated
sync occurs; push to apertium-languages, push to apertium-all
sync occurs; no pushes

Thoughts? It would be a ~3 line code change and easily generalizes.

shardulc commented 6 years ago

Nice! I think that's a pretty good way to do it. We need to remember to document this (non-trivial) discussion though.

sushain97 commented 6 years ago

Also, this sort of structure could be used to imply ignores. apertium-incubator would not sync in response to any changes to repos in its group, i.e. incubator, languages, nursery, etc. Two birds with one stone? This is somewhat like the levels you suggested but without the multiple servers.

apertium / apertium-on-github

Ignore metarepo events within each group #7