Getting in control of LuaJIT upstreams

[lukego]

Good news and bad news:

The bad news is that it is becoming more difficult to keep in sync with LuaJIT. The root upstream branch is moving forwards (luajit/luajit#v2.1) but this does not include the lua-mode of DynASM so we need to pull that separately from @capr's branch (capr/luajit#v2.1-dynasm-luamode-static), and then we also have our own modifications that we have made directly in the snabbswitch repo (e.g. traceprof profiler). Gone are the breezy days when we could simply use the code on Mike's v2.1 branch.

The good news is in two parts. First, this is a good problem to have i.e. a sign of vibrant development of LuaJIT. Second, keeping multiple source trees synchronized with the changes that they want to have is exactly what Git is for. So we should be able to sort this out cleanly with a suitable Git workflow.

Here is a proposal for such a workflow that will accommodate today's needs and also tomorrow's e.g. when more people are publishing branches of LuaJIT that contain changes we want and don't get from one of our existing upstreams.

First, here are the branches that we want to synchronize:

• luajit/luajit#v2.1 is the official upstream branch of LuaJIT. This is the default place that new features, optimizations, and fixes go. We need to be able to bring these into Snabb Switch directly.
• capr/luajit#v2.1-dynasm-luamode-static is @capr's branch where he adds the Lua mode of DynASM. This is an important feature that we need and that is not available anywhere else.
• snabbco/luajit#snabb is a new branch that we can (re)create for merging all the changes we want into a "Snabb" version of LuaJIT.
• snabbco/snabbswitch is the Snabb Switch repository where we use git subtree to import/export the version of LuaJIT that we are actually using.

How to sync these branches? Here is a picture of my proposal:

In words: The LuaJIT integration branch for Snabb Switch is snabbco/luajit#snabb. This branch pulls from two upstreams: the root branch and the dynasm-luamode branch to bring in new code from the outside world. It also merges bidirectionally with the subtree in the snabbswitch repo as the means to import/export all changes between Snabb Switch and the outside world.

This scheme depends on the upstreams being well behaved i.e. long-lived branches that never rebase.

Thoughts? Is this a way out of the current confusion? cc @capr @wingo @eugeneia @hb9cwp

See also luajit/luajit#108.

[eugeneia]

It doesn't eliminate the problem that snabbco/luajit#snabb needs dedicated maintenance. E.g. I'd still be in a test/ask-cosmin/repeat loop. The advantage would be that we don't have to sync LuaJIT and DynASM-LuaMode in the Snabb repository. :+1:

[lukego]

Thinking about this a bit more...

Can be that the snabbco/luajit#snabb repo is a mistake and that it is better to pull changes directly into the snabbswitch repo with git subtree pull. That way the changes will pass CI before they are merged. (If we did have a snabbco/luajit#snabb branch it could be strictly a downstream export of the code in the snabbswitch repo so that it is available to others.)

[lukego]

Closing: see that we have a working solution in #724 with using git subtree pull from multiple upstreams directly.