Forks welcome?

[lukego]

Question: Is it socially acceptable to create down-stream forks of LuaJIT and try to get traction with users and contributors? If so are there any best practices to keep in mind e.g. in terms of how such branches communicate with the world and interact with this repo?

I ask because this LuaJIT/LuaJIT Github repo is working very well as a conservative bug-fixes tree to pull from but it seems like more downstream branches could help foster experimentation and collaboration in the community. I ask the question because such forks can also lead to confusion and fragmentation so I would like to know what other people think. (I wonder if Mike's raptors comment was aimed at discouraging this kind of thing and whether we should refrain out of respect.)

I am particularly wondering whether a Linux-like distributed development model may work as a Plan B in case efforts to clone Mike Pall fail (#45). The vision is a large network of branches with emergent "A merges B" relationships to propagate good changes between them.

[neomantra]

There is already an example of a popular fork and I don't think it stirred up any controversy:

OpenResty uses its own LuaJIT fork and OpenResty itself is used by many other projects. I use this fork generally because I also use OpenResty.

I also had the (unfounded) impression that CloudFlare and agentzh use this fork extensively in test and production and that I can trust it for bleeding edge usage. For example, it had backed out the trace stitching support because they were seeing some issues. I really want to use the LJ_GC64 mode, but I also fear it. It sounds like a cop out, but I don't have time to do the proper QA needed. If that appeared in the OpenResty fork, I would adopt it immediately because I trust them.

While "hostile" forks aren't helpful (and that is not what you are suggesting), performing experiments or curating commits on other trees will increase the velocity of LuaJIT participation and improvement; this should be encouraged. Making your "A merges B" notion more concrete, incorporating these forks into "products" such as OpenResty and SnabbSwitch and having those forks adopt each other's commits will increase their coverage and reduce the fear of the changes' wider adoption.

[MageSlayer]

Hi guys

Maybe it's a good idea to think how to intensify development itself (crowd-funding perhaps)? I mean the amount of active qualified contributors is really low. Forking is easy, but who will do proper changes (with unit-tests and such)?

And I wouldn't call a fork something that just disables some functionality. LJ_GC64 is really necessary to be completed, imho.

[lukego]

@neomantra Thanks for the link to the OpenResty fork. I assume you are talking about the v2.1-agentzh branch? This does seem to contain some interesting commits not present on upstream:

$ git log --oneline --no-merges origin/v2.1..openresty/v2.1-agentzh
f074c64 Revert "Revert "Windows/x86: Add full exception interoperability.""
97369c4 removed the GCC 4 requirement for x86 for older systems like Solaris i386.
ec9b238 Revert "Windows/x86: Add full exception interoperability."
09ab905 Makefile: ensure we always install the symlink for "luajit".
aa8fc2e debugging: added an (expensive) assertion to check GC objects in current stack upon trace exiting. thanks Mike Pall.
003fedc Revert "applied a new versiono of the patch for DragonFlyBSD compatibility from lhmwzy in #2."
0342b38 fixed compilation errors on Solaris when -DLUA_USE_TRACE_LOGS is enabled.
1f3fe16 disabled trace stitching by default for now since it may trigger random lua stack corruptions when using with ngx_lua.
8dfae52 feature: jit.dump: output Lua source location after every BC.
6b44f18 bugfix: fixed build regression on i386 introduced by the LUA_USE_TRACE_LOGS feature.
c7c4fd0 feature: added internal memory-buffer-based trace entry/exit/start-recording event logging, mainly for debugging bugs in the JIT compiler. it requires -DLUA_USE_TRACE_LOGS when building.
1297695 feature: save g->jit_base to g->saved_jit_base before lj_err_throw clears g->jit_base which makes it impossible to get Lua backtrace in such states.
eb2c264 checked in Mike Pall's interim fix for the assertion failure "ir->o == IR_CONV && ir->op2 == ((IRT_NUM<<5)|IRT_INT)" in lj_snap_replay.
e74374d applied a new versiono of the patch for DragonFlyBSD compatibility from lhmwzy in #2.
696009d reverted the patch for DragonFlyBSD because it is wrong.
1a5fa91 relaxed the heuristic limit further for loopunroll.
e037dae bugfix: fixed a typo in commit 395fc489 for DragonFly BSD. thanks lhmwzy for the report.
395fc48 feature: applied John Marino's patch for compiling LuaJIT on DragonFlyBSD. thanks lhmwzy for #1.
913e838 turned off an optimization for 64-bit architectures when valgrind is enabled.
5f0aacf Revert "added Mike Pall's temporary workaround for fixing a segmentation fault when a lua error is thrown."
ed31d40 Revert "Revert "temporary work-around for the bogus "loop unroll limit reached" failures.""
c865570 added Mike Pall's temporary workaround for fixing a segmentation fault when a lua error is thrown.
9fbd55a Revert "temporary work-around for the bogus "loop unroll limit reached" failures."
6315a75 no clever memory over-reads when LUAJIT_USE_VALGRIND is defined.
aa04c32 fixed a regression in the last commit.
c885803 temporary work-around for the bogus "loop unroll limit reached" failures.
1672aa5 allow use of system malloc on x86_64.
3eeb56a updated .gitignore a bit.

I would find it very interesting to participate in an upstream community who are discussing/reviewing/merging these kinds of changes with each other. In Snabb-land we also have a branch with some additions to upstream but at the moment we just develop that inside our own repo as a git subtree. It would be cool to instead develop that on a dedicated Github repo and share/discuss/test changes together with other people. (For example I don't know how everybody else is managing to optimize their code without traceprof but I don't know a good way to share it with you all :-) snabbco/snabb#623).

Linux has an interesting model here. They just maintain a list of all the branches that people are maintaining and what their purposes is (e.g. collect stable bug fixes, collect optimizations for ARM, etc). There are thousands of these branches but keeping them in sync is only O(log N) work thanks to Git. Could be a good start for us, too?

[lukego]

@MageSlayer My feeling is that funding can speed things up but only really helps when you have a structure in place. I have sponsored some LuaJIT work and it feels much more productive when there is an upstream community to review and merge it.

I would love to see people successfully offering LuaJIT development services. The challenge I see is that their customers/employers will need to be confident that the software will land upstream in some well-maintained branch. This does not fundamentally have to be Mike's LuaJIT/LuaJIT#v2.1 branch but at the moment that is the only game in town.

Testing is an interesting topic. This is also a case where a network of forks can help. For example, if the branch openresty/luajit2#agentzh-v2.1 is what CloudFlare are using then that increases my confidence in any code that is merged there. On the Snabb side we also do a lot of testing. Our CI lab is a Hydra cluster of ~20 physical servers that run a machine-year worth of performance tests every month. These tests cover thousands of permutations of different software versions, configurations, workloads, dependency versions, hardware platforms, etc. On the one hand this is not a comprehensive test suite for LuaJIT but on the other hand it is a powerful tool for catching even relatively obscure bugs. It would be great if this could serve the LuaJIT community more directly too.

[MageSlayer]

My feeling is that funding can speed things up but only really helps when you have a structure in place.

And structure can appear when there is funding :)

I have sponsored some LuaJIT work and it feels unproductive if there is no upstream community to review and merge it.

That's exactly my point. However, I still fail to see how technical forking can solve organizational issue of lacking community. I mean having some community will inevitably lead to some form of branching/forking.

Testing is an interesting topic. This is also a case where a network of forks can help. For example, if the branch openresty/luajit2#agentzh-v2.1 is what CloudFlare are using then that increases my confidence in any code that is merged there. On the Snabb side we also do a lot of testing.

That said, that's your company private testing. Nobody sees it and nobody outside can make use of it, right? It is not useful for progressing e.g. GC64 branch. Let's put it another way - yesterday, I tried switching my project to GC64 and 10th of ~200 unit tests (testing pretty basic stuff, no performance, just calculation results checking) immediately crashed with assert failure. Still https://github.com/LuaJIT/LuaJIT-test-cleanup on GC64 completes without errors and that's the only official testing LuaJIT has so far.

Hopefully, you see what I am trying to say... or putting it straight (no offense) I am afraid of hostile forking based on unclear business interests. Making VM/language implementation testing private is the first step in that direction. LuaJIT project already suffers a lot because of semantic/implementation-specific unit test absence.

I really feel that someone/crowd-funding/some community should step in and fund some dirty work of ensuring quality for public interest => earning trust + attracting potential contributors.

[corsix]

Let's put it another way - yesterday, I tried switching my project to GC64 and 10th of ~200 unit tests (testing pretty basic stuff, no performance, just calculation results checking) immediately crashed with assert failure.

Is said project and test suite available publicly somewhere (or failing that, available under NDA)?

[MageSlayer]

Is said project and test suite available publicly somewhere (or failing that, available under NDA)?

It's a private project, and I am progressively narrowing the issue to a standalone test-case. So I'll publish a small test-case as soon as possible.

[lukego]

JFYI: Snabb test infrastructure is online at https://hydra.snabb.co/ and we would be more than happy to host LuaJIT test suites. The environment is nice for performance testing (bare metal servers, no background activity, disabled TurboBoost/hyperthreads/etc). Just give a whistle if somebody is interested in working there.

The test case definitions, including the complete software base and kernel settings of all the servers, is defined as Nix expressions in Github repo https://github.com/snabbco/snabblab-nixos.

Having said that, I also think that private testing by companies is valuable to the community. I am really happy to learn that CloudFlare have a branch containing patches that make LuaJIT work better for their purposes. I think that we would all benefit from more companies doing this kind of thing and then having a workflow for creating network effects e.g. merging each others' branches. Seems to work for Linux where the whole project moves forward from diverse commercial activities like Google shipping phones, Sony shipping TVs, etc. These organizations are focused on their own interests but they are not hostile and they do provide a net benefit to the wider community.

[lukego]

(Relatedly: Snabb is not a company but rather an open source project like LuaJIT. I used a company to bootstrap the development but these days it is co-developed by people at companies like Igalia, Deutsche Telekom, Juniper, Akamai, SWITCH, and also independent contractors/consultants/hackers. The Linux-style decentralized model is working for us and we all benefit from each others' activities even though we all have our own priorities. I will update the snabb.co website to reflect this transition from company to open source project over new year. Some backstory here if anybody is interested: https://github.com/snabbco/snabb/issues/738#issuecomment-180727157)

[lukego]

(Closing due to giving up waiting for guidance.)

[lukego]

Just another reflection:

This LuaJIT/LuaJIT repo is working very well for maintaining the master and v2.1 branches. However it is not really supporting the creation of new branches by new maintainers. So if we want to have new features and new maintainers then it seems like we need a new place for them to get started including asking newbie questions and making silly mistakes and pursuing features that may be dead-ends.

I would love for such a place to exist... I had actually imagined that this repo would be a "big tent" to accommodate all of us but I am giving up that idea after the feedback at https://github.com/LuaJIT/LuaJIT/pull/290#issuecomment-285192553. Could be the situation is more like Linux where LKML was for experts only and the Kernelnewbies community had to be created separately. I am not taking the initiative because I don't know if I am the only one who is interested.

If anybody does create a forum designed to foster growth of new maintainers then please drop me a note! Just now the nearest solution I can think of is to individually follow lots of people's private Github forks but that seems like too much effort and risk of low signal/noise ratio due to seeing all commits rather than only the ones proposed for discussion.