Alright, it's time to bash on my favorite language a little bit. Specifically the problem of libraries because Lua doesn't have many good ones and because it's a common ritual for someone to raise the issue on the Lua mailing list once a year and then nothing is done about it, a tradition that goes back many years on that list.
So I'm gonna make a list of problems that I haven't seen being addressed head-on on the mailing list, and understandably so, because, well, this is a criticism of just about everybody in this community, so it's not bound to create a lot of friends. But hey, we're not here to make friends, we're here to get some clarity on the situation and maybe that can help even more.
First, I don't use vanilla Lua, I use LuaJIT, which is what most of luapower is written for (luapower.com is my one-man answer to most of the issues on this list). Which brings me to my first point about writing libraries.
1. The Lua C API doesn't scale
If you're going to talk about why there aren't that many (good) Lua libraries around, you have to at least ask yourself how is the experience of writing one. To me that experience has always been pretty terrible. The problem here is that the Lua C API doesn't scale. It's a great API if your library has 5 functions because you won't notice that you just spent two hours binding them. But for binding 200 functions with objects and whatnot? Forget about it. You'll waste weeks. And you'll forever be haunted by the fear that sooner or later your Lua stack will get misaligned because there's no way you got that right 100%.
So what's the problem here? To me it's pretty clear. It's another instance of programming at the wrong level of abstraction.
So what can be done about it, use a binding generator? Those are like web frameworks, inflexible, opinionated and a bad case of inversion-of-control (is there ever a good case?), so no.
The focus should be on working to finish luaffi and then making bindings on top of that. Or finishing Titan and making bindings with that. Until one of these projects is stable and complete, don't even attempt to make another C/Lua library. Listen, if you don't have the right tool for doing something, then make the tool, don't compromise by using the wrong tool.
Lucky for me, I use LuaJIT so I can make bindings at the right level of abstraction and scale. This is how I wrote a binding for the Windows API (the UI system, not the easy parts) in 2 months at 22KLOC, a binding for cairo (which has hundreds of methods) at 3KLOC etc.
Now I know many people would like to use plain Lua because LuaJIT is big a scary black box (it is for me!) that is stuck at Lua 5.1/5.2 with no hopes of going beyond, which is why I recommend the above two projects instead of saying "use LuaJIT". I'm ok with LuaJIT but I can totally understand why many people are not. Which brings me to my next point...
2. Where is Lua going after 5.1?
Seems to me that Lua has lost its way since 5.1 (except for bug & consistency fixes most of which LuaJIT also incorporates), succumbing to the wishes of the bikeshedding crowd by adding features of little value instead of focusing on keeping the language stable. Let's go through some of them:
integers: what happened to the "doubles are as fast integers" argument? and yes, I've read their arguments and I cannot see any strong points in there (eg. machines with slow floats: you know what's even slower than floats on a slow machine? an interpreter!; bitwise ops: see next point, etc.).
bitwise operators: look, if you're bit fiddling in Lua you're doing it wrong. If you're doing SHA256 in Lua you already don't care about performance at all anyway. Even with JIT-compiled bit ops you won't get enough performance for tasks like this, so just stop trying and use C which is perfect for this.
utf-8 support: too little to be any good and probably done at the insistence of the community (because "it's 2020, man") not because the authors actually use it themselves. The contexts where you need to encode and decode utf-8 need megabytes of additional libraries and by that time you won't be using this puny utf-8 library anyway for many reasons (eg. you need a different treatment of invalid bytes, etc.).
Roberto's struct library: a really nice library, probably added just because of its good code-size-to-versatility factor, similar to the pattern matching library, nothing wrong here, except it would be made obsolete by a proper ffi library where you could define your own C structs.
to-be-closed variables: who asked for that? look, if I can't have ARC on limited resources (also integrated with exceptions, so basically RAII), I'll do my own freeing, thank you very much. Actually, forget RAII. Ah, screw the whole thing.
const: what the hell man, are we following script kiddie webdev trends now? what's next, optional types? enums? stop please.
making stdlib functions more metamethod-aware -- this was probably also done to satisfy the bikeshedding community's idealistic and impractical notions of consistency, the people who felt uneasy with the fact that not everything in Lua is an object and wanted more opportunities to slap a metamethod onto everything until even the semantics of basic operations are buried underneath their half-assed abstractions.
So what if these things have little value, why bashing on progress, right? Obviously it's because of the cost of tweaking or rewriting libraries all the time to support the latest version of the language. The deal with Lua breaking backwards compatibility all the time was that it needs to do that in order to improve the language, which is a good deal, but there needs to be hope that this will stop someday. And I don't see any sign of that happening even now when the returns are quickly diminishing.
The Lua version census done in January 2020 by Hisham which shows the popularity of Lua 5.1, LuaJIT 2.0 and LuaJIT 2.1 can be an indication of the importance of language stability (again, given the current context, I'm not saying they should've stayed at Lua 2.0 or anything).
3. Package managers just don't work
Building C code is a nightmare because nobody takes responsibility for improving it at the language level. The people responsible with the C language don't care because (they think) it's not their job (outside the scope they say). The C compiler people don't care because it's not their job either.
The library authors care the most, but they're left in the dirt by the first two crowds. So they have to spend a huge amounts of time making these big and complex build scripts in order to support the plethora of build systems and compilers out there. Sometimes the build scripts have more lines of code than the actual library. Ironically, all those additional lines of code do very little to improve the chances of a user getting a binary, especially when dependencies are involved.
20% of the support issues of very goddamn C project on github are build-related. For most other languages that percent is zero. In fact, I can't think of another activity in computing these days with a higher probability of failure than building C libraries. Yes, we're that dumb.
In this context, is no wonder that the package managers of dynamic languages that attempt to solve this problem fail miserably in practice. It doesn't matter if it's npm or LuaRocks, Windows or Linux, the failure rate is still very high.
So is there something a package manager can do about it? No, I don't think so, so stop trying. Specify for your language one compiler for each platform, make the simplest build scripts that you can make. In bash. No autotools, cmake, premake, ninja or any of that garbage. Look, do you want to get a binary or not? How much does actually getting a binary, compared to everything else, matters to you when you design a package manager?
Probably nobody noticed because nobody builds this way, but building 50 C files with one gcc command is actually faster than building only the one that you just changed and linking with the rest of the already compiled ones the way make does it (this is not surprising given that computers work very differently today than when make was made).
4. The libraries themselves
This is not Lua-specific but people generally tend to avoid openly criticizing other people's work, and rightfully so, since most people don't take criticism very well and get defensive about it. As a result, very little actual, specific, critical feedback is exchanged, and so a lot of libraries never grow past their initial use case. Let's do some of that criticism here:
OpenResty has a http client, a mysql client, a dns client, etc., but they only work within an OpenResty context because they're based on so-called cosockets which are exactly like normal Lua sockets on a coroutine-based epoll loop, but you can't use those libraries with LuaSocket out-of-the-box. Luckily, modifying them to work standalone is easy, but that's only because the two socket APIs are very similar and not because of any intention of the authors.
LuaFileSystem cannot open files with non-ANSI characters on Windows.
LuaSocket uses select() on Windows instead of IOCP.
LuaSocket crams various protocols in its distribution instead of developing them as standalone libraries working on a simple I/O interface and a line buffer.
LuaSec selects a single socket in the middle of the TLS handshake, basically blocking other sockets while doing so.
The most complete implementation of the http protocol doesn't even work on Windows at all (the one that uses cqueues for scheduling).
Do you see a pattern here with Windows? And why can no one make a protocol library that doesn't depend on a specific I/O or scheduling library?
Why Lua has no library ecosystem
Alright, it's time to bash on my favorite language a little bit. Specifically the problem of libraries because Lua doesn't have many good ones and because it's a common ritual for someone to raise the issue on the Lua mailing list once a year and then nothing is done about it, a tradition that goes back many years on that list.
So I'm gonna make a list of problems that I haven't seen being addressed head-on on the mailing list, and understandably so, because, well, this is a criticism of just about everybody in this community, so it's not bound to create a lot of friends. But hey, we're not here to make friends, we're here to get some clarity on the situation and maybe that can help even more.
First, I don't use vanilla Lua, I use LuaJIT, which is what most of luapower is written for (luapower.com is my one-man answer to most of the issues on this list). Which brings me to my first point about writing libraries.
1. The Lua C API doesn't scale
If you're going to talk about why there aren't that many (good) Lua libraries around, you have to at least ask yourself how is the experience of writing one. To me that experience has always been pretty terrible. The problem here is that the Lua C API doesn't scale. It's a great API if your library has 5 functions because you won't notice that you just spent two hours binding them. But for binding 200 functions with objects and whatnot? Forget about it. You'll waste weeks. And you'll forever be haunted by the fear that sooner or later your Lua stack will get misaligned because there's no way you got that right 100%.
So what's the problem here? To me it's pretty clear. It's another instance of programming at the wrong level of abstraction.
So what can be done about it, use a binding generator? Those are like web frameworks, inflexible, opinionated and a bad case of inversion-of-control (is there ever a good case?), so no.
The focus should be on working to finish luaffi and then making bindings on top of that. Or finishing Titan and making bindings with that. Until one of these projects is stable and complete, don't even attempt to make another C/Lua library. Listen, if you don't have the right tool for doing something, then make the tool, don't compromise by using the wrong tool.
Lucky for me, I use LuaJIT so I can make bindings at the right level of abstraction and scale. This is how I wrote a binding for the Windows API (the UI system, not the easy parts) in 2 months at 22KLOC, a binding for cairo (which has hundreds of methods) at 3KLOC etc.
Now I know many people would like to use plain Lua because LuaJIT is big a scary black box (it is for me!) that is stuck at Lua 5.1/5.2 with no hopes of going beyond, which is why I recommend the above two projects instead of saying "use LuaJIT". I'm ok with LuaJIT but I can totally understand why many people are not. Which brings me to my next point...
2. Where is Lua going after 5.1?
Seems to me that Lua has lost its way since 5.1 (except for bug & consistency fixes most of which LuaJIT also incorporates), succumbing to the wishes of the bikeshedding crowd by adding features of little value instead of focusing on keeping the language stable. Let's go through some of them:
integers: what happened to the "doubles are as fast integers" argument? and yes, I've read their arguments and I cannot see any strong points in there (eg. machines with slow floats: you know what's even slower than floats on a slow machine? an interpreter!; bitwise ops: see next point, etc.).
bitwise operators: look, if you're bit fiddling in Lua you're doing it wrong. If you're doing SHA256 in Lua you already don't care about performance at all anyway. Even with JIT-compiled bit ops you won't get enough performance for tasks like this, so just stop trying and use C which is perfect for this.
utf-8 support: too little to be any good and probably done at the insistence of the community (because "it's 2020, man") not because the authors actually use it themselves. The contexts where you need to encode and decode utf-8 need megabytes of additional libraries and by that time you won't be using this puny utf-8 library anyway for many reasons (eg. you need a different treatment of invalid bytes, etc.).
Roberto's struct library: a really nice library, probably added just because of its good code-size-to-versatility factor, similar to the pattern matching library, nothing wrong here, except it would be made obsolete by a proper ffi library where you could define your own C structs.
to-be-closed variables: who asked for that? look, if I can't have ARC on limited resources (also integrated with exceptions, so basically RAII), I'll do my own freeing, thank you very much. Actually, forget RAII. Ah, screw the whole thing.
const: what the hell man, are we following script kiddie webdev trends now? what's next, optional types? enums? stop please.
making stdlib functions more metamethod-aware -- this was probably also done to satisfy the bikeshedding community's idealistic and impractical notions of consistency, the people who felt uneasy with the fact that not everything in Lua is an object and wanted more opportunities to slap a metamethod onto everything until even the semantics of basic operations are buried underneath their half-assed abstractions.
So what if these things have little value, why bashing on progress, right? Obviously it's because of the cost of tweaking or rewriting libraries all the time to support the latest version of the language. The deal with Lua breaking backwards compatibility all the time was that it needs to do that in order to improve the language, which is a good deal, but there needs to be hope that this will stop someday. And I don't see any sign of that happening even now when the returns are quickly diminishing.
The Lua version census done in January 2020 by Hisham which shows the popularity of Lua 5.1, LuaJIT 2.0 and LuaJIT 2.1 can be an indication of the importance of language stability (again, given the current context, I'm not saying they should've stayed at Lua 2.0 or anything).
3. Package managers just don't work
Building C code is a nightmare because nobody takes responsibility for improving it at the language level. The people responsible with the C language don't care because (they think) it's not their job (outside the scope they say). The C compiler people don't care because it's not their job either.
The library authors care the most, but they're left in the dirt by the first two crowds. So they have to spend a huge amounts of time making these big and complex build scripts in order to support the plethora of build systems and compilers out there. Sometimes the build scripts have more lines of code than the actual library. Ironically, all those additional lines of code do very little to improve the chances of a user getting a binary, especially when dependencies are involved.
20% of the support issues of very goddamn C project on github are build-related. For most other languages that percent is zero. In fact, I can't think of another activity in computing these days with a higher probability of failure than building C libraries. Yes, we're that dumb.
In this context, is no wonder that the package managers of dynamic languages that attempt to solve this problem fail miserably in practice. It doesn't matter if it's npm or LuaRocks, Windows or Linux, the failure rate is still very high.
So is there something a package manager can do about it? No, I don't think so, so stop trying. Specify for your language one compiler for each platform, make the simplest build scripts that you can make. In bash. No autotools, cmake, premake, ninja or any of that garbage. Look, do you want to get a binary or not? How much does actually getting a binary, compared to everything else, matters to you when you design a package manager?
Probably nobody noticed because nobody builds this way, but building 50 C files with one gcc command is actually faster than building only the one that you just changed and linking with the rest of the already compiled ones the way
makedoes it (this is not surprising given that computers work very differently today than whenmakewas made).4. The libraries themselves
This is not Lua-specific but people generally tend to avoid openly criticizing other people's work, and rightfully so, since most people don't take criticism very well and get defensive about it. As a result, very little actual, specific, critical feedback is exchanged, and so a lot of libraries never grow past their initial use case. Let's do some of that criticism here:
OpenResty has a http client, a mysql client, a dns client, etc., but they only work within an OpenResty context because they're based on so-called cosockets which are exactly like normal Lua sockets on a coroutine-based epoll loop, but you can't use those libraries with LuaSocket out-of-the-box. Luckily, modifying them to work standalone is easy, but that's only because the two socket APIs are very similar and not because of any intention of the authors.
LuaFileSystem cannot open files with non-ANSI characters on Windows.
LuaSocket uses select() on Windows instead of IOCP.
LuaSocket crams various protocols in its distribution instead of developing them as standalone libraries working on a simple I/O interface and a line buffer.
LuaSec selects a single socket in the middle of the TLS handshake, basically blocking other sockets while doing so.
The most complete implementation of the http protocol doesn't even work on Windows at all (the one that uses cqueues for scheduling).
Do you see a pattern here with Windows? And why can no one make a protocol library that doesn't depend on a specific I/O or scheduling library?