Some suggestions

[brynne8]

Luapower is great and saved me a lot of time on configuring a cross-platform lua developping environment. In real use, I have the following ideas:

• Maybe I need a way to set up a luapower repo in local network, as it is never connected to internet. Is there an easy way to do this? Currently I could only mgit on my Internet, and copy it to a computer in local network. I don't want an all-in-one repo that cannot be separated.
• Additionally, I really expect mgit to resolve dependencies automatically, so that I don't have to write mgit clone luasec lua-headers luajit openssl socket to install luasec, I only write mgit clone luasec and all things are done.
• Your MySQL connector is the best one in Lua world I think. And I hope other database connection libraries could be added. And it's better not tied to nginx module. I have a strong need for connecting Oracle database.
• It seems libmariadb depends on zlib package, but it's not documented
• Some nice packages could be brought from ulua.io I think
• Copas and its relevant timerwheel and binaryheap is still a nice library to include. You could use your own heap if you don't favor Tieske's version. One reason I didn't migrate to socketloop is that it doesn't support UDP sockets. Another is it doesn't support an equivalent submodule with copas.limit

[capr]

Hi Alexander,

I'm glad you found luapower useful.

I'm not sure what you mean by setting up a luapower repo in local network. You mean you want to use git to push/pull in a LAN, like this?

About resolving dependencies automatically, that could be done by having the luapower script generate dependency files (one for each supported platform) that you would put in each package's repo, and then have mgit just clone all modules from that file. mgit can't do that all by itself since it has no knowledge of the dependency tree, in fact it doesn't even know anything about Lua at all, it's the 2000 LOC luapower.lua that knows all about that stuff. It's not hard to do, just something I didn't have a pressing need for I suppose so I haven't got around to do it.

About the mysql module, it may be more complete than the openresty one, but it drags with it the mysql client library which is big, and also note that it doesn't bind the async API yet so you can't schedule it with coroutines like you do with normal sockets. I plan on adding that API on the next project that involves mysql. It's the same with an Oracle driver really. This is how a lot of libraries came about, and frankly I don't understand why so few people build software like this (in reusable, independent layers).

About zlib dep of mariadb, you're right, thanks for the heads up (added to the WHAT file). Thing is binary dependencies still have to be declared manually because I haven't got around to implement a tool to extract the dependency list from dll/so/dylib (it should be all of them). I can't seem to find a cross-platform lib for that...

Which packages did you use from ulua?

Re Copas, I'm actually working to replace the whole network stack with pure Lua code. Not all of them published on the website yet but you can find them on github. That currently comprises:

• socket2: sockets + coroutine scheduler based on IOCP/epoll/kqueue (everybody says that you can't make a cross-platform API out of those because of "fundamental differences" between the scheduling models but I don't know what they're smoking).
• a binding of BearSSL's libtls API to replace LuaSec.
• http protocol that you can use with your preferred I/O library like all protocol libraries should be, plus a client and a server module that works with socketloop. This is actually on the website here https://luapower.com/http.

Re heap, you mean something like this https://luapower.com/heap ? I'm not sure what a timerwheel is but I assume it's a heap of timeouts to decide the minimum amount of timeout to poll for the purpose of scheduling multiple timers?

What does copas.limit do?

[brynne8]

I'm not sure what you mean by setting up a luapower repo in local network. You mean you want to use git to push/pull in a LAN, like this?

Just able to checkout single packages from local folder as if it were from GitHub would be fine. Other git operations are not necessary.

Which packages did you use from ulua?

ulua is a distro that includes binary packages like luapower, but currently it doesn't ship libraries with external dependencies, which is often a pain in building packages. While on the other hand, ulua has many packages from luarocks, where we have more freedom in choosing libs.

I might use some of them to process csv (ftcsv), to parse command line args (argparse), to do unit testing (busted) like these.

Re Copas, I'm actually working to replace the whole network stack with pure Lua code.

It's nice to replace the whole network stack in luapower.

Re heap, you mean something like this https://luapower.com/heap ?

yes

I'm not sure what a timerwheel is ...

Timerwheel is pure lua implementation of timers, using similar logic like copas. It's said to be efficient and simple in code.

but I assume it's a heap of timeouts to decide the minimum amount of timeout to poll for the purpose of scheduling multiple timers?

The latest copas seems to do so with Timerwheel and binaryheap.

What does copas.limit do?

It's a very useful utility in copas I think, which limits the amount of active requests, so that even I creates thousands of copas "threads" at once, it will keep the amount, and after one finishes, it will pick another request to send. If I don't limit the amount, the socket will become error in select(). And this way, I won't have to write additional sleep function or other treatment.

[capr]

Just able to checkout single packages from local folder as if it were from GitHub would be fine. Other git operations are not necessary.

Then I guess that's between you and git. If you can make git work, mgit will work too (mgit clone is only doing git fetch and git checkout nothing fancy).

Re ulua: for csv you could try https://luapower.com/csv, maybe that works for you (I've used it). The other libs that you mentioned are pure Lua and these are quite easy to bring into luapower. What you have to do is: 1) fork the project and clone it somewhere 2) rename the files to fit the luapower directory structure, i.e. rename README.md to argparse.md, move argparse.lua into the root folder of the repo, remove everything else (more info here https://luapower.com/get-involved) 3) commit/push it back to github 4) clone it into your luapower folder with mgit, eg. mgit clone git@github.com:AlexanderMisel/argparse. mgit doesn't care where the libs come from, in fact not a single one of them has to come from luapower, you can fork them all to your account and modify them at will, and clone them from your own account -- that's the core idea behind luapower actually, to be able to assemble a project from multiple places and yet stay under git at all times and even be able to make pull requests etc. to libs that are not yours directly from your luapower folder.

Re copas.limit the http client I mentioned has that feature (among other tuning parameters), but I guess you could add that at the socketloop layer too, just make a queue (just a Lua list) and resume client coroutines as other coroutines finish.

[brynne8]

Thanks for your detailed reply. Moreover, an iconv binding is needed for luapower I think, for conversion of charset. Currently I'm using this on windows.

And for lpeg, I would tend to use a JIT-ted version of it (https://github.com/starius/lpjit). It seems lacking of JIT, it even cannot beat libraries using pure lua on LuaJIT when parsing json, csv, etc. But lpjit seems inactive for years and rewritten a lot of code. Would you like to write a lpeg JIT, only need to rewrite lpeg vm I think, like the PyPeg?

[capr]

Re iconv, is there still anyone who's not using Unicode these days?

Re lpeg, I'm actually a bit disappointed by PEG in general and LPEG in particular. I mean I use it for some things and I like lpeg.re a lot as a DSL, and as far as parsing libraries go it's probably one of the best out there. Still I think that parsing libraries in general have a lot of disadvantages in practice that nobody talks about and are way over-hyped probably because people think that parsing by hand is hard or hard to maintain (it's not!) and they have no choice but to use PEG or something. So I use LPEG for small parsing jobs but I wouldn't use it as the parser of a compiler for instance.

[brynne8]

Re iconv, is there still anyone who's not using Unicode these days?

In windows, there are many. The default encoding for Windows 7, and even Windows 10 is not UTF-8, different with other operating systems. Windows 10 provided an option to use UTF-8 as default encoding, but not enabled by default.

LPEG ...

I like lpeg.re too and I currently, almost only use the re module of lpeg. I think that PEG rules make one more focused on the grammar itself, and make them rethink about order of rules when they use PEG to describe some grammar.

Parsing can be a lot of repetitive work and when I started helping Textadept the editor improve lexers, I really found it useful in defining lexers, and easy to read.

[capr]

The default encoding for Windows 7, and even Windows 10 is not UTF-8.

Not sure what you mean by default encoding. The Windows API comes in two variants: ANSI and UTF-16 and it also has a small API to convert between UTF-16 and UTF-8 which I use in all of https://luapower.com/winapi so that it you can use only utf-8 Lua strings throughout. I never had to worry about code pages with Windows since Windows XP.

[brynne8]

It's not about winapi. You know I am from China, and files and paths and even apps are encoded in GBK (cp936). We need to convert them to utf8.

[capr]

Oh I see. So for instance, using luapower's fs module can you open files with GBK-encoded file names if you pass them as utf-8? I'm asking because the fs module uses the widechar API internally as opposed to lfs which uses the ANSI one.

[brynne8]

Yes. I can open files when pass UTF-8 file path. Thanks. Maybe fs API could be improved so that we can use

local a = f:read('*a')

like the lua io lib to read content from the file. Or if there could be more examples on using fs module, on opening and read content of files into a lua string.

[capr]

added f:readall(), so ffi.string(f:readall()) should be equivalent to f:read'*a'.