Http tools for Finale Lua

[rpatters1]

Requirement

It is becoming apparent that there is a need to be able to download files from the web and manipulate them in various ways. This includes potentially updating script files and the like. One of the challenges is that it is very bad form to download files from the web in synchronous calls. An asynchronous approach would be ideal.

Describe the solution you'd like

I would like to keep the core RGP Lua functions clean from this for a few reasons:

• I am getting a bit overwhelmed, and this could become a moderately large project.
• By making it a separate Lua library, I don't have to be the only contributor.
• We can nevertheless include it in RGP Lua as a library to be built-in much as luasocket currently is (and add an option to load or not load it per script).
• Nevertheless, it will be possible to build it as an external dll or dylib and require the latest version without dependence on a new version release of RGP Lua.

Therefore I am proposing a new C++ repository, perhaps called lua-httptools to live in the finalelua organization. It will make use of OS-level http calls, or any other tools available to C++ development environments, to process http calls.

Describe alternatives you've considered

I have considered a built-in finenv function to download a file. But this is very problematic because

• it must be a synchronous call and it is also a one-off function specific to a narrow requirement. A synchronous call can lead to big problems if scripts start trying to download multi-megabyte (or gigabyte) files.
• it is in the RGP Lua codebase, which means only I can contribute to it. Yet my personal interest in this feature is low.
• it is dependent on the release cadence of RGP Lua. Given the history of Finale on Lua (think JW Lua and the 5-year gap of no updates), I think it is better to keep this separate.
• there is no compelling reason that this functionality should be part of finenv. It has nothing to do with Finale and will not be dependent on the PDK Framework.

I also looked for existing open-source projects. There are a fair number of them, but they are all quite heavy and have lots of dependencies. It's more work than I have interest in to figure them out, but it would probably be possible to get one of them to work if someone wants to try. This might be a good first step for an interested party.

Ultimately, I suspect that a C/C++ library that can directly access the OS apis and is more tailored to the needs of Finale lua will be a good compromise. It can be built into RGP Lua (as an automatic load-in to the Lua state) but it can also be loaded externally if needed.

Next Steps

If someone is interested, they can look into a 3rd party library and see if it can be made to work in RGP Lua. That might short-circuit the whole conversation.

I will migrate the code I have put into finenv into a shell library that can be built with RGP Lua but be independent as a code base. It will be very rudimentary and ultimately probably not the final API. But it will at least start the project.

We need to be thinking about how to structure the api. To start with, the library will have a single function, httptools.get_file_synchronous. Then more interested parties than I can start adding to it in ways that seem interesting to them. Initially it will not be included in RGP Lua until we can gauge the interest and utility. Instead, it will be built as an external library that can be required by any script that needs it. In fact, you'll be able to debug it through Finale as described at sample-clib. I intend to use that libary as my starting point.

@Nick-Mazuk could you set things up so that we can tag Jan Angermüller? I suspect he will be quite interested in this as well.

[Nick-Mazuk]

cc @JAngermueller

Last time I just emailed him. Will do that again.

[Nick-Mazuk]

My initial thoughts is that I like this proposal. I'd almost say that anything that isn't core to RGP Lua should probably be in a separate, public repository.

Do you have a strong preference towards using C or C++? In my opinion, Rust would be a better language for this project for a few reasons:

1) Rust is memory-safe, which to me is quite important in onboarding new developers who don't have that much experience in memory management. 2) The Rust language scales far better. While I've worked on very large C++ codebases before, a ton of tooling was needed to keep things under control. With Rust, none of that tooling is needed. 3) Rust has all the bells and whistles of a modern developer experience built-in. C and C++ do not. 4) If we wanted to create a monorepo for all the RGP Lua extensions (which I'd recommend), Rust is a lot easier to use. 5) Rust's build system is much, much easier to use and less error prone.

Rust also has the exact same runtime performance as C++, so performance should not be an issue.

There are just two main hurdles I see if using Rust:

1) A new language to learn for some people 2) We'd need to interop with C++ (though this is a previously solved problem)

https://firefox-source-docs.mozilla.org/writing-rust-code/cpp-interop.html

https://cxx.rs/

[rpatters1]

I know nothing about Rust, so I can't comment, except as follows:

• The Lua language is intimately tied to C/C++. Any mechanism for creating a Lua-callable library in Rust will add complexity. I don't know how to do it myself (obviously). Heck, the whole way Lua loads and interacts with a C library is quite idiosyncratic and Lua-specific, if easy enough to learn.
• The Lua<=>PDK Framework is all done with LuaBridge, which is 100% modern c++. (This was inherited from JW Lua.) If a Rust library ever needs to interact with PDK Framework classes, that will be a big challenge. (Whereas it is fairly easy in C++, as my sample code illustrates.)
• I don't know if I can easily build a Rust library into RGP Lua. Maybe. I just don't know. Does Rust have an XCode (Mac) and a Visual Studio (Win) presence? Those are the IDEs I use. Does it have external dependencies that might create installation headaches? (I mean, Windows C++ has a f*king version-dependent runtime that you can't count on being installed, so it's not a stretch to think that Rust would be at least as troublesome. At least with macOS, the C++ runtime is installed with the OS.)
• I probably will not be involved in any Rust projects personally. At least not to start with, so I wouldn't be starting this project like I could the C++ one. (But that's fine with me. I don't need to.)

C++ is not for the faint of heart. Having said that, I'm not sure how much onboarding there needs to be. Hopefully this project is limited in scope. This is a lua project, after all. 🤣

[rpatters1]

I did a little more research and realized that luasocket already has http running over luasocket. It should simply be a matter of installing these in a sub-directory called "socket" and then all the requires should work. (Similar to the "library" sub-directory we use.)

I need to make sure that RGP Lua is including all the C components.

[JAngermueller]

Including luasocket sounds like a good and easy solution to me. That is all I was looking for.

Currently my plug-ins on macOS use "curl" and on Windows PowerShell with WebClient.DownloadFile ( https://docs.microsoft.com/en-us/dotnet/api/system.net.webclient.downloadfile?view=net-6.0 ). This runs stable. But especially on Windows it leads to many black window pop-ups and sometimes the anti-virus software doesn't like these Powershell internet calls.

[rpatters1]

You can probably get luasocket to work without waiting for a release of RGP Lua. Building the full version in has its challenges, but I'm hoping they are surmountable.

[rpatters1]

In considering this further, I'm thinking it would be much cleaner not to have this embedded in RGP Lua at all. (Embedding more than the core luasocket library starts to get right messy, because of the mix of lua and c code.)

Right now the challenge with luasocket is accessing it in a way that isn't highly dependent on the end-user setup. Maybe the time would be better spend creating a full luasocket package that's easy to have self-contained with any script that needs it.

[rpatters1]

Following up on this, RGP Lua 0.64, when luasocket is requested, will load both socket.core and socket.lua into the global variable socket. This is more conformant with the actual luasocket installation, and it allows either of these to work:

local socket = require('socket')
local socket = require('socket.core')

I've run into both variants in various scripts that use luasocket.

In addition, when luasocket is requested, RGP Lua 0.64 will preload mime.core but will not load it into a variable. As far as I can tell, socket.core and mime.core are the only C modules in luasocket. This means that the rest of the luasocket code, which is all in lua, can be required externally by scripts as needed. This includes the http module.

[rpatters1]

To get it to work the way we'd like (so that the require statements for luasocket work the same as if it were installed with luarocks), we will need to do one of two things. I would like your input on which you prefer and any other ideas you might have.

• Install the lua sources in subdirectories as described on the luasocket installation page.
• Replace the require function as follows:

-- This require function would be pre-loaded. I'm showing here for now.
raw_require = require
function require(item)
    if item ~= "socket.core" then
        local _, end_index = item:find("^socket%.")
        if end_index then
            item = item:sub(end_index+1)
        end
    end
    return raw_require(item)
end

This allows all the luasocket lua sources to reside in an arbitrary directory. For my testing I'm pulling them straight out of the luasocket repo. The following code is working for me with script debugging enabled on the latest dev build of RGP Lua.

-- This require function would be pre-loaded. I'm showing here for now.
raw_require = require
function require(item)
    if item ~= "socket.core" then
        local _, end_index = item:find("^socket%.")
        if end_index then
            item = item:sub(end_index+1)
        end
    end
    return raw_require(item)
end

if finenv.IsRGPLua then
    require('mobdebug').start()
end

local socket_source_path = finenv.RunningLuaFolderPath() .. "../lua-source/luasocket/?.lua"
socket_source_path = socket_source_path .. ";" .. finenv.RunningLuaFolderPath() .. "../lua-source/luasocket/src/?.lua"
package.path = package.path .. ";" .. socket_source_path

local http = require 'http'

print("http:")
for k, v in pairs(http) do
    print(k, v)
end

local mime = require 'mime'

print("")
print("mime")
for k, v in pairs(mime) do
    print(k, v)
end

I would only replace the require function if the embedded luasocket were requested. What does everyone think about this?

[rpatters1]

I think the best approach for this is, if only debugging is requested ("Enable Debugging" in the configuration or finaleplugin.Debug = true in plugindef) then I will only load the core socket functions into socket. However, if finaleplugin.LoadLuaSocket = true is specified, then I will additionally pre-load mime.core and override the require function.

What about making the override of the require function depend on an additional option? Something like finaleplugin.FlattenSocketRequires = true that is only meaningful if LoadLuaSocket is true. I see this mainly as a tool for scripts that need to access http for things like checking for updates, so I would like to keep it pretty narrow and perhaps some barriers to entry aren't such a bad idea.

[rpatters1]

More followup. I realized we don't need an additional option in finaleplugin. The original require is being moved to a new global called __original_require. (My code has to have the original to be able to call it with the massaged library name.) So any script that wants the original require can simply put it back.

[rpatters1]

After working with @JAngermueller with luasec, I am coming around to the idea that it adds too much complexity for what we want. It is based on OpenSSL, and that in itself is a can of worms. I don't think we can count on it being compatible on every end-user machine, and we can't expect them to build the version we need.

So I'm back to the idea of a C++ module. It seems like my time would be better spent creating a simple file downloader using built-in OS tools than trying to get luasec (actually, OpenSSL) to work reliably in every environment where we'll need it. I think the module really must be C++ because Lua is so tightly integrated with C/C++. It won't be anything fancy anyway. (Also, modern c++ has solved most of those memory handling issues, and the debuggers have gotten quite good at pinpointing them in older code.)

Assuming that we may need to grow it past a simple https file downloader, what would be a good open-ended name for it? How about finaleos?

[rpatters1]

I have created a luaostools repository. The release binaries main branch should allow simple downloads of small-to-medium size files. If we ever need to download large files, they probably should be saved directly to disk, and that will require a different api.

I'm still working on cleaning up some of the error reporting, but the main branch should be stable enough to use.