Open rpatters1 opened 2 years ago
cc @JAngermueller
Last time I just emailed him. Will do that again.
My initial thoughts is that I like this proposal. I'd almost say that anything that isn't core to RGP Lua should probably be in a separate, public repository.
Do you have a strong preference towards using C or C++? In my opinion, Rust would be a better language for this project for a few reasons:
1) Rust is memory-safe, which to me is quite important in onboarding new developers who don't have that much experience in memory management. 2) The Rust language scales far better. While I've worked on very large C++ codebases before, a ton of tooling was needed to keep things under control. With Rust, none of that tooling is needed. 3) Rust has all the bells and whistles of a modern developer experience built-in. C and C++ do not. 4) If we wanted to create a monorepo for all the RGP Lua extensions (which I'd recommend), Rust is a lot easier to use. 5) Rust's build system is much, much easier to use and less error prone.
Rust also has the exact same runtime performance as C++, so performance should not be an issue.
There are just two main hurdles I see if using Rust:
1) A new language to learn for some people 2) We'd need to interop with C++ (though this is a previously solved problem)
https://firefox-source-docs.mozilla.org/writing-rust-code/cpp-interop.html
I know nothing about Rust, so I can't comment, except as follows:
LuaBridge
, which is 100% modern c++. (This was inherited from JW Lua.) If a Rust library ever needs to interact with PDK Framework classes, that will be a big challenge. (Whereas it is fairly easy in C++, as my sample code illustrates.)C++ is not for the faint of heart. Having said that, I'm not sure how much onboarding there needs to be. Hopefully this project is limited in scope. This is a lua project, after all. 🤣
I did a little more research and realized that luasocket already has http running over luasocket. It should simply be a matter of installing these in a sub-directory called "socket" and then all the requires should work. (Similar to the "library" sub-directory we use.)
I need to make sure that RGP Lua is including all the C components.
Including luasocket sounds like a good and easy solution to me. That is all I was looking for.
Currently my plug-ins on macOS use "curl" and on Windows PowerShell with WebClient.DownloadFile ( https://docs.microsoft.com/en-us/dotnet/api/system.net.webclient.downloadfile?view=net-6.0 ). This runs stable. But especially on Windows it leads to many black window pop-ups and sometimes the anti-virus software doesn't like these Powershell internet calls.
You can probably get luasocket to work without waiting for a release of RGP Lua. Building the full version in has its challenges, but I'm hoping they are surmountable.
In considering this further, I'm thinking it would be much cleaner not to have this embedded in RGP Lua at all. (Embedding more than the core luasocket library starts to get right messy, because of the mix of lua and c code.)
Right now the challenge with luasocket is accessing it in a way that isn't highly dependent on the end-user setup. Maybe the time would be better spend creating a full luasocket package that's easy to have self-contained with any script that needs it.
Following up on this, RGP Lua 0.64, when luasocket is requested, will load both socket.core
and socket.lua
into the global variable socket
. This is more conformant with the actual luasocket installation, and it allows either of these to work:
local socket = require('socket')
local socket = require('socket.core')
I've run into both variants in various scripts that use luasocket.
In addition, when luasocket is requested, RGP Lua 0.64 will preload mime.core
but will not load it into a variable. As far as I can tell, socket.core
and mime.core
are the only C modules in luasocket. This means that the rest of the luasocket code, which is all in lua, can be required externally by scripts as needed. This includes the http
module.
To get it to work the way we'd like (so that the require
statements for luasocket work the same as if it were installed with luarocks), we will need to do one of two things. I would like your input on which you prefer and any other ideas you might have.
require
function as follows:
-- This require function would be pre-loaded. I'm showing here for now.
raw_require = require
function require(item)
if item ~= "socket.core" then
local _, end_index = item:find("^socket%.")
if end_index then
item = item:sub(end_index+1)
end
end
return raw_require(item)
end
This allows all the luasocket lua sources to reside in an arbitrary directory. For my testing I'm pulling them straight out of the luasocket repo. The following code is working for me with script debugging enabled on the latest dev build of RGP Lua.
-- This require function would be pre-loaded. I'm showing here for now.
raw_require = require
function require(item)
if item ~= "socket.core" then
local _, end_index = item:find("^socket%.")
if end_index then
item = item:sub(end_index+1)
end
end
return raw_require(item)
end
if finenv.IsRGPLua then
require('mobdebug').start()
end
local socket_source_path = finenv.RunningLuaFolderPath() .. "../lua-source/luasocket/?.lua"
socket_source_path = socket_source_path .. ";" .. finenv.RunningLuaFolderPath() .. "../lua-source/luasocket/src/?.lua"
package.path = package.path .. ";" .. socket_source_path
local http = require 'http'
print("http:")
for k, v in pairs(http) do
print(k, v)
end
local mime = require 'mime'
print("")
print("mime")
for k, v in pairs(mime) do
print(k, v)
end
I would only replace the require
function if the embedded luasocket were requested. What does everyone think about this?
I think the best approach for this is, if only debugging is requested ("Enable Debugging" in the configuration or finaleplugin.Debug = true
in plugindef
) then I will only load the core socket functions into socket
. However, if finaleplugin.LoadLuaSocket = true
is specified, then I will additionally pre-load mime.core
and override the require
function.
What about making the override of the require
function depend on an additional option? Something like finaleplugin.FlattenSocketRequires = true
that is only meaningful if LoadLuaSocket
is true. I see this mainly as a tool for scripts that need to access http
for things like checking for updates, so I would like to keep it pretty narrow and perhaps some barriers to entry aren't such a bad idea.
More followup. I realized we don't need an additional option in finaleplugin
. The original require
is being moved to a new global called __original_require
. (My code has to have the original to be able to call it with the massaged library name.) So any script that wants the original require
can simply put it back.
After working with @JAngermueller with luasec
, I am coming around to the idea that it adds too much complexity for what we want. It is based on OpenSSL, and that in itself is a can of worms. I don't think we can count on it being compatible on every end-user machine, and we can't expect them to build the version we need.
So I'm back to the idea of a C++ module. It seems like my time would be better spent creating a simple file downloader using built-in OS tools than trying to get luasec
(actually, OpenSSL) to work reliably in every environment where we'll need it. I think the module really must be C++ because Lua is so tightly integrated with C/C++. It won't be anything fancy anyway. (Also, modern c++ has solved most of those memory handling issues, and the debuggers have gotten quite good at pinpointing them in older code.)
Assuming that we may need to grow it past a simple https file downloader, what would be a good open-ended name for it? How about finaleos
?
I have created a luaostools repository. The release binaries main
branch should allow simple downloads of small-to-medium size files. If we ever need to download large files, they probably should be saved directly to disk, and that will require a different api.
I'm still working on cleaning up some of the error reporting, but the main
branch should be stable enough to use.
Requirement
It is becoming apparent that there is a need to be able to download files from the web and manipulate them in various ways. This includes potentially updating script files and the like. One of the challenges is that it is very bad form to download files from the web in synchronous calls. An asynchronous approach would be ideal.
Describe the solution you'd like
I would like to keep the core RGP Lua functions clean from this for a few reasons:
luasocket
currently is (and add an option to load or not load it per script).dll
ordylib
andrequire
the latest version without dependence on a new version release of RGP Lua.Therefore I am proposing a new C++ repository, perhaps called
lua-httptools
to live in thefinalelua
organization. It will make use of OS-level http calls, or any other tools available to C++ development environments, to process http calls.Describe alternatives you've considered
I have considered a built-in
finenv
function to download a file. But this is very problematic becausefinenv
. It has nothing to do with Finale and will not be dependent on the PDK Framework.I also looked for existing open-source projects. There are a fair number of them, but they are all quite heavy and have lots of dependencies. It's more work than I have interest in to figure them out, but it would probably be possible to get one of them to work if someone wants to try. This might be a good first step for an interested party.
Ultimately, I suspect that a C/C++ library that can directly access the OS apis and is more tailored to the needs of Finale lua will be a good compromise. It can be built into RGP Lua (as an automatic load-in to the Lua state) but it can also be loaded externally if needed.
Next Steps
If someone is interested, they can look into a 3rd party library and see if it can be made to work in RGP Lua. That might short-circuit the whole conversation.
I will migrate the code I have put into
finenv
into a shell library that can be built with RGP Lua but be independent as a code base. It will be very rudimentary and ultimately probably not the final API. But it will at least start the project.We need to be thinking about how to structure the api. To start with, the library will have a single function,
httptools.get_file_synchronous
. Then more interested parties than I can start adding to it in ways that seem interesting to them. Initially it will not be included in RGP Lua until we can gauge the interest and utility. Instead, it will be built as an external library that can be required by any script that needs it. In fact, you'll be able to debug it through Finale as described at sample-clib. I intend to use that libary as my starting point.@Nick-Mazuk could you set things up so that we can tag Jan Angermüller? I suspect he will be quite interested in this as well.