rtc static compilation with dll files

[ShadyShenAli]

I'm using luart version

LuaRT 1.7.1 x64 - Windows programming framework for Lua.

I'm trying to statically build an executable using rtc. I've included all the related dll files in the folder and execution with luart works fine. However, when I execute

 rtc -s . h.lua

and execute the generated h.exe file, error occurred, saying socket/core.dll not found.

The folder structure looks like,

D:\base\libcrypto-1_1-x64.dll
D:\base\libssl-1_1-x64.dll
D:\base\ltn12.lua
D:\base\mime
D:\base\mime.lua
D:\base\socket
D:\base\socket.lua
D:\base\ssl
D:\base\ssl.dll
D:\base\ssl.lua
D:\base\mime\core.dll
D:\base\socket\core.dll
D:\base\socket\ftp.lua
D:\base\socket\headers.lua
D:\base\socket\http.lua
D:\base\socket\smtp.lua
D:\base\socket\tp.lua
D:\base\socket\url.lua
D:\base\ssl\https.lua

and error message is

socket.lua:12: module 'socket.core' not found:
        no field package.preload['socket.core']
        no file 'D:\lua\socket\core.lua'
        no file 'D:\lua\socket\core\init.lua'
        no file 'D:\socket\core.lua'
        no file 'D:\socket\core\init.lua'
        no file 'D:\..\modules\socket\core\socket\core.lua'
        no file 'D:\..\modules\socket\core\init.lua'
        no file '.\socket\core.lua'
        no file '.\socket\core.wlua'
        no file '.\socket\core\init.lua'
        no file 'D:\lua\socket\core.wlua'
        no file 'D:\socket\core.wlua'
        no file 'D:\..\modules\socket\core\socket\core.wlua'
        no file '.\socket\core.wlua'
        no file ''
        no file 'D:\socket\core.dll'
        no file 'D:\..\lib\lua\5.4\socket\core.dll'
        no file 'D:\..\modules\socket\core\socket\core.dll'
        no file 'D:\loadall.dll'
        no file '.\socket\core.dll'
        no file ''
        no file 'D:\socket.dll'
        no file 'D:\..\lib\lua\5.4\socket.dll'
        no file 'D:\..\modules\socket\socket.dll'
        no file 'D:\loadall.dll'
        no file '.\socket.dll'
        no file ''
        no embedded module 'socket/core.dll' found

The entire code is attached as base.zip

[samyeyo]

Building a static executable that uses embedded binary modules will throw those kind of runtime error.

Lua binary modules are depending on lua54.dll which is not found hence the error at runtime.

You have two options to make it work :

• Copy LuaRT\bin\lua54.dll next your generated executable and cross fingers (because using a static CRT and a dynamic one can lead to crashs)

• Use dynamic compilation (remove the -s flag)

[ShadyShenAli]

@samyeyo : thanks for the quick reply. Unfortunately option doesn't work. (same error message).

As to option 2, it works. But the folder structure looks quite messy. Is it possible to place those dependent .lua and .dll files inside a sub-folder?

It seems that I can put .lua files under ./lua, e.g. for socket.http, I can put it under base/lua/socket/http.lua but for those dlls, such as socket/core.dll, if they are put under /lua/socket/core.dll, it will not be picked. Any workaround for this?

[samyeyo]

You can embed the folder structure with Dynamic compilation too, it should work

[ShadyShenAli]

I managed to make it work without embedded.

Now my app folder has the following structure

D:\BASE
│   h.exe
│   h.lua
│   libcrypto-1_1-x64.dll
│   libssl-1_1-x64.dll
│   lua54.dll
│
└───lua
    │   ltn12.lua
    │   mime.lua
    │   socket.lua
    │   ssl.dll
    │   ssl.lua
    │
    ├───mime
    │       core.dll
    │
    ├───socket
    │       core.dll
    │       ftp.lua
    │       headers.lua
    │       http.lua
    │       smtp.lua
    │       tp.lua
    │       url.lua
    │
    └───ssl
            https.lua

And I modify my h.lua file by adding the following in the beginning,

package.path = ";.\\lua\\?.lua;"..package.path;
package.cpath=";.\\lua\\?.dll;"..package.cpath;

However, embedded still doesn't work. I created the embedded exe file via

base> rtc -c lua h.lua

and move it to the root folder together with the lua54.dll and execute h.exe. The following error occurs

h.lua:4: module 'socket.http' not found:
        no field package.preload['socket.http']
        no file ''
        no file '.\lua\socket\http.lua'
        no file 'D:\lua\socket\http.lua'
        no file 'D:\lua\socket\http\init.lua'
        no file 'D:\socket\http.lua'
        no file 'D:\socket\http\init.lua'
        no file 'D:\..\modules\socket\http\socket\http.lua'
        no file 'D:\..\modules\socket\http\init.lua'
        no file '.\socket\http.lua'
        no file '.\socket\http.wlua'
        no file '.\socket\http\init.lua'
        no file 'D:\lua\socket\http.wlua'
        no file 'D:\socket\http.wlua'
        no file 'D:\..\modules\socket\http\socket\http.wlua'
        no file '.\socket\http.wlua'
        no file ''
        no file ''
        no file '.\lua\socket\http.dll'
        no file 'D:\socket\http.dll'
        no file 'D:\..\lib\lua\5.4\socket\http.dll'
        no file 'D:\..\modules\socket\http\socket\http.dll'
        no file 'D:\loadall.dll'
        no file '.\socket\http.dll'
        no file ''
        no file ''
        no file '.\lua\socket.dll'
        no file 'D:\socket.dll'
        no file 'D:\..\lib\lua\5.4\socket.dll'
        no file 'D:\..\modules\socket\socket.dll'
        no file 'D:\loadall.dll'
        no file '.\socket.dll'
        no file ''
        no embedded module 'socket/http.dll' found

The last seems suggest the embed binary is not proper recognized.

Any idea?

[samyeyo]

base> rtc -c lua h.lua

You should use instead : rtc -c . h.lua

Because your modules are inside the subfolder lua

If you put lua the content of the folder will be put as the root folder inside the executable (there will be only mime, ssl and socket subfolders)

The embedded content is a virtual filesystem.

rtc uses the directory content as the root of this filesystem (not as a subfolder)

[ShadyShenAli]

I'm afraid that doesn't work.

Reproduceable steps

current folder structure

D:\TEST
|   libcrypto-1_1-x64.dll
|   libssl-1_1-x64.dll
|   lua54.dll
|
\---httpTest
    |   h.lua
    |   libcrypto-1_1-x64.dll
    |   libssl-1_1-x64.dll
    |
    \---lua
        |   ltn12.lua
        |   mime.lua
        |   socket.lua
        |   ssl.dll
        |   ssl.lua
        |
        +---mime
        |       core.dll
        |
        +---socket
        |       core.dll
        |       ftp.lua
        |       headers.lua
        |       http.lua
        |       smtp.lua
        |       tp.lua
        |       url.lua
        |
        \---ssl
                https.lua

running httpTest>luart h.lua is successful. Now remove the two dlls from httpTest folder.

compile it dynamically linked with embedding,

httpTest> rtc -c . h.lua
copy h.exe ..
cd ..
h.exe

will yield the following error

h.lua:4: module 'socket.http' not found:
        no field package.preload['socket.http']
        no file ''
        no file '.\lua\socket\http.lua'
        no file 'D:\test\lua\socket\http.lua'
        no file 'D:\test\lua\socket\http\init.lua'
        no file 'D:\test\socket\http.lua'
        no file 'D:\test\socket\http\init.lua'
        no file 'D:\test\..\modules\socket\http\socket\http.lua'
        no file 'D:\test\..\modules\socket\http\init.lua'
        no file '.\socket\http.lua'
        no file '.\socket\http.wlua'
        no file '.\socket\http\init.lua'
        no file 'D:\test\lua\socket\http.wlua'
        no file 'D:\test\socket\http.wlua'
        no file 'D:\test\..\modules\socket\http\socket\http.wlua'
        no file '.\socket\http.wlua'
        no file ''
        no file ''
        no file '.\lua\socket\http.dll'
        no file 'D:\test\socket\http.dll'
        no file 'D:\test\..\lib\lua\5.4\socket\http.dll'
        no file 'D:\test\..\modules\socket\http\socket\http.dll'
        no file 'D:\test\loadall.dll'
        no file '.\socket\http.dll'
        no file ''
        no file ''
        no file '.\lua\socket.dll'
        no file 'D:\test\socket.dll'
        no file 'D:\test\..\lib\lua\5.4\socket.dll'
        no file 'D:\test\..\modules\socket\socket.dll'
        no file 'D:\test\loadall.dll'
        no file '.\socket.dll'
        no file ''
        no embedded module 'socket/http.dll' found

It seems that "./lua/?.lua" in package.path is not recognized when looking in embedded content

Therefore I tested again by changing the virtual root to the lua folder. i.e.

HTTPTEST> rtc -c lua h.lua
copy h.exe ..
cd ..
h.exe

This time, lua module is found, while c module is not. Error message:

socket.lua:12: module 'socket.core' not found:
        no field package.preload['socket.core']
        no file ''
        no file '.\lua\socket\core.lua'
        no file 'D:\test\lua\socket\core.lua'
        no file 'D:\test\lua\socket\core\init.lua'
        no file 'D:\test\socket\core.lua'
        no file 'D:\test\socket\core\init.lua'
        no file 'D:\test\..\modules\socket\core\socket\core.lua'
        no file 'D:\test\..\modules\socket\core\init.lua'
        no file '.\socket\core.lua'
        no file '.\socket\core.wlua'
        no file '.\socket\core\init.lua'
        no file 'D:\test\lua\socket\core.wlua'
        no file 'D:\test\socket\core.wlua'
        no file 'D:\test\..\modules\socket\core\socket\core.wlua'
        no file '.\socket\core.wlua'
        no file ''
        no file ''
        no file '.\lua\socket\core.dll'
        no file 'D:\test\socket\core.dll'
        no file 'D:\test\..\lib\lua\5.4\socket\core.dll'
        no file 'D:\test\..\modules\socket\core\socket\core.dll'
        no file 'D:\test\loadall.dll'
        no file '.\socket\core.dll'
        no file ''
        no file ''
        no file '.\lua\socket.dll'
        no file 'D:\test\socket.dll'
        no file 'D:\test\..\lib\lua\5.4\socket.dll'
        no file 'D:\test\..\modules\socket\socket.dll'
        no file 'D:\test\loadall.dll'
        no file '.\socket.dll'
        no file ''
        no embedded module 'socket/core.dll' found

Any idea?

The source file attached for your reference test.zip

[samyeyo]

I will investigate this more sorry. Thank you for lo this precise feedback

[samyeyo]

You are right, it was a bug. It's fixed and will be available in next LuaRT release. It was due to the fact I didn't implemented the Lua module search fully. It will now respect package.path / package.cpath strings and LUA_PATH / LUA_CPATH environment variables.

I have tested your example it now compiles and run seamlessly.

Thank you for your precise feedback on this !

[samyeyo]

Should have been fixed in LuaRT 1.8.0

[ShadyShenAli]

@samyeyo : which command-line command do you use that makes it work with 1.8.0 version? considering the following folder structure

D:\HTTPTEST
│   libcrypto-1_1-x64.dll
│   libssl-1_1-x64.dll
│   lua54.dll
│
└───src
    │   h.lua
    │
    └───lua
        │   ltn12.lua
        │   mime.lua
        │   socket.lua
        │   ssl.dll
        │   ssl.lua
        │
        ├───mime
        │       core.dll
        │
        ├───socket
        │       core.dll
        │       ftp.lua
        │       headers.lua
        │       http.lua
        │       smtp.lua
        │       tp.lua
        │       url.lua
        │
        └───ssl
                https.lua

And cwd is src.

case a: If I use rtc -c . h.lua, when the h.exe is run under httpTest folder, it shows error

h.lua:4: module 'socket.http' not found
...
        no embedded module 'socket/http.dll' found

case b: If I use rtc -c lua h.lua, and h.exe is run in httpTest folder, it shows error

socket.lua:12: module 'socket.core' not found:
...
        no embedded module 'socket/core.dll' found

Considering in h.lua,

package.path = ";.\\lua\\?.lua;"..package.path;
package.cpath=";.\\lua\\?.dll;"..package.cpath;

<embed_root>\\lua\\socket\\http.lua is not checked for module ```socket.http" in the case a, as is expected.

[ShadyShenAli]

are you sure the new code is in the released version? it seems that the new error message should be something like luaL_error(L, "error loading module '%s' from embedded file '%s'", lua_tostring(L, 1), filename); rather than lua_pushfstring(L, "no embedded module '%s.dll' found", modname);

[samyeyo]

Ok I have found the issues.

First issue: Manage your DLL binary dependencies of your Lua binary modules The LuaSec ssl.dll module is linked to the openSSL DLL libraries libcrypto-1_1-x64.dll and libssl-1_1-x64.dll So when your Lua script requires this embedded binary module, Windows cannot find this two dependencies as they are embedded.

Solution You will need to extract manualy this two dependencies at the start of your Lua script :

if embed then
    embed.File("libcrypto-1_1-x64.dll")
    embed.File("libssl-1_1-x64.dll")
end

Second issue: rtc don't support Lua binary modules that creates sub modules In this case, this is a rtc bug. It is fixed now on the main branch, available in the LuaRT 1.8.0 release page (binaries have been updated)

Thank you again for your report

[samyeyo]

I will work on loading embedded Lua binary modules even if they have DLL dependencies.

The main goal will be to autoload those DLL dependencies if they are embedded too (as for libcrypto-1_1-x64.dll and libssl-1_1-x64.dll in your case, so end users won't need to take care of manualy extract those dependencies.