Pack assets directory to binnary resource

[xland]

Is it possible to pack assets directory to binnary resource?

What Sciter(https://sciter.com/) did:

Run commond packfolder.exe ui resources.cpp -v "resources" and then generate code like this:

const unsigned char resources[] = {
0x53,0x41,0x72,0x00,0x43,0x00,0xff,0xff,0x03,0x00,0xff,0xff,0x6e,0x00,0xff,0xff,0x04,0x00,0xff,0xff,0x66,
//...

What Electron(https://github.com/electron/electron) and yue(https://github.com/yue/yue) did:

Pack assets directory to a zip(like) file without compression but support random access. See:https://github.com/electron/asar

[mikke89]

Hi, yes, this is certainly possible. Please also see my answer here: https://github.com/mikke89/RmlUi/discussions/299#discussioncomment-2527065

We don't help you pack your assets. It's quite common to make your own simple binary formats, otherwise I'm sure there are tons of tools already for this purpose.

Once you have your asset manager in order you would typically provide your own custom file interface so that RmlUi can read the files from your asset or resource manager.

[xland]

For people who read this in the future.

libzip will be a good choice. https://github.com/nih-at/libzip

It can load zip archive from stream( void* data)

[mikke89]

Right, there is also PhysicsFS which might be a good fit for some users: https://icculus.org/physfs/

[xland]

@mikke89

NW.js just attache the resource zip file to the end of executable file (connect them together). Not define a variable in the rc file to pack the resource zip file. This is very helpful for those who only develop ui files without a Visual Studio environment. But I don't know how the exe reads the resources attached to its tail at runtime. Could you give me some sugesstions?

NW.js (https://nwjs.readthedocs.io/en/latest/For%20Users/Package%20and%20Distribute/)

On Windows you can even hide the zip file by appending the zip file to the end of nw.exe. You can run following command on Windows to achieve this: copy /b nw.exe+package.nw app.exe

[mikke89]

There are many approaches to embed binary files into the executable. The most portable way is to convert the binary to a char array and just embed that into the source code. Hopefully we will get #embed in the standard soon to make that process simpler. And then there are platform-specific methods as you suggest but I'm not familiar with those, but I'm sure there are plenty of resources on this elsewhere.

[xland]

Read exe resource in FileInterface. Some ugly code, but it works.

Please kindly advise.

// #include <Windows.h>
// #include "../../resource.h"
static std::map<int, size_t> resourceSizeMap;
Rml::FileHandle ShellFileInterface::Open(const Rml::String& path)
{
    if (path.substr(0, 6) != "assets") {
        FILE* fp = fopen(path.c_str(), "rb");
        return (Rml::FileHandle)fp;
    }
    if (path == "assets/banner.rml") return IDR_UI1;
    else if (path == "assets/customInstall.rml") return IDR_UI2;
    else if (path == "assets/iconfont.ttf") return IDR_UI3;
    else if (path == "assets/installOption.rml") return IDR_UI4;
    else if (path == "assets/installPanel.rml") return IDR_UI5;
    else if (path == "assets/topBar.rml") return IDR_UI6;
    else if (path == "assets/banner1.tga") return IDR_UI7;
    else if (path == "assets/banner2.tga") return IDR_UI8;
    else return NULL;
}
void ShellFileInterface::Close(Rml::FileHandle file)
{
    if (file > IDR_UI8) {
        fclose((FILE*)file);
    }   
}
size_t ShellFileInterface::Read(void* buffer, size_t size, Rml::FileHandle file)
{
    if (file > IDR_UI8) {
        return fread(buffer, 1, size, (FILE*)file);
    }
    HMODULE instance = ::GetModuleHandle(NULL);
    HRSRC resID = ::FindResource(instance, MAKEINTRESOURCE(file), L"ui");
    if (resID == 0) {
        return 0;
    }
    HGLOBAL res = ::LoadResource(instance, resID);
    if (res == 0) return 0;
    LPVOID resData = ::LockResource(res);
    memcpy(buffer, resData, size);
    return size;

}
bool ShellFileInterface::Seek(Rml::FileHandle file, long offset, int origin)
{
    if (file > IDR_UI8) {
        return fseek((FILE*)file, offset, origin) == 0;
    }
    return true;
}
size_t ShellFileInterface::Tell(Rml::FileHandle file)
{
    if (file > IDR_UI8) {
        return ftell((FILE*)file);
    }
    if (resourceSizeMap[file] > 0) return resourceSizeMap[file];
    HMODULE instance = ::GetModuleHandle(NULL);
    HRSRC resID = ::FindResource(instance, MAKEINTRESOURCE(file), L"ui");
    if (resID != 0) {
        size_t resSize = ::SizeofResource(instance, resID);
        resourceSizeMap[file] = resSize;
        return resSize;
    }   
    return 0;
}

[mikke89]

Good to hear you got it working, that seems like a good start! While I'm not familiar with this part of the Windows API, I do have some feedback:

• I would be a bit worried about collisions between the returned fopen handles and the IDR_ values.
• It seems like the IDR resources do not implement seeking, this could lead to trouble.
• Instead, I would put all the state in a struct, which could contain fopen handle, IDR value, size, and seeking state. Then you could simply allocate this struct and return its pointer as the file handle. Then there is no need for the lookup map either.

[xland]

@mikke89

IDR_ values is defined in the resource.h

#define IDR_UI1                         112
#define IDR_UI2                         115
#define IDR_UI3                         116
#define IDR_UI4                         117
#define IDR_UI5                         118
#define IDR_UI6                         121
#define IDR_UI7                         122
#define IDR_UI8                         123

These values reference to the exe file resource.

When RmlUi request assets/***.rml I make the FileInterface read the exe resource And give the resource data back to RmlUi So that FileInterface don't need to read assets/***.rml files from disk.

1: IDR_ values is less than 200 And a normal Rml::FileHandle pointer is much more bigger than this. An example:2217787826080 So maybe There will be no conflict between them. Yes it is ugly.

2: I found that Seek function and Tell function are called twice by RmlUi. They are only used to determine the size of the assets/***.rml files. No matter how big the file is, RmlUi reads all the data of it at one time. Is it true? If it is, all I need to do is give the size of the resource in the Tell function. And RmlUi will send the size value to Read function. Then I read the resource data in this function.

3: FileInterface is defined in RmlUi\Core I think a average developer would be wise not to change the definition here. How do I make Open function return a custom struct without changing the definition here?

[mikke89]

I don't mean you should modify the file interface. Remember that the user handle can contain any address, so you can easily pass a new object that way. Here is an example (consider it pseudo code):

struct FileData {
FILE* file_handle;
int idr_number;
size_t seek_state;
size_t length;
};

Rml::FileHandle ShellFileInterface::Open(const Rml::String& path)
{
    FileData* data = new FileData{};
    if (path.substr(0, 6) != "assets") {
        data->file_handle = fopen(path.c_str(), "rb");
    }
    else {
        data->idr_number = /* ...*/;
    }
    return (Rml::FileHandle)data;
}

And then in the other functions interpret the handle as a data pointer i.e. (FileData*)file_handle. This way you can easily solve all the concerns I had previously. While Seek may not be needed right now (I'm not sure whether we do that now) it is better to implement the whole interface in case we decide to use it a some point.

[xland]

I see what you mean. Thank you very much.

[xland]

I made a installer software by RmlUi. Which like NSIS or InnoSetup,but more beautiful. https://www.zhihu.com/zvideo/1514698419817459714

[mikke89]

Hey, that looks really good. Great effort!

If you have a screenshot I could add it to the gallery if you'd like. :)

[xland]

I'll be very happy to see my app on the RmlUi official website. Images are here: installer_Imgs.zip

[mikke89]

Thanks! I just added one of the screenshots to the gallery. Let me know if you want me to change the attribution or add a link.

[xland]

It's ok.

[xland]

It's time to close this issue.