amigaos4 (big-endian) port

[kas1e]

Trying to port OpenClaw to amigaos4 (so PPC cpus, big-endian), and while able to build binary, it runs, but then fail on loading of CLAW.REZ, returning NULL.

I checked and CLAW.REZ is here, opens by main binary, read, etc, but still libwap return NULL in WAP_LoadRezArchive(), which mean that it think its invalid data => probabaly issues with structures and big/little endian differences ?

So question is, any luck libwap have some love and big-endian support ?:) Or at least, some pointing on what is wrong can help too.

Thanks!

[pjasicek]

Hi, OpenClaw was never designed with support for big-endian architectures in mind 😄

The libwap library and zip file library would have to be redesigned with this in mind I think. I am not really sure how much work that would be to be honest. If you could backtrack where exactly does the WAP_LoadRezArchive() return NULL I might be able to make an estimate of the effort it would take. But as I said, it was never designed around support both endiannes types.

[kas1e]

If you mean classic libzip library, that one if i remember right have big endian support already..

For libwap, at moment i tried to do conversion to big endian via SDL_SwapLE32, like this:

Changing that in WAP_LoadRezArchive():

// Read rez file version
    FileRead(&(rezArchive->version), fileStream, 127);
    // Read rez payload offset (offset of first directory)
    FileRead(&(rezArchive->rootDirectory->offset), fileStream, 131);
    // 
    FileRead(&(rezArchive->rootDirectory->size), fileStream, 135);

on that:

// Read rez file version
    FileRead(&(rezArchive->version), fileStream, 127);
   rezArchive->version = SDL_SwapLE32(rezArchive->version);
    // Read rez payload offset (offset of first directory)
    FileRead(&(rezArchive->rootDirectory->offset), fileStream, 131);
    rezArchive->rootDirectory->offset = SDL_SwapLE32(rezArchive->rootDirectory->offset);
    // 
    FileRead(&(rezArchive->rootDirectory->size), fileStream, 135);
    rezArchive->rootDirectory->size = SDL_SwapLE32(rezArchive->rootDirectory->size);

But only that change make it all freezes..

Maybe that 127 bytes header also need adjustements as well as some other stuff too ?

I also added few printfs to WAP_LoadRezArchive() , so to see where it return NULL (as i see there just 2 places where return NULLs : 1 in case of no file found and 1 in case wrong data inside). So, file opens fine, but fail when do "if (expectedRezArchiveSize != actualLoadedFileSize)" check.

[pjasicek]

Hmm, I guess then what would solve this would be making FileRead and all other reading generics to be endian agnostic. But I don't have any device which has big endian, so there would be no way for me to test it at the moment. I would have to find some VM which has big endian architecture.

[kas1e]

Yeah seems so..

Is only FileRead are real generic function which used offten, or there is more ? I foind only that one at moment:

template<typename T>
static void FileRead(T* dest, std::ifstream* file, int32_t offset)
{
    file->seekg(offset, std::ios::beg);
    file->read((char*)dest, sizeof(*dest));
}

[pjasicek]

https://github.com/pjasicek/OpenClaw/blob/master/libwap/IO.h

This is used for parsing wwd files I think, I did not write this one.

[kas1e]

That part probabaly already endian agnostic , see at top:

inline bool system_is_big_endian() {
    int n = 1;
    return *(char *)&n != 1;
}

and it used in read_write_impl() .. So probabaly roots at least endian aware (i hope :) )

[pjasicek]

Oh yeah, so that's not a problem then. Then just put the same thing in the FileRead method and hope for it to work 😄

[kas1e]

Ok simple adding that:

template<typename T>
static void FileRead(T* dest, std::ifstream* file, int32_t offset)
{
    file->seekg(offset, std::ios::beg);
    file->read((char*)dest, sizeof(*dest));
    *dest = SDL_SwapLE32(*dest);            
}

Make that loading of game pass futher (so at least it not NULL anymore when load CLAW.REZ data into the memory, so maybe all fine there), but then it fail to load ASSETS.ZIP things with words " [InitializeResources] Failed to initialize resource cachce from resource file: ASSETS.ZIP ".

Full log currently looks like this:

4/0.Work:games/openclaw> openclaw 
INFO: Looking for: PROGDIR:config.xml
INFO: Expecting config.xml in path: config.xml
INFO: Loaded with:
    Config File: config.xml
    Saves File: SAVES.XML
INFO: >>>>> Initializing display...
INFO: Display successfully initialized.
INFO: >>>>> Initializing audio...
INFO: Audio successfully initialized.
INFO: >>>>> Initializing font...
INFO: Font successfully initialized...
INFO: >>>>> Initializing resource cache...
INFO_AMIGAOS4: this time not NULL , we seems to load CLAW.REZ fine!
ERROR: [InitializeResources] Failed to initialize resource cachce from resource file: ASSETS.ZIP
ERROR: [RunGameEngine] Failed to initialize. Exiting. 

[pjasicek]

is the ASSETS.ZIP file in the same folder as CLAW.REZ ?

[pjasicek]

Even if it is, it will fail.

https://github.com/pjasicek/OpenClaw/blob/master/OpenClaw/Engine/Resource/ZipFile.cpp#L156

Whole ZipFile.cpp is taken from Game Coding Complete 4 book and just modified so it runs under linux by me.

It's not really big/little endian agnostic.

[kas1e]

Yeah, file there, and yeah, seems ZipFile.cpp need fixes too .. Probabaly that first fread() before the line you point out need to be fixed, like this maybe swapping the same that &dh structure..

[pjasicek]

If you get stuck let me know, I'll try to help :wink:

[kas1e]

Yeah, help need it,i suck at code expectually when it come to all those conversion things with arrays and co :)

One of my friends who help me with some other things back in the past, saying that this FileRead()'s fix probabaly wasn't right, and proper one will be something like this:

inline void littleBigEndian (void *x, int sz) {
    unsigned char *toConvert = reinterpret_cast<unsigned char *>(x);
    unsigned char tmp;
    for (size_t i = 0; i < sz/2; ++i) {
        tmp = toConvert[i];
        toConvert[i] = toConvert[sz - i - 1];
        toConvert[sz - i - 1] = tmp;
    }
}
template<typename T>
static void FileRead(T* dest, std::ifstream* file, int32_t offset)
{
    file->seekg(offset, std::ios::beg);
    file->read((char*)dest, sizeof(*dest));
    littleBigEndian(dest, sizeof(*dest));
}

But, that If FileRead is only used for single elements that needs Big/Little endian reodoring, and never for Array and things like that (he haven't checked),

But that still didn't fix 127 bytes array, if this need to be fixed.

As for ZipFile.cpp, currently dunno how to fix it..

[kas1e]

Also as probabaly we firstly need to be sure that CLAW.REZ opens/reads/etc fine , i was able to build libwap_tests binary too , but when i run just like "$ libwap_tests CLAW.REZ" it says "no tests run". Dunno if i running it run at all ?

[kas1e]

I going a bit futher: in zipfile.cpp, i doing that:

    fseek(m_pFile, -(int)sizeof(dh), SEEK_END);
    long dhOffset = ftell(m_pFile);
    memset(&dh, 0, sizeof(dh));
    if (fread(&dh, sizeof(dh), 1, m_pFile) != 1)
        return false;

#ifdef __BIG_ENDIAN__
    dh.sig = SDL_SwapBE32(dh.sig);
    dh.nDisk = SDL_SwapBE16(dh.nDisk);
    dh.nStartDisk = SDL_SwapBE16(dh.nStartDisk);
    dh.nDirEntries = SDL_SwapBE16(dh.nDirEntries);
    dh.totalDirEntries = SDL_SwapBE16(dh.totalDirEntries);
    dh.dirSize = SDL_SwapBE32(dh.dirSize);
    dh.dirOffset = SDL_SwapBE32(dh.dirOffset);
    dh.cmntLen = SDL_SwapBE16(dh.cmntLen);
#endif

    printf("amigaos4 zipfile.cpp: dh.sig = %x\n",dh.sig);
    printf("amigaos4 zipfile.cpp: before check on zipheader signature\n");
    // Check
    if (dh.sig != TZipDirHeader::SIGNATURE)
        return false;

I.e. conver strcuture of zipheader to big endian , and then i go futher.

Now i still fail in the !pCustomCache->Init() from BaseGameApp.cpp, and trying to see what it calling next after zip inititialisation..

[pjasicek]

I'd help but I don't have any big endian device / vm ... but what you are doing makes sense to me

[kas1e]

Was away from home, so continue again :) Doing that which in previous post passed things futher, but still i again have error "Failed to initialize resource cachce from resource file ASSETS.ZIP" , while ZipFile::Init() return success now for sure.

I.e. BaseGameApp::InitializeResources() starts, going till that part:

    if (!pCustomCache->Init())
    {
        LOG_ERROR("Failed to initialize resource cachce from resource file: " + customArchivePath);
        return false;
    }

And fail with that error. From brief look i not very much understand c++ code, so dunno what functions it call now .. That pCustomCache() declared as :

ResourceCache* pCustomCache = new ResourceCache(50, pCustomArchive, CUSTOM_RESOURCE);

So its again ResourceCache , which probabaly call something else now ?

[pjasicek]

It is still the problem with the ZipFile - it is the pCustomArchive

[kas1e]

What i mean is what function called next after zip:init(). zip:readfile() are next one which need fixing ? I just didnt find any function called pCustomArchive() (if that what you mean)

[pjasicek]

pCustomArchive is just a variable, not a function..

The initialization fails in this method: https://github.com/pjasicek/OpenClaw/blob/master/OpenClaw/Engine/Resource/ZipFile.cpp#L136

you need to put debug logs there and see where it returns false and figure out why.

[kas1e]

That strange, as zip::init seems passed fine, that what i have in output now, when put bunch of prinfs in zip::init():

INFO: Looking for: PROGDIR:config.xml
INFO: Expecting config.xml in path: config.xml
INFO: Loaded with:
    Config File: config.xml
    Saves File: SAVES.XML
INFO: >>>>> Initializing display...
INFO: Display successfully initialized.
INFO: >>>>> Initializing audio...
INFO: Audio successfully initialized.
INFO: >>>>> Initializing font...
INFO: Font successfully initialized...
INFO: >>>>> Initializing resource cache...
this time we load that CLAW.REZ fine !
amigaos4 basegameapp.cpp: till that moment initialisation going fine
amigaos4 zipfile.cpp: dh.sig = 6054b50
amigaos4 zipfile.cpp: before check on zipheader signature
amigaos4 zipfile.cpp: after check on zipheader signature
amigaos4 zipfile.cpp: before processing each entry
amigaos4 zipfile.cpp: before < dh.nDirEntries && success loop
amigaos4 zipfile.cpp: Init return success, horray!
ERROR: [InitializeResources] Failed to initialize resource cachce from resource file: ASSETS.ZIP
amigaos4: initialize resource failed in !pCustomCache->Init()
amigaos4: damn! we can'tinitialize resources !
ERROR: [RunGameEngine] Failed to initialize. Exiting.

I.e. init return success for sure first time, but then, it fail when doing so next time ? It is probabaly something else, not zip::init(), as it should then fail first time too when open ASSETS.ZIP first time..

Will try to invistigate more