unable to create character when using gamefilesd2

[mewmew]

I tried running DGEngine with the Diablo II game files, however I can't seem to get it working quite right.

I've tried both rev 6c499d91c7d91ce5f1715ccf31212acf8748e363 and v0.3.7 (as mentioned in the Wiki that Diablo II support was added in this version).

As of v0.3.7, DGEngine can run using Diablo 2 gamefiles.

The game launches fine, and does display graphics from Diablo II, however, I seem to get stuck at the character selection screen, where I cannot enter a character name or get further.

After launching the game with ./DGEngine gamefilesd2, no fonts are displayed in the main menu.

Also once you click on the character selection screen, I can't seem to get any further. I've tried to use the mouse, to click Enter and to enter random characters as to give a character name, but to no success thus far.

This is what my DGEngine directory looks like with the extracted Diablo 2 MPQs.

.editorconfig*
.git/
.gitattributes*
.gitignore*
android/
build/
BUILD.txt*
cmake_modules/
CMakeLists.txt*
d2char/
d2data/
d2exp/
d2music/
d2sfx/
d2speech/
d2video/
d2xmusic/
d2xtalk/
d2xvideo/
DGEngine*
DGEngine.sln*
DGEngine.vcxproj*
DIABDAT.MPQ
fantasycore/
fantasycore.zip
gamefilesd/
gamefilesd2/
gamefilesdex/
gamefilesflare/
gamefileshf/
LICENSE.GPL3.txt*
LICENSE.txt*
LICENSE.Zlib.txt*
main.json*
README.md*
src/

Note, this is what is reported to the terminal:

$ ./DGEngine gamefilesd2
Uniform "elapsedTime" not found in shader
Uniform "loading" not found in shader
Uniform "mousePosition" not found in shader
Uniform "textureSize" not found in shader
Uniform "pixelSize" not found in shader

Any idea what I may need to do to get this working?

Cheers, Robin

[ghost]

check if the full paths in gamefilesd2/res/fonts.json match the ones in d2data/data/local/font/LATIN. maybe it's the case sensitivity. when DGEngine fails to load something, you get something like in that screenshot.

[mewmew]

Awesome! Fixing the font worked.

The issue was that I had d2data/data/local/font/LATIN as d2data/data/local/font/latin.

What I used to extract the Diablo II MPQs was Zezula's MPQ editor, with the listfiles for MPQ archives at http://www.zezula.net/download/listfiles.zip

I tried to simply rename latin to LATIN and that made the fonts work in the main menu.

Now I can create a new characters and start to load a game before it crashes. It crashes during the loading screen, probably due to some other file not being properly loaded.

What tool are you using to extract the MPQ files? Just so I can use something with consistent results.

Cheers, Robin

Edit: progress thus far:

working fonts in main menu.

working fonts in character selection screen.

crash during loading.

u@x220 ~/D/d/DGEngine> ./DGEngine gamefilesd2
Uniform "elapsedTime" not found in shader
Uniform "loading" not found in shader
Uniform "mousePosition" not found in shader
Uniform "textureSize" not found in shader
Uniform "pixelSize" not found in shader
fish: “./DGEngine gamefilesd2” terminated by signal SIGSEGV (Address boundary error)

back trace at the crash:

(rr) bt
#0  0x000055730d78c91e in LevelCell::setTileIndex(unsigned long, int) ()
#1  0x000055730d78a366 in LevelMap::setD2Area(int, int, DS1::Decoder&)::{lambda(std::unordered_map<int, DS1::Cell, std::hash<int>, std::equal_to<int>, std::allocator<std::pair<int const, DS1::Cell> > >, int, int, std::set<int, std::less<int>, std::allocator<int> > const&)#1}::operator()(std::unordered_map<int, DS1::Cell, std::hash<int>, std::equal_to<int>, std::allocator<std::pair<int const, DS1::Cell> > >, int, int, std::set<int, std::less<int>, std::allocator<int> > const&) const ()
#2  0x000055730d78a6a7 in LevelMap::setD2Area(int, int, DS1::Decoder&) ()
#3  0x000055730d8f9bd3 in Parser::parseMapObj(rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> > const*, rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> > const&, LevelMap&, PairXY<int> const&, int, bool) ()
#4  0x000055730d8fa179 in Parser::parseMap(rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> > const*, rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> > const&, LevelMap&, PairXY<int> const&, int, bool, int) ()
#5  0x000055730d8fa3de in Parser::parseMap(rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> > const*, rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> > const&, LevelMap&, int) ()
#6  0x000055730d8fa97a in Parser::parseLevelMap(Game&, rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> > const*, rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> > const&, Level&) ()
#7  0x000055730d8fba23 in Parser::parseLevel(Game&, rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> > const&) ()
#8  0x000055730d8aead2 in Parser::parseDocumentElem(Game&, unsigned short, rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> > const&, ReplaceVars&, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>&) ()
#9  0x000055730d8ad880 in Parser::parseDocumentElemHelper(Game&, unsigned short, rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> > const&, ReplaceVars&, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>&) ()
#10 0x000055730d8ad70c in Parser::parseDocument(Game&, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>, rapidjson::CrtAllocator> const&, ReplaceVars) ()
#11 0x000055730d8ad5ca in Parser::parseJson(Game&, std::basic_string_view<char, std::char_traits<char> >) ()
#12 0x000055730d8ad173 in Parser::parseFile(Game&, rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> > const&) ()
#13 0x000055730d8aed5d in Parser::parseDocumentElem(Game&, unsigned short, rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> > const&, ReplaceVars&, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>&) ()
#14 0x000055730d8ad880 in Parser::parseDocumentElemHelper(Game&, unsigned short, rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> > const&, ReplaceVars&, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>&) ()
#15 0x000055730d8ad70c in Parser::parseDocument(Game&, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>, rapidjson::CrtAllocator> const&, ReplaceVars) ()
#16 0x000055730d8ad5ca in Parser::parseJson(Game&, std::basic_string_view<char, std::char_traits<char> >) ()
#17 0x000055730d8acb44 in Parser::parseFile(Game&, std::basic_string_view<char, std::char_traits<char> >) ()
#18 0x000055730d8aed3f in Parser::parseDocumentElem(Game&, unsigned short, rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> > const&, ReplaceVars&, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>&) ()

[ghost]

I use the same program (but windows is case insensitive). The listfiles contain this:

DATA\LOCAL\Font\Latin\Font16.dc6
DATA\LOCAL\Font\Latin\Font16.tbl
data\local\FONT\LATIN\font16.txt
DATA\LOCAL\Font\Latin\Font24.dc6
DATA\LOCAL\Font\Latin\Font24.tbl
data\local\FONT\LATIN\font24.txt
DATA\LOCAL\Font\Latin\Font30.dc6
DATA\LOCAL\Font\Latin\Font30.tbl
data\local\FONT\LATIN\font30.txt
DATA\LOCAL\Font\Latin\Font42.dc6
DATA\LOCAL\Font\Latin\Font42.tbl
data\local\FONT\LATIN\font42.txt
DATA\LOCAL\Font\Latin\Font6.dc6
DATA\LOCAL\Font\Latin\Font6.tbl
DATA\LOCAL\Font\Latin\Font8.dc6
DATA\LOCAL\Font\Latin\Font8.tbl
data\local\FONT\LATIN\FONTER.TBL

So some files are upper and others are lower. This is the problem, I think. you can try to build physfs using an external stormlib so you can load from the MPQ files directly and avoid these problems.

https://github.com/dgengin/physfs/blob/master/README2.txt

I haven't tried this recently, so I don't know if it works without any extra steps.

[mewmew]

I think. you can try to build physfs using an external stormlib so you can load from the MPQ files directly and avoid these problems.

I have physfs installed with MPQ support (using the AUR package).

[ghost]

You have to rebuild it with the external full StormLib with D2 MPQ support. I'm attaching the names I have. d2data.txt d2char.txt

[mewmew]

Thanks!

[mewmew]

I extracted d2data and d2char using your listfiles, Seem to get the same crash though.

(gdb) bt
#0  0x000055555567891e in LevelCell::setTileIndex(unsigned long, int) ()
#1  0x0000555555676366 in LevelMap::setD2Area(int, int, DS1::Decoder&)::{lambda(std::unordered_map<int, DS1::Cell, std::hash<int>, std::equal_to<int>, std::allocator<std::pair<int const, DS1::Cell> > >, int, int, std::set<int, std::less<int>, std::allocator<int> > const&)#1}::operator()(std::unordered_map<int, DS1::Cell, std::hash<int>, std::equal_to<int>, std::allocator<std::pair<int const, DS1::Cell> > >, int, int, std::set<int, std::less<int>, std::allocator<int> > const&) const ()
#2  0x00005555556766a7 in LevelMap::setD2Area(int, int, DS1::Decoder&) ()
#3  0x00005555557e5bd3 in Parser::parseMapObj(rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> > const*, rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> > const&, LevelMap&, PairXY<int> const&, int, bool) ()
#4  0x00005555557e6179 in Parser::parseMap(rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> > const*, rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> > const&, LevelMap&, PairXY<int> const&, int, bool, int) ()
#5  0x00005555557e63de in Parser::parseMap(rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> > const*, rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> > const&, LevelMap&, int) ()
#6  0x00005555557e697a in Parser::parseLevelMap(Game&, rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> > const*, rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> > const&, Level&) ()
#7  0x00005555557e7a23 in Parser::parseLevel(Game&, rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> > const&) ()

[ghost]

Try and delete the save and create a new one.

[mewmew]

Great! Removing the save file worked.

Now in game works:

[ghost]

That screenshot is not the latest version. you should see the D2's first map.

[mewmew]

Hmm, seems like I'm back at crashing.

Pulled to the latest version.

u@x220 ~/D/d/DGEngine> ./build/DGEngine gamefilesd2
Uniform "elapsedTime" not found in shader
Uniform "loading" not found in shader
Uniform "mousePosition" not found in shader
Uniform "textureSize" not found in shader
Uniform "pixelSize" not found in shader
fish: “./build/DGEngine gamefilesd2” terminated by signal SIGSEGV (Address boundary error)
u@x220 ~/D/d/DGEngine> git branch
  issue-21
* master
u@x220 ~/D/d/DGEngine> git pull
Already up to date.

Note, same crash as before:

#0  0x000055555567891e in LevelCell::setTileIndex(unsigned long, int) ()
#1  0x0000555555676366 in LevelMap::setD2Area(int, int, DS1::Decoder&)::{lambda(std::unordered_map<int, DS1::Cell, std::hash<int>, std::equal_to<int>, std::allocator<std::pair<int const, DS1::Cell> > >, int, int, std::set<int, std::less<int>, std::allocator<int> > const&)#1}::operator()(std::unordered_map<int, DS1::Cell, std::hash<int>, std::equal_to<int>, std::allocator<std::pair<int const, DS1::Cell> > >, int, int, std::set<int, std::less<int>, std::allocator<int> > const&) const ()
#2  0x00005555556766a7 in LevelMap::setD2Area(int, int, DS1::Decoder&) ()
#3  0x00005555557e5bd3 in Parser::parseMapObj(rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> > const*, rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> > const&, LevelMap&, PairXY<int> const&, int, bool) ()

[mewmew]

I tried again, and I keep hitting the crash now whenever I try to log into the game.

For reference, this is the extracted MPQ files I have on my computer.

https://srv-file5.gofile.io/download/33iwQg/mpq_files.txt

This is on rev 6c499d91c7d91ce5f1715ccf31212acf8748e363, running with ./build/DGEngine gamefilesd2.

[mewmew]

@dgengin On a related note, how much work would it be to have DGEngine read files in an in-case sensitive manner?

Seeing as this seems to be an issue that keeps re-appearing, it may be worth to solve it once and for all.

[ghost]

I'll have to see what it is. It might be the map being loaded for town that doesn't have the same case.

As for fixing this, it can't be done. The paths and names are hashed to get the files in the MPQ file, so you have to use the correct path, otherwise the file won't be found in the MPQ file. on windows, it's not a problem to use an extracted archive because it's case insensitive.

[mewmew]

As for fixing this, it can't be done. The paths and names are hashed to get the files in the MPQ file, so you have to use the correct path, otherwise the file won't be found in the MPQ file. on windows, it's not a problem to use an extracted archive because it's case insensitive.

I know that to extract these files you need to use backslashes instead of slashes since that is part of the hash info for the MPQ files. However, the hash algorithm always uppercases the names before hashing them, so it does not consider casing.

What I was referring to was that if you have extracted all files (but they have the wrong casing once extracted) that DGEngine would be able to locate say Font16.dc6 using both the file path DATA\LOCAL\Font\Latin\Font16.dc6 and data\local\Font\Latin\Font16.dc6 or something like that.

I know that checking all possible casings would be 2^n where n is the length of the path. However, perhaps it would be possible to check both the expected casing e.g. DATA\LOCAL\Font\Latin\Font16.dc6 and the lowercase version of the same casing e.g. data\local\font\latin\font16.dc6 on the file system?

[mewmew]

For reference, this is how OpenDiablo2 is intending to handle case-insensitive matching of file names. From https://github.com/OpenDiablo2/OpenDiablo2/issues/71

Problem Different versions use different capitalization of the mpq files. Because linux is case-sensitive, this makes it so that most linux players will have to manually change all of the mpq names to match the case in their specific release.

Solution We should have logic that first checks for the file, but if the file is not found, it should do a directory listing of the mpq path, and do a case-insensitive match to find the actual proper casing for that file for this version, and use that instead.

[predator8bit]

Isn't it possible to extract the mpq archive into a directory and convert everything to lowercase during the process?

Kinda like how innoextract has a lowercase flag.

[ghost]

The last commit should try and load all lowercase path if it fails to load at first.

rename to lowercase all d2* folders recursively:

find d2* -depth -exec rename 's/(.*)\/([^\/]*)/$1\/\L$2/' {} \;

[mewmew]

Thanks! Really happy to see lowercase support geing added.

I synced with the latest version of DGEngine, however, now I get a different crash:

u@x220 ~/D/d/DGEngine> ./build/DGEngine gamefilesd2
fish: “./build/DGEngine gamefilesd2” terminated by signal SIGSEGV (Address boundary error)

This is on rev dgengin/physfs@9543c6d3b299f4423d385508db2355c86cbd138c and v0.4.8 of DGEngine.

u@x220 ~/D/d/DGEngine> gdb --args ./build/DGEngine gamefilesd2
GNU gdb (GDB) 8.3.1
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./build/DGEngine...
(No debugging symbols found in ./build/DGEngine)
(gdb) run
Starting program: /home/u/Desktop/diablo/DGEngine/build/DGEngine gamefilesd2
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
[New Thread 0x7fffec43e700 (LWP 25162)]

Thread 1 "DGEngine" received signal SIGSEGV, Segmentation fault.
0x00007ffff627afe3 in PHYSFS_fileLength () from /usr/lib/libphysfs.so.1
(gdb) bt
#0  0x00007ffff627afe3 in PHYSFS_fileLength () from /usr/lib/libphysfs.so.1
#1  0x000055555561e3e0 in sf::PhysFSStream::getSize() ()
#2  0x00005555555b699f in FileUtils::readText[abi:cxx11](char const*) ()
#3  0x00005555557c15cd in Parser::parseGame(Game&, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >) ()
#4  0x00005555555b9245 in Game::load(std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >) ()
#5  0x000055555558826a in main ()

[ghost]

Yes, I got that one too but ignored it thinking it was just me.

This last one works.

[mewmew]

Thanks you so much!!

Now it's working :)

Really impressive how far DGEngine has progressed!