64-bit support

[Hello71]

Hopefully this is not actually too hard on the DT side. In theory, the only changes needed are to change VIRTADDR to be typedefed to void * and the memory layout parsers to be changed. However, there are probably a lot of places where VIRTADDRs are stuffed into ints which will end badly.

One question is though whether we should support connecting to 32-bit DF from 64-bit DT and vice versa. I'm inclined to say "no, that's stupid, why would you use 32-bit DF on 64-bit OS", but maybe there are arguments to the contrary.

[splintermind]

I've started work on this in the x64 branch. I'm not sure what adjustments need to be made in the pro for the qmake stuff yet though. Testing has been limited without offsets yet, so there will be more work for sure.

There's still going to be the need for a 32-bit DT while DF supports 32-bit so keeping two builds is probably easiest... although there may need to be some tweaking for the updater and/or layout directory structure to split 32 vs 64 layouts.

[Hello71]

I've started work on this in the x64 branch. I'm not sure what adjustments need to be made in the pro for the qmake stuff yet though. Testing has been limited without offsets yet, so there will be more work for sure.

I'm strongly in favor of moving to CMake entirely, because qmake sucks (more than CMake anyways). I started a local branch for it but got distracted by proper OS X support.

There's still going to be the need for a 32-bit DT while DF supports 32-bit so keeping two builds is probably easiest... although there may need to be some tweaking for the updater and/or layout directory structure to split 32 vs 64 layouts.

right, I'm saying that there are two ways to handle 32-bit support:

1. compile DT in 64-bit mode and select uint32_t or void * depending on the bitness of the DF
2. compile DT in 32-bit mode and always use void *

I think option 2 is easier.

[lethosor]

Related to option 1, would a union like this work?

union df_ptr { 
  uint32_t ptr32;
  uint64_t ptr64;
}

Alternatively, maybe using uint64_ts for all pointers and shoving 32-bit pointers into the first 4 bytes would be possible, although compilers tend to complain about uint64_t-to-uint32_t casts. I'm not really familiar with how DT handles this stuff already, so apologies if any of that would be too hard to implement.

[nifisher]

No, that would result in a union sized incorrectly for 32 bit.

edit: Hello71 is correct, this statement makes no sense.

[Hello71]

No, that would result in a union sized incorrectly for 32 bit.

wat

[nifisher]

In hindsight I'm not sure what I meant by that statement. The union would be 8bytes long regardless of architecture, but other than wasting space on 32bit, accessing the 32bit pointer field would work correctly. I'm guessing I had a brain fart and was thinking about accessing the union without specifying which field. Regardless, it still won't work because you need to specify the field.

{
unint32 foo[128];
df_ptr bar = foo;

/*code here*/
}

How do you put the address of "foo" into "bar" in an architecture independent manner using that union? Even if you set the pointer (say by always writing to the 64bit field), how do you access it without knowing what architecture you are building for?

[Hello71]

you do it... based on the bitness of the connected process? we are already doing highly arch-specific hacks (including but not limited to injecting mmap via ptrace which requires knowing arch syscall convention), so one more is not such a big difference

[nifisher]

you are correct, but that means you are back to where you were before and the union itself doesn't add anything. In fact, just using a plain pointer that isn't of a fixed width (unless needed) would likely be way less of a headache across the board anyhow.

though I am wary to fully commit to that statement. It would make my day job obscenely easier, but I can't remember what edge cases that would cause pain with.

[Hello71]

sigh, that's option 2 of my 2 options above.

[Hello71]

Alternatively, maybe using uint64_ts for all pointers and shoving 32-bit pointers into the first 4 bytes would be possible, although compilers tend to complain about uint64_t-to-uint32_t casts.

narrowing conversions are legal but you would need to be sure in the case of pointers to do (uint32_t)*ptr instead of *(uint32_t*)ptr which is undefined behavior. a union is the way to go in the case of option 1.

[nifisher]

Hello71, yes, but my response was initially in reply to lethosor, so I had neglected to read back further up during our conversation

[Hello71]

I think I might roll a patch combining all the changes I want to do some time in the next few days, supporting Linux only.

[ubergarm]

@Hello71, thanks for working on this. I just pulled your forks latest changes and have been testing tonight.

I got it working after starting a new fortress and was able to correctly set some labors, but DT starts crashing after getting my first migration wave.

Here is output of GDB:

GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.04) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copyi
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from DwarfTherapist...(no debugging symbols found)...do
(gdb) run
Starting program: /df/DwarfTherapist
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.
[New Thread 0x7fffebcc5700 (LWP 431)]
2016-Nov-27 04:36:30.559 INFO   core    Dwarf Therapist "37.0.0" starti
2016-Nov-27 04:36:30.559 INFO   core    Runtime QT Version 5.5.1 [/buil
process 427: D-Bus library appears to be incorrectly set up; failed to
 not length 0, with no other text
See the manual page for dbus-uuidgen to correct this issue.
[New Thread 0x7fffea3e2700 (LWP 433)]
2016-Nov-27 04:36:30.916 INFO   core    beginning to read settings [/bu
2016-Nov-27 04:36:30.925 INFO   core    finished reading settings [/bui
[New Thread 0x7fffe1417700 (LWP 434)]
2016-Nov-27 04:36:37.562 INFO   core    attempting connection to running DF game [/build/src/mainwindow.cpp:403] (connect_to_df)
2016-Nov-27 04:36:37.568 INFO   core    adding valid layout "v0.42.06 linux" checksum: "0x6ffd260f" SHA: "10203b4ffa91ab6cc1e479dd6b809ef214cbb0fe" [/build/src/dfinstance.cpp:104] (DFInstance)
2016-Nov-27 04:36:37.571 INFO   core    adding valid layout "v0.42.05"
checksum: "0xde7a048b" SHA: "9e0f2039f008546517997c08c41f318f98a79d7b"
[/build/src/dfinstance.cpp:104] (DFInstance)
2016-Nov-27 04:36:37.573 INFO   core    adding valid layout "v0.42.04"
checksum: "0xddd87cd8" SHA: "cbce922a4dae18a4e59e5d904da96d01a5be7b2e"
[/build/src/dfinstance.cpp:104] (DFInstance)
2016-Nov-27 04:36:37.576 INFO   core    adding valid layout "v0.42.03"
checksum: "0x1313e22c" SHA: "d81b90aafff9af84b4d6723a600f61099ea32da6"
[/build/src/dfinstance.cpp:104] (DFInstance)
2016-Nov-27 04:36:37.578 INFO   core    adding valid layout "v0.42.02"
checksum: "0x79b0a9a1" SHA: "ec6a3c43d97c096f997bdd22e39921b065326ad9"
[/build/src/dfinstance.cpp:104] (DFInstance)
2016-Nov-27 04:36:37.580 INFO   core    adding valid layout "v0.42.01"
checksum: "0x5feb615a" SHA: "00f2d47192b17c96b0787205b7b241ae15184233"
[/build/src/dfinstance.cpp:104] (DFInstance)
2016-Nov-27 04:36:37.583 INFO   core    adding valid layout "v0.43.03"
checksum: "0xdf4ffde3" SHA: "a79c9a734d3af0631e91a2c1aad5285c65ab200f"
[/build/src/dfinstance.cpp:104] (DFInstance)
2016-Nov-27 04:36:37.585 INFO   core    adding valid layout "v0.43.02"
checksum: "0x3c048bd0" SHA: "efe236ee1921468def7d24f1018afcd912ca5042"
[/build/src/dfinstance.cpp:104] (DFInstance)
2016-Nov-27 04:36:37.588 INFO   core    adding valid layout "v0.43.05 linux64" checksum: "0xaf8cc86c" SHA: "97dc4619028e83d69d60a5d1f13eb70dca2fd800" [/build/src/dfinstance.cpp:104] (DFInstance)
2016-Nov-27 04:36:37.589 INFO   core    Dwarf fortress path: "/df/df_linux" [/build/src/dfinstancelinux.cpp:291] (find_running_copy)
2016-Nov-27 04:36:37.689 INFO   core    Setting memory layout for DF checksum "0xaf8cc86c" [/build/src/dfinstance.cpp:1043] (set_memory_layout)
2016-Nov-27 04:36:37.690 INFO   core    Detected Dwarf Fortress version "v0.43.05 linux64" using MemoryLayout from "/df/share/memory_layouts/linux/0.43.05-linux64-therapist.ini" [/build/src/dfinstance.cpp:1048] (set_memory_layout)
2016-Nov-27 04:36:37.691 INFO   core    Connection to DF version "v0.43.05 linux64" established. [/build/src/mainwindow.cpp:427] (connect_to_df)
2016-Nov-27 04:36:38.654 INFO   core    Loaded 12 views from disk [/build/src/viewmanager.cpp:174] (reload_views)
2016-Nov-27 04:36:38.659 INFO   core    group_by now set to 0  for view  "Labors Full" [/build/src/dwarfmodel.cpp:780] (set_group_by)
2016-Nov-27 04:36:38.660 INFO   core    "redrew views in 4ms" [/build/src/viewmanager.cpp:321] (draw_views)
2016-Nov-27 04:36:41.955 INFO   core    current year: 153 [/build/src/dfinstance.cpp:746] (refresh_data)
Thread 1 "DwarfTherapist" received signal SIGSEGV, Segmentation fault.
__memset_sse2 () at ../sysdeps/x86_64/multiarch/../memset.S:117
117     ../sysdeps/x86_64/multiarch/../memset.S: No such file or directory.
(gdb) up
#1  0x00000000007c263b in DFInstanceLinux::read_raw(unsigned long long, unsigned long long, void*) ()
(gdb) up
#2  0x000000000058e3c3 in QVector<unsigned long long> DFInstance::enum_vec<unsigned long long>(unsigned long long) ()
(gdb) up
#3  0x00000000005708c2 in DFInstance::enumerate_vector(unsigned long long) ()
(gdb) up
#4  0x00000000006255be in FortressEntity::read_entity() ()

My build environment is described in my fork's Dockerfile. I've also added the memfile.

You can recreate using my docker newb pack

Let me know where and how best to give you feedback. I might be able to submit patches if you're interested (and i get up to speed on the specific challenges here)...

Thanks!

[Hello71]

What do you mean, "added the memfile"? If you mean the save, where? Otherwise, compile with cmake -DCMAKE_BUILD_TYPE=Debug, and do gdb, bt full.

[ubergarm]

By memfile I meant the 64bit Dwarf Therapist Memory Map you suggested in the bay12forums. Not saved games.

So I rebuilt your latest commit 680274b with debugging flags enabled and seems to be better. I can at least set some labors. However if I wait a while and read again it eventually crashes. backtrace debugging here

I'll get these new dwarves busy filling the quantum stockpiles and keep you posted! Thanks!

PS I updated ubergarm/dwarf-fortress docker image with DwarfFortress build 680274b if any other intrepid folks want to give it a try.

[telnoratti]

I was feeling intrepid, so I tried the docker image @ubergarm provided. It worked okay until a dwarf died. After this it kept crashing each time it opened. backtrace and crashlog.

I was able to mitigate the problem by saving and continue playing, but when I opened up the job manager DT crashed and couldn't be opened again without crashing.

EDIT: added links to backtrace and log

[ubergarm]

@telnoratti I appreciate your intrepidness! I just built a new container from the the latest patches to see if it helps anything out.

I'd love to get this going before Toady One does the next release, lol.

Just pushed a new ubergarm/dwarf-fortress docker image. Sorry no automated builds, haven't figured out best way to build/vendor everything in one place.

This update has DwarfTherapist binary built from 49b792b7

Just tested that it starts up and connects. Hopefully it keeps working with migrations and dead dwarves!

Cheers have a good weekend!

[pastellexists]

@ubergarm how do I get the 64bit patch without Ironhand? Installation instructions at all?

[ubergarm]

You may be better off with the latest Linux lazy newb pack which caught up to df 0.43.05 recently I believe:

http://www.bay12forums.com/smf/index.php?topic=163211.0

Otherwise, you can try building Hello71's dwarf therapist fork yourself like I did here:

https://github.com/ubergarm/Dwarf-Therapist/blob/DF2016/Dockerfile On Aug 4, 2017 11:31 AM, "SimplyStars" notifications@github.com wrote:

@ubergarm https://github.com/ubergarm how do I get the 64bit patch without Ironhand? Installation instructions at all?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/splintermind/Dwarf-Therapist/issues/313#issuecomment-320279450, or mute the thread https://github.com/notifications/unsubscribe-auth/AEgWYT5z1Q5wjHFxjjSZA_8fGnQC4j4xks5sUzlcgaJpZM4JM1fa .

[woj1993]

@splintermind We have 64 bit memory layout of DF 0.44.02. Meybe we should compile ver 38 of DT with support of 64 bit and relase it as beta. What do you thing?

[bendlas]

Hi, I've recently updated the DT derivation in nixos, in order to be able to use it with 43.05. Similar to the LNP, I needed to use @Hello71's branch, to get a working version. I have no stake in the cmake vs qmake question, as the according change in the recipe was rather easy (https://github.com/NixOS/nixpkgs/commit/d6ee81e912c5dbfb7db23387c31996b258d0193f#diff-0ca3a9429c3b2c3144eefa9018953779R20), but either way it would be awesome to get the fixes for https://github.com/splintermind/Dwarf-Therapist/issues/311 and https://github.com/splintermind/Dwarf-Therapist/pull/323 into the main branch, to free people to get 44.02 support going ;-)