Closed FriskTheFallenHuman closed 1 year ago
Merging #113 into this issue:
Is your feature request related to a problem? Please describe. Slightly; Memory usage runs dangerously close to the 32-bit limit, especially on macOS due to texture memory allocation issues.
Describe the solution you'd like Overhaul 32-bit specific code so that it does not rely on a specific memory size, and add 64-bit code with #ifdefs in areas that are not possible to generalize.
Describe alternatives you've considered Fix the memory fragmentation issues on the platforms on which this is an issue.
Additional context Very High texture quality is disabled on macOS for this reason. ValveSoftware/Source-1-Games#2238 contains a lot of pertinent information. Additionally, both CS:GO and Garry's Mod have been ported to 64-bits.
this allows the engine use more resources of the current machine
This is a common misconception. The only thing that a 64-bit client would enable is going past the 4GB RAM limit. It would not improve performance, or utilize your hardware more efficiently. Considering TF2 rarely approaches even half of that limit, it's probably not worth the considerable amount of effort that would be involved. Resources are best spent elsewhere.
Replacing RFC with "help wanted" because we REALLY do want x64 builds, but this isn't something me or coms want to work on because it's a giant pain
I've been doing some experimenting with compiling for x64. Not preserving functionality like use of intrinsic functions yet, just a simple full build. To that end I've gotten most of the way through the process, but I've hit a bit of a roadblock in that tf2 seems to use a custom version of the steamworks sdk, which we need x64 binaries for.
What seems custom about it? Are you sure it's not just using an older version?
Looking through the files included in public/steam, it's got a mixture of header files that can't be found in any single version of the public SDKs. An example of this is: "isteamcontroller.h" was introduced in V125 of the public steam API, but the file "isteamgamestats.h" was removed in V118, so there's no public version where these two files are in the same SDK version.
Edit: I've also run a search through more modern versions of the API, and there's no direct port of the old functionality lost from "isteamgamestats.h" - There may be a way to reproduce the same functionality in there somewhere, but simple fixes, like the functionality just being moved into different files doesn't seem to be a thing.
Edit 2: Quick update, I've found a version of steam_api64 that works for most of what we need for steam, but we're still missing x64 binaries for steamnetworkingsockets. I'm still investigating.
New update:
I've managed to get the dedicated server to compile, but so far I've been unable to build an x64 version of vphysics.dll.
The client needs an x64 version of steamnetworkingsockets in order to compile (Or it requires a rewrite to remove reliance on that dll).
I don't think removing the reliance on steamnetworkingsockets is possible, as technically even opening a map via the console starts up a server.
@treacherousfiend I was thinking on this, and wouldn't the same functionality be available via the more modern steam API versions? Ever since they made steam relays publicly usable, wouldn't this essentially reproduce what steamnetworkingsockets used to do?
Trying to update to a modern version of the steam API would introduce other problems (namely, game stats) but it's a thought!
I wouldn't know that, i'm mostly just going off of some basic source engine knowledge, so removing anything network related from tf2 seems like a bad idea. I want to say that the dedicated server is even just a headless TF2 exe
You'd think so, but thus far, the dedicated server hasn't required any linking to steamnetworkingsockets. I'm not saying it won't somewhere down the chain, but it's not linking it in the same way the client does at least.
It'll be something to look into, might end up being an optimization of its own depending on how its implemented.
Wanted to a add a new update on progress here: I've managed to compile a broken 64 bit dedicated server. We load all necessary dlls for the server to run, but we can't properly load assets.
Current obstacles:
VPhysics builds are completely broken. I only got it to compile by removing offending code causing errors. This will almost definitely just break things, but getting VPhysics builds to work is another task in itself.
DmxLoader is broken, as asset serialization & deserialization relies on the sizeof certain objects, which contain pointers, so the size of those types are no longer correct. I hacked past it by only saving 32 bits of pointers allocated for these objects so we could get compiling, but obviously this will just straight up break, so we need a real solution for this problem.
There are several other places where compile time assert sizeofs are failing, but I've not yet investigated the consequences of these asserts yet, but the assumption is these are also breaking.
The Client still needs an x64 solution for steamnetworkingsockets.lib/dll
Need to write a whole TON of vpc scripts for x64.
Update: Been out for most of the week, so haven't been able to work on this too much. However it seems as though we are now successfully loading assets on the dedicated server. Now we're crashing due to some memory allocations in the console. So slowly making progress!
The Client still needs an x64 solution for steamnetworkingsockets.lib/dll
CS:GO should have a 64-bit copy of steamnetworkingsockets, but I haven't checked.
CS:GO is 32 bit as well. Even if it wasn't it would need to include .lib files, which it doesn't. =(
I'm pretty sure CSGO is only 32 bit on Windows.
https://www.phoronix.com/scan.php?page=news_item&px=CSGO-64bit-Linux
Does this matter? It's from the latest update.
Does this matter? It's from the latest update.
Yeah, this makes TF2 only offline :(
No, it doesn't. They just updated to the new Steamworks API for Steam datagrams, which has been included in the main API. We can't really use new APIs unless we update all of our Steam usage, which is a huge feat.
Unfortunately, it's increasingly looking like it may be necessary to update the steam API used.
I've gotten the dedicated server to the point where we're trying to create a server using the steam libraries. Right now it crashes somewhere inside the current version of steam_client64.dll.
I'll continue working on it, see if I can spot anything.
@Zionner for testing, you might want to use Goldberg emulator instead of legit steamworks API for testing. On linux we used it when steamworks API wasn't working, and it did contain all the features TF2 needed (afaik).
Another Update for you guys: I got past the steamworks problem, by disabling inventory support. I'll look into using Goldberg as @melvyn2 suggested, when I need to use that API.
I'm at the point where I'm just slowly fixing up 64-bit to 32-bit data loss, as they come up. We have started loading entities on the dedicated server though, so I'm hoping that we will have a first test of how broken this stuff is soon. :P
It's been a while since I updated this, but here's where I'm at:
There's some nasty corruption going on somewhere in the studio model data, that's holding up progress. Due to various reasons I've not been able to spend quite as much time on this the past few weeks, but I am still trying to figure out the problem, and will update here when I make progress.
It's been quite a while since I've update this, so here's a new one:
I've been in/out of the hospital for the past month, and this will probably continue into January. I still hope I can find some time to continue on this, but not much progress is being made at the moment.
@Zionner Hope you are in a good health and mood!
Feel free to publish your x86-64 branch here. As a one who tried to port Source 2007 on x86-64 (and even get main menu and gui running successfully :)) can help you a bit.
Main problems to solve:
size_t
and stored / restored from disk, so on x86-64 we should use unsigned int
to be portable. Also some APIs are x86 only - ex. consuming / returning pointers as ints / unsigned ints. If we rewrite to x86-64, than we break ABI. So special care should be done to allow playing with x86-64 clients on x86 servers and vice versa.size_t
and ptrdiff_t
as memory sizes, indices and so on.uintptr_t
or intptr_t
types. In other case we got truncations on x86-64.GetWindowLong/SetWindowLong
are deprecated and should be replaced with x86-64 counterparts.intptr_t
or uintptr_t
instead of int
for pointers to action handlers, as it stores address of handler.Unfortunately, if vphysics issue will not be solved, there is no sense in porting to x86-64.
missing required ivp sdk, hence, no way for vphysics on x86-64. Need to reverse engineer existing steam version for Linux (check hypothesis it has debug symbols inside), or implement missing functionality. Looks like included ivp sources are from Source 2007 (outdated, but better than nothing).
https://github.com/HurricanePootis/team-comtress-2/commits/vphysics-rev
To chime in about the reversed vphysics- it's partially complete. Most of it works but ragdoll physics is the only thing that doesn't iirc.
To chime in about the reversed vphysics- it's partially complete. Most of it works but ragdoll physics is the only thing that doesn't iirc.
May be it makes sense in finish reverse engineering?
Personally i'm not strong in assembler, and looks like you are using some high level Hydra tool. But still, if help is needed, ready to jump in.
For anybody who interested in porting to x86-64:
To chime in about the reversed vphysics- it's partially complete. Most of it works but ragdoll physics is the only thing that doesn't iirc.
May be it makes sense in finish reverse engineering?
Personally i'm not strong in assembler, and looks like you are using some high level Hydra tool. But still, if help is needed, ready to jump in.
The one doing almost all of the reversing is crackbomber - I'm not doing any reversing (just working out the bugs for the linux port).
Small tip: add /windows
to vpc flags for project generation, and generated Visual Studio solution will have both Win32 and x64 platforms. See https://github.com/mastercomfig/team-comtress-2/blob/master/external/vpc/utils/vpc/main.cpp#L644
Also interesting to note, that vpc's $WIN32
platform doesn't work for x86-64 build on Windows. You should explicitly mark vpc item as $WIN64
, like $WIN32 || $WIN64
or just $WIN64
. It leads to a huge amount of changes in vpc scripts to support x86-64 build, because almost all $WIN32
code is also $WIN64
.
For example, let's see how Microsoft defines _WIN32
in x64 builds: _WIN32 Defined as 1 when the compilation target is 32-bit ARM, 64-bit ARM, x86, or x64. Otherwise, undefined.
(https://docs.microsoft.com/en-us/cpp/preprocessor/predefined-macros). So made a hypothesis that it is easier to fix vpc project generation for x86-64 by assuming $WIN32
is also enabled when $WIN64
platform defined, instead of manually replacing almost all $WIN32
(~116 cases) with $WIN32 || $WIN64
.
Summary: VPC platform | Was before fix | Become after fix |
---|---|---|
$WIN32 |
x86 only | both x86 and x86-64 |
$WIN64 |
x86-64 only | x86-64 only |
$WIN32 or $WIN64 |
both x86 and x86-64 | both x86 and x86-64 |
$WIN32 and !$WIN64 |
x86 only | x86 only |
In order to be compatible with original vpc behavior we can do this under special command line flag, smth
like /definewin32forwin64
.
Pros: less changes in original vpc scripts, easier to reason about for regular windows dev (as Windows SDK headers do). Cons: automated solution can miss some edge cases, changes in vpc can lead to regressions.
What do you think, should we change VPC, or just replace $WIN32
(~116 cases) with $WIN32 || $WIN64
where needed?
The $WINDOWS
VPC macro seems to be defined for both arches.
The
$WINDOWS
VPC macro seems to be defined for both arches.
Thanks! Checked, it does.
Replacing $WIN32
(~116 cases) with $WINDOWS
looks like an easier & safer task than changing VPC, even if i personally prefer the latest one :).
Now it is possible to have both old x86 and new x86-64 bins as as single pack. Player can just select which one he will run.
Since there's been some movement here, I thought I'd update you all on my end:
I've been going through chemo for the past month, and will be for the next few months. So I've not had much time, or energy to work on this.
I'll do my best to at at least get a branch up of my changes. A couple of the changes are iffy (Specifically, the changes around studiomtl serialization) But 99% of the changes have been fixing pointer truncations, replaced some (not all) of the ASM & SIMD stuff, and fixing x64 compiler errors. Last I worked on it, x64 was compiling (Server side only), you could start a server a begin loading a map. The map loading would crash trying to deal with animations.
It's worth noting that the x64 changes are on a pretty old version of comtress (See my fork for the last time it was update).
Since there's been some movement here, I thought I'd update you all on my end:
I've been going through chemo for the past month, and will be for the next few months. So I've not had much time, or energy to work on this.
I'll do my best to at at least get a branch up of my changes. A couple of the changes are iffy (Specifically, the changes around studiomtl serialization) But 99% of the changes have been fixing pointer truncations, replaced some (not all) of the ASM & SIMD stuff, and fixing x64 compiler errors. Last I worked on it, x64 was compiling (Server side only), you could start a server a begin loading a map. The map loading would crash trying to deal with animations.
It's worth noting that the x64 changes are on a pretty old version of comtress (See my fork for the last time it was update).
Thank you for the update!
What do you think about me cherry-picking your commits into the https://github.com/mastercomfig/team-comtress-2/pull/554 (with all the credits, of course)? A lot of work has been done by you, so it makes sense to not loose any bit of it.
P.S. Wish you to fight against illness and win. Waiting for your final PR approval in a next few months :)
I've committed the changes I managed to get through at: https://github.com/Zionner/team-comtress-2/tree/x64
I've committed the changes I managed to get through at: https://github.com/Zionner/team-comtress-2/tree/x64
Nice work. Like how commits are atomic and easy to follow and cherry-pick.
Progress update:
Managed to compile most of the bins in x86-64 mode on Windows. Thanks to @Zionner for huge job of porting the things on the server part.
Now updates for client/server:
socket
, steam_api
and steamnetworkingsockets
libs with back x64 binaries to build client, server, engine, inputsystem, etc. Will merge with @Zionner fixes, so some dependencies will be satisfied. Resolvable?libpng
to build in x64. Will use thirdparty/libpng-1.5.2
as a reference. Resolvable.Required to find lib/bin for SDL 2.0.2. Will use
thirdparty/SDL2as a reference.
Found one, now use precompiled. Resolvable.Required to find lib/bin for open_vr <= 0.9.1.
Found one, now use precompiled. Resolvable.Required to find lib/bin for bink (binkw64).
Found one, now use precompiled. Resolvable. Need to check, likely engine doesn't depend on it at all, so just remove.Missed x64 dependencies:
libpng
is in the source tree, check the vphysics-rev
branch on how to compile it (check build.sh
).
Good news on vphysics - @CRACKbomber has said that he's continuing the reversing.
Is your feature request related to a problem? Please describe. One thing that would be good for team comtress 2 is x64 builds, this allows the engine use more resources of the current machine and it could improve the performance and stability.
Describe the solution you'd like The leak's groups.vgc has a group for x64 bits, another mention of it is on source_dll_win32_base.vpc
Describe alternatives you've considered nope
Additional context iirc this is only used for mac os builds.