Open GrimSqueaker opened 3 years ago
After pull your code my Visual Studio 2019 generates this erros, I will be find problems, Tom.
MEMSET: identificator not found
Problem is in 5fb70fe0 Fix: Fix Cmake after splitting files
When I remove engine/defs.h from sub_main.h it is ok.(defs.h is already included by Basic.h), Tom.
I actualize Visual Studio settings to run 64bit version, game drop after begin - in sound initialization:
Sorry, I noticed the MS build failing yesterday but it was too late in the evening for me to fix it.
I had problems yesterday merging my 64bit branch, because it had conflicts with the Bit -> int_t changes as well as the introduction of separate files. Now I am back with 212 compilation errors in GCC, which I will also comment out via the preprocessor, before I will start to fix my way into the main menu. This should be possible and not too hard. I had this already working in my previous copy of the repo. What comes after this, I don't know. Your array->structs changes probably help a lot to get further.
In addition to rewriting fields into structures, we will also have to write structure converters that contain pointers, they must be converted when loading or saving data (usually saves). I would put them in the special FilesConvert.cpp / .h files. The simplest will be to have a copy of the data structure, where they will be instead of int32_t pointers. Raw data would be loaded into them, and then copied to a pointer structure. (Recorded pointers are most often zero)
typedef { uint8_t a uint8_t* b } structureX
typedef { uint8_t a uint32_t b } shadowX
LoadWithConvertX: LoadFromFile(shadow,file) structureX.a=shadowX.a structureX.b=shadowX.b
or
SaveWithConvertX: shadowX.a=structureX.a shadowX.b=structureX.b SaveToFile(shadow,file)
Tom.
Yes, just by compiling as 64 bit the structure that is saved as in-game save (type_D41A0_BYTESTR_0) increases in size from 224791 to 245115. Some of that might be fixed by using int32_t instead of int, etc. But for other stuff we will need converters.
I am not yet as proficient as you with all the existing structures. In order to prevent me from breaking essential stuff, I have implemented for this particular structure a unit test that is now also running as part of the GitHub Linux CI action. If you have more structures that should never change in size or other constraints that we definitely have to keep, please feel free to enhance the unit tests.
If I find time, I could also try to make the CMake setup - including unit tests - run on Windows. That would be a big improvement if we could use the same build system. But it would change the way you and @thobbsinteractive are working. It would mean that the VS solution is not checked in to the repository anymore but instead before you can work on the code you would have to generate it via CMake. I understand that this complicates some things but simplifies others at the same time. For pure Windows developers this is a seemingly unnecessary step, but for example it would get rid of the multiple checked in build directories (CMake would prepare all necessary files when generating the solution), it would allow to generate for XCode on MacOS in the future, and much more.
I think you can actually use the cmake in visual studio. Least I remember reading it somewhere.
In other news I was going to work this weekend on in game resolution, but I was wondering, should we also be replacing the normal int with int32_t etc...? My guess is yes?
On Sat, 13 Feb 2021, 00:46 GrimSqueaker, notifications@github.com wrote:
Yes, just by compiling as 64 bit the structure that is saved as in-game save (type_D41A0_BYTESTR_0) increases in size from 224791 to 245115. Some of that might be fixed by using int32_t instead of int, etc. But for other stuff we will need converters.
I am not yet as proficient as you with all the existing structures. In order to prevent me from breaking essential stuff, I have implemented for this particular structure a unit test that is now also running as part of the GitHub Linux CI action. If you have more structures that should never change in size or other constraints that we definitely have to keep, please feel free to enhance the unit tests.
If I find time, I could also try to make the CMake setup - including unit tests - run on Windows. That would be a big improvement if we could use the same build system. But it would change the way you and @thobbsinteractive https://github.com/thobbsinteractive are working. It would mean that the VS solution is not checked in to the repository anymore but instead before you can work on the code you would have to generate it via CMake. I understand that this complicates some things but simplifies others at the same time. For pure Windows developers this is a seemingly unnecessary step, but for example it would get rid of the multiple checked in build directories (CMake would prepare all necessary files when generating the solution), it would allow to generate for XCode on MacOS in the future, and much more.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/GrimSqueaker/remc2/issues/36#issuecomment-778516393, or unsubscribe https://github.com/notifications/unsubscribe-auth/AETQQFG34Q56ZTRG4H3KWRTS6W4WJANCNFSM4XJXHU5Q .
In a lot of places int32_t would help with the 64 bit build. But a naive find-and-replace would also make the code cumbersome to read, because simple loop counters etc. can safely stay int. So, I guess, I would just change it, where it actually has to be the correct size. I have started with some fixes for that.
Higher game resolution would be a great enhancement. It also seems that that the game speed depends on the window resolution. If I start it in 640x480 (game and window) everything is unplayable fast. If I use 1920x1080 as window resolution with scaling the game feels just about right.
Yes, it should be possible to open the CMakeLists.txt as if was is a solution and Visual Studio does some magic to hide the behind-hte-scenes from you. But in some places the CMake setup probably is too Linux specific until I generalise it.
Intresting, Tom put in a delay between frame renders to ensure the game speed did not exceed 60 FPS. I wonder if it is OS specific somehow.
Anyway good night
On Sat, 13 Feb 2021, 01:50 GrimSqueaker, notifications@github.com wrote:
In a lot of places int32_t would help with the 64 bit build. But a naive find-and-replace would also make the code cumbersome to read, because simple loop counters etc. can safely stay int. So, I guess, I would just change it, where it actually has to be the correct size. I have started with some fixes for that.
Higher game resolution would be a great enhancement. It also seems that that the game speed depends on the window resolution. If I start it in 640x480 (game and window) everything is unplayable fast. If I use 1920x1080 as window resolution with scaling the game feels just about right.
Yes, it should be possible to open the CMakeLists.txt as if was is a solution and Visual Studio does some magic to hide the behind-hte-scenes from you. But in some places the CMake setup probably is too Linux specific until I generalise it.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/GrimSqueaker/remc2/issues/36#issuecomment-778532405, or unsubscribe https://github.com/notifications/unsubscribe-auth/AETQQFHRA5ZRIMSXRLVRTPTS6XEG3ANCNFSM4XJXHU5Q .
Hi I would automatically select int on int32_t only in structures, Elsewhere it has to be done manually. I don't know if I can break CMake in a visual studio. We can try it, if it doesn't work I can always copy .sln and .vcproj from older revisions. Game frame looks like this: -compute all events -draw all entites -if it was faster than the set value, wait
In other words, the game rendering is slow and SDL rendering output too. Mybe we can optimize slow pixel by pixel rendering in VGA_Blit in port_sdl_vga_mouse.cpp.
Tom.
Hi Tom.
I think the slowness is caused by the upscaling in vgablit as it's nice an quick if the render window is set to 640.
Once I get higher resolutions running I will make sure the VGAblit is 1:1 so the speed should be nice and fast. I did try using SDL to upscale the buffer to the window, but while it could impressively shrink the output i could not get it to upscale :)
On Sat, 13 Feb 2021, 11:43 turican0, notifications@github.com wrote:
Hi I would automatically select int on int32_t only in structures, Elsewhere it has to be done manually. I don't know if I can break CMake in a visual studio. We can try it, if it doesn't work I can always copy .sln and .vcproj from older revisions. Game frame looks like this: -compute all events -draw all entites -if it was faster than the set value, wait
In other words, the game rendering is slow and SDL rendering output too. Mybe we can optimize slow pixel by pixel rendering in VGA_Blit in port_sdl_vga_mouse.cpp.
Tom.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/GrimSqueaker/remc2/issues/36#issuecomment-778598106, or unsubscribe https://github.com/notifications/unsubscribe-auth/AETQQFGP3PMWTKJYBW6GDNTS6ZJWNANCNFSM4XJXHU5Q .
Hi Tim, did you check in the enhancedassets directory? Seems like it is missing at the moment? I will adjust CMake to use the new layout. Cheers, Sebastian
I thought I had, but the Windows build is broken for the same reason. I will double check
https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail Virus-free. www.avast.com https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
On Sat, Feb 13, 2021 at 1:49 PM GrimSqueaker notifications@github.com wrote:
Hi Tim, did you check in the enhancedassets directory? Seems like it is missing at the moment? I will adjust CMake to use the new layout. Cheers, Sebastian
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/GrimSqueaker/remc2/issues/36#issuecomment-778614448, or unsubscribe https://github.com/notifications/unsubscribe-auth/AETQQFGQI2MHTRKA7U7J3Z3S6ZYM3ANCNFSM4XJXHU5Q .
Yeah it was my fault, I had to add the enhancedassets folder to the gitignore exclusions (as we need the wildcard paths to match the different output dirs)
On Sat, Feb 13, 2021 at 1:58 PM Timothy Hobbs thobbsinteractive@gmail.com wrote:
I thought I had, but the Windows build is broken for the same reason. I will double check
https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail Virus-free. www.avast.com https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail <#m_4952600300728034341_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
On Sat, Feb 13, 2021 at 1:49 PM GrimSqueaker notifications@github.com wrote:
Hi Tim, did you check in the enhancedassets directory? Seems like it is missing at the moment? I will adjust CMake to use the new layout. Cheers, Sebastian
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/GrimSqueaker/remc2/issues/36#issuecomment-778614448, or unsubscribe https://github.com/notifications/unsubscribe-auth/AETQQFGQI2MHTRKA7U7J3Z3S6ZYM3ANCNFSM4XJXHU5Q .
Thanks for the quick fix. :-)
I fix many problems for x64 compilation. For next work I must write convert mechanism for translate array<>struct save files, (otherwise I would damage load save fixes for original game compactibility - this is necessarily for debugging) Tom.
Nice work Tom. I am still working on the resolution increase, I think I am running into a problem with the frame buffering, but given enough time I think I can get around it.
I just got my Raspberry Pi 400, only to discover its a 32 bit OS (unless in install the beta version). But I am looking forward to getting this running in Linux and one day upgrading to 64bit and trying out both you and Sebastians' hard work.
Kind regards
Tim
On Thu, Feb 18, 2021 at 9:32 PM turican0 notifications@github.com wrote:
I fix many problems for x64 compilation. For next work I must write convert mechanism for translate array<>struct save files, (otherwise I would damage load save fixes for original game compactibility
- this is necessarily for debugging) Tom.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/GrimSqueaker/remc2/issues/36#issuecomment-781616894, or unsubscribe https://github.com/notifications/unsubscribe-auth/AETQQFCFKOL6QY2VH4LNMPTS7V2NFANCNFSM4XJXHU5Q .
Hi,
thanks for the 64bit work, Tom. But the current PR crashes in 32bit when starting the first level. I had no time to investigate further today (because of the Perseverance landing live stream :-). Maybe tomorrow I can take a closer look. It crashes in DataFileIO::Read with a strange "malloc(): unsorted double linked list corrupted" error. This seems to origin in glibc with either a heap overflow or more likely a heap corruption by writing to unwanted addresses. I guess, I will have to bisect the git commits to find the change that introduced the regression.
Tell me if you need help setting stuff up on Linux, Tim. I like helping people and I am glad getting some feedback and improving the Linux CMake setup so that it not just "works for me".
Cheers, Sebastian
Hi Sebatian, I watched Perseverance landing too, and I wonder what he can do :) In my system is starting first level in 32bit ok, please try to find out the details, or correct the error if you find the cause. Maybe anythink with sub_83E80_freemem4 or x_DWORD_E9C38_smalltit(backing up this buffer to other variables is sometimes weird). Tom.
You can compare code with origin disassembled code in remc2-dev/remc2/memimages/NETHERW.c for finding errors, too. Tom.
Well I just discovered one interesting thing about this game, the in game render is interlaced. So it renders half the screen then the other half in different frames (Menus are done separately). This was an old trick to double FPS. So I might be able to drop that, it would mean halving the render code as a lot was duplicated in the reverse engineering.
Anyway I am really looking forward to seeing the HD video from the Mars landing. It was all buffered up so is coming back in bits and pieces, but we finally will have HD video with sound of the landing! Very cool.
I will try to continue setting up my new Pi next week. I pulled the branch, but the build failed after the cmake command likely because of missing SDL stuff (which the instructions to say to get). Its a 32bit OS but an ARM processor so I think it will create new challenges for the compilation. I guess I find out when a carry on with the instructions.
Tim
On Fri, Feb 19, 2021 at 6:47 PM turican0 notifications@github.com wrote:
You can compare code with origin disassembled code in remc2-dev/remc2/memimages/NETHERW.c for finding errors, too. Tom.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/GrimSqueaker/remc2/issues/36#issuecomment-782232219, or unsubscribe https://github.com/notifications/unsubscribe-auth/AETQQFGKFYUQPEXTOAP3HHLS72P4JANCNFSM4XJXHU5Q .
I found out that I get a memory error when switching the resolution 640-> 320, I'll try to decide and maybe the problem will disappear for you, Tom.
I test two implementation of void sub_83E80_freemem4(uint8_t ptr)//264e80 1: if (ptr != NULL) { free((void)ptr); ptr = NULL; } and 2: free((void)ptr);
Both runs with errors.
At now is better remove this code. Later we can rewrite malloc/free to malloc_safety/free_safety with simple garbage collector. (or change the whole allocation method)
See if it works now.
Tom.
With your latest development branch I can now start the first level again (in 32bit), but it crashes when I build a castle. I'll try to investigate.
If you are interested, this is the callstack:
I would like to find a solution for this before I merge it, because otherwise I might not be able to test things out anymore.
Yes, the 32bit ARM might behave quite differently, but I have never tried it. You can check the list of dependencies from the Linux CI which is running on Ubuntu. And both, Ubuntu and Raspbian, are based on Debian. So, I would assume that the packages are identically named.
The next Perseverance live stream is announced for 22 Feb at 8:00 CET. Maybe there will be some video available already. Currently, Madrid is receiving 2Mb/s from MRO probably used as a relay (https://eyes.nasa.gov/dsn/dsn.html). This is so cool.
Ah, sorry. My mistake. I should have checked the code first. It is triggered by the preprocessor flag "TEST_x64". I will disable this on Linux and merge the PR.
This is ok, allert_error is only my margin for problematic part of code(when game gone to this, game crash, and we can analyze/fix it). I look at it.
In x64 I have made progress. Particly rendered frame of game(frame before full fade in):
Hi Tim, interlaced render is not visible, may be game this feature turn on only on slow machines(in original code, in remc2 is set best quality by default). Or can't be this code needed for blur effect.? Tom.
Begin of first level in x64 playable!!!
I think I might have discovered the old VR render from the original games code. The render was interlaced but also side-by-side (which I thought was an addressing issue) . If you look at the code to set the viewport it sets two render buffer pointers, one at the start of the global render buffer, and a second at start - viewport width. Being this second pointer is not used as much it maybe just for the VR or other renders. I will keep on exploring. I am keen to get this working
On Sun, Feb 21, 2021 at 11:10 AM turican0 notifications@github.com wrote:
Hi Tim, interlaced render is not visible, may be game this feature turn on only on slow machines(in original code, in remc2 is set best quality by default). Or can't be this code needed for blur effect.? Tom.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/GrimSqueaker/remc2/issues/36#issuecomment-782832170, or unsubscribe https://github.com/notifications/unsubscribe-auth/AETQQFFK2X4R6VZDCE2HSTLTADL2RANCNFSM4XJXHU5Q .
x64 mostly fixed. I finish level 1 and play level 2 without errors. At now we must play whole game for bug finding, Tom.
Wow. Now I have to get this working on Linux, because the compiler is much more picky in what is valid code. But as compared to what you have achieved making it work on Linux is a minor issue.
This VR support sounds cool.
I stand corrected. It is working right away. The first level is completely playable in 64bit on Linux. Amazing work and thanks to you, Tom. :+1:
This raises the question: Will we need 32 bit in the future? At least for CMake it would simplify things, if I remove special cases and only compile for the system native bitness. It would complicate Linux CI if we insist on verifying 32bit compatibility.
32-bit compatibility will probably always be useful for easier debugging, it may not be available on all platforms, but at least for me it is important for 32-bit windows.
Hi Tom,
while working on deleting dead code I stumbled upon this code
v63 = (char*)x_InterlockedExchange((long*)&v64, (signed __int32)(v57 + 1));
*v63 = v62;
in sub_63670_draw_minimap_a
which is only called in 320 resolution. The problem is that x_InterlockedExchange
always returns 0 and is dereferenced. This might explain your crash in 320.
In other news, I have now removed around 23k lines of dead code that has not been called from anywhere. I will bring this to the dev branch soon.
Cheers, Sebastian
Nice work Seb. I can't wait to merge that into my branch.
Kind regards
Tim
On Tue, 23 Feb 2021, 23:25 GrimSqueaker, notifications@github.com wrote:
Hi Tom,
while working on deleting dead code I stumbled upon this code
v63 = (char*)x_InterlockedExchange((long*)&v64, (signed __int32)(v57 + 1)); *v63 = v62;
in sub_63670_draw_minimap_a which is only called in 320 resolution. The problem is that x_InterlockedExchange always returns 0 and is dereferenced. This might explain your crash in 320.
In other news, I have now removed around 23k lines of dead code that has not been called from anywhere. I will bring this to the dev branch soon.
Cheers, Sebastian
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/GrimSqueaker/remc2/issues/36#issuecomment-784557059, or unsubscribe https://github.com/notifications/unsubscribe-auth/AETQQFFKQXBLYOEIR7Z6DU3TAQTMBANCNFSM4XJXHU5Q .
Good job Seb, for x_InterlockedExchange, can use this code? (I read https://docs.microsoft.com/en-us/windows/win32/api/winnt/nf-winnt-interlockedexchange) We maybe must change long to other type.
long x_InterlockedExchange(long volatile Target, long Value) { long temp = Target; *Target = Value; return temp; }; Tom.
I fix x64 in sub_63670_draw_minimap_a, but in level 1 this function not used, I test it later, Tom.
Ok, would have been a lucky try. Is there anything threaded going on that really requires locks? Because otherwise we could just remove this.
Edit: By "remove this" I mean "replace it by a standard assignment".
We don't need locks, it just wants to test if it works well, and then simplify it (remove the x_InterlockedExchange). But it's not important now, it only makes sense in terms of code cleanup (which would like to cleanup comprehensively anyway). Tom.
[image: 1080_Screen.png] Hey Guys, it took a while but I finally got HD Working on my resolution-upgrade branch!
I still have to fix the menu drawing, mouse navigation and map render, but it works. It renders very slowly at the moment which could be because software renders are slow (hence the original move to 3D Hardware for games) or we just need to adjust the delay added to make the game playable. Either it took me about 8 hours and tracking down a memory leak to find the one array variable I finally needed to change to make this work, so I'm celebrating.
Hope your changes are going just as well
Kind regards
Tim
On Thu, Feb 25, 2021 at 8:06 AM turican0 notifications@github.com wrote:
We don't need locks, it just wants to test if it works well, and then simplify it (remove the x_InterlockedExchange). But it's not important now, it only makes sense in terms of code cleanup (which would like to cleanup comprehensively anyway). Tom.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/GrimSqueaker/remc2/issues/36#issuecomment-785668424, or unsubscribe https://github.com/notifications/unsubscribe-auth/AETQQFG23QBD7GHPU4NM3VDTAXZGTANCNFSM4XJXHU5Q .
Hi Tim, good job. You could re-attach the screenshot for a better understanding? I don't see it in your post, Thanks, Tom.
Hi Tom, I replied via Email I guess that is why it did not appear here. When I fix the menus etc... I'll send a better one.
Hey, this looks great. Good job. And on the way you probably learned a lot to untangle some of the rendering code an data structures.
I hope so. I'm going to work on scaling up the menus next, but at some point I want to extract the render code into more modular methods that use parameters rather than global types to make them more portable. I have done a bit of that in my branch, but there is more to do.
I have a feeling that changing the speed delay won't speed up the frame rate as it was written to measure the draw time and deduct it from the delay. I think I will need to implement double buffering to speed up the render, we allowed for it in memory allocation in engine.h but at the moment we use the same pointer for the graphics buffer and don't flip it. We should be writing from one address while "displaying" (copying to the SDL window) from the other.
Eitherway I hope to make good progress this week and test out the Linux build too.
Tim
On Sun, 28 Feb 2021, 23:47 GrimSqueaker, notifications@github.com wrote:
Hey, this looks great. Good job. And on the way you probably learned a lot to untangle some of the rendering code an data structures.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/GrimSqueaker/remc2/issues/36#issuecomment-787538657, or unsubscribe https://github.com/notifications/unsubscribe-auth/AETQQFA5VDMBPRXNNZUZWPLTBLBXNANCNFSM4XJXHU5Q .
It is super!!! Would it help us to disassemble Populous III code? It should be based on the same engine and may use DirectX / OpenGL, but I'm not sure. Tom.
Hi Tom,
As I remember, Populous 3 used open GL and DirectX (depending on your config) I did not know it was based on the same Engine. The refactoring will make replacing the Graphics code possible, but my goals are to keep the old render for the time being, increase the draw distance and get multiplayer working. Then produce a community patch. Open GL comes next along with the editor and larger maps. Then hopefully after all that Magic Carpet 1.
Ironically it will be easier to rewrite in Open GL code as Open GL has set render pipeline steps that are established design patterns, the current software render code does not "seem" to obey those steps.
On Mon, 1 Mar 2021 at 07:17, turican0 notifications@github.com wrote:
It is super!!! Would it help us to disassemble Populous III code? It should be based on the same engine and may use DirectX / OpenGL, but I'm not sure. Tom.
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/GrimSqueaker/remc2/issues/36#issuecomment-787678012, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANV67ULBPGG5OFRQKIXVFQTTBMWPHANCNFSM4XJXHU5Q .
I add popTB.c and D3DPopTB.c - disassempled Populous III to memimages, you can inspire it for creation D3D or OpenGL rendering, Tom.
Thanks Tom I will take a look once I have sorted out the in game menus. I have discovered that the slow frame rate is due to the Software Render. We already double buffer with SDL.
I can hopefully quickly fix this if I use the interlacing code I found for the VR render and multi-thread the render with one extra thread to draw alternate rows. Obviously I would make the multi-threading render a configurable and option.
Do you guys have any suggestions for multi-platform threading libraries or will the standard C++ 11 ones work? As I remember in Windows you can force the OS to dedicate a cpu-core to a thread rather than allowing the OS to decide where it is executed. I am wondering if that is possible in a multi-platform libraries?
I will start with trying to build the code in 64 bit. In order to get going, I will to the same thing as with the Linux build by introducing conditional compilation with "FIXME" notes and printouts. In my previous clone of the repo I got to the first level with that approach but it only could render the map and not the 3d world. I think if we consequently move forward with structures and rewrite those pointer to int assignments we can get it working at some point.
For this I have introduced the preprocessor define
COMPILE_FOR_64BIT
that can be set in CMake by passing-DUSE_64_BIT=True
on the command line. It would be good if somebody could also introduce this into the VS solution, since I am not proficient in VS.The benefit of this preprocessor flag is that I can leave the 32bit version unchanged and after having it compile with 64 bit can start to iteratively only fix those code locations that are triggered when starting to play the game.