search

SethRobinson / proton

Proton SDK: Seth's GL/GLES messy multi-platform C++ game SDK. Can output to Windows, OS X, iOS, Android, Linux (Raspbian too), HTML5, Flash
Other
79 stars 16 forks source link

[RFC] Project Generation / Build System #35

Open mempler opened 8 months ago

mempler commented 8 months ago

The issue

Right now, ProtonSDK is dependent on multiple projects for various platforms, which require a lot of work to maintain and requires a lot of duplication between platforms.

having to constantly switch between devices just to make sure you added everything correctly is really cumbersome.

My proposal

So, my proposal would be to have a project generator like CMake or premake5 to replace the current, mainly duplicate build systems for all the platforms.

Some possibilities

project generation

build systems:

Possible solution

The best choice would be moving forward with CMake, as it is natively supported by almost all IDEs, thus really easy to integrate and is basically industry standard at this point.

and we could also recycle the current implementation that is used by linux & pis.

Request for comment (rfc)

Right now, this is merely a proposal and may or may not be implemented, so i rather would have a discussion if this is even wanted / needed.

mempler commented 8 months ago

I do have my own private fork of ProtonSDK that has been fully ported to CMake, allthough not really ready to use outside my own projects, given it's more my own style and less cmake's style and it also does not contain any media compilation, its still handled by an external script right now.

an example CMakeLists.txt of one of my projects would look like this:

cmake_minimum_required (VERSION 3.8)
project("Pokemon")

ProtonSDK_Project(
    TARGET Pokemon

    CAPABILITIES
        "NETWORK"

        "AUDIO_FMOD"
        "TEXTSCANNER"

        "ENTITY_UTILITIES"

    ENTITY_COMPONENTS
        "HTTPComponent"
        "LogDisplayComponent"

    SOURCES
        "App.cpp"

        "World/Tile.cpp"
        "World/TileExtra.cpp"
        "World/World.cpp"
        "World/WorldRenderer.cpp"
        "World/WorldTileMap.cpp"

        "Utils/Device.cpp"

        "GUI/LogOverlay.cpp"
        "GUI/Button.cpp"
        "GUI/Menu/MainMenu.cpp"
        "GUI/Menu/AboutMenu.cpp"
        "GUI/Menu/OptionsMenu.cpp"

        "Component/DownloadFileComponent.cpp"
        "Component/GameLogicComponent.cpp"
    )
mempler commented 8 months ago

My current approach to handle this issue consists of 4 files and depends on vcpkg to reduce third-party code in shared folder

For reference `proton/CMake/Project.cmake` ```cmake function("ProtonSDK_Project") set(options CONSOLE) set(oneValueArgs TARGET) set(multiValueArgs SOURCES CAPABILITIES ENTITY_COMPONENTS DEFINES) cmake_parse_arguments(PROTONSDK_EXECUTABLE "${options}" "${oneValueArgs}" "${multiValueArgs}" ${ARGN}) # Adjust for source locality # ------------------------------------------------------------------------- set(__sources) foreach(source ${PROTONSDK_EXECUTABLE_SOURCES}) list( APPEND __sources "${CMAKE_CURRENT_SOURCE_DIR}/source/${source}" ) endforeach() set(__libs) set(__definitions) set(__include_dirs) message("HEY: ${PROTONSDK_EXECUTABLE_CONSOLE}") if(PROTONSDK_EXECUTABLE_CONSOLE) # Console applications # --------------------------------------------------------------------- list( APPEND __sources "${PROTONSDK_ROOT}/PlatformSetup.cpp" "${PROTONSDK_ROOT}/util/Variant.cpp" "${PROTONSDK_ROOT}/Manager/VariantDB.cpp" "${PROTONSDK_ROOT}/FileSystem/FileManager.cpp" "${PROTONSDK_ROOT}/FileSystem/StreamingInstance.cpp" "${PROTONSDK_ROOT}/FileSystem/StreamingInstanceFile.cpp" "${PROTONSDK_ROOT}/Math/rtRect.cpp" "${PROTONSDK_ROOT}/Renderer/SoftSurface.cpp" "${PROTONSDK_ROOT}/util/MathUtils.cpp" "${PROTONSDK_ROOT}/util/TextScanner.cpp" "${PROTONSDK_ROOT}/util/CRandom.cpp" "${PROTONSDK_ROOT}/util/MiscUtils.cpp" "${PROTONSDK_ROOT}/util/ResourceUtils.cpp" "${PROTONSDK_ROOT}/ClanLib-2.0/Sources/Core/Math/angle.cpp" "${PROTONSDK_ROOT}/ClanLib-2.0/Sources/Core/Math/mat3.cpp" "${PROTONSDK_ROOT}/ClanLib-2.0/Sources/Core/Math/mat4.cpp" "${PROTONSDK_ROOT}/ClanLib-2.0/Sources/Core/Math/rect.cpp" "${PROTONSDK_ROOT}/ClanLib-2.0/Sources/Core/Math/vec2.cpp" "${PROTONSDK_ROOT}/ClanLib-2.0/Sources/Core/Math/vec3.cpp" "${PROTONSDK_ROOT}/ClanLib-2.0/Sources/Core/Math/vec4.cpp" ) list( APPEND __definitions -D"_CONSOLE" ) else() # Graphical applications # --------------------------------------------------------------------- list( APPEND __sources "${PROTONSDK_ROOT}/BaseApp.cpp" "${PROTONSDK_ROOT}/PlatformSetup.cpp" # "${PROTONSDK_ROOT}/SDL/SDL2Main.cpp" "${PROTONSDK_ROOT}/util/VideoModeSelector.cpp" "${PROTONSDK_ROOT}/util/PassThroughPointerEventHandler.cpp" "${PROTONSDK_ROOT}/util/TouchDeviceEmulatorPointerEventHandler.cpp" "${PROTONSDK_ROOT}/util/Variant.cpp" "${PROTONSDK_ROOT}/Manager/VariantDB.cpp" "${PROTONSDK_ROOT}/Audio/AudioManager.cpp" "${PROTONSDK_ROOT}/FileSystem/FileManager.cpp" "${PROTONSDK_ROOT}/FileSystem/StreamingInstance.cpp" "${PROTONSDK_ROOT}/FileSystem/StreamingInstanceFile.cpp" "${PROTONSDK_ROOT}/GUI/RTFont.cpp" "${PROTONSDK_ROOT}/Manager/Console.cpp" "${PROTONSDK_ROOT}/Manager/GameTimer.cpp" "${PROTONSDK_ROOT}/Manager/MessageManager.cpp" "${PROTONSDK_ROOT}/Manager/ResourceManager.cpp" "${PROTONSDK_ROOT}/Math/rtRect.cpp" "${PROTONSDK_ROOT}/Renderer/Surface.cpp" "${PROTONSDK_ROOT}/Renderer/SoftSurface.cpp" "${PROTONSDK_ROOT}/Renderer/SurfaceAnim.cpp" "${PROTONSDK_ROOT}/Renderer/RenderBatcher.cpp" "${PROTONSDK_ROOT}/util/MathUtils.cpp" "${PROTONSDK_ROOT}/util/CRandom.cpp" "${PROTONSDK_ROOT}/util/MiscUtils.cpp" "${PROTONSDK_ROOT}/util/ResourceUtils.cpp" "${PROTONSDK_ROOT}/util/RenderUtils.cpp" "${PROTONSDK_ROOT}/util/GLESUtils.cpp" "${PROTONSDK_ROOT}/Entity/Entity.cpp" "${PROTONSDK_ROOT}/Entity/Component.cpp" "${PROTONSDK_ROOT}/ClanLib-2.0/Sources/Core/Math/angle.cpp" "${PROTONSDK_ROOT}/ClanLib-2.0/Sources/Core/Math/mat3.cpp" "${PROTONSDK_ROOT}/ClanLib-2.0/Sources/Core/Math/mat4.cpp" "${PROTONSDK_ROOT}/ClanLib-2.0/Sources/Core/Math/rect.cpp" "${PROTONSDK_ROOT}/ClanLib-2.0/Sources/Core/Math/vec2.cpp" "${PROTONSDK_ROOT}/ClanLib-2.0/Sources/Core/Math/vec3.cpp" "${PROTONSDK_ROOT}/ClanLib-2.0/Sources/Core/Math/vec4.cpp" ) endif() # Common # ------------------------------------------------------------------------- list( APPEND __include_dirs "${PROTONSDK_ROOT}" "${PROTONSDK_ROOT}/ClanLib-2.0/Sources" "${CMAKE_CURRENT_SOURCE_DIR}/source" ) list( APPEND __definitions ${PROTONSDK_EXECUTABLE_DEFINES} -D"BOOST_BIND_GLOBAL_PLACEHOLDERS" ) # Third-party libraries # ------------------------------------------------------------------------- find_package(Boost REQUIRED) find_package(ZLIB REQUIRED) find_package(CURL CONFIG REQUIRED) find_package(PNG REQUIRED) find_package(OpenGL REQUIRED COMPONENTS OpenGL) find_package(SDL2 CONFIG REQUIRED) find_package(libjpeg-turbo CONFIG REQUIRED) list( APPEND __libs ZLIB::ZLIB ) if(NOT PROTONSDK_EXECUTABLE_CONSOLE) # Graphical applications # --------------------------------------------------------------------- list( APPEND __libs PNG::PNG OpenGL::GL $,SDL2::SDL2,SDL2::SDL2-static> ) list( APPEND __definitions -D"C_GL_MODE" ) endif() # Capabilities # ------------------------------------------------------------------------- if("NETWORK" IN_LIST PROTONSDK_EXECUTABLE_CAPABILITIES) list( APPEND __sources "${PROTONSDK_ROOT}/Network/NetHTTP.cpp" "${PROTONSDK_ROOT}/Network/NetHTTP_libCURL.cpp" "${PROTONSDK_ROOT}/Network/NetUtils.cpp" "${PROTONSDK_ROOT}/Network/NetSocket.cpp" ) list( APPEND __definitions -D"RT_USE_LIBCURL" ) list( APPEND __libs CURL::libcurl ) endif() if("NETWORK_ENET" IN_LIST PROTONSDK_EXECUTABLE_CAPABILITIES) list( APPEND __libs enet ) endif() if("AUDIO_FMOD" IN_LIST PROTONSDK_EXECUTABLE_CAPABILITIES) list( APPEND __libs fmod ) list( APPEND __sources "${PROTONSDK_ROOT}/Audio/AudioManagerFMOD.cpp" ) endif() if("TEXTSCANNER" IN_LIST PROTONSDK_EXECUTABLE_CAPABILITIES) list( APPEND __sources "${PROTONSDK_ROOT}/util/TextScanner.cpp" ) endif() if("ENTITY_UTILITIES" IN_LIST PROTONSDK_EXECUTABLE_CAPABILITIES) list( APPEND __sources "${PROTONSDK_ROOT}/Entity/EntityUtils.cpp" ) # Entity utils require some components list( APPEND PROTONSDK_EXECUTABLE_ENTITY_COMPONENTS "FocusInputComponent" "FocusUpdateComponent" "FocusRenderComponent" "TouchHandlerComponent" "TouchDragComponent" "Button2DComponent" "EmitVirtualKeyComponent" "SliderComponent" "TyperComponent" "ScrollComponent" "FilterInputComponent" "SelectButtonWithCustomInputComponent" "TextBoxRenderComponent" "RenderScissorComponent" "ScrollBarRenderComponent" "RectRenderComponent" "UnderlineRenderComponent" "InputTextRenderComponent" "TextRenderComponent" "OverlayRenderComponent" "InterpolateComponent" ) endif() if("JPEG" IN_LIST PROTONSDK_EXECUTABLE_CAPABILITIES) list( APPEND __sources "${PROTONSDK_ROOT}/Renderer/JPGSurfaceLoader.cpp" ) list( APPEND __definitions -D"RT_JPG_SUPPORT" ) list( APPEND __libs $,libjpeg-turbo::turbojpeg,libjpeg-turbo::turbojpeg-static> ) endif() if("PNG" IN_LIST PROTONSDK_EXECUTABLE_CAPABILITIES) list( APPEND __definitions -D"RT_PNG_SUPPORT" ) list( APPEND __libs PNG::PNG ) endif() # Platform specific sources # ------------------------------------------------------------------------- # Linux # ------------------------------------------------------------------------- if(UNIX AND NOT APPLE) list( APPEND __sources "${PROTONSDK_ROOT}/linux/LinuxUtils.cpp" ) list( APPEND __definitions -D"RTLINUX" -D"PLATFORM_LINUX" ) # ------------------------------------------------------------------------- # Apple # ------------------------------------------------------------------------- elseif (UNIX AND APPLE) list( APPEND __sources "${PROTONSDK_ROOT}/OSX/OSXUtils.mm" ) list( APPEND __libs "-framework Cocoa" ) if(NOT PROTONSDK_EXECUTABLE_CONSOLE) list( APPEND __sources "${PROTONSDK_ROOT}/OSX/app/main.m" "${PROTONSDK_ROOT}/OSX/app/MainController.mm" "${PROTONSDK_ROOT}/OSX/app/MyApplication.mm" "${PROTONSDK_ROOT}/OSX/app/MyOpenGLView.mm" ) list( APPEND __libs "-framework CoreFoundation" "-framework OpenGL" "-framework QuartzCore" "-framework AppKit" "-framework Foundation" ) endif() list( APPEND __definitions -D"RTOSX" -D"PLATFORM_OSX" ) endif() # Entity Components # ------------------------------------------------------------------------- foreach(component ${PROTONSDK_EXECUTABLE_ENTITY_COMPONENTS}) list( APPEND __sources "${PROTONSDK_ROOT}/Entity/${component}.cpp" ) endforeach() # Executable # ------------------------------------------------------------------------- # if (UNIX AND APPLE) # add_executable( # ${PROTONSDK_EXECUTABLE_TARGET} # MACOSX_BUNDLE # ${__sources} # ) # else() # add_executable( # ${PROTONSDK_EXECUTABLE_TARGET} # ${__sources} # ) # endif() add_executable( ${PROTONSDK_EXECUTABLE_TARGET} ${__sources} ) target_include_directories( ${PROTONSDK_EXECUTABLE_TARGET} PRIVATE ${__include_dirs} ) target_link_libraries( ${PROTONSDK_EXECUTABLE_TARGET} PRIVATE ${__libs} ) target_compile_definitions( ${PROTONSDK_EXECUTABLE_TARGET} PRIVATE ${__definitions} ) # Disable all warnings # ------------------------------------------------------------------------- target_compile_options( ${PROTONSDK_EXECUTABLE_TARGET} PRIVATE -w ) endfunction() ```
For reference `proton/CMakeLists.txt` ```cmake cmake_minimum_required (VERSION 3.8) project("ProtonSDK") set("PROTONSDK_ROOT" "${CMAKE_CURRENT_SOURCE_DIR}/shared") set(CMAKE_C_STANDARD 11) set(CMAKE_CXX_STANDARD 11) include("CMake/Project.cmake") add_subdirectory(Pokemon) if(NOT APPLE) add_subdirectory(tools/RTPack) endif() add_subdirectory(lib/enet) add_subdirectory(lib/fmod) ```
For reference `proton/vcpkg.json` ```json { "$schema": "https://raw.githubusercontent.com/microsoft/vcpkg-tool/main/docs/vcpkg.schema.json", "dependencies": [ "sdl2", "curl", "zlib", "opengl", "libjpeg-turbo", "libpng", "rapidxml", "boost-signals2" ] } ```
For reference `proton/CMakePresets.json` ```json { "version": 6, "cmakeMinimumRequired": { "major": 3, "minor": 23, "patch": 0 }, "configurePresets": [ { "name": "debug", "displayName": "Debug Config", "generator": "Ninja", "binaryDir": "${sourceDir}/build/Debug", "toolchainFile": "./vcpkg/scripts/buildsystems/vcpkg.cmake", "cacheVariables": { "CMAKE_BUILD_TYPE": "Debug", "CMAKE_EXPORT_COMPILE_COMMANDS": "ON" } }, { "name": "release", "displayName": "Release Config", "generator": "Ninja", "binaryDir": "${sourceDir}/build/Release", "toolchainFile": "./vcpkg/scripts/buildsystems/vcpkg.cmake", "cacheVariables": { "CMAKE_BUILD_TYPE": "Release", "CMAKE_EXPORT_COMPILE_COMMANDS": "ON" } }, { "name": "debug-xcode", "displayName": "Debug XCode Config", "generator": "Xcode", "binaryDir": "${sourceDir}/build/Debug-XCode", "toolchainFile": "./vcpkg/scripts/buildsystems/vcpkg.cmake", "cacheVariables": { "CMAKE_BUILD_TYPE": "Debug", "CMAKE_EXPORT_COMPILE_COMMANDS": "ON" } } ], "buildPresets": [ { "name": "debug", "configurePreset": "debug" }, { "name": "release", "configurePreset": "release" } ] } ```
SethRobinson commented 8 months ago

Thanks for the input.

I agree with this, a single CMake to rule them all would be great. I primarily develop in Visual Studio and I think its internal direct CMake support (no need to create .sln files from the CMakeList.txt I guess) is probably far enough along now that it can could replace my current workflow, if I can figure out all the flags etc to duplicate how I have it now.

Some random thoughts:

mempler commented 8 months ago

Actually my work is based on shared/linux/proton.cmake, it just so happens that in the end that i rewrote everything lol!

the reason i did that is so projects dont infect each other projects with the root CMakeLists.txt i had within proton itself, so i could straight up use rtpack to compile media without having to rely on prebuilt binaries.

allthough this did bring another issue where it does multiple compilation on the same file, due to different #defines.

I've used vcpkg in the past (as seen in reference material) which helped me reduce a lot of third-party imports, which cluttered my initial cmake port.

only issue with that approach though is that it'll break existing build systems (except visual studio, as vs2019+ has builtin vcpkg support). So, only VS solutions would have a gradient porting phase :/

though this is less of an issue if the existing third-party code sticks around until cmake is fully embraced.

SethRobinson commented 8 months ago

I'd never heard of vcpkg so read up on it, hmm, looks pretty good and does seem to support the platforms I care about (or almost, see below) and is open source.

I think leaving the messy source tree (boost/jpg/png/etc) is necessary for now, as I'm sure nobody would appreciate old projects suddenly breaking. (although I think I am using packages/libs for linux/ios already for the most part)

I don't mind if the existing CMake support breaks though as it's only used for linux and I don't mind rejiggering my apps. I also don't mind requiring Visual Studio '22+ for Windows.

A good test would be a CMake/vcpkg setup for say, RTSimpleApp that works for Win/html5/linux/mac/ios/android. Probably quite challenging to get all those platforms working, especially the xcode/android studio parts. Do you work with all those platforms by any chance? :P

Note: vcpkg might not support mac/ios stuff completely without workarounds.

mempler commented 7 months ago

I think leaving the messy source tree (boost/jpg/png/etc) is necessary for now, as I'm sure nobody would appreciate old projects suddenly breaking. (although I think I am using packages/libs for linux/ios already for the most part)

Maybe a gradient deprecation notice would be nice. like a simple #pragma warning in the main files that are affected by this change.

Note: vcpkg might not support mac/ios stuff completely without workarounds.

Oh right, i haven't really developed for iOS and mac in general, i'm not familiar with the API.

A good test would be a CMake/vcpkg setup for say, RTSimpleApp that works for Win/html5/linux/mac/ios/android. Probably quite challenging to get all those platforms working, especially the xcode/android studio parts. Do you work with all those platforms by any chance? :P

I did work with Win32, Linux and macOS on that project, given most of my code is based on those example projects lol

but i haven't gotten android and iOS to run.

About HTML5, does it use emscripten to compile to wasm or plain js ? i really didn't use it within proton, so no testing on my end there.

iProgramMC commented 7 months ago

I agree with the sentiment, but Proton is sort of designed to "add stuff as you go", which means you can exclude certain files from compilation to cut down on size. For example, you can add LinearParticle support if you need it, or keep it out if you don't. So putting all the shared Proton stuff inside of a CMakeLists.txt for itself isn't a great idea.

Android does use CMake to build, but it would be nice for Linux to do the same. As for iOS and Mac OS, I don't believe you can get away without using Xcode. But I could be wrong and they actually support CMake to build apps targeting those platforms, that would be nice.

Anyways, I don't think adding Mempler to the project as a collaborator directly is a good idea, as it's just not worth it, and they can already make pull requests which are good enough for most cases.

mempler commented 7 months ago

As for iOS and Mac OS, I don't believe you can get away without using Xcode. But I could be wrong and they actually support CMake to build apps targeting those platforms, that would be nice.

CMake can generate Xcode projects, so that is not really an issue.

Anyways, I don't think adding Mempler to the project as a collaborator directly is a good idea, as it's just not worth it, and they can already make pull requests which are good enough for most cases.

I agree with this, adding me: someone who did not work actively with this repository yet as an contributor would not be an good idea, given how much damage I could do.

SethRobinson commented 7 months ago

CMake can generate Xcode projects, so that is not really an issue.

Yeah, that's kind of how Unity 3D works. I mean, they don't use CMake internally as far as I know but they do just export a xcode file and it kind of works. I guess for tricky projects that have a lot of customized xcode settings you could always generate it, then keep using that as well.

Android Studio seems like it might have some internal CMake support but.. hrm, not sure.

About HTML5, does it use emscripten to compile to wasm or plain js ? i really didn't use it within proton, so no testing on my end there.

I just looked (currently it's all done from a single batch file basically) and it does create .wasm files. It's actually a pretty easy system to get to work and the files are quite tiny compared to Unity/etc.

Well, easy except none of html5 examples actually compiled (the boost .cpp files had to be removed), I just fixed them, will commit it in a sec.

Anyways, I don't think adding Mempler to the project as a collaborator directly is a good idea, as it's just not worth it, and they can already make pull requests which are good enough for most cases.

Meh, I think it's fine. :)

Here's my reasoning:

Proton SDK needs all the help it can get to stay current and "building" so worth taking some risks. :) It's not trying to be Unity or Godot, I think it exists in a unique space for people who like extremely portable flexible low level stuff.

iProgramMC commented 7 months ago

Android Studio seems like it might have some internal CMake support but.. hrm, not sure.

Of course it does. That's how you created the AndroidGradle projects :)

it's not a huge deal to rollback git a bit.

Well, as long as you keep a local and decently up to date copy on your hard drive. If you get an HD crash and a contributor suddenly decides to force push an empty repository (unless you have turned it off, which you really should!), boom, source's inaccessible, which is a pain, but not a total one because I believe git could recover it as it's merely an "orphaned branch".

SethRobinson commented 7 months ago

Of course it does. That's how you created the AndroidGradle projects :)

Oops, yeah you're right - maybe it wouldn't be too hard to make it use a global app-level CMakeLists.txt instead. Probably the build.gradle/manifest/android studio part of stuff should just stay how they are (I don't think CMake can create that stuff anyway like it can with Xcode projects), it's a crazy world with the java-end of things.

Well, as long as you keep a local and decently up to date copy on your hard drive. If you get an HD crash and a contributor suddenly decides to force push an empty repository (unless you have turned it off, which you really should!)

Damn, gonna go find that option and turn it off, don't want a contributor to be able to trash repo history

iProgramMC commented 7 months ago

maybe it wouldn't be too hard to make it use a global app-level CMakeLists.txt instead.

I've done it before.

That being said I agree that the CMakeLists should be global at app level, not at the level of the whole Proton SDK. As mentioned before, Proton is supposed to allow you to remove certain cpp files that you don't need (e.g. in RTBareBones, you don't compile in the entire set of entity components).

gonna go find that option and turn it off

You probably have seen a banner on top of the repo if you have not already turned off force pushing.