OpenGEX format as alternative for MD5mesh/MD5anim

[Methelina]

I want to suggest the OpenGEX http://opengex.org/ universal efficient container for the assets for RBDoom. It has a important advantages in compartment with DAE/FBX in compatibility, standardization and human-readability. Text-based file format designed to facilitate the transfer of complex scene data between Modern applications such as modelling tools and game engines.

Implementation of this feature will share the engine for the more potential users and asset-creators. That uses modern versions of Blender, Maya and Max in the studios or home and have no views of possibilities to work with md5 or its specific disadvantages.

The use of this format have a comfortable paradigms with direct transfer the data without third party middle-ware. From tool to the engine with same standard and specification without loosing data from different/homemade plugins and software connectors. This way save user from hi-versioning and bittroting issues from the thirdparty or outdated plugins that have different specification between importing and exporting software.

The export plugins posted here support the following features: Hierarchical scene organization (node trees). Factored node transforms where necessary (e.g., splitting into position, rotation, and scale). Object instancing (multiple nodes can reference the same object). Geometry objects, light objects, and camera objects. Multiple materials per mesh with independent binding per instance. Multiple vertex colors and texture coordinate sets per mesh. Skinned meshes (skeleton, bind-pose transforms, bone influence weighting data). Morphed meshes (also known as blend shapes or shape keys). Keyframe animation for all node transforms and morph weights. Linear, Bézier, and TCB animation curves. Material colors and textures (diffuse, specular, normal, emission, opacity, transparency). Texture coordinate transforms.

It has supportable official export-plugins for – Blender 2.72. – 3dsMax 2014 – Maya 2014

http://opengex.org/OpenGex-Import.zip Import-code example

Comparation with collada http://opengex.org/comparison.html

Documentation and specification http://opengex.org/OpenGEX.1.1.2.pdf

[spannerman79]

I'm all for open formats & FOSS. Code wise I have no idea how this would effect the assets needed to play Doom etc (assets outlined via https://github.com/RobertBeckebans/RBDOOM-3-BFG/blob/master/README.txt#L205 )

[Methelina]

You mean the existed one? If yes, there no need to change the existed assets to the new format. OpenGEX can be used to import raw data in alternative way that will be parsed-out in the engine, doom-friendly inner structure and used same way as an old one. Same as did in Unity3D engine for example, where all data import is raw to prevent data loss and than parsed out and setted up in the editor in the engine-native look.

------- UP for example in same manner of doom md5 ideology. parsing the file without animation nodes can be read as BindPose object and the animation-nodes with frame from 0 to N as the animation files md5anim and so on in very automatic way. The read specific data can be stored in the map container for example for the modular relative way in a configuration node or something like that.

[BielBdeLuna]

seems pretty interesting!

[BielBdeLuna]

doom3 assets could be opened because the code would remain in the engine, just as we have DAE this doesn't mean we loss md5 support, just like we could add OpenGEX to the code.

[Methelina]

indeed, that's what I'm talk about :) I hope Robert will be interested about this feature request. I can make a test animated model for example tests if it will be needed

[BielBdeLuna]

I wonder if we could use DAE for static meshes, and OGEX for skinned meshes? but maybe both can serve both purposes?

[Methelina]

DAE can be used for static and for skinned meshes same as the openGEX for animated and static data too. My idea went from the differences between ogex and collada from the practical experience. OpenGEX better than dae in many ways https://github.com/RobertBeckebans/RBDOOM-3-BFG/issues/url The truth about use of collada for animated assets will raise when you will try to export assets from maya or max into the engine it will makes a very awkward results because of plugins for max and maya have their own specific issues and version-oriented differences with the import code of the engine library. We tried to work with dae some day, its perfect for static meshes but with skinned assets it comes to the unbearable hell... DAE is based on XML and sometimes parsing of xml code will run into the funny results, mb im wrong but in my experience of development of engines - many ways run into the FBX when artists want to see their assets in the game without glitches and with animation. But here we cant use FBX by the close source and licensing. Thats why i suggest to use the openGEX

[BielBdeLuna]

ok

[BielBdeLuna]

With OpenGEX can we have separated animation files from the mesh (just like md5, maybe have an OGEX file for the skinned mesh, and another OGEX file with the animation related "nodes") so we can have several meshes animated from the same animation files?

---

If I've read it correctly we could have OGEX files for static models with less "nodes" because we don't need any of the animation stuff isn't it? so those OGEX files for static models could weight less than the ones for skinned meshes isn't it?

[Methelina]

Yes it can be separated files, for sure it more depends of how engine organize imported data than from openGEX. If the engine able to set the animation files to he different meshes - why not? It can be used for openGEX too. And all data can be inside the one ogex file, bindpose and all animation sequences. Anyway on the engine every animation frame range and mesh will be separated entity.

And yes no animation in file = less nodes. Less weight it clearly seen if you open any ogex file with the gedit or notepad.

[RobertBeckebans]

OpenGEX is indeed an interesting alternative to DAE for static meshes however it is more or less unsupported by its creator. Doom 3 .md5mesh lacks support for different smoothing groups and doesn't import any custom normals or tangent spaces. Adding OpenGEX support even for animations wouldn't be difficult but time consuming. I work with the C4 engine on another project and we even don't use OpenGEX although it was invented for that engine. DAE just works there without the trouble to install any OpenGEX importers. It might be very easy to add OpenGEX support if its done using the open asset import library. I think it added support for it a while ago. My DAE importer is also ripped off from Assimp.

[motorsep]

I am missing the point - why do you need smoothing groups when you can either split edges on export (when using Blender; don't know much about other 3D apps in that regard) or simply rely on normal maps and self-shadowing.

The major downside of md5 is lack of exporters for MAX/Maya.

[BielBdeLuna]

On the other hand if OpenGEX is unmaintained is also unmaintained in assimp. isn't it? so seeing that OpenGEX seems an easy implementation as a stand alone, and assimp. a really complicated implementation, I think we could implement just the OpenGEX library with far more simplicity than the whole assimp.

what do you think?

[Methelina]

Assimp its an interesting solution or mb better to try assimp for some case than only ogex. Need the comparation run from the skilled coders that can reveal what will be better :) 'm just around to have all loss-less blacl-magic-less support for modern maya\max users such im and people i know.

[RobertBeckebans]

The easiest way to implement and verify OpenGEX support would be to automatically export .md5mesh/anim files to OpenGEX so they can be reimported. That way it would be easy to see how much OpenGEX matches the existing skeletal animation system. I only work with Blender because that is the only 3D app that I mastered more or less. I can understand that it might be quite frustating to work with .md5mesh if those plugins haven't been updated for years and don't support the newest versions. Actually I think OpenGEX is the nicest text based format since OBJ and a lot more powerful.

[motorsep]

My Blender 2 MD5 plugin works with latest Blender ;)

[BielBdeLuna]

I'm trying to port OpenGEX to IdTechX

[motorsep]

@Illasera Hopefully they will listen to you :)

Btw, BFG engine binarizes md5model/anim files for loading in runtime, so memory footprint becomes much smaller. So in reality there is no need to re-invent the wheel. Just need to maintain Blender's add-on ( https://github.com/motorsep/blender-idtech4-md5 ) and fix Maya's original tool and maybe write 3DS MAX plugin (at this point I think majority of people who are still messing with id Tech 4 / BFG use Blender anyway).

[BielBdeLuna]

but the idea is to add more options that md5 doesn't allow, of course it wouldn't be needed if we just wanted to keep using the features of md5

[motorsep]

The same question that has been coming up through the history is practicality. There hasn't been any development recently that offered any new characters for any of Doom 3 mods (not counting TDM and a few older mods).

Now, if no one makes any characters using current capacity of the engine (which is more than plenty to make something cool), no one will care for new model format, regardless of extra features it offers.

So is it practical to put efforts into new format ? No. However, open source development generally is not about practicality. It's about "it's freedom, I want to do it and no one can stop me". :)

So, I'll just see what happens :) After all, development on idTech front has been dragging recently.

[BielBdeLuna]

that's true sorry OpenGEX is for OTE, but we could add it too for RBDoom3BFG

[BielBdeLuna]

the fact that we get OpenGEX into RBDoom3BFG doesn't mean we loose MD5 functionality, and about the "goal" of the project, is it this strict? and any ways it's Robert's project and he defines what his project ends up being.

[BielBdeLuna]

yes, but since we want to add functionality to the engine, MD5 doesn't provide all the bells and whistles we might need in the future, so what do we do? we stick with MD5 and never evolve form this?

[BielBdeLuna]

no, Unity and Unreal are not open source

[aFoxNamedMorris]

@BielBdeLuna If you want an open engine with the kind of features you want, you might look at Armory.

[motorsep]

eeehh, I am out of here....

but before I do, let's face the facts. What evolution are you talking about?! RBD3BFG has been inactive for a loooong time. OTE never progressed beyond initial state. It's been years in the making and nothing really happened.

I released our engine quite a few month ago and only this morning I got a message from someone who just barely tried compiling it (and succeeded nevertheless :P ). No one even bothered to make minimum required assets for it. Not to mention attempted to make a game with it (despite the fact that's out of all forks out there it's the most production ready fork)

So the facts point to only one path for idtech 4 / BFG - stagnation and decline.

Btw, besides releasing engine, I also released Blender add-on I used in production. So if someone would grab all that and bothered to actually make a stater project, open source engine would make a big leap. People need to be able to build games with limitations imposed by exiting engine and once they hit the ceiling and max out the potential, then maybe new features should be added.

However, open source is never about being practical.

[BielBdeLuna]

@motorsep yeah we all know you position on this it has never been different @aFoxNamedMorris very interesting, all implemented in Blender game engine isn't it? it looks great! but idTech 4 brings a more matured base.

[BielBdeLuna]

that's the reason behind adding this to OTE to get all those properties that do not exist in md5.

you've scrapped the scripting language for a c99 implementation? great! we're planing a lot of new features on the OTE camp also.

and about graphics and speed yes indeed, we also plan big changes on them too.

[RobertBeckebans]

BTW if you want OpenGex support you can also try the Tombstone engine which has been released as version 1.0 this week. OpenGex was developed for it. Tombstone comes with full source code and its code is as clean as the Doom 3 code but it has a shader editor that is almost as powerful as the UE4 material editor. It also has a useful World Editor compared to id Tech 4.

[motorsep]

What BFG engine really needs (to make it useful to indies) is improvements in the tools, some new tools, improved performance for scripts (hell, Blueprint in UE4 is much faster and it's a script too), more C++ stuff exposed to DoomScript, 100% working multiplayer, documentation and starter assets / examples.

I am 100% certain that providing what I outlined above will spark more work done with the engine than chewing same old story about a need in better model formats and better graphics.

Note that I am not talking about RBD3BFG, I am talking about forks aimed at providing standalone game engine for indie game dev.

[BielBdeLuna]

we are already working on that too, but works takes time

[BielBdeLuna]

at the moment there is working ceGUI but the scripts for ceGUI for the menus and the HUD and others need to be created, and there is also bringing back the original GUI. As well as finding a way to restore the tools with a cross compile graphics library ( probably imGUI ). Then there is also the creation of the assets needed for a minimum working base. And lots of improvements to game code.

[motorsep]

@Illasera I am not working on anything using idTech 4 / BFG at this time. I was working on a commercial game using our own fork, but that didn't pan out and I released the engine (and Blender tools): https://github.com/motorsep/StormEngine2 hoping that someone might pick it up and continue improving it.

I personally (and from my hands-on experience in the industry) see OTE's team approach as faulty. With lack of resourced and tiny pool of talent they are trying to make a crossplatform engine based on RBDoom3BFG. A pool of people who will use that engine is so small, that it doesn't even make sense to do what OTE team is doing. Our engine (Storm Engine 2) runs on Windows/Linux, with tools working on Windows only. Since statistically there are more devs on Windows, chances that someone will pickup and work with Storm Engine 2 are greater than with OTE, since Storm Engine 2 just works and lets you build Windows/Linux FPS game.

And that's why I keep saying - get Storm Engine 2 and roll with it. Once you have someone making and releasing commercial game with it (hell, even a free but complete game), you get some PR which would allow you to do something like Patreon going and get funding to move tools from MFC to Qt, making toolset cross platform.

If users come first, then FOSS philosophy shouldn't blind you to the fact that devs want to work with stable engine with decent tools, docs and examples. Even if engine has limited features.

Anyway, going into offtopic again.

Btw, there is fhDoom https://github.com/eXistence/fhDOOM/ which is good old Doom with fancy rendering features. Also in the state with no active development going and author not willing to go beyond his own scope of tasks.

That's the thing with all these forks. Everyone is doing it for their own reason. fhDoom has own project that is powered by their fork. RBDoom3 is to power Doom 3 BFG with best visuals possible. StormEngine 2 is there because I ran out of funds and developers (essentially shutting down my project), OTE is there for ambitions of its team.

None of the projects/forks out there is aimed at "let's make robust engine devs can use to build FPS games". Probably for a good reason - can't really compete or even compliment UE4 / Unity / CE (although I do miss some aspects of working with idTech 4, including performance on my old PC).

[BielBdeLuna]

Why limited to FPS, the engine is more capable than just FPS. But yeah, go ahead, use StormEngine 2 ( it is a great engine ) Motorsep will support you on his engine.

[motorsep]

@BielBdeLuna Because FPS is what Doom 3 engine was made for. It's not suited for any other types of games. Hell, even FPS capacity is crippled to Doom 3 feature set.

So instead of sitting on the clouds never making any significant progress with OTE, why don't you guys make something feasible, within realistic reach and then expand on it?! Reminds me so much of Star Citizen game.

And no, I am not gonna support anyone using our engine (maybe with a word of advice only). As a matter of fact, one can grab it as-is, make and release a game with it (FPS). Everything is in there and working.

[RobertBeckebans]

@Illasera actually I'm also surprised how many work on forks of D3 BFG that I wasn't aware of. I don't have big plans for this engine at the moment. I want to finish and stabilize the SSAO and Screen Space Global Illumination and expose the new renderer features like HDR to the graphics menu. I still have problems with the bileratal filtering there. I'm also interested in adding Screen Space Reflections to mix them with the environment probes. That's pretty much it at the moment. It's not much work. I have another fork of the old D3 with the BFG renderer however I didn't upload it because I simply don't want to support it because of lack of time. Otherwise I released almost everything BFG based to the public and I think it was a good move by Motorsep to share the Steel Storm 2 engine. It has plenty cool additions although it is based on an older version of RBDOOM-3-BFG. I'm not further interested in putting hundreds to thousands of unpaid hours into this stuff. The latest renderer features of my fork where developed when I was on vacation or when I had double check some algorithms that I implemented in another engine.

I have been working full time on another project based on the C4 engine for several years and I mastered its renderer internals on a similar grade like D3. Right now, I'm evaluating Tombstone (the successor) for commercial purposes.

D3 BFG is nice. It has a fast renderer, rock solid game code that runs in separate thread and superb shooter input feedback. It just feels right. You can do pretty much everything D3 and Quake 4 has to offer and if you need anything game logic wise then it is very likely that you can find it int the D3 or Quake 4 game code SDKs. D3 and Quake 4 are great for learning how stuff can be done in a shooter game. D3 BFG even has its own physics engine that supports rag-dolls and inverse kinematics. And with my renderer goodies like PBR texture support you can give your assets a look that won't look bad compared to UE4 or Unity. If you have performance problems with the Doom scripting system then I can only give you the advice to solve it like Quake 4 and Rage have done it: Move the AI and weapon code to C++. John Carmack admitted that the scripting stuff was overused in D3.

However the tools really suck and only Motorsep's team tried hard to fix it. The D3Radiant is pure crap for me and the DarkRadiant is worse IMHO. I added Collada support because all .ase or .lwo Blender exporters where broken or unsupported on that time. Even if I finish my direct Blender JSON map exporter which already works to a certain degree ... then I still have to deal with all flaws of Blender to make more complex levels. Having complex Blender scenes with linking into other .blend files does not work too well and you have to consider things like missing texture compression.

On contrary Tombstone allows me to do every shit I want with working tools and it has support for asynchronous and recursive loading of worlds (AWESOME). Something even UE4 doesn't support properly. Its horror shooter sample game code even features most things D3 has to offer including a rock solid save game system. Basic stuff that you have to implement yourself with UE4.

I think it is the best that everybody releases their stuff in early stages or at least present it and share ideas or project goals so we can benefit from each other.

[wtfbbqhax]

I've only very recently decided that I would be using DOOM-3-BFG for the purpose of re-writing an Quake 3 mod with a community who still continues to make assets.

Regarding the #344 OP of adding OpenGEX, the answer is really obviously that Doom3 needs an pluggable abstraction for model formats. Similarly, I will need to add md3 and iqm support but would additionally look at other formats for which assets are commonly distributed.

@motorsep

That's the thing with all these forks. Everyone is doing it for their own reason. fhDoom has own project that is powered by their fork. RBDoom3 is to power Doom 3 BFG with best visuals possible. StormEngine 2 is there because I ran out of funds and developers (essentially shutting down my project), OTE is there for ambitions of its team.

It looks like you've dropped the entire git-history of your engine. This means it cannot be merged with any other DOOM-3-BFG fork. If possible, you should fix this.

What BFG engine really needs (to make it useful to indies) is improvements in the tools, some new tools, improved performance for scripts (hell, Blueprint in UE4 is much faster and it's a script too), more C++ stuff exposed to DoomScript, 100% working multiplayer, documentation and starter assets / examples.

+1, blueprint editor is cool.

There are many of these visual programming styled engines popping up.

I disagree with DoomScript, as well as every other proprietary language ID Software keeps insisting on. Replacing these with big-boy languages which just work, have large community module index at hand all maintained outside the engine is the only way to go anymore.

If it were me LuaJit dll + a modified luarocks for installing packages into game's paths then sprinkle in some in-game lua editor/console.

[wtfbbqhax]

Everyone seems to have their own disjoint idea's and needs. Bringing it all together in a non-biased forum might actually help the scene grow.

Does anyone disagree with this?

[motorsep]

I think what needs to happen is someone needs to get one fork as foundation and merge good stuff from other forks making Doom 3 BFG 100% moddable ( even if tools are Windows-only), decent looking and well performing ob Windows and Linux.

Thinking that BFG engine (any fork, or all forks put together) can beat commercial engines out there when it comes to serious development is naive and plain silly. That's actually the biggest reason people switched from Quake ports and Doom 3 ports to UE4/Unity/CE/Tombstone/you-name-it.

[BielBdeLuna]

no one tries to beat anything here

[motorsep]

@Illasera

That's where you are wrong. Why on earth remove TGA support?! That's why BFG (and UE4, etc.) still support TGA as any image editor can save into TGA. Engine itself compresses it into whatever appropriate format (one of the DXT formats) and saves into .bimage container. Artists don't have to worry about messing with comes compressing BS. So, TGA support stayed from Doom 3 and they added compression and binarization on top of that.

From what I gathered ASTC is only supported on mobile platforms (namely only on Samsung top models and some other few top models of some smartphones manufacturers). No desktop (OpenGL) game uses ASTC compression.

There isn't much you can replace in BFG engine with "better" algorithms. That's what I see often in FOSS community. id Software had a pool of talent who did their research in the field, and then on top of that they did field tests and found most optimal way of doing things. Especially on rendering side.

I had a privilege to work with ex-iD and ex-HumanHead programmers on Storm Engine 2 and it was clear that these ideas of ripping parts of the engine and replacing them with something else was not trivial thing to do. As a matter of fact I was advised either live with what engine is (and improve/fix what was achievable to improve and not go into such depth as replacing physics with Bullet, or adding bunch of rendering stuff as it's not feasible to achieve in the scope of time and resources that were available) or switch to Unity / UE4 while we are not too deep into development.

Even implementing stuff from whitepapers available online is not going to yield desired performance because whitepapers are not final implementations. That's for the subjects of performance of many forks with improved graphics vs Crysis 3 or DOOM (2016). No one will share exact optimizations and algorithms that yield high performance rendering and physics.

That's why I keep bumping heads with OTE team because their approach is not only impractical (as well as removing things and replacing them), but will result if failure to deliver robust FOSS engine.

As for Flash, why replace what's not broken and that performs extremely well? I released source SWF files covering not only HUD, but also all menus and PDA. SWF files mostly contain graphics, and functionality is coded in C++ in the engine.

[BielBdeLuna]

@motorsep but this approach is based on the assumption that you would be wasting money implementing this or that, money that would be better inverted on a more modern engine in the creation of a product, but if the final intention is to have a FOSS engine developed by a community of people, that advise looses significance, because, yes one individual has a gigantic task ahead, but a whole community can deliver at their pace, the changes they want to add, isn't it? besides, why bump any heads on other's projects just out of good will? In OTE we've implemented ceGUI, but we still have SWF, and in time I'd like to implement doom3 GUI system back but with the c++ interactions from SWF, so what if I want to loose my time on that endeavour? it's not like I'm indebted to release something to any specified time-line, isn't it?

[wtfbbqhax]

@motorsep

think what needs to happen is someone needs to get one fork as foundation and merge good stuff from other forks making Doom 3 BFG 100% moddable ( even if tools are Windows-only), decent looking and well performing ob Windows and Linux.

@Illasera

Well , I would love that, We all have different takes on the engine, See if maybe we can come to an agreement on some things to replace the main pipeline functionality with, But rest assure, the forks will continue as i stated earlier, To each his/her own direction, And creating one bloated engine is a bad idea. I think a step in the right direction will be REPLACING existing things with improved algorithm, I know it sounds generic but its better to improve/replace than adding completely new functionality IN ADDITION to the old stuff.

Good point(s).

I'm not saying we need yet another fork- I'm saying that an agreed place to communicate ideas, or new projects, and to ask questions is missing.

Forks should be encouraged, and git makes it easy enough to pick and choose which branches/commits you want from anyone else. Knowing where to look is the difficult part.

[BielBdeLuna]

what about using the already existing idTech forums?

[motorsep]

lol, after all that has been said and brought in as an example, is there still anything else to discuss? o.O

[coldtobi]

@motorsep (as StormEngine has no issue tracker enabled apperanty) Would it be possible for you to relicese the assets to a free license? The CC one isn't as it does not allow modification and it is also not free because of prohibites commercial use... With that, as I understood it, those assets would be a great startting point of creating real free games, aren't they? (Sorry for hi-jacking here, but this thread brought my attention to the Storm Engine 2)

[motorsep]

@coldtobi Unfortunately not. There aren't too many assets and you can re-create them from scratch by simply looking at them (and ascii defs can be copied/pasted as there is no way to say whether you made them from scratch or copied them :P ).

The only thing that is complex is SWF files, but yet again, you can re-create them from scratch in no time given that the source is readily available.

[bryansee]

@RobertBeckebans I have started work on OpenGEX on my Extreme-RBDOOM-3-BFG fork, I've implemented also the OpenDDL, although it's in preliminary stage.

[RobertBeckebans]

There has been a lot of discussion on the id Tech 4 Discord recently about how to get models into Doom 3. The IcedTech port has a branch that supports a .fbx -> .md6mesh/.md6anim workflow through assimp that I will adopt. I also want to note that I added .dae support in 2012 to get models from Blender directly to work with RBDOOM-3-BFG for static models. I had to discover recently that this code still works perfectly fine in 2020. If you are interested in an updated MD5 script for Blender 2.8 then I can recommend the script by KozGit: https://github.com/KozGit/Blender-2.8-MD5-import-export-addon