Tremulous compatibility

[illwieckz]

On the topic on Trem compatibility. Trem compatibility is very much needed a very much lot (emphasis is on purpose). I will favor any PR on that purpose.

We may only miss 2 or 3 features to load unmodified pk3 tremulous map:

• [ ] a way to automatically load an extra pak if the pak is a pk3, something like fs_extralegacypaks that would work like fs_extrapaks but only if the pak is a pk3, one would just set this cvar to res-tremulous and enable legacy paks and then it's done, people can upload pk3 to their unvanquished server.
• [ ] a better implementation of https://github.com/Unvanquished/Unvanquished/pull/1512 .
• [ ] maybe the same for .arena if we start to parse it for the map loading screen, no need today as the only code parsing it was removed.
• [x] in the future it is expected the game will produce the navmesh itself, so there would no need to repack maps for navmeshes, we already have that code ( https://github.com/Unvanquished/Unvanquished/pull/1646 ) but it is disabled by default because it is not ready for production use yet.

The minimap would be missing but we can live with it.

It's probably very possible to achieve unmodified tremulous pk3 loading for 0.54.

Many huge efforts were done that makes it possible (it also helped many other purposes, for example the dpk renaming made easier to implement our vfs in NetRadiant):

• [x] some work by TimePath, like fs_legacypaks if I'm right, initially purposed for Xonotic but very reusable for legacy Tremulous maps.
• [x] re-enable the ability to load not versionned pak (pk3) and versionned paks (dpk): https://github.com/DaemonEngine/Daemon/pull/14
• [x] properly handle autodownload of not-versionned pak: https://github.com/DaemonEngine/Daemon/pull/14
• [x] many renderer fixes and material parsing rewrite I done like https://github.com/DaemonEngine/Daemon/pull/184, there are still some edge cases where our engine is not 100% compatible with q3 behaviours but for at least one of the remaining incompatibility that's because our engine does not have the same interpretation of an undefined behavior when the mapper is doing a mistake, reaching such compatibility level is already an huge achievement.

Note that I don't mind that much if tremulous unmodified pk3 are not listed in client server list if at least server owners can make them playable just by uploading a pk3 to a server but that probably already works anyway.

I consider that in a perfect world there are only two meaningful usages of legacy maps.

• loaded unmodified without any effort done,
• first class ports like InterstellarOasis were maps may even be recompiled with newer compiler features, etc.

The half-ported maps like @bmorel and Sweet do are very useful and I'm very grateful to them because it provides map now when the people need them. This was and it is the best thing to do to take care of our tiny player base and community. We're not living in a perfect world so doing that is really good. 👍️ 👏️🤝️👌️🤗️💯🥇🥈🥉🏅🎖️

But at the same time I'm super sad at seeing them spending so much time just at renaming files and doing things they would not have to do if we properly supported legacy maps out of the box. I consider we waste their precious time and I consider myself very responsible at not providing them a game that does not require them wasting their time to load a legacy map.

That's why I favor efforts to make unmodified legacy map loading a reality (like automatic repeater replacement with drills https://github.com/Unvanquished/Unvanquished/pull/1953 ).

[DolceTriade]

Imo, All of this compatibility is just not worth having in the engine. It would be better to create a standalone tool that is capable of re-arranging a trem map to better map to Unvanquished. I like the idea of creating a tremcompat pk3 for commonly referenced textures and shaders and other misc files, but I don't believe we should litter the Unvanquished code base with more tech debt. It's bad enough as it is :(

It seems as if if you have to download a pk3 and copy it to the correct location anyways, it shouldn't be terribly difficult to run an additional step ./fixupmap.sh or something.

[ghost]

but I don't believe we should litter the Unvanquished code base with more tech debt.

Well, i'd say that many of the tech debt we have actually comes from fresher code than trem, which was never finished. slipher did a great job at finishing or continuing several of those things (rmlui, despite still not at last version, getting rid of old cvar stuff) but there's also the C++ Vec3 API (that I have fixed in my local glm move) and CBSE, notably (damage entities, and probably other stuff as well).

What is certain is, it's much easier to read and understand trem's map-entity system than unvanquished's, so I can't help but smile when I read that trem support is a cause for tech debt. In any case, and as I said, I'm ok if that code does not go in unv's codebase. It will only mean that instances which support the many trem maps as much as possible will have to be a fork. With current development rate of unv, I have to say that maintaining a fork is even not that hard. As long as one does not try too hard to get things merged, at least, because this implies maintenance of many small branches for months which is overall a huge pain.

Not sure it it's worth mentioning it, but getting a saner sgame codebase is one of my personal goals. Ideally, when the work will be done, I'd like to take time to actually write some documentation, which is another thing I feel is lacking more in unv than in trem (this is a general feeling, but I might be wrong: neighbor's grass is always greener, after all).

[illwieckz]

Having the engine and the game code more compatible with legacy assets makes the idea of porting other Q3 mods more appealing. Most of the “legacy way” of doing things were shared among Q3 mods, loading old .arena path and old levelshot path would just enables the compatibility for loading assets for a handful of mods. For example with my WIP branch to load old Tremulous levelshot, Q3Rally levelshot displays properly.

This contributes positively to the opinion that it may be a good idea to port other games on Dæmon and rebase gamecode over Unvanquished.

Right now the experience is “meh, they broke compatibility, I don't even know if it's doable to bring back it and how much hard it would be, better look at something else”.

[illwieckz]

And for the complexity of the compat mode, what's missing is basically:

if exists(meta/castle/castle.tga)
    load(meta/castle/castle.tga)
else if exists(levelshots/castle.tga)
    load(levelshots/castle.tga)
endif

if exists(meta/castle/castle.arena)
    load(meta/castle/castle.arena)
else if exists(scripts/castle.arena)
    load(scripts/castle.arena)
endif

Server owners can already enable legacy paks with fs_legacypaks and set res-tremulous as fs_extrapaks. The idea of fs_extralegacypaks is just to not uselessly load res-tremulous on non-legacy maps.

[slipher]

I think a conversion script could handle a lot of trivial tasks more conveniently than the engine/gamelogic:

• Rename the BSP if necessary
• Move the levelshot
• Convert repeaters in the layout to drills (or just delete them)
• Add a DEPS file referencing a Tremulous pak
• Fix case sensitivity issues

For more complex things it may make sense to implement built-in compatibility. I think the q3 shader support in the engine makes sense. Compatibility with Tremulous map entities could go either way (not sure if those things have a trivial rewriting or not).

[illwieckz]

Convert repeaters in the layout to drills (or just delete them)

This is not trivial to do outside the game, though esquirel (part of Urcheon) was written to make it possible (then near-to-trivial). But anyway, I have wrote many scripts to modify maps and this contributes to my opinion it's not the good way to do things. In fact trivial things are better done in game if trivial, and non-trivial things (like entities) may be easier to do in game, and may even become trivial to do in game.

No one game will ever port to Dæmon if we say they have to mass-repackage their thousands of contributed maps, not saying that can be a legal grey area. I cannot see how “it's better to mass-repackage”, and this is confirmed by my experience of writing converters and editing tools since years.

-rwxrwxr-x 1 illwieckz illwieckz  184 2015-02-26 20:14:05.009774519 +0000 add-deps
-rwxrwxr-x 1 illwieckz illwieckz  205 2015-02-27 21:07:31.533367940 +0000 asset_convertor
-rwxrwxr-x 1 illwieckz illwieckz  164 2015-02-24 21:53:22.754333545 +0000 build_arena
-rwxrwxr-x 1 illwieckz illwieckz  206 2015-02-25 02:20:47.006289022 +0000 build_mapstuff
-rwxrwxr-x 1 illwieckz illwieckz 1471 2015-02-26 19:47:20.173527359 +0000 create_fixes
-rwxrwxr-x 1 illwieckz illwieckz 2617 2015-03-01 03:44:31.131628161 +0000 daemonmap_helper
-rwxrwxr-x 1 illwieckz illwieckz  113 2015-02-27 03:34:37.312445225 +0000 drop_ext_shader
-rwxrwxr-x 1 illwieckz illwieckz   76 2015-06-20 21:28:38.571442541 +0000 dump_lm
-rwxrwxr-x 1 illwieckz illwieckz  410 2015-02-25 18:52:52.444454363 +0000 file_unicity
-rwxrwxr-x 1 illwieckz illwieckz  662 2015-03-04 02:42:00.832778213 +0000 filter
-rwxrwxr-x 1 illwieckz illwieckz  125 2015-03-04 21:09:29.788187808 +0000 find_obsolete_entities
-rwxrwxr-x 1 illwieckz illwieckz   93 2015-03-03 00:54:24.969295042 +0000 fix_entities
-rwxrwxr-x 1 illwieckz illwieckz   67 2015-02-25 21:40:18.440246438 +0000 fix_perms
-rwxrwxr-x 1 illwieckz illwieckz   88 2015-03-03 00:27:44.345799918 +0000 fix_weapons
-rwxrwxr-x 1 illwieckz illwieckz  159 2015-02-25 21:48:08.757999966 +0000 get_dir
-rwxrwxr-x 1 illwieckz illwieckz  296 2015-02-24 21:36:42.821872827 +0000 get_longnames
-rwxrwxr-x 1 illwieckz illwieckz   97 2015-02-24 21:47:55.362286993 +0000 get_mapdir
-rwxrwxr-x 1 illwieckz illwieckz   66 2015-02-24 21:21:32.274527856 +0000 list_maps
-rwxrwxr-x 1 illwieckz illwieckz   66 2015-03-02 19:41:44.570163917 +0000 list_res
-rwxrwxr-x 1 illwieckz illwieckz  123 2015-03-01 10:43:13.586354126 +0000 list_trem_textures
-rwxrwxr-x 1 illwieckz illwieckz  369 2015-02-25 19:07:54.375705912 +0000 list_unique_textures
-rwxrwxr-x 1 illwieckz illwieckz  167 2015-03-06 03:31:31.548323946 +0000 list_versions
-rwxrwxr-x 1 illwieckz illwieckz  190 2015-02-16 02:56:42.498714727 +0000 make_weblist
-rwxrwxr-x 1 illwieckz illwieckz  412 2015-06-20 21:18:51.408944216 +0000 migrate_to_bspdir
-rwxrwxr-x 1 illwieckz illwieckz 2406 2015-03-05 05:40:20.799955833 +0000 mk_build
-rwxrwxr-x 1 illwieckz illwieckz 5886 2015-03-05 19:34:06.254211939 +0000 mk_release
-rwxrwxr-x 1 illwieckz illwieckz  372 2015-02-26 03:07:43.013022408 +0000 patch_pk3_textures
-rwxrwxr-x 1 illwieckz illwieckz  218 2015-02-26 22:05:33.546093910 +0000 patch_pk3s
-rwxrwxr-x 1 illwieckz illwieckz  212 2015-02-26 03:26:16.106643226 +0000 recup_shaders
-rwxrwxr-x 1 illwieckz illwieckz  147 2015-03-01 11:50:28.429220847 +0000 refix_textures
-rwxrwxr-x 1 illwieckz illwieckz  221 2015-03-01 08:20:20.967055658 +0000 sanitize
-rwxrwxr-x 1 illwieckz illwieckz  290 2015-02-26 00:28:01.859154188 +0000 unpack_bsp
-rwxrwxr-x 1 illwieckz illwieckz 1181 2015-02-25 19:37:31.413817686 +0000 write_texture_replacement

Those are some scripts I wrote in 2015 (more than seven years ago).

I just stopped doing this in 2015 and I don't want to come back at it.The only local quick&dirty script I maintain is a script to import a bunch of pk3 into a git history when I want to do a “first class port” with InterstellarOasis.

This is not a road to success.

The only script I would consider to mass repack .pk3 is if we want to optimize them, that means doing things like:

• convert sounds to opus
• crunch what can be crunched
• export internal lightmaps so they can be converted to webp
• deal with internal lightmaps and external light style lightmaps conflict to make them all external and optimize them all
• fix model textures in bsp if needed
• fix bad separators like \ instead of /
• fix bad legacy common shaders that may impact navmesh generation
• deduplicate images
• etc.

But for someone not wanting to do that, it's better to load the pk3 as is.

And taking care of levelshot alternative path and replacing repeater by drill is far better done in game, really.

In fact even deciding the pakname isn't a trivial task to begin with and one cannot blindly do it on a mass-repackage run.

Just take this example:

map-bluedragon-1.1.0.pk3
map-bluedragon-b1-1.1.0.pk3
map-bluedragon-b2-1.1.0.pk3
map-bluedragon-b3-1.1.0.pk3
map-bluedragon-b4-1.1.0.pk3
map-bluedragon2_b1.pk3
map-bluedragon2_b2.pk3
map-bluedragon2_b3.pk3

You have to actually open a bunch of those map versions in game to understand that both b4-1.1.0 and 2_b3 are version strings and the name is bluedragon.

But then take this:

map-arachnid2-1.0.0.pk3
map-arachnid2-1.0.0.pk3
map-arachnid2-1.1.0.pk3

You have to know Tremulous history to know that arachnid2 is the map name and 2-1.1.0 is not the version. Because arachnid2 is a new map by OverFlow for Tremulous done in homage to the existing arachnid map done by Jex for Gloom. And arachnid2 isn't the 2nd version of Jex's map.

Also don't forget that part of those version strings can be baked in bsp name, so if one wants to properly name the map, he has to rename the bsp, rename the external lightmap directory, etc. In some situations it may even be needed to edit the q3map2_*.shader file if exists and the bsp itself to rename some paths in it.

[illwieckz]

Also, this is a side note, but usually taking care of compatibility with different assets helps to get a better design that benefits to anyone, including the core product.