pk3scan integration

[Mark90]

Ramblings of something I've worked on in the past, around January '15. Gonna continue where I left off, most of the text below is still fairly accurate.

The goal of pk3scan integration is to check pk3s on missing textures, name collisions of shaders/textures with other pk3s, check for weapons/start-stop timer on the maps, etc.

1. On the main server (which hosts database & website) there is already /home/warsow/racesow/maps with more than 3000 .pk3 files. Therefore we dont have to actually upload maps, instead we create a database which maintains knowledge/state of each map. Then we can simply flip a boolean to 'publish' maps once they have been approved. This should be accessible on a webpage in the admin interface.
2. The admin webpage should show:
   • maps approved (can be played on racesow gameservers)
   • maps in acceptance phase (can be played on a private map-acceptance gameserver)
     • Should display map statistics (shaders/textures missing, shaders/textures redefined, see 3)
   • maps in /home/warsow/racesow/maps that are new (not yet tested)
     • This should show .bsp names instead of .pk3 names
     • If a .bsp is contained in a .pk3 with multiple maps, all other .bsp should be moved to acceptance phase as well. All of them should be approved before making the .pk3 playable.
   • Overview of shaders/textures missing (so that texture packs can be assembled)
3. The admin webpage should enable admins to perform the following actions:
   • Approve a map -> Removes map from acceptance, sets map to 'approved'. (include admin id) -> Make map viewable on website, downloadable by users/servers -> Extract levelshot to ~/racesow/levelshots, track path in database.levelshots (see 4 for database)
   • Reject a map -> Removes map from acceptance, sets map to 'rejected' (include admin id + reason for rejecting)
   • Add a map -> Puts map in acceptance phase. This should parse the .pk3 file for:
     • redefined shader names (e.g. overriding data.pk3, or previously uploaded maps)
     • redefined texture files (e.g. overriding data.pk3, possibly check md5 hash)
     • redefined shader-texture dependencies (breaks the map that comes first in the alphabet)
     • missing shaders/textures
   • Upload texture_pure.pk3 or data_15_pure.pk3 stuff. Process like a regular map, but set 'pure' true in database.pk3files
4. The database tables on the main server:
   • pk3files (unique column filename, column 'md5hash', int column 'status' for approved/rejected/acceptance/new, boolean column 'badformat' for invalid pk3 file format, boolean column 'base' for data_15_pure.pk3/texture_pure.pk3, fk 'replaces' link to rejected pk3file)
   • bspfiles (fk 'pk3file', column 'filename', column md5hash)
   • levelshots (fk 'pk3file', column 'filename')
   • shaders (fk 'pk3file', column 'shadername') <- overview of shader definitions, makes it possible to detect collisions
   • textures (fk 'pk3file', column 'filename', unique column 'md5hash')
   • bsp_shaders (fk 'bspfile', fk 'shader') <- M2M for field shaders on 'bspfile'
   • shader_textures (fk 'shader', fk 'texture') <- M2M for field textures on 'shader'
5. The acceptance gameserver:
   • The API should ignore statistics reported by this server (player playtime, map playtime, racetimes)
   • Map coldrun if there are no unapproved maps
   • callvote map/maplist should return maps added for acceptance
6. The main server:
   • When the backend starts, it should check /home/warsow/racesow/maps for pk3's that are not yet in database.pk3files and add them with status 'new'
     • Validate pk3 zipfile format
     • Add new maps to database.pk3files
     • Search pk3 for .bsp files and add these to database.bspfiles
   • Install filelistener for /home/warsow/racesow/maps ? How to rsync approved pk3s from mainserver to gameservers without duplicating pk3's?

[Mark90]

After long thought, detecting shader conflicts among maps requires too much work which won't pay itself back. First of all, countless false errors will arise for "-wswfix" maps and newer versions of existing .bsp (as regulated through the pk3's version number). Second, upon finding an actual incorrect shader redefinition it is possible that the shader is actually the same (which will have to be verified manually), meaning there may not be a problem at all. And if there IS a difference between them, then there's the task of deciding which one should take precedence.

The functionality that will be implemented anyway is checking pk3's for missing textures/shaders, combined with a lookup of texture/data pk3's providing them.