[TF2-Beta] [Dedicated-Server] Audio files specified in .RES file not downloaded

Mecha-Weasel commented 11 years ago

I happened to load-up a custom map into a TF2-Beta dedicated-server. The custom .BSP file downloaded fine. SRCDS kind-of followed what was in the .RES file with the matching name. It downloaded the .TXT file specified in the .RES file. It downloaded the .RES file itself. However, it didn't download any of the .MP3 files that were also specified in the .RES file. Inside the .RES file the .TXT file and the .RES file itself are prefixed with "maps/". Inside the .RES file the .MP3 files are prefixed with "sound/".

If I manually copy the audio files into the proper folder in my client (C:\Program Files (x86)\Steam\steamapps\common\Team Fortress 2 Beta\tf\download\sound), everything works as expected - for me (obviously other players don't auto-download the required files).

This same custom map, .RES file, etc. works fine on non-Beta TF2 server.

matthewsor commented 11 years ago

We'll need the BSP file (and anything else it needs) so we can reproduce this.

zpostfacto commented 11 years ago

In addition to the .res file in question, can you confirm the client setting for cl_downloadfilter?

Mecha-Weasel commented 11 years ago

Sorry for late follow-up. This is the map from Game-Banana:

http://tf2.gamebanana.com/maps/154597

As far as the client-side "cl_downloadfilter" setting goes, it's set to "all".

Mecha-Weasel commented 11 years ago

PS: I can push-down the files using a SourceMod plugin on the TF2 beta, but the built-in functionality (sans-SM) didn't download the audio files listed in the matching .RES file.

zpostfacto commented 11 years ago

I was not able to reproduce this on a Windows dedicated server. Can you provide some more details. What OS's were your client and server? If you lowercase all the filenames, does it work?

zpostfacto commented 11 years ago

I tried it on a linux server and wasn't able to reproduce either. I have not tried a linux client, that might be it.

If Linux is involved, another possibility is that we are getting bitten by case sensitivity. (I notice that the files in the res list have mixed case.) Part of the filesystem changes in this build was to remove the automatic case fixing on linux (which was deathly slow). Now we assume that any file that is opened through our game's filesystem layer will be all lowercase if it exists loose on disk. It looks like the download functions are case preserving for the most part, so if your res list has the proper case, the downloading will work. But then when the client goes to access that file through the normal file access calls, it won't find it if the file was saved with mixed case.

That means pretty much all files in your game folder should be lowercase.

One more question: Are you using a fastdl server? Exactly what the consequences are for FastDL servers and the case mangling, that's going to depend on the case handling of the HTTP server you are using.

Mecha-Weasel commented 11 years ago

Sorry for the late reply. Yes, I was/am using FastDL path (i.e. URL to grab files from a web-site instead of in-game protocol download). Yes, hosting on Linux (Debian 6, 64-bit). Yes, I believe the file names of those resources may be mixed-case. I will verify and let you know.

On Tue, 2013/03/26 11:20 AM, Fletcher Dunn <notifications@github.com> wrote:

One more question: Are you using a fastdl server?

If you are using Linux, I have a pretty good feeling that we are getting bitten by case sensitivity. Part of the filesystem changes in this build was to remove the automatic case fixing on linux (which was deathly slow). Now we assume that any file that is opened through our game's filesystem layer will be all lowercase if it exists loose on disk..

That means pretty much all files in your game folder should be lowercase.

Now, exactly what the consequences are for FastDL servers, that's going to depend on the case handling of the HTTP server you are using.

Case sensitivity is from the devil. There has never been any reason in its favor that comes even remotely close to outweighting all the problems it causes.

Reply to this email directly or view it on GitHub: https://github.com/ValveSoftware/Source-1-Games/issues/194#issuecomment-15476717

Mecha-Weasel commented 11 years ago

The files are listed in mixed-case in the RES file, and they match exactly - both in the file system on the game server, and on the web-server where the FastDL URL points-to.

If it helps for testing, I can rename everything lower-case on the game server and the web-server.

But, would the references to them inside the BSP need to be updated to be all lower-case also? If so, for any map authors inadvertently using mixed-case or upper-case (including prior to now) ... ouch.

For this particular odd-ball map, I suppose I can bust-out EntSpy and hack that to work if needed - since its a map that's pretty-much abandoned by whoever initially made it, no big deal. I am more worried about other after-market/custom maps.

Although, I suppose many just compile everything directly into the BSP these days - and don't mess with RES files anymore anyway. I guess bandwidth is nearly infinite now anyway - compared to "back in the day".

Mecha-Weasel commented 11 years ago

PS: Totally agree on the case-sensitivity thing BTW. I am waiting for some variant of EXT file system to be adopted by the Linux kernel - that is case-preserving but isn't case sensitive (a'la NTFS). But, it's a religious issue. So, not holding my breath (any more).

zpostfacto commented 11 years ago

Welp, I am unable to reproduce the problem, and you are able to reproduce it. So can you please describe your server and client situations in more detail?

Exactly where did you extract the zip to on your server?
Are the filenames exactly as they appear in that .zip?
What OS is the client running?
What exactly does the client do when it fails to download the file?

Do you have a server setup with a public IP that I can try to connect to? (Please update your server to the one just released.

Regarding file references within .bsp files. We have all sorts of existing assets that we ship that contain references using mixed case, and it's not a problem. We lowercase the relative part of the filename before appending it to the search path and passing it on to the underlying filesystemn. So provided that the file is either in a VPK (which are not case sensitive), or is loose on disk lowercased, it shouldn't be a problem. That's how we handle all of our files, and it works great, without requiring us to search for files with all different combinations of uppercase and lowercase.

Mecha-Weasel commented 11 years ago

I haven't re-retried to "reproduce" the symptom lately. So, not sure if any later update fixed it or not. I can blow everything away and re-test if necessary. Regarding file paths, here's where everything is right now (before attempting to rename everything lower-case, etc.):

Here's what's in the RES file:

Resources { "maps/mario_kart_2012.res" file "maps/mario_kart_2012.txt" file "sound/mario_kart_2012/MarioKartBegin.mp3" file "sound/mario_kart_2012/BananaPhone-Raffi.mp3" file "sound/mario_kart_2012/CloseToTheEdit-ArtOfNoise.mp3" file "sound/mario_kart_2012/Hell-SquirrelNutZippers.mp3" file "sound/mario_kart_2012/KaosKiller-KapBambino.mp3" file "sound/mario_kart_2012/KinkoTheClown-OgdenEdsl.mp3" file "sound/mario_kart_2012/KnightRider-TVTheme.mp3" file "sound/mario_kart_2012/RobotRock-DaftPunk.mp3" file "sound/mario_kart_2012/Speeds-LainOST.mp3" file "sound/mario_kart_2012/StillAlive-EllenMcLain.mp3" file "sound/mario_kart_2012/WorkingMansTheme-LainOST.mp3" file }

Here's where things are in the file system on the Linux game server (including BZ2 files):

{INSTALL-DIRECTORY}/tf/maps/mario_kart_2012.bsp {INSTALL-DIRECTORY}/tf/maps/mario_kart_2012.bsp.bz2 {INSTALL-DIRECTORY}/tf/maps/mario_kart_2012.res {INSTALL-DIRECTORY}/tf/maps/mario_kart_2012.txt {INSTALL-DIRECTORY}/tf/sound/mario_kart_2012/MarioKartBegin.mp3 {INSTALL-DIRECTORY}/tf/sound/mario_kart_2012/BananaPhone-Raffi.mp3 {INSTALL-DIRECTORY}/tf/sound/mario_kart_2012/CloseToTheEdit-ArtOfNoise.mp3 {INSTALL-DIRECTORY}/tf/sound/mario_kart_2012/Hell-SquirrelNutZippers.mp3 {INSTALL-DIRECTORY}/tf/sound/mario_kart_2012/KaosKiller-KapBambino.mp3 {INSTALL-DIRECTORY}/tf/sound/mario_kart_2012/KinkoTheClown-OgdenEdsl.mp3 {INSTALL-DIRECTORY}/tf/sound/mario_kart_2012/KnightRider-TVTheme.mp3 {INSTALL-DIRECTORY}/tf/sound/mario_kart_2012/RobotRock-DaftPunk.mp3 {INSTALL-DIRECTORY}/tf/sound/mario_kart_2012/Speeds-LainOST.mp3 {INSTALL-DIRECTORY}/tf/sound/mario_kart_2012/StillAlive-EllenMcLain.mp3 {INSTALL-DIRECTORY}/tf/sound/mario_kart_2012/WorkingMansTheme-LainOST.mp3 {INSTALL-DIRECTORY}/tf/sound/mario_kart_2012/MarioKartBegin.mp3.bz2 {INSTALL-DIRECTORY}/tf/sound/mario_kart_2012/BananaPhone-Raffi.mp3.bz2 {INSTALL-DIRECTORY}/tf/sound/mario_kart_2012/CloseToTheEdit-ArtOfNoise.mp3.bz2 {INSTALL-DIRECTORY}/tf/sound/mario_kart_2012/Hell-SquirrelNutZippers.mp3.bz2 {INSTALL-DIRECTORY}/tf/sound/mario_kart_2012/KaosKiller-KapBambino.mp3.bz2 {INSTALL-DIRECTORY}/tf/sound/mario_kart_2012/KinkoTheClown-OgdenEdsl.mp3.bz2 {INSTALL-DIRECTORY}/tf/sound/mario_kart_2012/KnightRider-TVTheme.mp3.bz2 {INSTALL-DIRECTORY}/tf/sound/mario_kart_2012/RobotRock-DaftPunk.mp3.bz2 {INSTALL-DIRECTORY}/tf/sound/mario_kart_2012/Speeds-LainOST.mp3.bz2 {INSTALL-DIRECTORY}/tf/sound/mario_kart_2012/StillAlive-EllenMcLain.mp3.bz2 {INSTALL-DIRECTORY}/tf/sound/mario_kart_2012/WorkingMansTheme-LainOST.mp3.bz2

Here's where things are in the file system on the Apache web server (relative to URL): Note: Same web-server URL non-beta version of this same server uses fine:

{SV_DOWNLOADURL}/maps/mario_kart_2012.bsp {SV_DOWNLOADURL}/maps/mario_kart_2012.bsp.bz2 {SV_DOWNLOADURL}/maps/mario_kart_2012.res {SV_DOWNLOADURL}/maps/mario_kart_2012.txt {SV_DOWNLOADURL}/sound/mario_kart_2012/MarioKartBegin.mp3 {SV_DOWNLOADURL}/sound/mario_kart_2012/BananaPhone-Raffi.mp3 {SV_DOWNLOADURL}/sound/mario_kart_2012/CloseToTheEdit-ArtOfNoise.mp3 {SV_DOWNLOADURL}/sound/mario_kart_2012/Hell-SquirrelNutZippers.mp3 {SV_DOWNLOADURL}/sound/mario_kart_2012/KaosKiller-KapBambino.mp3 {SV_DOWNLOADURL}/sound/mario_kart_2012/KinkoTheClown-OgdenEdsl.mp3 {SV_DOWNLOADURL}/sound/mario_kart_2012/KnightRider-TVTheme.mp3 {SV_DOWNLOADURL}/sound/mario_kart_2012/RobotRock-DaftPunk.mp3 {SV_DOWNLOADURL}/sound/mario_kart_2012/Speeds-LainOST.mp3 {SV_DOWNLOADURL}/sound/mario_kart_2012/StillAlive-EllenMcLain.mp3 {SV_DOWNLOADURL}/sound/mario_kart_2012/WorkingMansTheme-LainOST.mp3 {SV_DOWNLOADURL}/sound/mario_kart_2012/MarioKartBegin.mp3.bz2 {SV_DOWNLOADURL}/sound/mario_kart_2012/BananaPhone-Raffi.mp3.bz2 {SV_DOWNLOADURL}/sound/mario_kart_2012/CloseToTheEdit-ArtOfNoise.mp3.bz2 {SV_DOWNLOADURL}/sound/mario_kart_2012/Hell-SquirrelNutZippers.mp3.bz2 {SV_DOWNLOADURL}/sound/mario_kart_2012/KaosKiller-KapBambino.mp3.bz2 {SV_DOWNLOADURL}/sound/mario_kart_2012/KinkoTheClown-OgdenEdsl.mp3.bz2 {SV_DOWNLOADURL}/sound/mario_kart_2012/KnightRider-TVTheme.mp3.bz2 {SV_DOWNLOADURL}/sound/mario_kart_2012/RobotRock-DaftPunk.mp3.bz2 {SV_DOWNLOADURL}/sound/mario_kart_2012/Speeds-LainOST.mp3.bz2 {SV_DOWNLOADURL}/sound/mario_kart_2012/StillAlive-EllenMcLain.mp3.bz2 {SV_DOWNLOADURL}/sound/mario_kart_2012/WorkingMansTheme-LainOST.mp3.bz2

Mecha-Weasel commented 11 years ago

Sorry missed this part:

"Do you have a server setup with a public IP that I can try to connect to? (Please update your server to the one just released."

Yes, but right now it's got SourceMod enabled on it, which is pushing things down fine. Let me know if you want to try hitting the beta server - I'll just disable SourceMod (probably MetaMod entirely) for the test. No big deal, that's what betas are for.

Mecha-Weasel commented 11 years ago

"What OS is the client running?"

Windws-7, 64-bit, Home-Premium.

"What exactly does the client do when it fails to download the file?"

Map loads, but music doesn't download, and when entities try to trigger the music to play, I think the console (of the client) logs that the file was not found. No crash or anything like that - sound just doesn't play.

Mecha-Weasel commented 11 years ago

Attempting to update to the latest BETA right now. Will leave MM:S & SM disabled for now, once I know its up and running (sans MM:S/SM) I will let you know connect info.

Mecha-Weasel commented 11 years ago

Will update tonight (after day job). Can't access server properly from my current location to update, disable MM:S/SM, etc.

Mecha-Weasel commented 11 years ago

Sorry, hit wrong-button. Didn't intend to "Close".

Mecha-Weasel commented 11 years ago

Looks like this might be resolved now. Maybe it was an issue with content download from the server prior to changing the download-folder structure conflicting? Here's what I did to test:

On the (Debian Linux) dedicated server:

Disabled SourceMod & MetaMod:Source on the (Linux) dedicated server.
Updated the (Linux) dedicated server to latest-release from today.
Started the server back up again.

On the (Windows-7) client computer:

Deleted the entire "C:\Program Files (x86)\Steam\steamapps\common\Team Fortress 2 Beta\tf\maps" folder from the (Windows) client machine.
Deleted the entire "C:\Program Files (x86)\Steam\steamapps\common\Team Fortress 2 Beta\tf\sounds" folder from the (Windows) client machine.
Signed-into Steam on the (Windows) client.
Let the TF2 beta (Windows) client-side update apply.
Started TF2 beta (Windows) client.
Connected to the (Linux) dedicated server.

Results on the (Windows-7) client computer:

The BSP file (within the BZ2) re-downloaded from the (Linux) dedicated server.
It took about 9 minutes to download just the map. Note: The BZ2 file is about 61-MB, the extracted BSP itself is about 89-MB.
I noticed that now the BSP, TXT and RES file went to the new "download" folder-structure on the client (C:\Program Files (x86)\Steam\steamapps\common\Team Fortress 2 Beta\tf\download\maps).
I noticed that the local copies (on the Windows client) are all now named in lower-case.
Correction: The BSP, TXT and RES always were lower-case, it was the sounds (MP3's) that were mixed-case.
All the other files specifided in the RES file (MP3's) came down auto-magically this time!
I noticed that now they also went to the new "download" folder-structure on the client (C:\Program Files (x86)\Steam\steamapps\common\Team * Fortress 2 Beta\tf\download\sound\whatever...)
I noticed that the local copies (on the Windows client) are all now named in lower-case.
All the sounds that were in the RES file got triggered as expected by the entities in the map!

Mecha-Weasel commented 11 years ago

PS: Server is still on-line. Currently still has MetaMod:Source & SourceMod disabled. Will probably re-enable those some time tomorrow evening to test-out any issues with them and the new build (not that any are expected).

Mecha-Weasel commented 11 years ago

Connection information if you still want to check-out the behavior is:

steam://connect/192.30.161.61:6901

But, it seems to be working as expected now. I will probably re-enable MM:S/SM later today.

ValveSoftware / Source-1-Games

[TF2-Beta] [Dedicated-Server] Audio files specified in .RES file not downloaded #194