About querying official Q3 master server

The-Gig commented 2 years ago

Hi, I noticed that in mods.ini you say that "masterServer defaults to master.ioquake3.org", because you are not able to get qstat to speak to master.quake3arena.com anymore, right?

I'm not used with qstat, anyway I did a few tries using the qstat.exe (2.11) which comes with MS: If I try qstat -q3m master.ioquake3.org:27950, I correctly get the ioquake3 server list... If I try qstat -q3m dpmaster.deathmask.net:27950, I correctly get OA server list... If I try qstat -q3m master.quake3arena.com:27950 or qstat -q3m master3.idsoftware.com:27950, I get "No response" message.

I asked on the OpenArena Discord #offtopic channel if someone had any idea about that, and Rdnt Cntrl (author of "Rat" OA mod) said that qstat sends getservers 68 empty full demo, and that way no response from those servers. getservers 68 empty full works, instead. getservers 68 empty demo works too. So it might just be the length of the query. He doesn't know whether it's possible to customize the query with qstat, so he made the test customizing an "oaquery" Python script he had, to make it work with q3. He posted the test script (https://cdn.discordapp.com/attachments/232325920822657024/1000064664966811648/oaquery-q3a.py; no warranty) and said ./oaquery-q3a.py -a should list all the non-empty servers.

I'm not used with python either, so I did instead try to see if I was able to understand how to customize qstat configuration (documentation http://manual.freeshell.org/qstat/qstatdoc.html) to modify the string, but I have not really been able to understand how to write it (I did a couple of tests, but probably I was doing it wrong). Probably you are more used with qstat config, do you want to give it a try?

I suppose it would be nice to be able to query the official q3 master server, as I guess the ioquake3 master server might be lacking some servers...

Side notes:

The server browser built in into Q3A (aka Multiplayer menu) works, and AFAIK it only queries master.quake3arena.com, hard-coded in the gui source code (afaik, the only way to make that server browser query another master server is to trick it with a DNS hack, like customizing your "hosts." file). Makes you wonder why they didn't allow it to query multiple master servers, considering servers have sv_master1 - sv_master5 customizable cvars to sent their heartbeat to up to 5 different master servers at once... Anyway, the point is that there should be some way to query the official server.
Considering the above, I have no idea why master3.idsoftware.com (currently 192.246.40.56) exists at all, considering the main server is "master.quake3arena.com" (currently 192.20.216.53).

torhus commented 2 years ago

As far as I can tell, the official Q3 masters are still down, as they have been for years. I can't get a response from them with that script either, just the OA servers. I don't have Q3 installed anymore, so I can't test that, but I don't see how it would work. The old master servers still have a DNS entry, but they don't respond to ping.

The-Gig commented 2 years ago

Wait, the fact that a server doesn't reply to PING requests doesn't necessarily imply the server is non-working, its firewall simply being configured to do not answer to ICMP requests is an explanation as probable as the other. As you can see from this screenshot taken from Process Monitor, as soon as I enter the "Multiplayer" screen, quake3 process sends and receives UDP packes with 192.20.216.53 (which is master.quake3arena.com) and then starts sending packets the various servers. udp

torhus commented 2 years ago

Alright, that is odd. Thanks for the information.

The-Gig commented 2 years ago

For the record, back in 2016 master.quake3arena.com really went down for some time, IDK if days, weeks or months (master3.idsoftware.com was working instead, and it was possible to use its IP with the hosts. file trick to make the built-in server browser work). But then it was fixed...

torhus commented 2 years ago

Yeah, there was a lot of discussion about there original master server, but I think it's long gone now.

The-Gig commented 2 years ago

What is "long gone"? The discussion, I guess, as the original master server seems to still be there and working, although we have to find out how to use it with qstat..

The-Gig commented 2 years ago

About Rdnt Cntrl's script, how did you manage to make it work, at least with OA? If I type python ./oaquery-q3a.py -a I get errors:

    server_addresses = query_master(addrs, args.timeout, args.retries, args.empty)
  File "D:\MonsterBrowser\oaquery-q3a.py", line 614, in query_master
    while not dispatcher.recv(timeout) and retries > 0:
  File "D:\MonsterBrowser\oaquery-q3a.py", line 390, in recv
    (data, raddr) = self._socket.recvfrom(MAX_MSGLEN, socket.MSG_DONTWAIT)
AttributeError: module 'socket' has no attribute 'MSG_DONTWAIT'. Did you mean: 'MSG_DONTROUTE'?

Update: rdnt ctrl says Windows doesn't have MSG_DONTWAIT. So, it looks like I'm out of luck.

Anyway, do you know how to customize the qstat's query string?

torhus commented 2 years ago

Right, I forgot the old masters are back up now.

I get the same errors you do with that script, but when I run it like oaquery-q3a.py master.quake3arena.com:27950, I get timeouts instead. However, it works fine in WSL. The problem is tcpdump doesn't seem to work there, so I can't see what it sends to the master.

It's possible that it could work with a custom qstat configuration, like I have got for WoP and Reaction. Maybe the master packet or the master query settings?

I'm moving this week, so I don't if I will be able to investigate this until later, but you are welcome to try.

The-Gig commented 2 years ago

Okay, I managed to run rdnt's script. I had to enable WLS (Windows Subsystem for Linux), install Ubuntu for WLS from Microsoft Store and launch the script. python3 oaquery-q3a.py -a actually shows players connected to each server, while python3 oaquery-q3a.py -m shows just the list of IP:PORT of the game servers. It's possible to specify server or server:port parameter to query a different master server.

And, depending from what you write in line 558 of the script, then the original q3a master server answers or not. As you try various combinations, empty full works, full demo works, empty full demo doesn't work. So, it may really seem master.quake3arena.com supports query strings shorter than those supported by master.ioquake3.org or dpmaster.deathmask.net, as Rdnt suggested.

I checked with Wireshark (analyzing network packets): with this text in the query packet immagine the original master server does not answer at all, it sends no packets back, while with this text it answers.

I have not been able to get any answer from master3.idsoftware.com; I have no idea about the reason, but that's less important than the original server I guess.

Still using wireshark, the packet sent by qstat -q3m master.quake3arena.com looks like this immagine and the server does not send any reply. How to shorten qstat query string? By the way, IDK what "demo" is meant for (maybe servers running the old Q3A demo, the one which contained 4 maps iirc? In that case, I guess omitting it should not be big deal)...

torhus commented 2 years ago

I tried this, in mods.ini:

[test]
mod=
masterServer=master.quake3arena.com:27950
qstatMasterServerType=testm
qstatConfigFile=test.cfg

test.cfg:

gametype TESTM new extend Q3M
    master query = empty full
end

And I get 688 servers. Unfortunately, MB crashes somwhere in the GeoIP code, the workaround is to disable that by renaming the GeoLite2-Country.mmdb file to something else. The crash is #9.

The-Gig commented 2 years ago

Before reading your last post, I had done a similar thing (using the test build which supports empty mod field as "all servers", Jul 20 2022), which seems to be the solution. This master server finds like 200 more working servers than the ioquake3 one (I guess in theory one could somehow export and compare the results to see how many of these are shared and how many are exclusive to one of the two master servers only)...

But the program does not crash for me. Does it happen to you as soon as you refresh the server list or when?

torhus commented 2 years ago

Right, I'm using a debug build, it's just an assert that gets triggered.

The-Gig commented 2 years ago

Although we already did the tests we needed with it, I write this for completeness. Rdnt Cntrl has updated the custom version of his script he made for us (to test q3 servers instead of oa servers), so it can also be run (with Python of course) on Windows directly, without the need for WSL. Here it is the new version: https://cdn.discordapp.com/attachments/232325920822657024/1001560788764471296/oaquery-q3a.py (in this version, the line where to mess with the "full/empty/demo" string is 560).

So, it can be run like python ./oaquery-q3a.py -m (get only server list), python ./oaquery-q3a.py -a (get informations from each server, such as current map and players), python ./oaquery-q3a.py -a master.quake3arena.com:27950 (ask the master server specified, which btw is the default one).

If run from a classic cmd command prompt or powershell console, text colors will not be interpreted correctly and will be shown as extra characters: (screenshot: https://media.discordapp.net/attachments/232325920822657024/1001578014401908907/unknown.png), but if run from the modern Windows Terminal (in either cmd or ps mode), the colors will be shown correctly (screenshot: https://media.discordapp.net/attachments/232325920822657024/1001586903914647582/unknown.png). Otherwise, it's possible to add --no-colors parameter to avoid the problem.

torhus commented 2 years ago

I can't download the script from that link, I get a 403.

The-Gig commented 2 years ago

Rdnt deleted the discord post with the attachment, but now he made a branch for the q3a version of the script inside the github repo, you can get it from there: https://github.com/rdntcntrl/oaquery/tree/q3a

torhus commented 2 years ago

This is basically fixed in 604565f, but the default configuration probably needs to be revised.

The-Gig commented 2 years ago

Interesting. Can I help?

torhus commented 2 years ago

I think it's probably ok the way it is now. If you want to check it out, you can use the build from the openarena branch. Delete or empty mods.ini to get MB to output the default configuration.

torhus commented 2 years ago

What I am unsure about is adding an entry for the Q3 master without mod filtering. There are different types of servers, like over one hundred Urban Terror servers. Wouldn't this be confusing to people if you can't actually join them with a regular Q3 client? I'm not sure if you just need some data files or if you need a different engine for these servers... 🤔

torhus commented 2 years ago

I tried this, in mods.ini:

[test]
mod=
masterServer=master.quake3arena.com:27950
qstatMasterServerType=testm
qstatConfigFile=test.cfg

test.cfg:

gametype TESTM new extend Q3M
    master query = empty full
end

The extra QStat configuration is no longer needed after updating to QStat 2.17, which does this correctly by default.

The-Gig commented 2 years ago

About qstat, very well!

About Urban Terror, if it does use the main q3 master server, I hope they still kept it working as a standard q3 mod and specific binaries just add some extra feature but aren't really a requirement. I hope to be able to do a bit of testing (connecting to an UT server from Q3A) this week.

torhus commented 2 years ago

I was able to join some UT servers after copying the q3ut4 folder over from my UT installation, and setting sv_pure 0.

The-Gig commented 2 years ago

Wondering how setting sv_pure might change the behavior of connecting to a server (in other words, acting as a client)... But there are still things mysterious to me in this game.

Anyway, my first tries to connect to q3ut4 servers have been a disappointment. But I want to do more tests. If they really take advantage of q3 master server to play a mod incompatible with stock q3 binaries, they are really reckless...

torhus commented 2 years ago

I suppose sv_pure didn't actually do anything, since that's a server command. It's been a while since I have had to deal with this.

Any servers in particular you have issues with?

The-Gig commented 2 years ago

I managed to have q3ut4 work with classic Q3A 1.32c binaries.

It looks like most UrT servers have automatic download either disabled or not correctly configured, hence I had to get the 1.4 GB zip from their website and extract q3ut4 folder into quake 3 folder. Well, I guess it makes sense because with old-fashioned slow UDP download it would take a lot of time to download it... and anyway manually downloading a mod was a normal way to have the job done back then in the glorious Q3A days. Some UrT servers still use some extra maps which are not part of the main package and considering they don't have autodownload properly enabled I cannot connect to them, but servers using stock UrT maps are okay. Another thing I had to do was to set _comhunkmegs 256, as I got "hunk" errors before. As it launched baseq3 and it then switched to the mod and I think that specific variable cannot be updated on-the-fly, I had to set the variable in baseq3.

Anyway, now I have confirm that it IS possible to play Urban Terror with classic Q3 binaries. So, the presence of UrT servers in "Q3 all servers" list should not be a problem, I think.

torhus commented 2 years ago

Yeah, I think it makes sense to include the list, it seems you get more or less the same list in the in-game browser. The question is if it makes sense to call it "Quake III Arena", or maybe something like "Quake III master".

The-Gig commented 2 years ago

Or maybe "Quake III all servers"/"Quake III all mods"? I'm not sure...

torhus commented 2 years ago

Good ideas, I think I will go with "Quake III (all servers)".

torhus / monster-browser

About querying official Q3 master server #8