Third-party API returns malformed A2S results

[aclist]

API may rarely return improperly byte-decoded modlist and mangle the output, concatenating multiple mods into a non-existent mod and assigning the wrong publishedfileid of some other content.

[SteveIDusa]

@aclist

Got a chance to get that Steam API, got it into the config, and confirmed that Add by IP did work and found 2 maps on the same ip on offer.

However, it seems I am back to an issue where I am getting missing .pbo errors. And for some reason on this particular server, it keeps telling me that the launch conditions are satisfied and launches the game without getting any mods. I even tried going back to the stable branch to see if that would fix it. But that did not resolve it either. I can find it in battlemetrics as ID 12270609, I can copy and paste in the ip from BM and it shows the two, but getting actual mods to display as needed in the browser is not happening at all. I even cleared the mods in the folders and took out all of the @(Mod Name) files. but I can not get this particular server to show the download links.

What am I forgetting?

Oh, its an hour or 2 later, it just occurred to me, it is a private personal server and not a rented one, could it be that they have a port that is locked down that DZGUI relys on to get the mod list?

[aclist]

Do you remember how the BM site was returning malformed mods at the top of the list?

Bohemia started obfuscating mod information over the server query protocol and sending it back encoded in a special format. I escalated the issue to BM and repeatedly begged them to acknowledge the issue, but, admittedly, it was pretty complicated, because Bohemia hadn't released any specifications about how the info was encoded.

During that time, we basically swapped to using another third party API (that was decoding the info sort of correctly) as a fallback for getting the modlist.

But that other API isn't exactly working in all cases, it seems. So we are just flip-flopping between different APIs trying to get the optimal result.

Since BM seems to be working as intended, I've restored the legacy code using the original method, and it should work on the server you stated. But do please test on other servers as usual and make sure that it's reliably returning the right info.

While I would really like to develop a native method without third party APIs, I don't really have the time or resources to quit my job and create that right now...it's a non-trivial undertaking. It would be nice to have something more robust that is not subject to the vagaries of third party APIs, but it's a big task.

[SteveIDusa]

@aclist

Oh, yeah, it may not be related to something I am doing or not doing.....OK. I will not pretend to even understand at this point how you are getting all of this information together, but I totally understand priorities need to be in the right place. So I will find these changes in the stable or testing branch?

And, since I can get into any modded servers at all, is a blessing all in it self, thank you again for the work that you have done!

[SteveIDusa]

@aclist

Just tried another join by IP with 135.125.189,121....Which in this case is a JohnMcLane's official (New Map) ALTERIA v.0.3 map and He was making the announcement so I suspect also a private server. It was announced this morning that he was opening it up for testing just for today. Although I can see it, even using the testing branch version of 2.6.0-rc3 it seems that it is doing the same thing as last night with the other server. Rather than opening the web browser with links for download, it just says the launch conditions are satisfied and then launches the game to a .pbo error.

I begin to suspect it has something to do with it being a private server rather than a rented server, but I may be missing other things to consider. I have such a limited knowledge of servers, but last night I know for a fact it was a private server as the owner was discording with my wife and myself and acknowledged that it was sitting just a few feet away from them. And I can not believe that someone that is designing some of the most popular maps for DayZ would be interested in paying a fee every month to do development, I mean perhaps, but why if you can do a private server with an older computer right in your home.

I just decided to take the IP that I got from his announcement and his given server name and try by Server ID number which is 16587403 and low and behold, I got a mod list to pop up in the Browser.........hmmmm

And I got in! Going to have to check the other server now....

Ok, so the Bullets and Beans server did produce the Mod list this morning which it did not last night.....Only issue is now a single mod called In-Vehicle Inventory ID# 2472500243, for some reason it will not load, only does .pbo error

[aclist]

Thanks for the additional info. That's a regression. I had neglected to update the connect-by-ip method while updating the other part. With connect-by-ip, you aren't providing any BM ID, so we have to use the other API.

Both methods should be working now. I'm not thrilled with depending on two different things for what should be essentially the same action (connect to a server), but I guess this whole project is just a series of workarounds anyway. It pains me to think we are resting on a house of cards, though.

I can confirm that 135.125.189.121 is working when using connect by IP on 2.6.0-rc4 and not adding a BM ID.

How is the new map?

I will continue investigating the Bullets and Beans issue.

[aclist]

I'm not seeing In-Vehicle Inventory reported in the modlist reported by the server. That's one that doesn't get downloaded and then the game client reports that it is missing?

[SteveIDusa]

@aclist

Right, it does not seem to get added to the list, and then when I go and subscribe manually it seems that DZGUI does not know that it exists, so that causes some type of conflict. I know it is there, I can see it, but for some reason it does not get loaded by DZGUI. For my wife on Windows 11, she had to also do a manual subscribe, but the DayZ Launcher allows here to go into the mod list and check a box to say load it. Of course that is not available in DZGUI, and loading up the DayZ launcher in Linux causes all of the subscribed mods to show as broken, so even if I tell it to load that mod, DZGUI still can not see it I guess.....

New map is shooting gallery at the moment. Spawn is highly contested, got killed a lot with little loot. After 4 or 5 hours of play we finally got it figured out that the best option is to run the coast and only hit houses for food and keep moving.....

[aclist]

Well, DZGUI has to parse the list of mods reported by the server as a single source of truth and prepare them that way. We can't just guess which mods might be correct or the user wants.

Found the problem: this one isn't showing up because the mod creator did not configure it correctly. https://steamcommunity.com/sharedfiles/filedetails/?id=2472500243

See all of the comments posted at the bottom. The meta.cpp contains a 0 instead of the correct UUID. Mod ID cannot be zero. This is consistent with the erroneous results we are getting back from the API.

The correct mod ID should be 2472500243.

It looks like the mod creator is unresponsive to this and has not fixed it.

[SteveIDusa]

@aclist

Thank you for doing the research to find the issue. I guess that server is off limits unless the owner wants to change the mod list.
But hey you have to do what will work and I for one am very happy with the results 99% of the time. Thank you again.

[aclist]

Unfortunately, there is no standardization with servers; it's a largely homebrew effort. Mods become discontinued or unsupported by the creator and are left in limbo. I'm not sure why it's possible to upload a mod with an empty ID, but I guess there is no integrity check performed. Ultimately, it's the responsibility of the server operator to audit these. I would suggest notifying the server operator that the mod is broken.

[SteveIDusa]

Already done, and they spoke to their small community and they agreed to remove it, Got several hours on last night. Is it possible to alter the meta.cpp file and get it to work? If so, I will try to talk the owner into making the change and see if that prompts me to get the correct mod.

[aclist]

I was thinking about that. If the server operator updates the ID to the correct one, it should report back correctly. Problem is, when you download the mod from Steam, it's still going to contain the wrong ID, so it's not going to be able to reconcile the local mod with the one the server expects unless we make an explicit exception for this mod ID and hardcode it to overwrite its ID. Which is bad practice, since we'd just be manually creating an exclusion for this particular mod, which wouldn't prevent the problem if another mod is improperly configured in the future.

[aclist]

One other thing you could do that would be much more effective here is for either the server operator or you to create and publish a duplicate of this mod with a correct ID, and start serving that one instead. It's easy to do. It seems that the author has abandoned the original one, along with other mods, so I wouldn't personally see that as a problem.

[SteveIDusa]

@aclist

I do not see anyway of doing that, do I just use the one that I subscribed too and repackage it? Sorry I have absolutely no experience with doing anything with mods except subscribing and using them. Where do I look to get the correct information on how to do it? Youtube, Google, ask you?

[aclist]

I have not done it for DayZ specifically, but have uploaded mods for other games. Generally the game ships with some interface that lets you push them directly to the Steam workshop and they get registered under your account. These are trivial to remove if something was done incorrectly. There are 20,000 mods for DayZ, and they get added and deleted all the time.

There is an application called DayZ Tools included with the DayZ license that is used to package mods. You would have to repackage the assets. This user on YouTube seems to have a whole playlist with different tutorials on doing it: https://www.youtube.com/watch?v=SPyYHHJR6oc&list=PLsFzoDww-uLyhL05aXaIA9DuqceJnrjwZ

It doesn't show up in your Steam library, but gets installed into "DayZ Tools/Bin/Launcher/DayZToolsLauncher.exe".

I haven't tried any of this myself, but it seems straightforward enough. I could maybe take a look at it when I have time.

You would have to run the below to install DayZ Tools from the depot:

steam steam://install/830640

[SteveIDusa]

Wonder if it will run under proton.....

[aclist]

You don't need the whole package, just two tools called DSUtils.exe and Publisher.exe. First one is used to sign the mod and the second is used to upload to Steam. I was able to get them working in Proton after a bunch of fiddling with installing the Microsoft Visual C++ Runtimes 2010, 2013, and 2019 into the Proton prefix. Steam needs to be running in the same Wine prefix as the tool for it to work. It takes a lot of fiddling (this used to be standard operating procedure for playing games in Wine), but is doable.

Just doing it on Windows would obviously be faster, though. Looks like you don't need to really repackage or change anything around, just update the meta.cpp file and sign and publish the pbos.

PITA but it is what it is. Just offering you alternatives here.

[SteveIDusa]

LOL, I finally dropped windows because Proton could launch and let me play a few of my favorite games with out fiddling with Wine or "Play on Linux" or Lutris. I have never had much luck with them....I guess I just am either missing prerequisite knowledge or understanding to fiddle my way through it. I will admit I once got the Epic Game Launcher installed with Lutris while GTA V was free on their system. Played for two hours and never picked it up again....not my style I guess.

I would be totally lost with setting up the MS stuff to run in the same prefix.....I get it can be done, but that is way beyond my current knowledge. It is kinda funny how it works too, I've noticed over the years, it is not really that something is being held secret from me, as much as I have to have the right experience at the right time with the right need or intent and suddenly I have a key to some knowledge that I did not have 3 years prior. Because I have tried reading tutorials and following video tutorials and I still just get lost.

I am however going to get my wife to take a look at those videos you linked, since she is still using Windows, and wants to make her own map....Maybe between her and I we will get it figured out...Thanks

[aclist]

I totally agree! Sometimes you have the right knowledge at the wrong time, or the right timing and the lack of knowledge. It's best to not try to force an issue, and just let it percolate in your brain. Then at some undefined point in the future, you find you have the right skills at the right time and it's an organic process built up through steady accumulation and exposure to new ideas. That's basically a fancy explanation for what we call "experience..."

I'm going to take a crack at either the server browser or mod downloads next and will let you know when a proof of concept is ready.

[aclist]

Stealth pre-release. Full server browser is now working. No setup required, no ID pre-entry or looking up servers in your browser. Added about 200 LOC plus some helpers to calculate radius of the Earth's surface and distance between latitudinal and longitudinal points. (Haversine formula)

You can filter by map, full/empty, daytime, nighttime, non-ASCII names, low population servers, and perform keyword search.

If you untick "All Maps," you will be given a list of maps to choose from, and it will return the full list of servers hosting that map.

Non-ASCII will suppress servers with fancy characters in the name (this also affects CJK languages and Cyrillic).

Keyword search lets you search for server(s) by name. This greatly speeds up narrowing down servers--for example, if you wanted to search for the "Bullets 'n' Beans" server, you could just type "beans" or "bullets". Also really helps if looking for servers with particular rulesets like "1PP" or "fresh wipe". Search is case insensitive and can be anything.

If no servers match the filter criteria, it will return to the main menu.

The server list gets loaded in and updated in real time. There are 20,000 servers broadcasting, but only about 3,000 of them active at a given time. Using proper filters to trim down your incoming list to <100 is the smart play here, but ~500 is still manageable. No pagination is needed. You can still load 3,000 if you like, but it takes awhile to update them all in place.

Also included is geolocation to show servers by proximity. This is the best sort criterion for sorting servers geographically proximate to you. Unfortunately, this is where we are running into the limitations of an unfunded project:

We are using a free geolocation database subset to calculate latitude and longitude. This database does not contain every IP on Earth. Currently, about 40% of the servers might come back with a distance of "unknown." This is because their IPs are not found in the database. This is bad because distance is the best predictor of whether a server will be good to play on (is a server in Germany or Djibouti?), so not having that info for every single server is not ideal. Secondly, in order to work around these unlisted ones, we have to do additional, computationally expensive pre-processing of the data to find nearest possible IPs, and this is slowing down the loading of the server list. If we could reliably check every IP without these workarounds, the whole table would load faster, and with 3,000 or more entries to deal with at times, every second saved is important here.

It seems it would cost about $400 for a one-time license to obtain a complete list of geolocation coordinates, with possible annual fees after that if we wanted to constantly ensure that every IP is up to date in the list on a rolling basis (to account for new servers coming up). Otherwise, getting the full list once and not updating it for the foreseeable future would still provide pretty good coverage.

Going forward, it looks like this server browser can replace all of the original functionality that depends on BM IDs and just let you search for/add servers on the fly. This would let us dispense with pre-registration and a lot of the finicky details. Currently, the new server browser lets you search and connect only. In order to keep a running list of servers you connected to and populate your favorites list via that, I would have to chuck out some of the legacy code and rewrite it to point to this new browser, as well as allow migration of old server lists into the new method under the hood. This should all be pretty OK to do, but it is a non-backwards-compatible change, so would bump the version to 3.0.0 at that time.

[aclist]

I forgot to mention that this now adds support for official servers.

[SteveIDusa]

@aclist

I got the notice of the change and tried early yesterday evening, but the browser server attempt ended in a "This requires Python" error and went back to the main navigation page. So I reverted back to the tried and true "Known" Server list and played for a bit. I kept having an issue on the server though that would require a disconnect and reconnect after a particular action. If I avoided that action the issue did not happen, only when I did that action. However this did mean that on an occasion my DayZ did a crash to desktop which meant restarting with DZGUI. I then got the new notification for the new new version, but I had disconnected at a place that I was not feeling safe, so I did not get a chance to retry.

At first I thought to myself, "Ah, now your just showing off!!" LOL, when I started to read the post above. World Radius? Geolocation? Why? Then of course I read more and realized what you were really doing is trying to get users the shortest distance to them or nearest server! HOLY CRAP! Really?

I want to make a suggestion, Even going to a full new version, Keep a "Legacy Process" link in the menu, for anyone that might find the old system useful. Because it is, and frankly though finicky, is still pretty darn concise. It is also a view into the process that you have undertaken and for some of us a (If murky) a little window into the process.

I cant wait to try the new browser though! WOW!

[aclist]

OK, first update, I managed to devise a more performant and accurate way of getting geolocation matches, so brought the inaccurate hits down from 40% to about 3% of all servers. I'm building an updated DB of the missing IPs by just adding them explicitly in order to bring it up to 100% coverage (as of this date).

Secondly, managed to improve the search algorithm itself so it's pretty speedy when doing the distance check.

As to the Python thing, well, I'm glad I included that warning as an afterthought. Python should be installed by default on almost all distributions, so I originally thought it would be fine to just drop you right in there, but I had an inkling it might be missing on certain setups. Here's what you need for your distro: https://linuxways.net/mint/how-to-install-python-on-linux-mint-20/

Other than that, no setup necessary.

As for features, yeah, I have to think about how to manage them. I don't mind keeping stuff, but the main menu is getting pretty busy. And I figured if you use the server browser, it would also passively build a "favorites" list of servers you connected to before, so you wouldn't have to add those by yourself. But I guess there might be someone out there who wants to add servers by ID and browse the web site...ya never know.

It's looking pretty great over here and provides a great way of discovering different servers and maps that are out there (there are 65 unique map types being hosted), so curious to see your feedback.

[SteveIDusa]

@aclist

Python 3 does indeed ship with mint by default and when we started the Python Course here at home with the kid, I was the only one that did not have to download and install it, while they on Windows did. I have noted, that on Windows they do not have to launch a python program with anything other than the name.py on the command line. While I on Linux Mint 20.3 and now after my reinstall Mint 21 have to launch python programs with "python3 name.py" at the command line.

I wonder if this is the crux of my systems result. I just tried it again, and got the same python requirement. As I have my lovely wife waiting for me to join I will have to come back to this a little later to review the update. Still looking forward to seeing all of the improvements.

[aclist]

OK, I am going to drop the Python requirement and use a small C program instead. The requirement was only for the distance calculation, not for the table itself.

[aclist]

OK, provided I did not screw anything up, the latest update should work out of the box and gives a 25% performance uplift over the Python version. I also added some extra error handling and added a method for the client to fetch the latest geolocation tables if they are out of date. Currently I have coverage for about 19000 servers with all distances accounted for.

[SteveIDusa]

@aclist

Ok, just downloaded 2.7.0-rc.7. Attempting to launch the "(NEW) Server Browser" option resulted in the same "Requires Python" error. Do I need to delete a config file or something? Thought it said it wrote a new one.....?

[aclist]

Sorry, I'm dumb! I forgot to remove that check despite changing the rest of it. This is why we have testing releases...

Should be fixed now.

[aclist]

Sorry, I must be running on little sleep. After I told you I had fixed it, it looks like I didn't actually commit the code. It was still sitting on my local machine when I looked at it today. Sorry! I just bumped the version to 2.7.0-rc.8.

[SteveIDusa]

@aclist

OK, Thanks, I tried last night and was like "Huh?", so I was going to come and ask this afternoon and find you figured out what was going to ask about. Will try as soon as I can!

[aclist]

Did you have any luck testing it out?

[SteveIDusa]

@aclist

I did try it once, but it seemed to be taking to long to respond, so I just killed the process in my System Monitor and then restarted and jumped into the game using the older process. I had not had a chance to get back to it until this evening.

So here is where I am at., it seems to launch as expected, after choosing the browser server and selecting options, the GUI disappears and the system monitor shows 4 dzgui processes, while waiting for the GUI to come back up 2 to 3 additional un-named processes pop up at the top of the process list with ID#'s but no names.

It seems to appear like it is doing something, but never displays anything to indicate that it is progressing. But I am not comfortable watching processes run with no progression displayed, so I kill it and move on.

[aclist]

Sorry to bug ya about it. I feel we are close here, and it's definitely working on my side, but it can be tough to test for all the different user environments.

What happens is this:

1. You select your filters
2. The GUI should disappear momentarily and a progress bar indicating the server list is being fetched should appear
3. After it completes, that progress bar should disappear and another progress bar should appear indicating that it's filtering the results
4. After a brief delay, the entire server list should appear as a table

Depending on the filters chosen, there could be some slight delay between 2 and 3. This is "technically" a bug, as the progress bars should be merged into one step, but I haven't adjusted that yet.

But it shouldn't just hang indefinitely, even if you selected a lot of servers. What are your search parameters? Have you tried searching by keyword to restrict it to just a few and seeing what happens?

What kind of CPU/memory do you have? I believe your system is newer than mine, so that's probably not it.

It strongly sounds like it's calculating the results in the background but not drawing the table for some reason. We need to find what is causing it to be hidden. Those subprocesses are just children of the table and the process filtering the servers.

Can you check the contents of $HOME/.local/share/dzgui/helpers and check for a file called latlon and ips.csv?

If they are both present, can you run ./latlon 40.71 -74.01 50.11 8.68? It should return 6203.0.

I'll try to keep thinking about what could be blocking it from drawing the table for you.

[aclist]

Another question, do you usually launch DZGUI from a shortcut or from the terminal? I can add some debug logging to print messages to the terminal indicating what phase of the process we are in while it runs, as this would help identify where it's locking up. It also occurs to me that if you are using a shortcut (not that there's anything wrong with that), it might be affecting the way subprocesses are getting spawned due to the permissions of the subshell.

[aclist]

Just tested on a Steam Deck and think I may have found the issue. Can you run sudo apt-cache policy libc6 and tell me what it reports?

[aclist]

I recompiled the distance checker against an older version of libc6 so that it supports Steam Deck and Ubuntu/Debian variants better. Seems to be working on the Steam Deck now. I did a package search for Linux Mint and I am fairly certain you are on libc6 2.31-0ubuntu9.9 at the highest, so this change should fix support. Whether it addresses the table going missing, not sure.

[SteveIDusa]

@aclist

Again specs are Ryzen 3700X, 16GBs RAM, GTX 1070,

Yeah, it looks like it's trying to do something, but I am not even getting the progress bars you mention,

When you published adding it to Steam, I converted to launching from Steam. I am getting more and more familiar with the command line, well because "Linux", but once you did that, I was "Yes! Please!"

I ran the apt-cache policy libc6 which produced this weather I used sudo or not: libc6: Installed: 2.35-0ubuntu3.1 Candidate: 2.35-0ubuntu3.1 Version table: *** 2.35-0ubuntu3.1 500 500 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages 100 /var/lib/dpkg/status 2.35-0ubuntu3 500 500 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages

Hope all of this helps!

[aclist]

OK, that does clear things up more. I'll make a note to always test it as a Steam shortcut in the future. I assumed that doing that through Steam Deck would suffice, but there seem to be some differences.

Looks your version of libc6 is pretty new, so no problems there. I noticed that the Steam Deck was running 2.33, so I recompiled the geolocation code to ensure it works on older versions.

As for the bug, I believe (touch wood) I found the issue. When I added it as a non-Steam game using the dzgui.desktop file located in $HOME/.local/share/applications/dzgui (I assume you added this one), there was a dumb bug where a │ character in the header was blocking creation of the table. I should have remembered that we had problems with this character before. Once I changed the text in the header, the issue cleared up instantly. I was able to troubleshoot this because I launched Steam itself from a terminal, so it was printing out debug information whenever an app was launched.

However, I found a secondary issue when exiting from the server list, where the process scanning server locations was blocking returning to the main menu, since it was waiting for all of the entries to finish getting scanned.

This doesn't happen when launching the script outside of Steam, but Steam is doing some stuff with forking processes and trying to spawn them in containers to ensure the runtime environment is sane, and it creates a situation where one thread can block the rest of the pipeline. I think "non-Steam game" was intended more for standalone applications and not something that forks and calls subshells, so it's kind of a pain here to make sure the subprocesses get cleaned up correctly after you return from each menu.

Anyway, instead of writing out the table entries in a pipe and listening for them via the menu, I backgrounded the process creating the table entries and just write them to a named pipe on a separate thread, with the table reading from those on demand. When we exit the table, the job responsible for writing the entries gets killed and stops blocking.

Thanks. I'm not exaggerating when I say that most of the functionality enjoyed by others in DZGUI is owed to you patiently testing this stuff.

[SteveIDusa]

Ok, was able to get to a server list, seems to be working as intended now, will attempt choosing a new server and testing mod downloads via the sever browser next login. It is looking very good! And WOW have you been busy!

[aclist]

That's good news! Let me know.

I have the automod installation working on my local build, but need to add more error handling still.

[SteveIDusa]

@aclist

Did a full Browser server join today! 57 Mods and it worked quite well, with the exception of....I really wish it were automated, man that took for ever! But it worked very well!! Thank you again for the hard work! Used RC20, then later in the day got updated to RC21

[aclist]

Very nice! Yipee! 57 mods, ouch. Automated is...coming. I am testing different distributions in a VM in order to have better coverage.

[aclist]

I'd say 2-4 days for automation. I am going to release the server browser to the stable branch first. I'm thinking of just making auto-install a default option instead of a toggle, and having it available regardless of the user's settings. There will be an initial check of whether the automation routine will work. If it doesn't, revert to manual mode. I think this is OK (probably), as it allows for fast-tracking people onto automation without burying it in the settings.

[aclist]

I'm going to close this thread, as it's getting rather long in the tooth and will be hard for other people to refer to, having over 93 comments in it. I've opened this "forum" as a place to vote on issues, discuss beta testing, propose features, etc., and anything else that doesn't fit into the category of bug: https://github.com/aclist/dztui/discussions

You can use this thread specifically to talk about the testing branch: https://github.com/aclist/dztui/discussions/35

It would be best to continue this conversation over there with regard to the auto mod installation.

See you on the other side!

[SteveIDusa]

Understood, thank you again!