HOF backup file problem

[Jirkaon]

Hello, It’s me, again. I have an issue to report. I’m using Win 10 and 11 (the issue is the same on both). If you start crawling the server, it goes ok, but if you want to save the crawled data and and restore it, only 346967 players will restore and you’re stuck. Can’t crawl again with the restored HOF. I already tried few things: Start crawling until about 450000 players are fetched, pause, wait till it stops, wait few seconds, backup HOF, wait few seconds, close and open program, login, wait, restore and only 346967 players will be restored. I tried to backup it in multiple different stages of fetched players (till about 800000). What I discovered was that under the 346967 players fetched everything was functionally ok, only above that number it was stuck. I always tried new session with fresh program (unzipped to different folder). Oh and the HOF file is always about 400 MB (never the same, but about 400 MB). Is there some option to send you a log file? If there is some hidden? I hope you understand it. Maybe I missed something so feel free to ask me.

[the-marenga]

Hey, thanks for the detailed report. I have no clue why it would break after that specific amount. There is some jank with backing up whilst crawling is active and a discrepancy for players with no equipment, but that does not seem to be what is happening here.

My best theory is this: Some players and HoF pages are outright broken, they will also break the official client. To deal with them, my tool just skips these players and pretends that they were successfully fetched, otherwise the tool would get stuck reattempting to fetch these invalid players forever. The amount of broken players/pages is maximum like 1 or 2 per server, so this should not matter. If I had to guess, after some point the crawling itself breaks in your case for all characters, not just invalid players for unknown reasons, but you have no indication of that, because the numbers keep increasing. When you backup & restore, the number displayed becomes the actual amount of fetched players, which is far lower, because it does not contain the players, that failed to load. This should not influence the continuation of the crawling, but it is my best guess.

Overall though, I think I have to basically rewrite this tool from scratch anyways. This was just meant as a tech demo for the underlying API, that I threw together in an afternoon. I did not expect anyone to actually use this. Currently the UI sucks and as you can tell, there are some bugs and the lack of logging is pretty bad in cases like this. I am pretty busy at the moment, but in the coming week I should be able to just remake this tool and handle a bunch of this stuff more professionally. Chances are, I might also just add the HoF sharing stuff, as I have some ideas on how to handle that without dealing with the privacy problems of that.

For the time being, can you tell me which server you are playing on, so I can reproduce the behaviour?

[Jirkaon]

Hi, thanks for the detailed answer. Now I understand little bit more of the “BTS” functions of your program. I’m not really code savvy so I can’t understand most of your code. Your explanation is closer to me.

Now I remembered something. You wrote about the bugged HOF pages. Sometimes it happens to me that if you want to browse the HOF on standard client (it doesn’t matter if it’s browser, Steam or mobile) you get few normally shown players and if you scroll for a little bit you get only - - - (or … , can’t remember now, but I don’t think it matters) instead of the standard player info. It will go forever, or at least for lots of pages (than I lose my patience :D). Can’t click on those. You need to reopen the HOF or sometimes relog to fix it. Could it be somewhat related? I think it’s the same behaviour as you described but more pronounced.

About the server… I was trying to not tell it (because Playa, you now how it is…), but… I will just tell you it’s the LAST fusion of CZECH servers. You will know which one I mean.

I’m looking forward for the redesigned version. But still… I was looking for some alternative to the og CrawlerNG. Yes, there are some alternatives even here on GitHub, but even though your is maybe the newest it is the richest in therms of features. You have my compliments for this masterpiece.

[the-marenga]

Now I remembered something. You wrote about the bugged HOF pages. Sometimes it happens to me that if you want to browse the HOF on standard client (it doesn’t matter if it’s browser, Steam or mobile) you get few normally shown players and if you scroll for a little bit you get only - - - (or … , can’t remember now, but I don’t think it matters) instead of the standard player info. It will go forever, or at least for lots of pages (than I lose my patience :D). Can’t click on those. You need to reopen the HOF or sometimes relog to fix it. Could it be somewhat related? I think it’s the same behaviour as you described but more pronounced.

Jup, it is exactly that. If you want some more technical information, the server actually sends back a hall of fame response, that ends in the middle of a player name and the official client just doesn't know how to handle that. I assume that is because someone registed a player with an invalid name and the server just gives up printing the name when it encounters the broken part of the name.

I was looking for some alternative to the og CrawlerNG

I tried to use that a few years ago, but I could not figure out what it was supposed to do. I assumed that it would do what this tool here does, but it just looked like a HoF viewer, when I tried it. If you have used it and know of any featues, that you would like to see from that, feel free to open another issue.

You have my compliments for this masterpiece.

Thanks 😄

[Jirkaon]

Oh, nice :D But I’m not sure about the invalid username as you wrote, because it happens to me even on some higher levels of players. Let’s say 400+. And as I said, reopening HOF or relog fixes it. Or did I miss something?

I was trying too, but never used it, because it was programmed for the old backend of S&F and never updated to the new one. That is what I found so maybe there was more to it, but who know. Only watched some videos how it worked. But in the end it did some parts of what this tool does only with different GUI. What I remember it was, as you said, only a crawler. Than you needed to fight the opponents manually or with help of MFBot to do it for you.

I’ll look more to it and try to find some new features, but I think that all features needed are already present in this tool.

[the-marenga]

I’m not sure about the invalid username as you wrote, because it happens to me even on some higher levels of players. Let’s say 400+

Interesting. I am pretty sure the broken username stuff is correct, because I could reliably reproduce it on specific HoF ranks. If it also happens on lvl 400+ accounts, that means there is more than one reason for this to happen. That is a bad sign in terms of being able to trust the server output and might mean, that a character session can become broken in some way just by looking at normal HoF pages... well... I guess I will have to do the logout/login stuff as well then to make sure. Thanks for the follow up

[the-marenga]

The new version is a total rewrite of the old code, which fixed a bunch of bugs. I am guessing, that your bug was fixed during that, but if it still happens to you, feel free to reopen this issue