Get list of players results in 500 internal server error

[thica]

Hi, sinnce some days, the collection of data does not work. It stuck with

`[....] Get list of players 500 Internal Server Error POST https://de6.elvenar.com/game/json?h=0ba1902da5f {'Accept': '/', 'Accept-Encoding': 'gzip, deflate, br', 'Accept-Language': 'en-US,en,q=0.5', 'Connection': 'keep-alive', 'Content-Length': '155', 'Content-Type': 'application/json', 'Cookie': 'ig_conv_last_site=https://de6.elvenar.com/game;sid=H12y5CIKUx-VnZd;req_page_info=game_v1;start_page_type=game;start_page_version=v1', 'Host': 'de6.elvenar.com', 'Os-type': 'browser', 'Referer': 'https://de6.elvenar.com/game', 'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 ' 'Goanna/4.8 Firefox 68.0 PaleMoon/29.4.2.1', 'X-Requested-With': 'ElvenarHaxeClient'} {'Server': 'nginx', 'Date': 'Fri, 20 May 2022 05:55:15 GMT', 'Content-Type': 'text/html', 'Content-Length': '2158', 'Connection': 'close', 'ETag': '"62825144-86e"'}

←[3A[←[38;5;1mFAIL←[0m] Traceback (most recent call last): File "fetchData.py", line 376, in main(username, passwd, country, world, prefix_path, guild_name, player_guild) File "fetchData.py", line 348, in main cartograph.initializePlayerList() File fetchData.py", line 234, in initializePlayerList self.getPlayerList() File "\fetchData.py", line 156, in getPlayerList raise Exception("Could not access player list") Exception: Could not access player list [....] Logout`

[Olmagzar]

Hi Carsten, thank you for reporting this issue.

I've never encountered this issue at this stage but I did when I wanted to download several cities at once. I ended up packing them by 25.

I believe the issue is related to returned data size. The server might return an error when the returned data is too big but I'm having struggle to precisely confirm that. Actually, the request sent to get the list of players was not limited in number of players (set to 999999 precisely) and I believe I never tried this on a world with as many players as yours have. This step succeed when I set the limit to 45000.

Note: I tried 49k but it sometimes fails and sometimes succeed, that might indicates it depends on the server load or that the list size varies but I doubt about this last hypothesis as it seems to varies too fast, succeeding at one time, failing a minute later and succeeding again another minute after. Changes happening that quick in the size of the list of players, not in the details of all cities, would be really weird in my opinion. I doubt players are deleted, restored, created that quick so the limit varies from 49000 to 49600 names. However, a server load increasing by several MB in a few minutes or seconds, reaching a limit with this request refusing it is more likely in my opinion.

I'll patch this to build the full list of cities to visit in several steps to respect an upper limit arbitrary set to 45k names per part of the list.

[thica]

Hi, thx for your support. I have used the logic of your code for a somehow different project. Maybe I could adjust it by myself, but without an API documention it is somehow difficult. Please let me know, if I provide support.

[Olmagzar]

Glad to hear this code is of some use to others :)

I thought I fixed this a few hours ago but, running the script on 'de' 'Sinya Arda', I found my new loop over the names created duplicates due to misunderstandings of the game API. It looks like requesting 5k players, providing an ID to start, as the game does when ones looking at its own rank with a list size of 8 and the player ID, is interpreted as "give me the next 5k players from page 5k (and ignored gibberish)" instead of giving the next 5k players from provided player ID.

I changed it to always request parts of 45k players instead of padding with the remaining players, and started the script again to check everything is good. From the local debbuging logs I have, the list of cities to visit is complete. I'm waiting the end of the journey now to check if I introduced an error trying to patch this bug.