List all players

[gilly42]

How to request a list of all players (to create a top10 rps player etc.)

[thekroko]

I think this issue has to be more descriptive on what we actually need/want. I have no intention of having an endpoint that lists ALL players, and I don't want to add a paginated endpoint either (no one is going to look at page 132 of the listing, except for a scraping robot).

What I can do is give you selective endpoints for specific purposes, i.e.

• Top 30 players by RP
• Top 30 players by LWRP
• [...]

[gilly42]

What argues against an endpoint to list ALL players? Genesis had a endpoint to grab a xml file with a snapshot of current herald data. I made a dynamic solution with backwards searchs of the top10 most lwrp ever and so on. History of level progress and so on. Was fun.

[thekroko]

I don't want it because it encourages scraping the herald for all data. I want to choose what we expose ourselves. I don't want users to scrape 90 000 characters every second.

[gilly42]

i guess you dont save all old data, or plan to implenet reports like, the fast leveler since server start. or who ever made most rps in 24hours ..... (btw thats why i send blue two msg over the forum, to ask him, if i can help on the herald on your side)

to prevent 90k requests every sec, you can create a endpoint for XML/JSON file which contains all current data for herald (every hour updated or so)

A "cointains all" file makes it possible to develop standalone "mini" heralds / client side (which could be super fast at search). and if the data is saved in a right way, its possible to make requests thats shows the development of guilds/players/realms over the time.

on the other side, if u really prevent scraping the data by bots, your should not implement guild data requests or prevent requests for the starter guilds. otherwise somebody can request the starter guild every few seconds and on this way he get a NEAR to all player list.

I interested in save the all the data and the changes over the time. I think this is a unique possibility, no game i know, offers that kind of data, for a complete historic look of the development of the players in the game.

EDIT: i've just found this site. http://sc.excidio.net/herald it is close to that what i plan to build, but this site does only 10 weeks backwards.

[thekroko]

We have aggregated long-time graphs of {RPs, Realms, Money, Class distributions, ..}. For that we don't need to expose all Players in one huge file.

[gilly42]

Yes thats my point. YOU have this data, but nobody else. Well it's your server, your rules. But all report/graphs/top10s/... which touches old data are up to you then and when you dont want to build them, they will never be created.

Btw: the Genesis xml file with all player and guild data (they are very detailed) is about 20MB (zipped), so huge is relative.

I only see this unique possibility to build a browseable history of uthgard. I think it would be awesome and many report we now not able to imagine will come when the data is avaible.

Lets take a realistic view on it. If you implenent TopXY, Guildviews or any other multiplayer-names-deliver-api-calls ist possible to get step by step a near-ALL-player-on-uthgard-list. Even if not, if someone creates a "enter your playername to start recording your uthgard development***" app in the End there will be an automated player-info-grab-bot (or many of them). You can try to limit the requests per second and so on, but why build limitations, why develop against the comunity, when you can remove the root problem by design.

***Maybe i'm the only one that have intrest in that data but i will build a historic uthgard record app with dynamic report/graphs anyway. The only question is, did i build it for all players (with our help), or have the players to add themself manually (and i automate the requests for them).

Do you have any argument against the full data expose, that could not be beaten by a little programming?

[thekroko]

I'd like the community to vote on it first, before deciding to publish all their data.

[gilly42]

I think this is a very good idea. Can you put the link to the vote here plz?

[mindvox-zz]

disclaimer: i am not here to harm any development. i am here because this looks like a developer thread (yet to have proof that all here are actually devs) and i don't see harm in making posts.

Claim and actual execution are two different things. OP (gilly42, heiko?) claims that he had created an xml solution to genesis herald. xml herald version that existed on genesis was a bad design that was completely out of date and not running for almost 6 months. all calls to fix it, fix bugs, make it more dynamic and robust were ignored/not resolved. The current herald was developed by database developer instead of website developer which is the right strategy since we are talking about stats and writing mysql scripts to extract them. efficiency of scripts and database structure are more important here than website design. claiming having fun and developing a good solution to a project that didn't work for ages is plain wrong.

Mindvox (Genesis Staff Member)

[gilly42]

I have thought a lot about this. If i understand it right you have two arguments against deliver a data. First you wont have too many requests on the api, the could be solution is a single json/xml file. The oder argument is that you not want to expose all data.

Maybe this could be a solution.

Instead of provide a file with all current char data, filter the data on chars which logged on in the last two week (or one week or one month) and only provide the filtered file. So you got both problems solved. The files does not contain all char data, and the size should be significant smaller than a full version.

[gilly42]

i have done a few report (still working on near everything) to have a look, whats i'm plan to build on uthgard data https://www.dayz.net/herald/

[gilly42]

btw @mindvox i'm not that guy you talking about at the genesis project.

[OliverRadini]

Was there any final decision on whether all player data would be exposed? I'm looking at building a guild profile page, but it would be more difficult to show members without an endpoint which makes player data accessible. Could it be cached, even just the basics of player data (name, level, race, class) would mean that we could have a more meaningful search, and list players that are in guilds etc.

[scrypt-kitty]

@mindvox said

efficiency of scripts and database structure are more important here than website design.

but @Donutttt and @burfo were talking in IRC yesterday and I don't see any reason that the Herald being built in this repo would have anything to do with database queries or stored procedures. That should all be done behind the Herald exposed API and performed by staff maintainers. We shouldn't (highly doubt we even would) have any kind of ability to directly query DB using SQL.

I'm with @gilly42 when it comes to JSON/XML - we should just call a defined endpoint and receive back data that we parse through to make the Herald. JSON would be much faster than XML but I believe this would also be left up to server maintainers to decide.

[OliverRadini]

I suppose the thing is that, at least from what I've heard from the devs, is that they want a front end built, and then they'll build what they can of an api for that. In this case I think there are a few people who would think that a list of players, whether paginated, cached, or whatever, would be useful. I certainly do, it doesn't have to be all player information, but if we had some way of querying a list of players which provided levels etc. it's be quite useful

[OliverRadini]

Still doesn't seem like much of a decision was reached with this. I'm currently building guild profiles on the basis that the guild data will contain a list of player names, and that we can then query for the data for each player as it's needed. This is working well so far with what I have built. The same principle could apply to things like global/realm RP rankings. There could be a cached list of player's names in order of RP; on the front-end, we paginate the player listings and only request player data as needed.

With regards to creating a listing of all player data... I feel that the best option would be a search endpoint, that can handle the matching of player names on the backend and return the best matches. Giphy is the one I've used most recently which does that. You just make a GET request with the search term, you get back an array with the relevant data. In any case my personal opinion is that a herald without a search feature is weak. On the IRC there have already been people talking about scraping the data, making calls every for every player in their guild every 15 minutes to keep player data on their guild's site up-to-date for instance (or something like that). 2 points from that:

1. It's awesome that people want to access Uthgard's data enough to setup something like that
2. People will find a way to scrape the data so it's far better to provide a way of making this efficient, or keeping the numbers of queries down.

imo the best way to prevent this kind of thing is to make the best herald possible. It'd have a search feature, at the very least.

[gilly42]

Yeah i love to have that, but like the kroko said

I'd like the community to vote on it first, before deciding to publish all their data.

And this may take a while.

Its still the same situation like Jun 2016, ok now more than only me whats the data. I quit to try to convinced thekroko to build an endpoint for all player data. I think they will get much load on their loadbalancer for (senseless) automated herald api requests.

but i'm not 100% uptodate, but do they update the data more than every 4h atm?

[OliverRadini]

I don't think an endpoint with every player's data is needed. But what is needed, if we want to create a good herald, is endpoints for:

1. Search (the logic of matching players can be handled on the back end and can be limited)
2. Rankings (RP, crafting skill etc.)

Rankings could be cached nightly and only be player names. We could paginate it so we only request as much data as is needed. I feel like rankings is the bare minimum as this is one of the main reasons, as far as I can tell, for people to use a herald.

[thekroko]

new endpoints: https://uthgard.org/herald/