search

ForsakenNGS / LogTracker

LogTracker (TBC Classic) Main Addon
MIT License
2 stars 6 forks source link

would you be willing to share the method you use to get the data from wcl? #6

Closed mspk7305 closed 1 year ago

mspk7305 commented 1 year ago

would you be willing to share the method you use to get the data from wcl?

ForsakenNGS commented 1 year ago

I'm using the official warcraft logs api (https://www.warcraftlogs.com/v2-api-docs/warcraft/) roughly like this:

Obviously thats a simplification, but should give a rough idea of my approach. The most important thing is reducing the number of api calls as far as possible and being a patreon for more api calls xD

mspk7305 commented 1 year ago

Thanks I already built my own bot to pull down data for my server every day and patch it into logtracker. It's up to 13 thousand players in the database already!

On Sat, Nov 19, 2022, 8:27 PM ForsakenNGS @.***> wrote:

I'm using the official warcraft logs api ( https://www.warcraftlogs.com/v2-api-docs/warcraft/) roughly like this:

  • Scan for new reports (GraphQL Report) every few minutes using the filters startTime, endTime and zoneId
    • Only scan for reports that are at least 12 hours old to avoid still active live-logs
    • Only include reports for relevant raids to reduce the required api calls
    • Add all players within the matching reports into a update queue (including the information which raid was logged)
  • Update the player logs (GraphQL Character) using the zoneRankings
    • If the player is not known at all, do a full update (all raid for both 10 and 25 players)
    • If the player is already known, do a partial update using the information from the queue (if a naxx25 report was scanned e.g. only update that)

Obviously thats a simplification, but should give a rough idea of my approach. The most important thing is reducing the number of api calls as far as possible and being a patreon for more api calls xD

— Reply to this email directly, view it on GitHub https://github.com/ForsakenNGS/LogTracker/issues/6#issuecomment-1321017104, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAM57BCY7MWKHBNOOTGB7E3WJGEAVANCNFSM6AAAAAARLSW5B4 . You are receiving this because you authored the thread.Message ID: @.***>

ForsakenNGS commented 1 year ago

If my server-side solution was more clean and documented I'd invite you in right away :D

Before sharing the code I would certainly have to make sure all my credentials are safe and at least the core workflows are documented, which is not an easy task. Especially the git publishing method is, well, let's call it not that sophisticated.

I'm currently at almost 1.5 million players and with my latest rework for update queues I expect it to double in the next 2-3 weeks.

Mind if I ask what issue made you work on your own grabber?

mspk7305 commented 1 year ago

Just lack of data for my server. As of last night I have about 14k in the database, but the process to retrieve them takes about 3 hours so I have it scheduled to run at 3AM every day on a server. Most of the import time is because it sleeps while it gets rate limited. It would be nice to be able to just get a dump of the WCL database but alas.... we have to resort to this clownshow nonsense.

On Mon, Nov 21, 2022 at 10:39 AM ForsakenNGS @.***> wrote:

If my server-side solution was more clean and documented I'd invite you in right away :D

Before sharing the code I would certainly have to make sure all my credentials are safe and at least the core workflows are documented, which is not an easy task. Especially the git publishing method is, well, let's call it not that sophisticated.

I'm currently at almost 1.5 million players and with my latest rework for update queues I expect it to double in the next 2-3 weeks.

Mind if I ask what issue made you work on your own grabber?

— Reply to this email directly, view it on GitHub https://github.com/ForsakenNGS/LogTracker/issues/6#issuecomment-1322344827, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAM57BEDSPL7PYPM64J6XZLWJOQTHANCNFSM6AAAAAARLSW5B4 . You are receiving this because you authored the thread.Message ID: @.***>