OpenSource-Deadlock-Tools / DevLock-Documents

Repository used for collecting any design notes and other written things that should be tracked and available for everyone to view
3 stars 1 forks source link

HLTV Match Data Pipeline POC #2

Open WanderingArbor opened 1 month ago

WanderingArbor commented 1 month ago

Provide a proof-of-concept for delivering raw match data from Valve's HLTV service by running "spectator injection" on matches in-progress. The service aims to read as many streams of match data as possible within reasonable rate limits to provide an ongoing repository of match event data for further work.

Ideal pipeline is to run a light rmq setup for stateful scraping and sanity.

The workflow as it stands as follows:

Image

This issue will be updated with more details as we go.

ankbhatia19 commented 1 month ago

RabbitMQ - Spectate Queue has been implemented in the Spectator Server repo, currently does not have an Error Queue implementation.

ankbhatia19 commented 1 month ago

Image

Posting this for self-reference