search

loda-lang / loda-cpp

Runtime and miner for the LODA language written in C++
https://loda-lang.org/
Apache License 2.0
21 stars 1 forks source link

Scalable OEIS proxy #67

Closed ckrause closed 2 years ago

ckrause commented 2 years ago

We want to reduce the traffic on the OEIS website for fetching stripped.gz, names.gz and the b-files. Miners should not fetch the files directly from OEIS, but from a LODA proxy which caches the files.

We would implement a simple proxy server that caches the files locally. For scalability: we add a look-up endpoint to the (central) API server. Every miner calls the look-up API once during start up. It fetches a list of servers (and their capabilities). It randomly selects one of the available servers and uses it for its lifetime (probably change after 12h or so). That would allow us to scale, also for submission of programs. It would also allow us blue/green deployments.

ckrause commented 2 years ago

We have an OEIS proxy now. It is not really scalable, but it caches the files, which is good enough for now.