DefiantLabs / cosmos-upgrades

a tool to search for scheduled cosmos upgrades
MIT License
4 stars 2 forks source link

Implement production ready setup with reader/writer worker split #30

Open pharr117 opened 11 months ago

pharr117 commented 11 months ago

The application is currently reliant on the Flask development mode setup since data is gathered and stored in single-process available Python global maps.

Flask is meant to be run behind some server type (sometimesa WSGI, gunicorn being one example). These servers spin up multiple instances of the Python program to help with load balancing and app runs.

The application data system is not prepared for this. Each gunicorn process would have its own global data store, meaning the app would be bashing RPC servers on a per-worker basis.

The application needs to be moved to a more production-ready state. One way would be to implement application modes for reader/writer process splitting such as:

  1. Reader mode -> a Flask application that reads from an external data store and responds to requests
  2. Write mode -> a Python application that writes to an external data store
  3. Data store layer -> A data storage location that both reader and writer interface with

A production deployment would then look like:

  1. A gunicorn server that spins up multiple Reader mode versions of the application
  2. A writer process that writes to the data store
  3. A shared data storage layer
pharr117 commented 11 months ago

Work had started here: https://github.com/DefiantLabs/cosmos-upgrades/tree/feat/production-ready-build

This was a very quick v1 of a possible reader/writer split that was working with a shared a FileSystem cache between the two workers with an example Docker Compose build linking the 2.

A more complex setup with an underlying database layer replacing the FileSystem cache may be a better move.