Harden core.blockstack.org service

[jackzampolin]

Discuss updating the core.blockstack.org service to run effectively on multiple hosts

[kantai]

Just to clarify -- I think the biggest issue here isn't documentation or cleaning up services that aren't running -- it's improving our current infrastructure. The core and node services both need some improvements, as performance can (and does) deteriorate on them with loads they should be capable of handling.

[jackzampolin]

I think replacing the node.blockstack.org service should be pretty straight forward, however hardening the core.blockstack.org service will be a bit more complicated.

Currently the service is tightly coupled to the VM that is running it and consists of a couple of different parts. I think we would require a couple of changes to make it more scalable:

• Make the long running jobs update the database (mongodb), and not dump files on disk. This would allow them to run as a continuous process and to run in a separate VM.
• Decouple the flask app. Currently the flask app is tightly coupled to uwsgi and the nginx instance. If we could configure the flask app with a mongodb_connection and a blockstack_api_connection then this could run in a separate VM and just expose a port.

If those two changes we made we could compose core.blockstack.org of the following services:

• blockstack-core: Either rely on node.blockstack.org or start a new array of blockstack-core nodes for this service to rely on
• blockstack api: These could connect to node.blockstack.org or share a VM/host with the blockstack-core nodes that support this service
• indexer(?): This would be a python process that runs continuously, updating the mongodb instance that the flask app draws from
• flask app: This app stitches together data from blockstack api and mongodb and responds to user requests.
• mongodb
• ngnix

These small changes would allow for a modular deployment where we could independently scale each of the components to deal with load. What do you think @kantai?

[kantai]

Decouple the flask app. Currently the flask app is tightly coupled to uwsgi and the nginx instance. If we could configure the flask app with a mongodb_connection and a blockstack_api_connection then this could run in a separate VM and just expose a port.

The flask app isn't coupled to uwsgi -- it could be run by any other python application server, but uwsgi is pretty easy to get running, and by using the uwsgi bindings in nginx, we get a little bit of a performance boost, though just running it in HTTP mode is fine too.

The way it is coupled right now, is we have

nginx -> flask app (running in uwsgi) -> api (6270) -> blockstackd
                          \-> mongodb

If we just configure the client.ini on the api node to point at node.blockstack.org, that decouples the blockstackd component. Decoupling the flask app from the api service is doable, but I'm not sure there's much benefit to be reaped there -- that api service doesn't do much CPU-wise and it's threaded.

Decoupling it from the mongodb instance is a great idea -- we could add an environment variable there to support this.

[jackzampolin]

Have a service running at core.technofractal.com I am testing for the replacement.

[jackzampolin]

This is complete and in an operational state: https://core.technofractal.com

All that needs to be done now is to get some certs and switch the DNS!

Full details and troubleshooting info -> https://github.com/blockstackinc/devops/tree/master/core.blockstack.org/docker

[jackzampolin]

DNS has been switched. Closing this issue.