[WIP]: remote ancients design sketches

[meowsbits]

Rel #103

core-geth is moving toward a feature providing a remote ancient storage interface. This issue is a place to brainstorm and evaluate design and use patterns.

Existing related behavior includes:

> ./build/bin/geth --help | grep data
   copydb                             Create a local chain from a target chaindata folder
   export-preimages                   Export the preimage database into an RLP stream
   import-preimages                   Import the preimage database from an RLP stream
   inspect                            Inspect the storage size for each type of data in the database
   removedb                           Remove blockchain and state databases
  --datadir value                     Data directory for the databases and keystore (default: "/home/ia/.ethereum")
  --datadir.ancient value             Data directory for ancient chain segments (default = inside chaindata)
  --keystore value                    Directory for the keystore (default = inside the datadir)
  --ethash.cachedir value             Directory to store the ethash verification caches (default = inside the datadir)
  --cache.database value              Percentage of cache memory allowance to use for database io (default: 50)
  --cache.noprefetch                  Disable heuristic state prefetch during block import (less CPU and disk IO, more time waiting for data)
  --ipcpath value                     Filename for IPC socket/pipe within the datadir (explicit paths escape it)
  --miner.extradata value             Block extra data set by the miner (default = client version)
  --metrics.influxdb                  Enable metrics export/push to an external InfluxDB database
  --metrics.influxdb.database value   InfluxDB database name to push reported metrics to (default: "geth")
  --metrics.influxdb.username value   Username to authorize access to the database (default: "test")
  --metrics.influxdb.password value   Password to authorize access to the database (default: "test")
  --snapshot                          Enables snapshot-database mode -- experimental work in progress feature

---

... and carry over @zcstarr 's preamble from the awkwardly replaced #131:

Summary: Currently the ancient store in core-geth is linked directly to writing data to disk. We have found that it might be beneficial to allow developers and users to have ancient storage that acts a service. The ancient storage service would provide the same functionality, but instead accept request over JSON-RPC. This will allow for people to write their own connectors for ancient storage. The full scope of current progress on this work is in the PR #103 . We need to define what the edges of the system for building custom freezer implementation should look like. Defining a developer and consumer workflow.

Acceptance Criteria The goal here is to define the interfaces that we expect users and developers to use when working with a custom remote freezer.

[meowsbits]

Idea / Codename: Same Same but Sockets (SSS)

Use the existing --datadir.ancient flag with "special" value handling to tell geth to NOT treat the value as a filepath, rather to enable a RemoteAncient RPC client using that endpoint.

Geth:

# Enable AncientRemote RPC server configured to use Websocket transport at given endpoint.
# Websocket transport is inferred from URL protocol.
$ geth --datadir.ancient=ws://localhost:8588

# Enable AncientRemote RPC server configured to use a socket at given endpoint.
# Socket transport is inferred from URL suffix.
$ geth --datadir.ancient=/tmp/ancient.ipc

Geth would be responsible for failing hard if the connection with the remote storage provider is not satisfied if and when geth attempts to freeze block data.

RemoteAncient service provider (example 1: S3):

$ go get https://github.com/etclabscore/ethereum-ancient-s3/cmd/...

$ env AWS_PROFILE=me AWS_REGION=us-east-2 ethereum-ancient-s3 --id=classic --websocket ws://localhost:8588

RemoteAncient service provider (example 2: Storj):

$ npm install -g ethereum-ancient-storj

$ ethereum-ancient-storj --foo=bar --socket /tmp/ancient.ipc

Where ethereum-ancient-s3 executable is built from package github.com/etclabscore/ethereum-ancient-s3/cmd, and that package optionally depends on github.com/ethereum/go-ethereum/rlp to decode the incoming block data. It is responsible for establishing a connection with geth via the endpoint and storing and retrieving block data. --id in the example is used to namespace the remote store per chain.

Behind the scenes:

ethereum-ancient-s3 is communicating with geth over JSON-RPC. Methods include ancient_append, ancient_truncate, ancient_get, ancient_n (n being the magnitude of the stored blocks, and thus representing the number of the block which may be next appended), ... etc; all and only the type AncientStore interface methods.

ethereum-ancient-s3 is responsible for using the id field of the RPC messages to ensured ordered management of requests.

[zcstarr]

I agree with the datadir specification of the client.

env AWS_PROFILE=me AWS_REGION=us-east-2 ethereum-ancient-s3 --id=classic --websocket ws://localhost:8588

I think we should leave off ids in the command and leave that up to the people running the processes to determine what is writing where.

Segmenting ethereum-ancient-s3/ ethereum-ancient-storj/ ethereum-ancient-xxxx comes at a cost

an alternative implementation would be to package things under a single umbrella

github.com/etclabscore/ethereum-ancient/services/s3/cmd
github.com/etclabscore/ethereum-ancient/services/storj/cmd
github.com/etclabscore/ethereum-ancient/services/ipfs/cmd

^^ The cost above is ultimate outside developer extensibility

If the goal is to create a platform that allows developers the ability to implement custom ancient stores, as well as provides a stable base to make easier to do this. I think the omni package has siginificant barriers that rule it out.

The omni ethereum-ancient package will require developers to then pr or fork etclabscore-ethereum-ancient services to then develop a new service, becoming a blocker to usage.

I think independent packages make the most sense in the form ethereum-ancient-xxxx

Thinking about it a little further it might make sense to make a base for developers to rely on that do the common things

Perhaps something more like this would make it easier to have specific impls

https://github.com/etclabscore/ethereum-ancient-s3
https://github.com/etclabscore/ethereum-ancient-storj
https://github.com/etclabscore/ethereum-ancient-swarm
... 

The hope here would be that the friction of doing the same thing for specific implementations could be lifted into a common package, that provides JSON-RPC interface and simple CMDLine parsing frame to add additional custom flags to. 

https://github.com/etclabscore/ethereum-ancient/cmd/... //includes cmdline parsing for common parameters and flags https://github.com/etclabscore/ethereum-ancient/server ..// includes framework interfaces for launching a server against the etheruem-ancient api

I think ideally as a developer you simply want to implement  ancient_append, ancient_truncate, ancient_get add a few custom flags and then run the service.

[meowsbits]

So maybe like ethereum-ancient-s3 --websocket ws://localhost:8588 --bucket=s3://my-etc-bucket ?

[meowsbits]

(And I would be a little hesistant to all ethereum-ancient-s3 a "client," maybe I guess I consider it a server?....)

[meowsbits]

I guess I'm inclined to prefer 1 remote ancient-store server executable -> 1 storage domain, eg. go-ethereum-ancient-s3 (maybe it's written in Golang, ethereum-ancient-storj, maybe that one's in Typescript..)?

The common overhead for these servers is

• implementing an RPC server that can support one or many of the geth-available transports, and
• implementing CLI logic.

I guess I don't see either of these as terribly costly, and see the "important" logic as the stuff that actually manages the interactions with whatever storage domain the server uses (ie. S3).

[zcstarr]

See epic long typing of comment , but yeah I think if we want to make the cost lower we just have a common package etheruem-ancient, that does all the scaffolding for hte client interface for you.

[zcstarr]

Then all you do is implement the interface for the methods and define your initialization for whatever it is you're doing. Then startup time for creating a new one of these can be mostly allocated towards the storage service details and not the wiring.

[meowsbits]

but yeah I think if we want to make the cost lower we just have a common package etheruem-ancient, that does all the scaffolding for hte client interface for you.

Indeed, if this fits a developer need I see it as a possible eventual spin-off. Go and Typescript and Javascript and Python, etc., will all have plenty of out-of-box ready options for CLI application scaffolding and RPC server frameworks. If developers need ethereum types and/or RLP decoders they can probably import those as well, either from github.com/ethereum/ or other vendors.

[zcstarr]

I guess I'm inclined to prefer 1 remote ancient-store server executable -> 1 storage domain, eg. go-ethereum-ancient-s3 (maybe it's written in Golang, ethereum-ancient-storj, maybe that one's in Typescript..)?

The common overhead for these servers is

• implementing an RPC server that can support one or many of the geth-available transports, and
• implementing CLI logic.

I guess I don't see either of these as terribly costly, and see the "important" logic as the stuff that actually manages the interactions with whatever storage domain the server uses (ie. S3). Totally one in typescript as well doesn't actually matter what the language is as long as it fits the interface.

[zcstarr]

I think so in terms of how we see this being deployed or used. I think we are likely to see people sidecar this in docker, the other thing to think about potentially is that we need a common package, because our logic is likely to grow a little more complex. We might do leader election for the writer of the data.

If the server logic is all on developers to implement it might be harder to get this to land. or come up with a scheme that works reasonably well, while shielding developers from this additional shared complexity.

[meowsbits]

the other thing to think about potentially is that we need a common package, because our logic is likely to grow a little more complex. We might do leader election for the writer of the data.

In geth? I am imagining geth as a Very Simple :tm: RPC client which is attempting to connect to an RPC server in order to put and get some data. One connection to one server. If that server, then, wants to scale up get :superhero: fancy, then it surely can. But I don't see any need for geth to do fancy stuff.

[meowsbits]

If the server logic is all on developers to implement it might be harder to get this to land.

I'm not convinced by this. Spinning up skeleton servers and CLI apps is gravy for even beginner-intermediate developers in every language I've worked with. Fancy servers handling huge loads can get more complex, but the anticipated load here of one client (or maybe you even service every single node on the Ethereum network, a whopping 10k clients...) is far from that. Boxing the ancient-store service(s) each as a single stand-alone app in any language might be a big selling point; do it however you want, and you get full creative ownership of the whole thing. As fancy or simple as you want. Just make sure your server handles these 6 JSON-RPC methods over at least 1 of 2 transports of your choice.

[zcstarr]

Geth no not geth, I'm talking about the ancient store, having the flexibility to not right data. There is the potential to have the independent ancient store servers if they are apart of a quorum do some sort of leader election. Doing that is easier to do or enable from a common interface.

I'm making a claim for ethereum-ancient to be a ancient store services framework, that handles cli and server logic.

ethereum-ancient-store-s3 essentially just inherits from ethereum-ancient-store package or uses the interfaces there and the server encapsulation to do ancient store stuff.

Of course there is nothing blocking anyone from doing something separate, because the client just reads and writes data.

Ultimately the server decides what persists to disk or goes wherever.

[meowsbits]

I'm talking about the ancient store, having the flexibility to not right data. There is the potential to have the independent ancient store servers if they are apart of a quorum do some sort of leader election. Doing that is easier to do or enable from a common interface

So this is a very fancy hypothetical ancient-store server (or servers). If developers want this, sure they can pull out common packages and refactor and build an entire ancient-store framework ecosystem if they want. But AFAIU it's far from necessary for getting of the ground.

[zcstarr]

I'm talking about the ancient store, having the flexibility to not right data. There is the potential to have the independent ancient store servers if they are apart of a quorum do some sort of leader election. Doing that is easier to do or enable from a common interface

So this is a very fancy hypothetical ancient-store server (or servers). If developers want this, sure they can pull out common packages and refactor and build an entire ancient-store framework ecosystem if they want. But AFAIU it's far from necessary for getting of the ground.

I think etheruem-ancient would probably only be built after the s3 store was built, and I'd use it to not copy pasta the s3 implementation for the storj implementation

[zcstarr]

So optional but a nice thing to make writing these easy.

[meowsbits]

OK, so I think we're agreed that core-geth wants to provide an RPC client. Maybe we should think a little more about the nitty-gritty details of that? (Configuration and CLI flags/options and defaults... these will be the mundane but important design decisions).

For example, establishing "special" values for --datadir.ancient would mean breaking some existing observable/usable edge cases, like if someone has an existing FS ancient store at /data/mydata.ipc, where /data/mydata.ipc is actually a normal dir. Weird, but possible. (There are workarounds/logic that can be added to handle this... Does the path already exist? Is it writable? Is it a directory? So then if it is writable but it's a directory, should geth... fail because it's not a socket? Or proceed with standard FS ancient store in that weirdly-named .ipc directory? Should it just complain and exit because it's confused?)

[zcstarr]

I tthink the breaking change doesn't really bother me super much but we could always add a flag for remote and ignore parsing it otherwise. so if remote is not enabled then we just write to it like a directory path.

Is it important that we don't add an additional flag ? I think if you're using this you're intentionally setting this up does it feel hidden if it is something like

--datadir.ancient = http://localhost:1337 --ancient.remote

Then it everything has distinct states it's more of a one to one mapping on (datadir.ancient, ancient.remote)

[meowsbits]

What about --ancient.remote=ws://myhost.net. Would need to be utils.CheckIncompatible'd with --datadir.ancient.

[zcstarr]

What about --ancient.remote=ws://myhost.net. Would need to be utils.CheckIncompatible'd with --datadir.ancient. I think this is problematic from a UX don't make me think twice perspective.

[meowsbits]

Or as we have it in the geth branch now (I think?) --datadir.ancient.remote=__

[zcstarr]

--datadir.ancient.remote=ws://myhost.net
--datadir.ancient = /var/foo/bar
``` is illegal probably makes the most sense it's explict then and it conveys the same association

[meowsbits]

This feature has been merged.