out-of-box symmetrical authentication plugin

[matthewdarwin]

When the authentication plugin system got re-written a few weeks ago, the out-of-box example plugins were lost. Someone has been using BASIC AUTH type plugin to protect their infrastructure as it runs in multiple datacentres.

Is there any way to have some out-of-the box sidecar auth plugin that just handles basic auth with a hard-coded password in the firehose config?

[maoueh]

@jubeless

[aasseman]

Bump! I'm stuck at v1.4.4 because of this.

[maoueh]

@aasseman I brought back secret:// support. For firehose-ethereum too, if you can compile from source and let me know, that would be great. Otherwise I'm going to make a release next week.

If you use Docker, I'll link the proper image soon.

[maoueh]

ghcr.io/streamingfast/firehose-ethereum:f2404c4

[aasseman]

I gradually updated all the way up to firehose-ethereum:v1.4.16-geth-v1.13.2-fh2.3 successfully. Then only to firehose-ethereum:f2404c4, but the reader-node instantly crashes (even before starting the Geth node it seems):

{"severity":"INFO","timestamp":"2023-10-01T14:10:06.764003418-04:00","logger":"fireeth","message":"starting with config file ''","logging.googleapis.com/labels":{}}
{"severity":"INFO","timestamp":"2023-10-01T14:10:06.764207924-04:00","logger":"fireeth","message":"launching applications: reader-node","logging.googleapis.com/labels":{}}
{"severity":"INFO","timestamp":"2023-10-01T14:10:06.764364156-04:00","logger":"fireeth","message":"building node arguments","node-type":"geth","node-role":"reader","logging.googleapis.com/labels":{}}
{"severity":"INFO","timestamp":"2023-10-01T14:10:06.76440767-04:00","logger":"reader","message":"adding superviser shutdown to plugins","plugin_name":"ToConsoleLogPlugin","logging.googleapis.com/labels":{}}
{"severity":"INFO","timestamp":"2023-10-01T14:10:06.764428301-04:00","logger":"reader","message":"registered log plugin","plugin count":1,"logging.googleapis.com/labels":{}}
{"severity":"INFO","timestamp":"2023-10-01T14:10:06.764446165-04:00","logger":"reader","message":"created geth superviser","superviser":{"binary":"geth","arguments":["--networkid=1","--datadir=/var/lib/geth","--ipcpath=/var/lib/fireeth/reader/ipc","--port=30305","--http","--http.api=eth,net,web3","--http.port=8547","--http.addr=0.0.0.0","--http.vhosts=*","--firehose-enabled","--cache=8192","--maxpeers=100","--syncmode","full","--metrics","--metrics.addr=0.0.0.0","--metrics.port=6061","--port=30310","--discovery.port=30310","--http.port=8545","--authrpc.port=8547","--authrpc.addr=0.0.0.0","--authrpc.vhosts=*","--authrpc.jwtsecret=/var/lib/jwt-eth/jwt.hex","--snapshot=true","--txlookuplimit=1000","--nat=extip:51.81.57.78"],"data_dir":"/var/lib/geth","ipc_file_path":"/var/lib/fireeth/reader/ipc","last_block_seen":0,"enode_str":""},"logging.googleapis.com/labels":{}}
{"severity":"INFO","timestamp":"2023-10-01T14:10:06.764468818-04:00","logger":"reader","message":"creating operator","options":{"Bootstrapper":null,"EnableSupervisorMonitoring":true,"ShutdownDelay":0},"logging.googleapis.com/labels":{}}
{"severity":"INFO","timestamp":"2023-10-01T14:10:06.764520733-04:00","logger":"reader","message":"parsing backup configs","configs":[],"factory_count":1,"logging.googleapis.com/labels":{}}
{"severity":"INFO","timestamp":"2023-10-01T14:10:06.764539106-04:00","logger":"reader","message":"parsing backup known factory","name":"gke-pvc-snapshot","logging.googleapis.com/labels":{}}
{"severity":"INFO","timestamp":"2023-10-01T14:10:06.764556676-04:00","logger":"reader","message":"backup config","config":[],"backup_module_count":0,"backup_schedule_count":0,"logging.googleapis.com/labels":{}}
{"severity":"INFO","timestamp":"2023-10-01T14:10:06.764606303-04:00","logger":"reader","message":"creating mindreader plugin","one_blocks_store_url":"s3://seaweedfs:8333/fireeth-one-blocks?region=none&insecure=true&access_key_id=any_no_empty_key&secret_access_key=any_no_empty_key","one_block_suffix":"default","working_directory":"/var/lib/fireeth/reader/work","start_block_num":0,"stop_block_num":0,"channel_capacity":100,"with_head_block_updater":true,"with_shutdown_func":true,"logging.googleapis.com/labels":{}}
{"severity":"INFO","timestamp":"2023-10-01T14:10:06.76473062-04:00","logger":"reader","message":"creating new mindreader plugin","logging.googleapis.com/labels":{}}
{"severity":"INFO","timestamp":"2023-10-01T14:10:06.764753699-04:00","logger":"reader","message":"adding superviser shutdown to plugins","plugin_name":"MindReaderPlugin","logging.googleapis.com/labels":{}}
{"severity":"INFO","timestamp":"2023-10-01T14:10:06.764770396-04:00","logger":"reader","message":"registered log plugin","plugin count":2,"logging.googleapis.com/labels":{}}
{"severity":"INFO","timestamp":"2023-10-01T14:10:06.76479352-04:00","logger":"reader","message":"adding superviser shutdown to plugins","plugin_name":"TrxPoolLogPlugin","logging.googleapis.com/labels":{}}
{"severity":"INFO","timestamp":"2023-10-01T14:10:06.764810549-04:00","logger":"reader","message":"registered log plugin","plugin count":3,"logging.googleapis.com/labels":{}}
{"severity":"INFO","timestamp":"2023-10-01T14:10:06.764883739-04:00","logger":"reader","message":"running node manager app","config":{"StartupDelay":0,"HTTPAddr":":13009","ConnectionWatchdog":false,"GRPCAddr":":13010"},"mindreader":true,"logging.googleapis.com/labels":{}}
{"severity":"INFO","timestamp":"2023-10-01T14:10:06.764957443-04:00","logger":"reader","message":"retrieved hostname from os","hostname":"reader-node-0","logging.googleapis.com/labels":{}}
{"severity":"INFO","timestamp":"2023-10-01T14:10:06.764979233-04:00","logger":"reader","message":"starting mindreader gRPC server","logging.googleapis.com/labels":{}}
{"severity":"INFO","timestamp":"2023-10-01T14:10:06.765252833-04:00","logger":"dgrpc","message":"standard server created","logging.googleapis.com/labels":{}}
{"severity":"INFO","timestamp":"2023-10-01T14:10:06.76545236-04:00","logger":"reader","message":"launching operator","logging.googleapis.com/labels":{}}
{"severity":"INFO","timestamp":"2023-10-01T14:10:06.765459118-04:00","logger":"reader","message":"launching operator HTTP server","http_listen_addr":":13009","logging.googleapis.com/labels":{}}
{"severity":"INFO","timestamp":"2023-10-01T14:10:06.765697404-04:00","logger":"reader","message":"launching gRPC server","listen_addr":":13010","logging.googleapis.com/labels":{}}
{"severity":"INFO","timestamp":"2023-10-01T14:10:06.765776772-04:00","logger":"reader","message":"serving gRPC","listen_addr":":13010","logging.googleapis.com/labels":{}}
{"severity":"INFO","timestamp":"2023-10-01T14:10:06.765780067-04:00","logger":"reader","message":"starting webserver","http_addr":":13009","logging.googleapis.com/labels":{}}
{"severity":"INFO","timestamp":"2023-10-01T14:10:06.765859258-04:00","logger":"reader","message":"operator ready to receive commands","logging.googleapis.com/labels":{}}
{"severity":"INFO","timestamp":"2023-10-01T14:10:06.765880071-04:00","logger":"reader","message":"received operator command","command":"start","params":null,"logging.googleapis.com/labels":{}}
{"severity":"INFO","timestamp":"2023-10-01T14:10:06.765907089-04:00","logger":"reader","message":"preparing for start","logging.googleapis.com/labels":{}}
{"severity":"INFO","timestamp":"2023-10-01T14:10:06.765924591-04:00","logger":"reader","message":"preparing to start chain","logging.googleapis.com/labels":{}}
{"severity":"INFO","timestamp":"2023-10-01T14:10:06.765942997-04:00","logger":"reader","message":"starting mindreader","logging.googleapis.com/labels":{}}
{"severity":"INFO","timestamp":"2023-10-01T14:10:06.766025073-04:00","logger":"reader","message":"creating new command instance and launch read loop","binary":"geth","arguments":["--networkid=1","--datadir=/var/lib/geth","--ipcpath=/var/lib/fireeth/reader/ipc","--port=30305","--http","--http.api=eth,net,web3","--http.port=8547","--http.addr=0.0.0.0","--http.vhosts=*","--firehose-enabled","--cache=8192","--maxpeers=100","--syncmode","full","--metrics","--metrics.addr=0.0.0.0","--metrics.port=6061","--port=30310","--discovery.port=30310","--http.port=8545","--authrpc.port=8547","--authrpc.addr=0.0.0.0","--authrpc.vhosts=*","--authrpc.jwtsecret=/var/lib/jwt-eth/jwt.hex","--snapshot=true","--txlookuplimit=1000","--nat=extip:51.81.57.78"],"logging.googleapis.com/labels":{}}
{"severity":"INFO","timestamp":"2023-10-01T14:10:06.766097789-04:00","logger":"reader","message":"successfully start service","logging.googleapis.com/labels":{}}
{"severity":"INFO","timestamp":"2023-10-01T14:10:06.766118241-04:00","logger":"reader","message":"operator ready to receive commands","logging.googleapis.com/labels":{}}
{"severity":"INFO","timestamp":"2023-10-01T14:10:06.76670182-04:00","logger":"reader","message":"starting consume flow","logging.googleapis.com/labels":{}}
{"severity":"INFO","timestamp":"2023-10-01T14:10:06.769679097-04:00","logger":"reader","message":"operator is terminating","error":"instance \"geth\" stopped (exit code: -1), shutting down","logging.googleapis.com/labels":{}}
{"severity":"INFO","timestamp":"2023-10-01T14:10:06.769725091-04:00","logger":"reader","message":"superviser is terminating","logging.googleapis.com/labels":{}}
{"severity":"INFO","timestamp":"2023-10-01T14:10:06.769745545-04:00","logger":"reader","message":"supervisor received a stop request, terminating node process","logging.googleapis.com/labels":{}}
{"severity":"INFO","timestamp":"2023-10-01T14:10:06.769776655-04:00","logger":"reader","message":"underlying process is not running, nothing to do","logging.googleapis.com/labels":{}}
{"severity":"INFO","timestamp":"2023-10-01T14:10:06.769794302-04:00","logger":"reader","message":"shutting down plugins","last_exit_code":-1,"logging.googleapis.com/labels":{}}
{"severity":"INFO","timestamp":"2023-10-01T14:10:06.769812746-04:00","logger":"reader","message":"stopping plugin","plugin_name":"ToConsoleLogPlugin","logging.googleapis.com/labels":{}}
{"severity":"INFO","timestamp":"2023-10-01T14:10:06.769829993-04:00","logger":"reader","message":"stopping plugin","plugin_name":"MindReaderPlugin","logging.googleapis.com/labels":{}}
{"severity":"INFO","timestamp":"2023-10-01T14:10:06.769846813-04:00","logger":"reader","message":"mindreader is stopping","logging.googleapis.com/labels":{}}
{"severity":"INFO","timestamp":"2023-10-01T14:10:06.769864835-04:00","logger":"reader","message":"waiting until consume read flow (i.e. blocks) is actually done processing blocks...","logging.googleapis.com/labels":{}}
{"severity":"INFO","timestamp":"2023-10-01T14:10:06.769882955-04:00","logger":"reader","message":"lines channel has been closed","logging.googleapis.com/labels":{}}
{"severity":"INFO","timestamp":"2023-10-01T14:10:06.769909825-04:00","logger":"reader","message":"reached end of console reader stream, nothing more to do","logging.googleapis.com/labels":{}}
{"severity":"INFO","timestamp":"2023-10-01T14:10:06.76992801-04:00","logger":"reader","message":"all blocks in channel were drained, exiting read flow","logging.googleapis.com/labels":{}}
{"severity":"INFO","timestamp":"2023-10-01T14:10:06.769952316-04:00","logger":"reader","message":"archiver selector is terminating","logging.googleapis.com/labels":{}}
{"severity":"INFO","timestamp":"2023-10-01T14:10:06.769975467-04:00","logger":"reader","message":"archiver selector is terminated","logging.googleapis.com/labels":{}}
{"severity":"INFO","timestamp":"2023-10-01T14:10:06.76999343-04:00","logger":"reader","message":"archiver Terminate done","logging.googleapis.com/labels":{}}
{"severity":"INFO","timestamp":"2023-10-01T14:10:06.770011184-04:00","logger":"reader","message":"consume read flow terminate","logging.googleapis.com/labels":{}}
{"severity":"INFO","timestamp":"2023-10-01T14:10:06.770029506-04:00","logger":"reader","message":"stopping plugin","plugin_name":"TrxPoolLogPlugin","logging.googleapis.com/labels":{}}
{"severity":"INFO","timestamp":"2023-10-01T14:10:06.77004663-04:00","logger":"reader","message":"all plugins closed","logging.googleapis.com/labels":{}}
{"severity":"INFO","timestamp":"2023-10-01T14:10:06.770063983-04:00","logger":"reader","message":"operator is waiting for superviser to shutdown","error":"instance \"geth\" stopped (exit code: -1), shutting down","logging.googleapis.com/labels":{}}
{"severity":"INFO","timestamp":"2023-10-01T14:10:06.770082181-04:00","logger":"reader","message":"operator done waiting for superviser to shutdown","error":"instance \"geth\" stopped (exit code: -1), shutting down","logging.googleapis.com/labels":{}}
{"severity":"INFO","timestamp":"2023-10-01T14:10:06.770100312-04:00","logger":"reader","message":"chain operator terminated shutting down mindreader app","logging.googleapis.com/labels":{}}
{"severity":"INFO","timestamp":"2023-10-01T14:10:06.770121092-04:00","logger":"reader","message":"operator ready to receive commands","logging.googleapis.com/labels":{}}
{"severity":"INFO","timestamp":"2023-10-01T14:10:06.77013892-04:00","logger":"reader","message":"superviser terminating, waiting for operator...","logging.googleapis.com/labels":{}}
{"severity":"ERROR","timestamp":"2023-10-01T14:10:06.770193292-04:00","logger":"fireeth","message":"\n################################################################\nFatal error in app reader-node:\n\ninstance \"geth\" stopped (exit code: -1), shutting down\n################################################################\n","logging.googleapis.com/labels":{},"serviceContext":{"service":"unknown"}}
{"severity":"INFO","timestamp":"2023-10-01T14:10:06.770260084-04:00","logger":"fireeth","message":"application reader-node shutdown unexpectedly, quitting","logging.googleapis.com/labels":{}}
{"severity":"INFO","timestamp":"2023-10-01T14:10:06.770286001-04:00","logger":"fireeth","message":"waiting for all apps termination...","logging.googleapis.com/labels":{}}
{"severity":"INFO","timestamp":"2023-10-01T14:10:06.770306507-04:00","logger":"fireeth","message":"all apps terminated gracefully","logging.googleapis.com/labels":{}}
Error: instance "geth" stopped (exit code: -1), shutting down
{"severity":"ERROR","timestamp":"2023-10-01T14:10:06.770352278-04:00","logger":"derr","message":"dfuse","error":"instance \"geth\" stopped (exit code: -1), shutting down","logging.googleapis.com/labels":{},"serviceContext":{"service":"unknown"}}

[aasseman]

Reverting back to firehose-ethereum:v1.4.16-geth-v1.13.2-fh2.3 fixes the reader-node, so the issue is indeed introduced in https://github.com/streamingfast/firehose-ethereum/commit/f2404c48c5a1ad7eed6ca8373da7c3fbd1428464.

[aasseman]

Tried again with the whole stack on firehose-ethereum:v1.4.16-geth-v1.13.2-fh2.3 except for the firehose on firehose-ethereum:f2404c4, and it works nicely. Happy with the correct auth token, and unhappy otherwise, as expected.

[maoueh]

Yeah sorry, you cannot use the image I linked with a reader-node because in the image I linked, the geth binary is not present.

Let me build a full image so you can test fully.

[maoueh]

There you go, same image as before but with geth bundled in so you can run it on a reader-node: ghcr.io/streamingfast/firehose-ethereum:f2404c4-geth-v1.13.2-fh2.3

[aasseman]

Thanks @maoueh ! I wrongly assumed that the container was built the same way as the other firehose-ethereum ones. Running just the firehose with the f2404c4 works fine too. I'll update to f2404c4-geth-v1.13.2-fh2.3 nevertheless.

[maoueh]

There is actually two kind of images built, "base" and "bundled" which contains base + chain's instrumented binary.

For our own infra needs, only reader-node is run from the bundled version (versions ending with -<fork>-<version>-<firehose-protocol>). The rest is all run through the "base" image.