[DevExp] Developer Workflow Experience & Improvements

Objective

Improve the developer experience for blockchain developers.

Origin Document

I encountered a handful of problems going through #359, ranging from bugs, issues and institutional knowledge that we have to capture and iterate on.

Goals

Reduce friction to onboard new developers
Share knowledge on tips & tricks for day-to-day tooling
Remove hidden expected undefined behaviour such as I know this doesn't work that others might not be aware of
Build small tools or add small pieces of documentation that can greatly make everyone more effective

Deliverables

[ ] [E2E Testing] Make it easier to run a single E2E test. Formulating the following CLI command is non-trivial and there is an opportunity to have a make helper to just put in a fuzzy matched name
```
go test -v -tags=test,e2e -count=1 -run \
TestFeatures/Tokenomics_Namespaces/Basic_tokenomics_validation_that_Supplier_mint_equals_Application_burn \
./e2e/tests/...
```
[ ] [Tooling] Make it easier to run any poktrolld command w/ default flags during development. Formulating the following CLI command is non-trivial and there is an opportunity to have a make helper to just put in a fuzzy matched name.
```
poktrolld query application show-application pokt1mrqt5f7qh8uxs27cjm9t7v9e74a9vvdnq5jva4 --home=./localnet/poktrolld --node=tcp://127.0.0.1:36657
```
[x] [Infra] Make it easier to maintain state across LocalNet restarts. Currently, every time we click Trigger Update, the blockchain restarts from height=1
[x] [Infra] Enable hot reloading. This is not in place today unless we click Trigger Update explicitly.
[x] [Infra] Document the order of operations if the Sequencer is restarted. Specifically, relayminers and appgateservers need to have Trigger Update clicked AFTER the sequencer is back up for things to function normally
[ ] [Logging] Document how on-chain logging should be done as a best practice. For example
```
logger = logger.With(
    "session_id", sessionHeader.GetSessionId(),
    "session_end_height", sessionHeader.GetSessionEndBlockHeight(),
    "supplier", supplierAddr)
logger.Info("validated the submitProof message ")
```
Consider which logger should be used:
1. The default cosmos-sdk logger is not very practical when used at the "debug" level as it's overwhelmed with cosmos-sdk specific output.
2. We should prefer using the "debug" level for any log that doesn't require node operator attention (e.g. handling invalid requests). I think this represents (or it should) the majority of logger calls.
Search each module for usages of logger.Info() and logger.Error(). Places where an error is logged and returned are prime examples of mis-leveled logs.

[x] [Debugging] Make it easier to sift through logs. Specifically:

[ ] Document how to filter them
[ ] Understand how to parse the JSON from logger
[ ] Understand how to parse the stack trace from on-chain code

[ ] Understand why sequencer showed no errors even through the relayminer show a panic tx timeout Screenshot 2024-02-13 at 4 41 06 PM

{"level":"error","error":"with hash c82ba5dc57ad0a561b39a75e7205989ae17bb59eef76ada2997456355bde22c1: recovered: runtime error: invalid memory address or nil pointer dereference\nstack:\ngoroutine 58 [running]:\nruntime/debug.Stack()\n\t/Users/olshansky/.asdf/installs/go/1.20.10/go/src/runtime/debug/stack.go:24 +0x64\ngithub.com/cosmos/cosmos-sdk/baseapp.newDefaultRecoveryMiddleware.func1({0x32c10c0, 0x6072a00})\n\t/Users/olshansky/go/pkg/mod/github.com/rollkit/cosmos-sdk@v0.47.3-rollkit-v0.10.6-no-fraud-proofs/baseapp/recovery.go:71 +0x2c\ngithub.com/cosmos/cosmos-sdk/baseapp.newRecoveryMiddleware.func1({0x32c10c0, 0x6072a00})\n\t/Users/olshansky/go/pkg/mod/github.com/rollkit/cosmos-sdk@v0.47.3-rollkit-v0.10.6-no-fraud-proofs/baseapp/recovery.go:39 +0x4c\ngithub.com/cosmos/cosmos-sdk/baseapp.processRecovery({0x32c10c0, 0x6072a00}, 0x4000cd6270)\n\t/Users/olshansky/go/pkg/mod/github.com/rollkit/cosmos-sdk@v0.47.3-rollkit-v0.10.6-no-fraud-proofs/baseapp/recovery.go:28 +0x50\ngithub.com/cosmos/cosmos-sdk/baseapp.processRecovery({0x32c10c0, 0x6072a00}, 0x400e1a2e28)\n\t/Users/olshansky/go/pkg/mod/github.com/rollkit/cosmos-sdk@v0.47.3-rollkit-v0.10.6-no-fraud-proofs/baseapp/recovery.go:33 +0xb8\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).runTx.func1()\n\t/Users/olshansky/go/pkg/mod/github.com/rollkit/cosmos-sdk@v0.47.3-rollkit-v0.10.6-no-fraud-proofs/baseapp/baseapp.go:641 +0xc0\npanic({0x32c10c0, 0x6072a00})\n\t/Users/olshansky/.asdf/installs/go/1.20.10/go/src/runtime/panic.go:890 +0x21c\ngithub.com/pokt-network/poktroll/x/tokenomics/keeper.Keeper.SettleSessionAccounting({{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {{0x0, 0x0}, 0x0, {0x0, ...}, ...}, ...}, ...)\n\t/Users/olshansky/workspace/pocket/poktroll/x/tokenomics/keeper/settle_session_accounting.go:83 +0x938\ngithub.com/pokt-network/poktroll/x/supplier/keeper.msgServer.SubmitProof({{{0x47846d8, 0x4000d8a000}, {0x4748ee0, 0x4001962640}, {0x4748ee0, 0x0}, {{0x47846d8, 0x4000d8a000}, 0x400000e010, {0x4748ee0, ...}, ...}, ...}}, ...)\n\t/Users/olshansky/workspace/pocket/poktroll/x/supplier/keeper/msg_server_submit_proof.go:99 +0x7c8\ngithub.com/pokt-network/poktroll/x/supplier/types._Msg_SubmitProof_Handler.func1({0x476ac38, 0x400e1b2480}, {0x36c53e0, 0x400e199e90})\n\t/Users/olshansky/workspace/pocket/poktroll/x/supplier/types/tx.pb.go:617 +0xb0\ngithub.com/cosmos/cosmos-sdk/baseapp.(*MsgServiceRouter).RegisterService.func2.1({0x476ac38, 0x400e1b2480}, {0x400bf79b48, 0x1aea8}, 0x400e1a6580, 0x400e1a2318)\n\t/Users/olshansky/go/pkg/mod/github.com/rollkit/cosmos-sdk@v0.47.3-rollkit-v0.10.6-no-fraud-proofs/baseapp/msg_service_router.go:118 +0xe8\ngithub.com/pokt-network/poktroll/x/supplier/types._Msg_SubmitProof_Handler({0x370eb60, 0x4000541ec0}, {0x476d2f0, 0x400e1a6580}, 0x4286990, 0x400dd9dea0)\n\t/Users/olshansky/workspace/pocket/poktroll/x/supplier/types/tx.pb.go:619 +0x1cc\ngithub.com/cosmos/cosmos-sdk/baseapp.(*MsgServiceRouter).RegisterService.func2({{0x476abc8, 0x40002e4390}, {0x47854d8, 0x400e18e600}, {{0xb, 0x0}, {0x40024a0b50, 0x8}, 0xbd, {0x284b905d, ...}, ...}, ...}, ...)\n\t/Users/olshansky/go/pkg/mod/github.com/rollkit/cosmos-sdk@v0.47.3-rollkit-v0.10.6-no-fraud-proofs/baseapp/msg_service_router.go:139 +0x43c\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).runMsgs(_, {{0x476abc8, 0x40002e4390}, {0x47854d8, 0x400e18e600}, {{0xb, 0x0}, {0x40024a0b50, 0x8}, 0xbd, ...}, ...}, ...)\n\t/Users/olshansky/go/pkg/mod/github.com/rollkit/cosmos-sdk@v0.47.3-rollkit-v0.10.6-no-fraud-proofs/baseapp/baseapp.go:800 +0x190\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).runTx(0x400107fa40, 0x3, {0x400daa2000, 0x492, 0x492})\n\t/Users/olshansky/go/pkg/mod/github.com/rollkit/cosmos-sdk@v0.47.3-rollkit-v0.10.6-no-fraud-proofs/baseapp/baseapp.go:743 +0xd74\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).DeliverTx(0x400107fa40, {{0x400daa2000, 0x492, 0x492}})\n\t/Users/olshansky/go/pkg/mod/github.com/rollkit/cosmos-sdk@v0.47.3-rollkit-v0.10.6-no-fraud-proofs/baseapp/abci.go:409 +0x184\ngithub.com/cometbft/cometbft/abci/client.(*localClient).DeliverTxAsync(0x40024d6000, {{0x400daa2000, 0x492, 0x492}})\n\t/Users/olshansky/go/pkg/mod/github.com/cometbft/cometbft@v0.37.2/abci/client/local_client.go:82 +0xc8\ngithub.com/cometbft/cometbft/proxy.(*appConnConsensus).DeliverTxAsync(0x40024a9560, {{0x400daa2000, 0x492, 0x492}})\n\t/Users/olshansky/go/pkg/mod/github.com/cometbft/cometbft@v0.37.2/proxy/app_conn.go:106 +0x1e0\ngithub.com/rollkit/rollkit/state.(*BlockExecutor).execute(_, {_, _}, {{{0xb, 0x0}, {0x377b78d, 0x6}}, {0x40024a0b50, 0x8}, 0x1, ...}, ...)\n\t/Users/olshansky/go/pkg/mod/github.com/rollkit/rollkit@v0.10.6/state/executor.go:337 +0x544\ngithub.com/rollkit/rollkit/state.(*BlockExecutor).ApplyBlock(_, {_, _}, {{{0xb, 0x0}, {0x377b78d, 0x6}}, {0x40024a0b50, 0x8}, 0x1, ...}, ...)\n\t/Users/olshansky/go/pkg/mod/github.com/rollkit/rollkit@v0.10.6/state/executor.go:140 +0x104\ngithub.com/rollkit/rollkit/block.(*Manager).applyBlock(_, {_, _}, _)\n\t/Users/olshansky/go/pkg/mod/github.com/rollkit/rollkit@v0.10.6/block/manager.go:788 +0x140\ngithub.com/rollkit/rollkit/block.(*Manager).publishBlock(0x4000db8600, {0x476ab90, 0x400246ee60})\n\t/Users/olshansky/go/pkg/mod/github.com/rollkit/rollkit@v0.10.6/block/manager.go:632 +0xac0\ngithub.com/rollkit/rollkit/block.(*Manager).AggregationLoop(0x4000db8600, {0x476ab90, 0x400246ee60}, 0x0)\n\t/Users/olshansky/go/pkg/m

Non-goals / Non-deliverables

Refactor any existing code
Add new on-chain or off-chain functionality
Build non-trivial tooling

General deliverables

[ ] Comments: Add/update TODOs and comments alongside the source code so it is easier to follow.
[ ] Testing: Add new tests (unit and/or E2E) to the test suite.
[ ] Makefile: Add new targets to the Makefile to make the new functionality easier to use.
[ ] Documentation: Update architectural or development READMEs; use mermaid diagrams where appropriate.

Creator: @Olshansk Co-Owners: @red-0ne @okdas

pokt-network / poktroll

[DevExp] Developer Workflow Experience & Improvements #384