Increase Hackability: `zingo-cli` regtest mode

[zancas]

Issues that add friction to development on this repo:

• It's not obvious where the wallet file is stored/what it is named.
• It's not obvious where the log files are stored
• There's no obvious pattern for a "Test Setup"

User Story:

A developer wants to work on this repo. They read a README, execute a minimum number of commands and have a "test cli" setup.

The test cli setup includes:

    * A concise README that lets the developer know what they need to know (including):
         - location/name/format of log files (an example line is sufficient to understand format)
         - `zcashd` config/datadirs that are in locations that are clearly described
         - `lightwalletd` config/datadirs that are clearly named/located
    * A locally running `zcashd`, in regtest mode, with:
         - easily reproduced config/datadirs
    * A locally running `lightwalletd` built from zcash main that is exclusively communicating with the local regtest-zcashd, and the test `zingo-cli`
    * a running `zingo-cli` attached exclusively to the locally running `lightwalletd` as described above.

Once I have read the README, I understand the zingo-cli regtest mode setup.

Moving forward I can git worktree add ../FRESHCHECKOUT, and in that repo run a single command, the result is a running zingo-cli as described above.

[dannasessha]

Updating the README seems straightforward enough, but I don't understand, specifcally, how you would want to handle the last part of your ask.

Moving forward I can git worktree add ../FRESHCHECKOUT, and in that repo run a single command, the result is a running zingo-cli as described above.

Would this involve a zcashd (and lightwalletd) as submodules? I would think the sync time with such a setup would be really very significant.

If not, I would think instructions in the README would be opinionated as to where these repos would exist in a filesystem, and perhaps a script could be provided to work with this recipe, but that individual adjustments could be taken by developers with the understanding they might have to tweak specific commands, which seems to run a bit counter to your ask here.

I like the more flexible option of providing a recipe but not dictating or turning this into a larger repo (I also dislike submodules).

Or am I missing another possibilty here?

[zancas]

I am -1 on submodules for this work. Submodules could be part of nice solution, but are unnecessary for the basic tool.

The correct answer is an opinionated README, and a script that fails-on-start if the expected locations are unavailable.

[dannasessha]

Alright, sounds good, thanks for clarifying.

[zancas]

I think we should focus on understanding the current logging systems/patterns so we correctly migrate all logs when running in regtest mode.

[AloeareV]

Also, note that the code currently supports connecting to mainnet or testnet only. Adding support for a regtest network would be a new feature.

[zancas]

To refine the scope here. We intend to add a --regtest flag to zingo-cli. Probably that flag will require the user to provide zcashd and lightwalletd binaries as additional arguments, and fail helpfully in their absence.

[dannasessha]

I worked with this for a while today, and interestingly enough, I could not get zcashd's most recent commit on main to run in regtest mode at all!

For me, testnet and main net work fine, other flags to the cli work fine, but when I either run --regtest on the cli invocation, or point to a config containing regtest=1 I run into a problem.

Several times it gave me an error and suggested running -reindex and reindex-chain. Trying these flags, they would both hang. After running those and cancelling, now when running the regtest versions no error prints, and the process just hangs. (I've let it run for several minutes, ps aux reports the process running, using some smaller amount of CPU [about 6%] and a possibly reasonable seeming amout of virtual memory [3.3 GB - zcashd running on main net is showing me 6.5 GB], but cumulative CPU time does not increase [I see stoppage at 0:15]).

After a clean and rebuild, I am seeing the same.

Can anyone else confirm or deny this behavior by trying locally to run regtest? I could open an issue at the zcash repo but ideally I don't want to do that if I'm the only one for whom this is happening.

[AloeareV]

What flags are you using? Try the command that hangs, but add -printtoconsole and see if that tells you anything?

[zancas]

Please paste the exact hanging command, and the exact error here.

[zancas]

When you launch a (set) of zcashd's in regtest mode (4 are launched by default), the length of the blockchain is 0, by default. The nodes aren't doing much so they don't consume much memory/cpu, resources. @AloeareV hypothesized that regtest mode is running. After my local experiment I agree.

I also agree that running with the -printtoconsole flag should be very informative.

Finally, once you kick up a zcashd in regtest you should be able to communicate with it via zcash-cli. An obvious thing to do is use the regtest-specific generate # command.

[dannasessha]

I confirmed that it wasn't running the way I had it set up before ("custom" config file name, passed with the zcashd invocation to point the daemon at it) - I was getting error: couldn't connect to server: unknown (code -1) from the cli side.

With @AloeareV 's help I was able to get it to a working state. One of the things I did was moved aside my normal wallet.dat and zcash.conf files, and moved in my regtest specific file. The other thing I did was add nuparam lines.

For future reference my config file looks like this now.

     1 regtest=1
     2 nuparams=5ba81b19:1 # Overwinter
     3 nuparams=76b809bb:1 # Sapling
     4 nuparams=2bb40e60:1 # Blossom
     5 nuparams=f5b9230b:1 # Heartwood
     6 nuparams=e9ff75a6:1 # Canopy
     7 nuparams=c2d6d0b4:1 # NU5
     8
     9 #testnet=1
    10 txindex=1
    11 insightexplorer=1
    12 experimentalfeatures=1
    13 rpcuser=xxxxxx
    14 rpcpassword=xxxxxx

Everything below #testnet=1 is there because Aloe suggested I'll need that for lightwalletd connectivity.

[zancas]

The lightwalletd params listed above can be found here:

https://github.com/zcash/lightwalletd/#zcashd

[zancas]

This page also has a lot of useful content:

https://zcash.readthedocs.io/en/latest/rtd_pages/lightwalletd.html

[zancas]

As I mentioned elsewhere, I think that a sane way to establish a base state with respect to zcashd is this:

Assuming a local clone and that you're starting in the root of the clone, with the commit of interest (master) checked out: (1) git worktree add ../$EXPT_NAME (2) cd ../$EXPT_NAME (3) paru -S python-simplejson (4) ./zcutil/clean.sh && ./zcutil/build.sh -j$(nproc) && ./qa/zcash/full_test_suite.py --nozmq

The downsides are that this takes a while and consumes a bit of bandwidth and disk-space, but it also "restates-your-assumptions" to speak.

It's a way of establishing a clean work-bench.

[dannasessha]

Unfortunately, I am not able to pass all tests (by running ./qa/zcash/full_test_suite.py) on a clean-built zcash/master or zingolabs/NO_zmq branches (a minor, targeted, modification from zcash/main), with or without flags.

Running the tests without flags seems to pass most tests, but I see

Running stage rpc
=================

ERROR: "import zmq" failed. Use --nozmq to run without the ZMQ tests.To run zmq tests, see dependency info in /qa/README.md.
Traceback (most recent call last):
  File "/home/ha/zcashes/zcash/qa/pull-tester/rpc-tests.py", line 455, in <module>
    main()
  File "/home/ha/zcashes/zcash/qa/pull-tester/rpc-tests.py", line 223, in main
    import zmq
ModuleNotFoundError: No module named 'zmq'

and then quits with !!! One or more test stages failed !!!

Running the command above fails and prints full_test_suite.py: error: unrecognized arguments: --nozmq, likewise for a single hyphen version, following the NO_zmq branch.

[dannasessha]

Done with PR number 26 over on new-name repo.