goyle
commented
3 years ago Description
Goal of this issue is to test the compact_chain API method.
Prerequisites
- GRIN-Node
Test Procedure
- Run GRIN-Node in Owner API listener mode
- Check how much memory is occupied by the chain data
- Run the
compact_chainAPI method - Check how much memory is occupied by the chain data again
Expected Result
We would expect would expect the chain to be compacted and take less space. Output should match the example. Include all cURL requests and responses.
Include the exact version of your grin-node and also your environment
uname -aEnvironment
OS: Debian 10 \
Grin Node: grin 5.0.0-rc.1 \
System Info: Linux debian 4.19.0-13-amd64 #1 SMP Debian 4.19.160-2 (2020-11-28) x86_64 GNU/Linux
Preface
For my first test run, running the compact_chain API method on a brand new node has no observable effect on the disk usage of the chain data. My second test run was more successful when I used the archive node I first ran in issue #5.
Steps
1: Building the GRIN-Node
Dropdown of the full steps for building a GRIN-Node with archive mode.
1. Download GRIN-Node v5.0.0-rc.1. ```shell $ wget https://github.com/mimblewimble/grin/archive/v5.0.0-rc.1.tar.gz ``` 1. 1. Extract `v5.0.0-rc.1.tar.gz`. ```shell $ tar -xvf v5.0.0-rc.1.tar.gz ``` * Output should be as follows. ``` grin-5.0.0-rc.1/ grin-5.0.0-rc.1/.cargo/ grin-5.0.0-rc.1/.cargo/config grin-5.0.0-rc.1/.ci/ grin-5.0.0-rc.1/.ci/general-jobs grin-5.0.0-rc.1/.ci/release.yml grin-5.0.0-rc.1/.ci/test.yml grin-5.0.0-rc.1/.ci/windows-release.yml grin-5.0.0-rc.1/.editorconfig grin-5.0.0-rc.1/.github/ ... ``` 1. Install Rust. ```shell $ curl https://sh.rustup.rs -sSf | sh; source $HOME/.cargo/env ``` * Proceed with installation with default profile. ``` default host triple: x86_64-unknown-linux-gnu default toolchain: stable (default) profile: default modify PATH variable: yes ``` * Output should be as follows. ``` stable-x86_64-unknown-linux-gnu installed - rustc 1.48.0 (7eac88abb 2020-11-16) ``` 1. Download dependencies, including `libcursesw5`. ``` # apt install build-essential git tor cmake git libgit2-dev clang libncursesw5 libncurses5-dev libncursesw5-dev zlib1g-dev pkg-config libssl-dev llvm ``` 1. Build GRIN-Node v5.0.0-rc.1. ```shell $ cd grin-5.0.0-rc.1/ $ cargo build --release ``` 1. Configure the nodes to save its logs and chain data in the current directory. ```shell $ cd target/release/ $ ./grin server config ``` * The output will be as follows. ``` grin-server.toml file configured and created in current directory ``` 1. Enable archive mode (line 62) and debug mode (line 237) in the `grin-server.toml` file. ```toml #run the node in "full archive" mode (default is fast-sync, pruned node) archive-mode = true ``` ```toml #log level for file: Error, Warning, Info, Debug, Trace file_log_level = "Debug" ``` 1. Run the GRIN-Node and wait until it is fully synced. ```shell ./grin ```2: Running GRIN-Node in Owner API Listener Mode
- Initialize the GRIN-Node. The node will start in the owner API listener mode by default.
$ cd grin-5.0.0-rc.1/target/release/ $ ./grin - The TUI should now appear and your GRIN-Node will be running.
3: Checking how much memory is occupied by the chain data
- Change to the directory where
chain_datais, which should be in~/.grin/main/, If you ran./grin server configafter building the node, it should be in~/grin-5.0.0-rc.1/target/release/.$ cd ~/.grin/main/ - Run the following command.
$ du -c chain_data/ - Save the output.
4: Running the compact_chain API method
- Open up a new terminal.
-
Create a new bash script to run the cURL request.
compact_chain.sh#!/bin/bash # Get the grin node foreign api password secret=$(cat ~/.grin/main/.api_secret) # Call the node's foreign api "compact_chain" method curl -ugrin:$secret localhost:3413/v2/owner -d '{"jsonrpc": "2.0", "method": "compact_chain", "params": [], "id": 1}' -v -w "\n" - Run the bash script.
$ bash compact_chain.sh -
The API response will appear as expected at the end of a long verbose cURL output.
Expected response of running the
compact_chainmethod:{ "id": 1, "jsonrpc": "2.0", "result": { "Ok": null } }
5: Checking how much memory is occupied by the chain data again
- Once again, go to the directory where
chain_datais and run theducommand.$ du -c chain_data/ - Save the output and compare the data.
Final Results
Checking chain_data size before running the compact_chain API method
$ du -c chain_data/Output:
469580 chain_data/txhashset/kernel
303084 chain_data/txhashset/rangeproof
62996 chain_data/txhashset/output
835664 chain_data/txhashset
1484 chain_data/peer
117232 chain_data/header/sync_head
117260 chain_data/header/header_head
234496 chain_data/header
794268 chain_data/lmdb
1865916 chain_data/
1865916 totalTUI view of the GRIN-Node:
┌──────────────────────────────────────────────────────────────────────────────┐
│Grin Version 5.0.0-rc.1 [Mainnet] │
└──────────────────────────────────────────────────────────────────────────────┘
┌──────────────────┐┌──────────────────────────────────────────────────────────┐
│Basic Status ││ │
│Peers and Sync ││ Current Status: Running │
│Mining ││ Connected Peers: 8 │
│Logs ││ Disk Usage (GB): 1.911 │
│Version Info ││ -------------------------------------------------------- │
│ ││ Header Tip Hash: 00049c208bde... │
│ ││ Header Chain Height: 1026121 │
│ ││ Header Cumulative Difficulty: 1738799139276499 │
│ ││ Header Tip Timestamp: 2020-12-31 11:52:27 UTC │
│ ││ -------------------------------------------------------- │
│ ││ Chain Tip Hash: 00049c208bde... │
│ ││ Chain Height: 1026121 │
│ ││ Chain Cumulative Difficulty: 1738799139276499 │
│ ││ Chain Tip Timestamp: 2020-12-31 11:52:27 UTC │
│------------------││ -------------------------------------------------------- │
│Tab/Arrow : Cycle ││ Transaction Pool Size: 0 (0) │
│Enter : Select││ Stem Pool Size: 0 (0) │
│Esc : Back ││ -------------------------------------------------------- │
│Q : Quit ││ │
└──────────────────┘└──────────────────────────────────────────────────────────┘Running the compact_chain.sh bash script
$ bash compact_chain.sh Output:
* Expire in 0 ms for 6 (transfer 0x563fb6e71f90)
* Expire in 1 ms for 1 (transfer 0x563fb6e71f90)
* Expire in 0 ms for 1 (transfer 0x563fb6e71f90)
* Expire in 1 ms for 1 (transfer 0x563fb6e71f90)
* Expire in 0 ms for 1 (transfer 0x563fb6e71f90)
* Expire in 0 ms for 1 (transfer 0x563fb6e71f90)
* Expire in 1 ms for 1 (transfer 0x563fb6e71f90)
* Expire in 0 ms for 1 (transfer 0x563fb6e71f90)
......
* Expire in 0 ms for 1 (transfer 0x563fb6e71f90)
* Expire in 0 ms for 1 (transfer 0x563fb6e71f90)
* Expire in 2 ms for 1 (transfer 0x563fb6e71f90)
* Expire in 0 ms for 1 (transfer 0x563fb6e71f90)
* Expire in 0 ms for 1 (transfer 0x563fb6e71f90)
* Expire in 0 ms for 1 (transfer 0x563fb6e71f90)
* Trying ::1...
* TCP_NODELAY set
* Expire in 149999 ms for 3 (transfer 0x563fb6e71f90)
* Expire in 200 ms for 4 (transfer 0x563fb6e71f90)
* connect to ::1 port 3413 failed: Connection refused
* Trying 127.0.0.1...
* TCP_NODELAY set
* Expire in 149999 ms for 3 (transfer 0x563fb6e71f90)
* Connected to localhost (127.0.0.1) port 3413 (#0)
* Server auth using Basic with user 'grin'
> POST /v2/owner HTTP/1.1
> Host: localhost:3413
> Authorization: Basic Z3JpbjpxUEgxT1JzcE1TWTliRk5yZXZvWA==
> User-Agent: curl/7.64.0
> Accept: */*
> Content-Length: 68
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 68 out of 68 bytes
< HTTP/1.1 200 OK
< access-control-allow-origin: *
< access-control-allow-headers: Content-Type, Authorization
< content-type: application/json
< content-length: 67
< date: Thu, 31 Dec 2020 11:53:43 GMT
<
{
"id": 1,
"jsonrpc": "2.0",
"result": {
"Ok": null
}
* Connection #0 to host localhost left intact
}The API response at the end of the verbose output matches the expected response.
{
"id": 1,
"jsonrpc": "2.0",
"result": {
"Ok": null
}
}Checking the chain_data size again after it finished compacting
The disk usage shows the expected results, which is a decrease from the original chain_data size.
Compared to the initial chain_data output, the rangeproof and output part of the txhashset data decreased in size.
$ du -c chain_data/Output:
469580 chain_data/txhashset/kernel
302500 chain_data/txhashset/rangeproof
62920 chain_data/txhashset/output
835004 chain_data/txhashset
1484 chain_data/peer
117232 chain_data/header/sync_head
117260 chain_data/header/header_head
234496 chain_data/header
794268 chain_data/lmdb
1865256 chain_data/
1865256 totalTUI view of the GRIN-Node:
┌──────────────────────────────────────────────────────────────────────────────┐
│Grin Version 5.0.0-rc.1 [Mainnet] │
└──────────────────────────────────────────────────────────────────────────────┘
┌──────────────────┐┌──────────────────────────────────────────────────────────┐
│Basic Status ││ │
│Peers and Sync ││ Current Status: Running │
│Mining ││ Connected Peers: 8 │
│Logs ││ Disk Usage (GB): 1.910 │
│Version Info ││ -------------------------------------------------------- │
│ ││ Header Tip Hash: 000207bfa946... │
│ ││ Header Chain Height: 1026122 │
│ ││ Header Cumulative Difficulty: 1738799310705629 │
│ ││ Header Tip Timestamp: 2020-12-31 11:52:46 UTC │
│ ││ -------------------------------------------------------- │
│ ││ Chain Tip Hash: 000207bfa946... │
│ ││ Chain Height: 1026122 │
│ ││ Chain Cumulative Difficulty: 1738799310705629 │
│ ││ Chain Tip Timestamp: 2020-12-31 11:52:46 UTC │
│------------------││ -------------------------------------------------------- │
│Tab/Arrow : Cycle ││ Transaction Pool Size: 0 (0) │
│Enter : Select││ Stem Pool Size: 0 (0) │
│Esc : Back ││ -------------------------------------------------------- │
│Q : Quit ││ │
└──────────────────┘└──────────────────────────────────────────────────────────┘Comparison of the chain_data disk usage
| 1 | 2 | |
|---|---|---|
| kernel | 469580 | 469580 |
| rangeproof | 303084 | 302500 |
| output | 62996 | 62920 |
| peer | 1484 | 1484 |
| sync_head | 117232 | 117232 |
| header_head | 117260 | 117260 |
| lmdb | 794268 | 794268 |
Conclusion
Testing on an archive node that was already running for several days shows a successful output. The compact_chain API method seems to work correctly, although it may require a slow buildup of new chain data to be able to successfully compact itself.
marekyggdrasil
Description Goal of this issue is to test the compact_chain API method
Prerequisites
Test procedure
Expected result:
We would expect would expect the chain to be compacted and take less space. Output should match the example. Include all cURL requests and responses.
Include the exact version of your grin-node and also your environment