The legacypoliteia tool is a CLI tool that will be used to convert and
import legacy git backend data into the tstore backend. The tool will
have two commands: convert and import.
The convert command converts the git backend data into tstore backend
data types and saves the converted data locally to JSON files. The
import command will import these JSON files into the tstore backend.
This commit adds the legacypoliteia convert command.
The convert command walks a politeia git repo and performs the
conversion on each proposal found in the repo. This is not a simple
conversion and some tstore types require retrieving data from external
APIs. Listed below are the performance bottlenecks of the tool and the
external API calls that are required.
Parsing the git commit log for commit timestamps so that each cast
vote can be associated with the timestamp of when it was committed to
the ballots journal. Exact timestamps of when a vote was cast does not
exist, so this is the closest substitute.
Retrieving the largest commitment address for each vote from dcrdata.
This address is used to verify the vote signature and is part of the
ticketvote plugin vote structure, but was not part of the git backend
vote structure.
Retrieving the user IDs from the politeia API using the public key
that was used to create signatures for various pieces of data
(records, comments, etc). These user IDs were not saved as part of the
git backend data structures, but are part of the tstore and plugin
data structures.
All of the legacy data types can be found in the tool's gitbe package.
The convert.go file contains all of the data conversions in an easy to
audit format.
A proposal that was voted on takes <30 seconds to convert. A proposal
that was not voted on takes 1-2 seconds to convert. The legacy politeia
data can be fully converted in ~50 minutes. The tool uses <100 MiB of
memory during execution.
Example output for a proposal conversion is shown below.
Found 115 legacy git proposals
Converting proposal 95a14094485c92ed3f578b650bd76c5f8c3fd6392650c16bd4ae37e6167c040d (1/115)
RecordMetadata
Token : 95a14094485c92ed3f578b650bd76c5f8c3fd6392650c16bd4ae37e6167c040d
Version : 1
Iteration: 2
State : vetted
Status : public
Timestamp: 1616835515
Merkle : 62fdd81f3e15c8fcd4fa85d27d88147a321ca75e1f4ec9eaf126ab58c4121b59
Files
index.md
Proposal metadata
Name : Video Content Production for Decred Phase 3
LegacyToken: 95a14094485c92ed3f578b650bd76c5f8c3fd6392650c16bd4ae37e6167c040d
Vote metadata
Link by: 0
Link to:
User metadata
User ID : 65706d2f-f82b-4aa5-b4eb-4b1140f4f41f
PublicKey: 1d3a6202cf928d4c53ea9a4cc3df59f819d2ffc6a2d703bdd6ea001a7bc460f3
Signature: 20667734b151ddb34608fb817377d8f41c25c81d4e43ca89b3561d707f9d4a847287d986441261cd4162c311e4ae16665dddf00d69c80344206efaed8eceb80a
Status changes
Token : 95a14094485c92ed3f578b650bd76c5f8c3fd6392650c16bd4ae37e6167c040d
Version : 1
Status : public
PublicKey: 1b7eefd4769c37397eb670a443c3f1827cb77a49d3ca812cd8e53b7344238346
Signature: df4991c2854c3d927bb8e2cd79eb5e77d972e42913943c22af90d9f2c54676b36a9ce8952cdbecf40d56569716c8d7b2bea8c18e05e3c61468c77cba559bc107
Reason :
Timestamp: 1616835515
Comments
Parsed 6 comment adds
Parsed 0 comment dels
Parsed 16 comment votes
AuthDetails
Token : 95a14094485c92ed3f578b650bd76c5f8c3fd6392650c16bd4ae37e6167c040d
Version : 1
Action : authorize
PublicKey: 1d3a6202cf928d4c53ea9a4cc3df59f819d2ffc6a2d703bdd6ea001a7bc460f3
Signature: 79d5edfbe0242bfdad00a15590c56d676a0bcad5c91b8ff3d2ab3d502a6ab2898245aed33f23953698441a1865175c3b6268a67c2f64795ddead4f9f0633f00d
Timestamp: 1617506203
Receipt : ed27fbbfba4ff31a125b7ebb4ed759b9bd0c6c31b0624cc718babb4b1ba2aaafcfdfc916c86940ce6513f9061b29d20b68282ab3b7cf497b3ba82868c41ec100
Vote details
Token : 95a14094485c92ed3f578b650bd76c5f8c3fd6392650c16bd4ae37e6167c040d
Version : 1
Type : 1
Mask : 3
Duration : 2016
Quorum : 20
Pass : 60
Options : [{ID:no Description:Don't approve proposal Bit:1} {ID:yes Description:Approve proposal Bit:2}]
Parent :
Start height: 543233
Start hash : 00000000000000000a8e52fa0c35b3ce498e38621b7dac59924cc0e28e8f6c3e
End height : 545505
Cast votes
Parsed 16334 vote journal entries
Retrieving commitment addresses from dcrdata...
Retrieved addresses 16334/16334
Parsing the vote timestamps from the git logs...
Parsing ballot journal commit 165/165
16334 vote timestamps found
1 : 337
2 : 15997
Total: 16334
Verifying cast vote signature 16334/16334
The legacypoliteia tool is a CLI tool that will be used to convert and import legacy git backend data into the tstore backend. The tool will have two commands: convert and import.
The convert command converts the git backend data into tstore backend data types and saves the converted data locally to JSON files. The import command will import these JSON files into the tstore backend.
This commit adds the legacypoliteia convert command.
The convert command walks a politeia git repo and performs the conversion on each proposal found in the repo. This is not a simple conversion and some tstore types require retrieving data from external APIs. Listed below are the performance bottlenecks of the tool and the external API calls that are required.
Parsing the git commit log for commit timestamps so that each cast vote can be associated with the timestamp of when it was committed to the ballots journal. Exact timestamps of when a vote was cast does not exist, so this is the closest substitute.
Retrieving the largest commitment address for each vote from dcrdata. This address is used to verify the vote signature and is part of the ticketvote plugin vote structure, but was not part of the git backend vote structure.
Retrieving the user IDs from the politeia API using the public key that was used to create signatures for various pieces of data (records, comments, etc). These user IDs were not saved as part of the git backend data structures, but are part of the tstore and plugin data structures.
All of the legacy data types can be found in the tool's gitbe package. The convert.go file contains all of the data conversions in an easy to audit format.
A proposal that was voted on takes <30 seconds to convert. A proposal that was not voted on takes 1-2 seconds to convert. The legacy politeia data can be fully converted in ~50 minutes. The tool uses <100 MiB of memory during execution.
Example output for a proposal conversion is shown below.