search

OriginProtocol / origin-js

We've moved to a monorepo: https://github.com/OriginProtocol/origin
MIT License
81 stars 33 forks source link

Script for one-time data migration #148

Closed micahalcorn closed 6 years ago

micahalcorn commented 6 years ago

We need to be able to deploy with new contracts. If we do, we won't be able to see all of the listings and purchases that have occurred on Rinkeby and Ropsten. Make a script that can be used (hopefully only once) to extract all of the existing data using the IPFS hashes, create the new contracts, and maintain the proper owner information. @wanderingstan can offer more insight if there are questions.

ambertch commented 6 years ago

@wanderingstan for clarification, is this a script to migrate the data currently on v0.1 (which used the former Listing.sol stored in the demo-dapp repository) to the v0.2 deployment, or for migrating data in the future whenever a new version of Listing.sol/ListingsRegistry.sol/Purchase.sol/(identity contracts) etc. is deployed?

From #106 it seems to be the latter

wanderingstan commented 6 years ago

You're right, this is for migrating data out of the ancient Listing.sol (v0.1) into our new multi-contract world. We considered that to be a different beast from #106:

106 is more about how we can put a process in place and what design decisions we need in our contracts to ensure upgradeable contracts / data migration.

wanderingstan commented 6 years ago

For anyone looking at this.

Code for old contract is here.

Contract location on test nets: On Rinkeby: 0x94dE52186b535cB06cA31dEb1fBd4541A824aC6d On Ropsten: 0xE66c9c6168d14bE4C3c145f91890740CbDf9EC8B

wanderingstan commented 6 years ago

Awesome work @ambertch!

For others, his code is in branch here: https://github.com/OriginProtocol/origin-js/tree/issue/148_script_data_migration/scripts/v0.1_migration

In discord he asks:

One issue I've encountered is that upon listing creation, the message sender is made the lister.... although we have a bunch of lister accounts that need to be the lister of the newly created listings How can we set the lister of a listing?

Damn that's a good point! Right now the creator of a listing can only be the wallet address that calls ListingsRegistry.create(). We'd have to add a solidity method for creating a listing on behalf of another account. We can certainly add that in for now, but I'm not sure if it's something we'd want in the final version.

For the config file, I agree they're bad to commit in general, but having a sample config file is always a good idea.

Sample config copy-pasted from discord:

{
    "fromNetworkName": "Ropsten",
    "toNetworkName": "Local",
    "fromGateway": "https://ropsten.infura.io/",
    "toGateway": "http://localhost:8545/",
    "fromListingAddress_v0_1": "0xE66c9c6168d14bE4C3c145f91890740CbDf9EC8B",
    "toListingsRegistryAddress_v0_2":"0x8e4c131b37383e431b9cd0635d3cf9f3f628edae"
}

I suggest we just go ahead with all the migrated listings coming from the wallet of whoever runs the migrate script, and open discussion around adding extra method(s) for creating listings with other owner ID's.

I'll play around with this some today, but looks like this is exactly what we want!

tyleryasaka commented 6 years ago

Yep, this is the tricky part. As @wanderingstan mentioned in Discord, I think the best approach long term is to separate the data and the logic of listing contracts, so that the logic can be upgraded without having to move data.

I'm fine with your suggestion @wanderingstan , for the current migration.

ambertch commented 6 years ago

Checklist

Prep

Dry Run (on 0.5.8)

Migration (on 0.5.10)

Follow-up

ambertch commented 6 years ago

As @wanderingstan posted, the migration code is in this branch: https://github.com/OriginProtocol/origin-js/tree/issue/148_script_data_migration/scripts/v0.1_migration

My JavaScript is a bit rusty, and so any feedback is appreciated!

ambertch commented 6 years ago

Localhost test run:

amberts-MacBook-10:v0.1_migration ambertho$ node migrate.js -c ./conf-test.json -d ./ropsten_listings.json -a read
Reading listings from: Ropsten - Gateway: https://ropsten.infura.io
--------------------------------------------
Source contract address: 0xE66c9c6168d14bE4C3c145f91890740CbDf9EC8B
Found 49 listings.
Retrieved 49 listings from source contract.
Wrote 49 listings to data file: ./ropsten_listings.json
amberts-MacBook-10:v0.1_migration ambertho$ node migrate.js -c ./conf-test.json -d ./ropsten_listings.json -a write
Creating listings on: Local - Gateway: http://localhost:8545
--------------------------------------------
Read 49 listings to migrate from data file.
Destination contract address: 0xf25186b5081ff5ce73482ad761db0eb0d25abfbf
Gas multiplier: 1.2
# Confirmations to wait: 6
Starting # listings in ListingsRegistry: 5
--------------------------------------------
Submitting listing: 0
   nonce: 0
   estimated gas: 588044
Submitting listing: 1
   nonce: 1
   estimated gas: 587967
...
Submitting listing: 47
   nonce: 47
   estimated gas: 587967
Submitting listing: 48
   nonce: 48
   estimated gas: 606044
--------------------------------------------
Submitted 49 listings
<<<<<<<<<<<<<< polling >>>>>>>>>>>>>>>
    current block: 79
    recieved receipts for: 49 listings.
    submitted: 0 confirmed: 44 mined: 5
<<<<<<<<<<<<<< polling >>>>>>>>>>>>>>>
    current block: 80
    recieved receipts for: 5 listings.
    submitted: 0 confirmed: 45 mined: 4
<<<<<<<<<<<<<< polling >>>>>>>>>>>>>>>
    current block: 82
    recieved receipts for: 4 listings.
    submitted: 0 confirmed: 47 mined: 2
<<<<<<<<<<<<<< polling >>>>>>>>>>>>>>>
    current block: 83
    recieved receipts for: 2 listings.
    submitted: 0 confirmed: 48 mined: 1
<<<<<<<<<<<<<< polling >>>>>>>>>>>>>>>
    current block: 86
    recieved receipts for: 1 listings.
    submitted: 0 confirmed: 49 mined: 0
--------------------------------------------
Listings have 6 confirmations.
Ending # listings in ListingsRegistry: 54 (49) created
--------------------------------------------
Results:
0 listing currently in submitted state: 
0 listings currently in mined state: 
49 listings with 6 confirmations: 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48
0 errors:
ambertch commented 6 years ago

Ropsten test run:

So that listings can be batched in the same block, instead of creating a listing and waiting for a transaction receipt to be returned from web3.js, the code queries for the nonce, signs and sends the transaction, then repeats the process for the next listing upon receiving the transaction hash. This assumed that the web3.js call to get the account's transaction count (including pending ones) would always return the current nonce.

I started getting errors pertaining to insufficient gas or incorrect nonces. Initially I thought that the web3.js method to estimate gas price was undershooting how much gas was required by the transactions. I figured this could happen, and the migration script has a configurable multiplier to account for variations in gas price not accounted for by web3.js. However, the issue persisted and I also noticed that regardless of how high I set the multiplier, I'd start getting errors X submissions into the migration.

Looking at the nonces, I saw that they increased monotonically until there were error(s), and in the transactions with errors the nonce would be 3-5 counts less than the nonce of the last successfully submitted transaction.

One possible explanation for this behavior might be: when transactions are submitted in the dev environment, there is one node running and so as long as it records that a transaction has been submitted, the count of all transactions (including pending ones) - the nonce - would be accurately reported by the web3.js call.

On testnet, however, the transactions go to a mining node's mempool and then are propagated across the network. It could be that requests for nonces can go to different nodes, and that a request for a nonce may get handled by a node which all previously submitted transactions have not yet propagated to.

I got the dry run to complete without errors by submitting a transaction every 20 seconds (more than one block time).

With the insufficient gas error, I don't know how mining nodes behave when they receive many transactions from an address within a block time. Since there are on the order of a couple hundred transactions total per block, it's possible that miners only allow an address to submit a certain number of transactions per block(?). The error message could also be a red herring. edit: the error is from the network thinking that the transaction submitted with a lower nonce is a replacement transaction for the first one, and has to be submitted with >10% greater gas. Retrying is the correct approach for the migration

Either spacing the transactions far apart or only submitting a transaction after getting the receipt for the previous one will probably work, but though I'd share my findings trying to parallelize the migration. edit: it was later discovered that the better way to submit many transactions at once would be to precalculate the nonces using an offset (ex. if the transactions would be sent in batches of 50, query for the base nonce, assign the next 50 count to transactions). web3.js.batch() probably does this as well

amberts-MacBook-10:v0.1_migration ambertho$ node migrate.js -c ./conf-ropsten.json -d ./ropsten_listings.json -a write
Creating listings on: Ropsten - Gateway: https://ropsten.infura.io
--------------------------------------------
Read 49 listings to migrate from data file.
Destination contract address: 0xac824c89eab1b2e1b92f0e1675b49d5ddaee06fe
Gas multiplier: 2
# Confirmations to wait: 6
Starting # listings in ListingsRegistry: 127
--------------------------------------------
Submitting listing: 0
   nonce: 91
   estimated gas: 844058
Submitting listing: 1
   nonce: 92
   estimated gas: 843930
Submitting listing: 2
   nonce: 93
   estimated gas: 843930
Submitting listing: 3
   nonce: 94
   estimated gas: 843930
...
Submitting listing: 47
   nonce: 138
   estimated gas: 843930
Submitting listing: 48
   nonce: 139
   estimated gas: 874058
--------------------------------------------
Submitted 49 listings
<<<<<<<<<<<<<< polling >>>>>>>>>>>>>>>
    current block: 3236899
    recieved receipts for: 46 listings.
    submitted: 3 | mined: 4 | confirmed: 42
<<<<<<<<<<<<<< polling >>>>>>>>>>>>>>>
    current block: 3236899
    recieved receipts for: 4 listings.
    submitted: 3 | mined: 4 | confirmed: 42
<<<<<<<<<<<<<< polling >>>>>>>>>>>>>>>
    current block: 3236899
    recieved receipts for: 4 listings.
    submitted: 3 | mined: 4 | confirmed: 42
<<<<<<<<<<<<<< polling >>>>>>>>>>>>>>>
    current block: 3236899
    recieved receipts for: 6 listings.
    submitted: 1 | mined: 6 | confirmed: 42
<<<<<<<<<<<<<< polling >>>>>>>>>>>>>>>
    current block: 3236900
    recieved receipts for: 7 listings.
    submitted: 0 | mined: 6 | confirmed: 43
<<<<<<<<<<<<<< polling >>>>>>>>>>>>>>>
    current block: 3236902
    recieved receipts for: 6 listings.
    submitted: 0 | mined: 5 | confirmed: 44
<<<<<<<<<<<<<< polling >>>>>>>>>>>>>>>
    current block: 3236902
    recieved receipts for: 5 listings.
    submitted: 0 | mined: 5 | confirmed: 44
<<<<<<<<<<<<<< polling >>>>>>>>>>>>>>>
    current block: 3236903
    recieved receipts for: 5 listings.
    submitted: 0 | mined: 4 | confirmed: 45
<<<<<<<<<<<<<< polling >>>>>>>>>>>>>>>
    current block: 3236903
    recieved receipts for: 4 listings.
    submitted: 0 | mined: 4 | confirmed: 45
<<<<<<<<<<<<<< polling >>>>>>>>>>>>>>>
    current block: 3236904
    recieved receipts for: 4 listings.
    submitted: 0 | mined: 3 | confirmed: 46
<<<<<<<<<<<<<< polling >>>>>>>>>>>>>>>
    current block: 3236904
    recieved receipts for: 3 listings.
    submitted: 0 | mined: 3 | confirmed: 46
<<<<<<<<<<<<<< polling >>>>>>>>>>>>>>>
    current block: 3236905
    recieved receipts for: 3 listings.
    submitted: 0 | mined: 1 | confirmed: 48
<<<<<<<<<<<<<< polling >>>>>>>>>>>>>>>
    current block: 3236905
    recieved receipts for: 1 listings.
    submitted: 0 | mined: 1 | confirmed: 48
<<<<<<<<<<<<<< polling >>>>>>>>>>>>>>>
    current block: 3236905
    recieved receipts for: 1 listings.
    submitted: 0 | mined: 1 | confirmed: 48
<<<<<<<<<<<<<< polling >>>>>>>>>>>>>>>
    current block: 3236905
    recieved receipts for: 1 listings.
    submitted: 0 | mined: 1 | confirmed: 48
<<<<<<<<<<<<<< polling >>>>>>>>>>>>>>>
    current block: 3236906
    recieved receipts for: 1 listings.
    submitted: 0 | mined: 0 | confirmed: 49
--------------------------------------------
Listings have 6 confirmations.
Ending # listings in ListingsRegistry: 176 (49 created)
--------------------------------------------
Results:
0 listing currently in submitted state: 
0 listings currently in mined state: 
49 listings migrated: 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48
0 errors:
ambertch commented 6 years ago

Rinkeby test run:

Added retries when creating listings. From some testing on Ropsten and Rinkeby, no transaction required more than 4 retries, typically 1 or 2 in the event of an error.

amberts-MacBook-10:v0.1_migration ambertho$ node migrate.js -c ./conf-rinkeby-dryrun.json -d ./rinkeby_listings.json -a write
Creating listings on: Rinkeby - Gateway: https://rinkeby.infura.io
--------------------------------------------
Read 283 listings to migrate from data file.
Destination contract address: 0xbc16a0193de4cbbb39c188850df711340d151b3f
Gas multiplier: 2
# Confirmations to wait: 6
Creating listings using account: 0x94654F3904277585EA3FD882Cf61433e2d28f52b
Starting # listings in ListingsRegistry: 19
--------------------------------------------
Submitting listing: 0
   nonce: 0
   estimated gas: 874186
Submitting listing: 1
   nonce: 1
   estimated gas: 873930
...
Submitting listing: 21
   nonce: 21
   estimated gas: 844058
Submitting listing: 22
   nonce: 5
   estimated gas: 874186
error submitting transaction: Error: Returned error: replacement transaction underpriced
    Retrying after 10 seconds
Submitting listing: 22
   nonce: 4
   estimated gas: 874186
error submitting transaction: Error: Returned error: replacement transaction underpriced
    Retrying after 15 seconds
Submitting listing: 22
   nonce: 22
   estimated gas: 874186
...
Submitting listing: 281
   nonce: 281
   estimated gas: 843930
Submitting listing: 282
   nonce: 282
   estimated gas: 874058
--------------------------------------------
Submitted 283 listings
<<<<<<<<<<<<<< polling >>>>>>>>>>>>>>>
    current block: 2285975
    recieved receipts for: 270 listings.
    submitted: 13 | mined: 36 | confirmed: 234
<<<<<<<<<<<<<< polling >>>>>>>>>>>>>>>
    current block: 2285976
    recieved receipts for: 36 listings.
    submitted: 13 | mined: 36 | confirmed: 234
<<<<<<<<<<<<<< polling >>>>>>>>>>>>>>>
    current block: 2285977
    recieved receipts for: 49 listings.
    submitted: 0 | mined: 39 | confirmed: 244
<<<<<<<<<<<<<< polling >>>>>>>>>>>>>>>
    current block: 2285977
    recieved receipts for: 39 listings.
    submitted: 0 | mined: 39 | confirmed: 244
<<<<<<<<<<<<<< polling >>>>>>>>>>>>>>>
    current block: 2285978
    recieved receipts for: 39 listings.
    submitted: 0 | mined: 29 | confirmed: 254
<<<<<<<<<<<<<< polling >>>>>>>>>>>>>>>
    current block: 2285979
    recieved receipts for: 29 listings.
    submitted: 0 | mined: 22 | confirmed: 261
<<<<<<<<<<<<<< polling >>>>>>>>>>>>>>>
    current block: 2285980
    recieved receipts for: 22 listings.
    submitted: 0 | mined: 13 | confirmed: 270
<<<<<<<<<<<<<< polling >>>>>>>>>>>>>>>
    current block: 2285980
    recieved receipts for: 13 listings.
    submitted: 0 | mined: 13 | confirmed: 270
<<<<<<<<<<<<<< polling >>>>>>>>>>>>>>>
    current block: 2285981
    recieved receipts for: 13 listings.
    submitted: 0 | mined: 13 | confirmed: 270
<<<<<<<<<<<<<< polling >>>>>>>>>>>>>>>
    current block: 2285982
    recieved receipts for: 13 listings.
    submitted: 0 | mined: 0 | confirmed: 283
--------------------------------------------
Listings have 6 confirmations.
Ending # listings in ListingsRegistry: 302 (283 created)
--------------------------------------------
Results:
0 listing currently in submitted state: 
0 listings currently in mined state: 
283 listings migrated: 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,205,206,207,208,209,210,211,212,213,214,215,216,217,218,219,220,221,222,223,224,225,226,227,228,229,230,231,232,233,234,235,236,237,238,239,240,241,242,243,244,245,246,247,248,249,250,251,252,253,254,255,256,257,258,259,260,261,262,263,264,265,266,267,268,269,270,271,272,273,274,275,276,277,278,279,280,281,282
0 errors:
ambertch commented 6 years ago
ambertch commented 6 years ago

On testnet, however, the transactions go to a mining node's mempool and then are propagated across the network. It could be that requests for nonces can go to different nodes, and that a request for a nonce may get handled by a node which all previously submitted transactions have not yet propagated to.

Some evidence of propagation effects when transactions are submitted in quick succession:

Submitting listing: 84 nonce: 476 estimated gas: 342632 Submitting listing: 85 nonce: 456 estimated gas: 324171 error submitting transaction: Error: Returned error: replacement transaction underpriced Retrying after 10 seconds Submitting listing: 85 nonce: 453 estimated gas: 324171 error submitting transaction: Error: Returned error: replacement transaction underpriced Retrying after 15 seconds Submitting listing: 85 nonce: 464 estimated gas: 324171 error submitting transaction: Error: Returned error: replacement transaction underpriced Retrying after 22.5 seconds Submitting listing: 85 nonce: 473 estimated gas: 324171 error submitting transaction: Error: Returned error: replacement transaction underpriced Retrying after 33.75 seconds Submitting listing: 85 nonce: 476 estimated gas: 324171 error submitting transaction: Error: Returned error: replacement transaction underpriced Retrying after 50.625 seconds Submitting listing: 85 nonce: 477 estimated gas: 324171

In this instance, when 4 retries were required before the network returned a transaction hash, it can be seen that the nonce generally increases during the retries (this is a count that includes pending transactions).

I don't know if there are sticky sessions for web3.js calls the way a proxy like NGINX or a load balancer can forward to the same node or backend across requests. So it could be that multiple requests made by the same client can go to different nodes.

ambertch commented 6 years ago

Since the non-network related errors that were observed are related to nonces, moving data between contracts might be best done via batching (where the nonce can be calculated from an offset rather than querying), or just not relying on querying the network for that value.

Web3.js has a batching feature.

ambertch commented 6 years ago

A couple of thoughts I had during implementation, about moving data around the network:

Neither of these are an issue at low volume, although the likelihood of occurrence increases with volume.

DanielVF commented 6 years ago

Long term we won't be moving contracts in mass like this, so we shouldn't have to worry about gas prices and migrations.

--

Since we build our Listing/Purchase contracts as state machines, and the Ethereum VM only runs one transaction at a time, for now we don't have to worry about resolving conflicting requests. The first transaction will grab the available time, and subsequent transactions trying reserve the same time will fail. This would of course change if we allowed multiple simultaneous reservation requests for overlapping time slots.

Similarly, we don't have to worry about locking on admin transactions either, since the whole admin transaction will go through and complete or fail before any other transactions are run.

ambertch commented 6 years ago

and the Ethereum VM only runs one transaction at a time, for now we don't have to worry about resolving conflicting requests.

Right, though I wonder what the behavior is when the transactions are submitted to the network within a block time. Both in terms of what happens with the mining node that initially receives the transaction, and the other mining nodes the transaction is propogaged to (as the node that receives the transaction isn’t necessarily the one which will mine the block, or it’s block can become an uncle block).

If the mining node is building the block plus validating the next state as transactions come in, then any conflicting transactions would be detected when they arrive at the node. I would guess this is the case, since a miner would want to have the block compiled in the least amount of time, so that it can spend the most amount of time hashing.

Another thing that seems like it would happen is that a transaction arrives at one mining node at the beginning of a blocktime, and another one near the end of that blocktime, and the latter is the one that produces the block that the network comes to consesus on for that blocktime.

So there are multiple ways that overlapping transactions can take a while to detect, since they can arrive at different mining nodes at different times, be included in the current block or a subsequent one(s), be included in the block that ends up being the one on the chain or one that becomes an uncle block, etc.

Of course all of this is pertaining to PoW, there is that recent talk on Casper Nick posted :)

DanielVF commented 6 years ago

It's completely up to the miner what order they put transactions in. A miner can do whatever they want there, and they can even completely not include any given pending transaction for all of time and eternity.

However, once a miner orders transactions for a block, each transaction then runs one at a time, all on its own. Then, after the miner has run the transactions, then the miner starts hashing on that result. Transactions aren't being added to a block during the hashing process.

ambertch commented 6 years ago

once a miner orders transactions for a block, each transaction then runs one at a time

That’s my understanding as well, that the state change occurs all at once. I was wondering whether the transactions would be validated (but the state change not applied) while the miner was putting transactions into the block.

A miner could, although what if there are a bunch of transactions with low gas fees that arrive throughout the blocktime, then some high gas fee ones arrive near the end of the blocktime. The miner would want to include those high gas fee transactions, and if there was a “staging” state change the effect of any already included transactions that are removed from the block would have to be reverted.

ambertch commented 6 years ago

Updated the script to use HDWallet (mnemonic based accounts) rather than a private key

ambertch commented 6 years ago

Operating manual:

After filling out the configuration, execute the script in read mode:

amberts-MacBook-98:v0.1_migration ambertho$ node migrate.js -c ./conf-test.json -d ./rinkeby_listings.json -a read
Reading listings from: Rinkeby - Gateway: https://rinkeby.infura.io/
--------------------------------------------
Source contract address: 0x94dE52186b535cB06cA31dEb1fBd4541A824aC6d
Found 289 listings.
Retrieved 289 listings from source contract.
Wrote 289 listings to data file: ./rinkeby_listings.json

yay! now we have a json file backup of the listings.

then, execute the script in write mode: amberts-MacBook-98:v0.1_migration ambertho$ node migrate.js -c ./conf-test.json -d ./ropsten_listings.json -a write

If there are duplicates the script will flag them:

Creating listings on: Local - Gateway: http://localhost:8545/
--------------------------------------------
Read 49 listings from data file.
Destination contract address: 0x8f0483125fcb9aaaefa9209d8e9d7b9c8b9fb90f
Gas multiplier: 1.2
# Confirmations to wait: 6
Creating listings using account: 0x627306090abab3a6e1400e9345bc60c78a8bef57
--------------------------------------------
Starting # listings in ListingsRegistry: 288
Checking for duplicates...
    Found duplicate listings: 0,1,2,3,4
Please remove these entries from the datafile before running the migration.

Pretty self explanatory, just remove the duplicate listings. Optionally, backup the datafile before doing so. Rerun the script.

Checking for duplicates...
No duplicate listings found.

Press any key to start the migration.

The script will start scrolling output as it sends transactions, checks confirmations, and prints the results:

<<<<<<<<<<<<<< polling >>>>>>>>>>>>>>>
    current block: 371
    recieved receipts for: 1 listings.
    submitted: 0 | mined: 0 | confirmed: 44
--------------------------------------------
Listings have 6 confirmations.
Ending # listings in ListingsRegistry: 332 (44 created)
--------------------------------------------
Results:
0 listing currently in submitted state: 
0 listings currently in mined state: 
44 listings migrated: 5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48
0 errors: 
--------------------------------------------
Checking data...
Checksums of the created listings and the ones in the datafile match.

If the script needs to be stopped mid run or errors out, CTRL+C will print the output. The writes are pretty much atomic as far as I can tell (if you get a transaction receipt, the listing will get created. The next time the script is run, the duplicates check will tell you which listings have been written.

There's a special case here: if the script needs to be run multiple times, only the first run will have the accurate number of starting listings (since subsequent runs will pick up the listings that have already been migrated). This line (https://github.com/OriginProtocol/origin-js/blob/issue/148_script_data_migration/scripts/v0.1_migration/migrate.js#L382) "startingNumListings" has to be set to the original number from the first run (if the ListingsRegistry was freshly created, it will be 0). Not doing so won't affect the actual migration, but the data integrity check at the end of the script uses the starting number of listings to query for the migrated data.

wanderingstan commented 6 years ago

DRY RUN Deploy.

Here is contract info for 2nd deploy of our contracts on Rinkeby for dry run: 

~/Documents/Origin/origin-js/contracts$ npx truffle migrate --network rinkeby | tee 0.6.1_rinkeby_migration_dry_run.log
Compiling ./contracts/Listing.sol...
Compiling ./contracts/ListingsRegistry.sol...
Compiling ./contracts/Migrations.sol...
Compiling ./contracts/Purchase.sol...
Compiling ./contracts/PurchaseLibrary.sol...
Compiling ./contracts/UserRegistry.sol...
Compiling ./contracts/identity/ClaimHolder.sol...
Compiling ./contracts/identity/ClaimHolderPresigned.sol...
Compiling ./contracts/identity/ClaimHolderRegistered.sol...
Compiling ./contracts/identity/ClaimVerifier.sol...
Compiling ./contracts/identity/ERC725.sol...
Compiling ./contracts/identity/ERC735.sol...
Compiling ./contracts/identity/KeyHolder.sol...
Compiling ./contracts/identity/OriginIdentity.sol...
Writing artifacts to ./build/contracts

Using network 'rinkeby'.

Running migration: 1_initial_migration.js
  Deploying Migrations...
  ... 0x3926c4d67da7f8905432d13e5c3a8bc4f80e8d6cd8346afab4ed3bbec4169cba
  Migrations: 0x2b002d6ae372ba066aae7131a7fee7884b21a5ea
Saving successful migration to network...
  ... 0xb386f029d28d508e85ee559fb20b406758b90c00721cfbd633c959cc47a03bb7
Saving artifacts...
Running migration: 2_deploy_contracts.js
  Running step...
  Deploying PurchaseLibrary...
  ... 0xffce9767c3b97d562cfa96ac494d8210986bbfb6866c12964cf10ea8461434c0
  PurchaseLibrary: 0x7b2470e8944ebec939fdc2c544a320be3cf27997
  Linking PurchaseLibrary to ListingsRegistry
  Linking PurchaseLibrary to Listing
  Deploying ListingsRegistry...
  ... 0x58e08018f9ffd280d81842345fe09c6055815e1b9dc3b7c61bb481c3c73bd1bf
  ListingsRegistry: 0x91346523646df332a08583762f3f68ca9e3a305b
  Deploying UserRegistry...
  ... 0x91f0305daff35b1e150b5c9d87aea5d777efb2b6dc891308c4ef7c6b8753fd0f
  UserRegistry: 0x13f1944513fa9e8159ed2e8248c84dca40a7cfae
  Deploying OriginIdentity...
  ... 0xdc6ace05c7b1d25ea30fba6ebc8fdb6126a30abf24133631425556a1309d451d
  OriginIdentity: 0xb7dd3588f28892a453881e8cbf50e420e38d28ca
Saving successful migration to network...
  ... 0x621305fd5f5a2de4dcca2e2571a58b5b282402237fb81b26aca5200832285421
Saving artifacts...
Running migration: 3_create_sample_listings.js
  Running step...
default_account:       0x1a5c29c94d03c4c8f7414564cbd57295d61e898f
a_seller_account:      0x564aae0251d49d1f8d4d8d9e5da08f8cceff9ef2
a_buyer_account:       0x47d4e5afd1624ea63cd7ace59d5dc1d87cff3bb5
another_buyer_account: 0xb7cc090637f6af85e3ed8cc623a5efe95b90b9f9
  ... 0xe77c20118948b05ca35e104c062756aa8da33ea0c1ac2cf153c6ae2c1cc943a5
  ... 0x49c5977c0cdc6140ae65ecb0871c7b0b4894e72a88fbb1d44ac10ef30d97cc75
  ... 0x3821ca3876ece975051488c2fb7b7c692244a421c79a3dfa4236071bbc331a30
  ... 0xe86a1c527a14535be1b3574013f7e7590f7dfe1993f080f17795b12d5d2216dd
  ... 0x92310f18b78f1fbac68546b97ce763a209f3dfa1f3f71450a6b48c88560d745c
Saving successful migration to network...
  ... 0xc704a390d19994329dab7fcdbac5b6791c5f7d27a994a20295aba53d4ee81a9c
Saving artifacts...
Running migration: 4_add_sample_issuers.js
  Running step...
  ... 0x6c726c13991954d37012954f81ae9d5ab6569c0e84771edfaf251f62a2d2ae78
  ... 0xcf8c06b0987a703a6710eba0323e3a2543268cae2aab7bcc85d3f700baa9446e
Saving successful migration to network...
  ... 0xe130c7076ffd34b1b2471e1dece97868b36ff04ad2fbdc3c2cdabeb9baef00d6