search

mimblewimble / grin

Minimal implementation of the Mimblewimble protocol.
https://grin.mw/
Apache License 2.0
5.04k stars 992 forks source link

AlreadySpent at block height 26126 #540

Closed antiochp closed 6 years ago

antiochp commented 6 years ago

My local node (with some additional debug logging added) - 

Dec 22 09:23:45.223 DEBG sumtree: apply_block: pos - Ok(63685), Commitment(08fa2d8f35a266bdf05fc5cf585ed9135ce380b10037f80c7ea75243feba5471de)
Dec 22 09:23:45.223 DEBG pmmr: prune: pos - 63685, index - 26126
Dec 22 09:23:45.223 DEBG pmmr: prune: nothing at that pos in backend
Dec 22 09:23:45.223 DEBG Error returned, discarding sumtree extension.
Dec 22 09:23:45.223 INFO Rejected block 486c233e at 26126: AlreadySpent(Commitment(08fa2d8f35a266bdf05fc5cf585ed9135ce380b10037f80c7ea75243feba5471de))
Dec 22 09:23:45.224 DEBG Block 486c233e refused by chain: AlreadySpent(Commitment(08fa2d8f35a266bdf05fc5cf585ed9135ce380b10037f80c7ea75243feba5471de))

On my local node there is no UTXO for this commitment - 

curl "127.0.0.1:13413/v1/chain/utxos/byids?id=08fa2d8f35a266bdf05fc5cf585ed9135ce380b10037f80c7ea75243feba5471de"
[]

On a different node (that processed this block successfully) - 

curl "127.0.0.1:13413/v1/chain/utxos/byids?id=08fa2d8f35a266bdf05fc5cf585ed9135ce380b10037f80c7ea75243feba5471de"
[{"output_type":"Transaction","commit":[8,250,45,143,53,162,102,189,240,95,197,207,88,94,217,19,92,227,128,177,0,55,248,12,126,167,82,67,254,186,84,113,222],"switch_commit_hash":null,"proof":null,"height":22912,"lock_height":0}]

Is this actually a UTXO? It got spent in block 26126 on some nodes (at least according to nodes that processed that processed this block)?

cc/ @ignopeverell

On the node that did process this block we see a lot of this - 

Dec 21 16:03:26.025 DEBG Rewind sumtrees to header 486c233e at 26126
Dec 21 16:04:50.775 DEBG validate_block: forked_block: 486c233e at 26126
Dec 21 16:04:50.775 DEBG Rewind sumtrees to header 486c233e at 26126
Dec 21 16:06:09.066 DEBG validate_block: forked_block: 486c233e at 26126
Dec 21 16:06:09.066 DEBG Rewind sumtrees to header 486c233e at 26126
Dec 21 16:06:20.580 DEBG validate_block: forked_block: 486c233e at 26126
Dec 21 16:06:20.580 DEBG Rewind sumtrees to header 486c233e at 26126

So the network forked at this block.

We can see this block on grinexplorer here - https://grinexplorer.net/block/486c233ea3bf09de730522c7bc83309b74625414736ce436ecc97b263ffff621

antiochp commented 6 years ago

I'm really confused as to why the API is showing this commitment as Unspent when it is an input to the block (on node that successfully processed the block).

antiochp commented 6 years ago 
Block 0ef7f6ab at 26127
https://grinexplorer.net/block/0ef7f6abcfc30bb9aef115f8060c298117d013608f12c79b9b51a36641ab8c3e
Inputs -
093d2e911bae6f79601fd7e73feb85ea5701245e47afafe1cfe93efde7af47c100
0845d893a6af224d9322a09ce3571efc3eb45702ef2a8c37119a226c2a3459d28e
08c50eb154106a427a68c40b6a825e19b82659fa2502137e1818e077f6cdf42f83
098263feaf6dfb8418faf5302fd91d1f67bbd4b731479872092d7a3c17b878840d

curl "127.0.0.1:13413/v1/chain/utxos/byids?id=093d2e911bae6f79601fd7e73feb85ea5701245e47afafe1cfe93efde7af47c100,0845d893a6af224d9322a09ce3571efc3eb45702ef2a8c37119a226c2a3459d28e,08c50eb154106a427a68c40b6a825e19b82659fa2502137e1818e077f6cdf42f83,098263feaf6dfb8418faf5302fd91d1f67bbd4b731479872092d7a3c17b878840d"
[] # no results, all spent as expected

Block 486c233e at 26126 (trouble maker) 
https://grinexplorer.net/block/486c233ea3bf09de730522c7bc83309b74625414736ce436ecc97b263ffff621
Inputs -
09c234b3eb0befb7ce18053b1b1ec606077472cea945043ad54bea7168774a651a
085889c7b1eb836b5da450990ac90fa3bb46efc7a845aa996ef1c3a3d6cc31ca00
0946913aed491c666c3b61f641405db74cabb5c1f3d55b3aa09ef352ee0fed7708
093d5e239aab86ef4b8a53a2d290673e3aff77f3dda2bb0c4c6db706e9c97ce753
08fa2d8f35a266bdf05fc5cf585ed9135ce380b10037f80c7ea75243feba5471de
08822c722b337569331ac7d124111a399b8df22fa86f70c81dd210c31b7f4b82fc

curl "127.0.0.1:13413/v1/chain/utxos/byids?id=09c234b3eb0befb7ce18053b1b1ec606077472cea945043ad54bea7168774a651a,085889c7b1eb836b5da450990ac90fa3bb46efc7a845aa996ef1c3a3d6cc31ca00,0946913aed491c666c3b61f641405db74cabb5c1f3d55b3aa09ef352ee0fed7708,093d5e239aab86ef4b8a53a2d290673e3aff77f3dda2bb0c4c6db706e9c97ce753,08fa2d8f35a266bdf05fc5cf585ed9135ce380b10037f80c7ea75243feba5471de,08822c722b337569331ac7d124111a399b8df22fa86f70c81dd210c31b7f4b82fc"
[{"output_type":"Transaction","commit":[9,194,52,179,235,11,239,183,206,24,5,59,27,30,198,6,7,116,114,206,169,69,4,58,213,75,234,113,104,119,74,101,26],"switch_commit_hash":null,"proof":null,"height":21836,"lock_height":0},
 {"output_type":"Transaction","commit":[8,88,137,199,177,235,131,107,93,164,80,153,10,201,15,163,187,70,239,199,168,69,170,153,110,241,195,163,214,204,49,202,0],"switch_commit_hash":null,"proof":null,"height":21881,"lock_height":0},
 {"output_type":"Transaction","commit":[9,70,145,58,237,73,28,102,108,59,97,246,65,64,93,183,76,171,181,193,243,213,91,58,160,158,243,82,238,15,237,119,8],"switch_commit_hash":null,"proof":null,"height":22005,"lock_height":0},
 {"output_type":"Transaction","commit":[9,61,94,35,154,171,134,239,75,138,83,162,210,144,103,62,58,255,119,243,221,162,187,12,76,109,183,6,233,201,124,231,83],"switch_commit_hash":null,"proof":null,"height":21882,"lock_height":0},
 {"output_type":"Transaction","commit":[8,250,45,143,53,162,102,189,240,95,197,207,88,94,217,19,92,227,128,177,0,55,248,12,126,167,82,67,254,186,84,113,222],"switch_commit_hash":null,"proof":null,"height":22912,"lock_height":0},
 {"output_type":"Transaction","commit":[8,130,44,114,43,51,117,105,51,26,199,209,36,17,26,57,155,141,242,47,168,111,112,200,29,210,16,195,27,127,75,130,252],"switch_commit_hash":null,"proof":null,"height":22983,"lock_height":0}]

# So all these inputs are Unspent UTXOs on this node (even though it processed this block).
antiochp commented 6 years ago

Picked a random block in grinexplorer with some inputs.

https://grinexplorer.net/block/8b4e10c13d03aa7f193a8fe10a4d8fb03cbbd06097a15de7c120a5e0cffa2dc2

Inputs - 
081afb5e713d468240235254e8e8a61d2f59450f9ed223efc2eecddd6b3f203abe
09839250d5fcec12c6d9b8ef45a5432f7c5df58e628beb6d79676ec7b00fb3766a
0976ffd725be8a46df4c28c4d7891936d959e60c5c948988710d81b7a78b751a76
08968409706814d16b225b399dc92561f1d8c0e9d87690ed4189b74264e6accdd3
082426658bd94c9529f65f1e0f5e2b03faa6c9be500b5ef40f157909550b53e747
091457f7b74c2087b8b5b40a20407dc6b498309fc5fb72d97b7ae70f30f594ed34
08b84aaba97cbc7bfa68735cb8393c054dc4088ae5ba4f7eac6d57c23c786cce90
09407722f60f45330aa0f9cc7e38bd76543cf8911ee1fe087bdcdb79beceb77ddc
First input does appear to be spent according to UTXO API - 
curl "127.0.0.1:13413/v1/chain/utxos/byids?id=081afb5e713d468240235254e8e8a61d2f59450f9ed223efc2eecddd6b3f203abe"
[]

But second input is showing up in UTXO API - 
curl "127.0.0.1:13413/v1/chain/utxos/byids?id=09839250d5fcec12c6d9b8ef45a5432f7c5df58e628beb6d79676ec7b00fb3766a"
[{"output_type":"Transaction","commit":[9,131,146,80,213,252,236,18,198,217,184,239,69,165,67,47,124,93,245,142,98,139,235,109,121,103,110,199,176,15,179,118,106],"switch_commit_hash":null,"proof":null,"height":23260,"lock_height":0}]
antiochp commented 6 years ago

Get the "unspent" tx output via the API - 

curl "127.0.0.1:13413/v1/chain/utxos/byids?id=09839250d5fcec12c6d9b8ef45a5432f7c5df58e628beb6d79676ec7b00fb3766a"
[{"output_type":"Transaction","commit":[9,131,146,80,213,252,236,18,198,217,184,239,69,165,67,47,124,93,245,142,98,139,235,109,121,103,110,199,176,15,179,118,106],"switch_commit_hash":null,"proof":null,"height":23260,"lock_height":0}]

Get the block via the API (commitment is in inputs list) -

curl "127.0.0.1:13413/v1/blocks/8b4e10c13d03aa7f193a8fe10a4d8fb03cbbd06097a15de7c120a5e0cffa2dc2"
{
    "header": {
        "hash": "8b4e10c13d03aa7f193a8fe10a4d8fb03cbbd06097a15de7c120a5e0cffa2dc2",
        "version": 1,
        "height": 23261,
        "previous": "8815e21e268a9220ddfefbf63b817fe9248a835c2c718b0d3494dc38000a4dab",
        "timestamp": "2017-12-19T14:16:58Z",
        "utxo_root": "006cb23bf30680abbfb3eaf85039e61df8d4bf649106fb8f5d278f02995dd795",
        "range_proof_root": "06d8b131ef81ad9bd351d8baf3d255a991982895f0842f1f31a14125940b2f76",
        "kernel_root": "5cff5005e175498540f5f37ee5dd30b098edf128ea5fcbacd9cc91f4059a8742",
        "nonce": 2423480446745879586,
        "difficulty": 132,
        "total_difficulty": 177459755
    },
    "inputs": [
        "081afb5e713d468240235254e8e8a61d2f59450f9ed223efc2eecddd6b3f203abe",
        "09839250d5fcec12c6d9b8ef45a5432f7c5df58e628beb6d79676ec7b00fb3766a",
        "0976ffd725be8a46df4c28c4d7891936d959e60c5c948988710d81b7a78b751a76",
        "08968409706814d16b225b399dc92561f1d8c0e9d87690ed4189b74264e6accdd3",
        "082426658bd94c9529f65f1e0f5e2b03faa6c9be500b5ef40f157909550b53e747",
        "091457f7b74c2087b8b5b40a20407dc6b498309fc5fb72d97b7ae70f30f594ed34",
        "08b84aaba97cbc7bfa68735cb8393c054dc4088ae5ba4f7eac6d57c23c786cce90",
        "09407722f60f45330aa0f9cc7e38bd76543cf8911ee1fe087bdcdb79beceb77ddc"
    ],
    "outputs": [
        {
            ...
    }
    ]
}
antiochp commented 6 years ago

Wrote a python "reconciliation" script that compares the inputs/outputs from blocks on the chain to the known UTXO set.

https://github.com/antiochp/grin-recon

Looks for the following -

This is what it outputs when run against my seed node (currently in sync with grinexplorer.net) -

--------------------
Inputs:  15989
Outputs:  47075
*** Duplicate (sets) of inputs:  16
*** Duplicate (sets) of outputs:  2
*** Spent yet still in UTXO set:  210
*** Unspent but missing from UTXO set:  31086
--------------------

Not immediately sure what to make of this. ¯_(ツ)_/¯

ignopeverell commented 6 years ago

Looks like there's something wrong with the remove logs on the node that accepted that block. I would look under .grin/sumtrees/utxo what that looks like. I added an additional save to pmmr_rm_log.bin so after each rewind the whole thing is saved in a file postfixed with the block number. That doesn't get used (only pmmr_rm_log.bin is) but it's useful for debugging. The size of each file should be increasing as the block heights increase (each spent output position is in there).

If you really want to go deeper you can look at those with hexdump. Each spent UTXO is a u64 for the position followed by a u32 for the block number at which it was spent.

heunglee commented 6 years ago

May https://github.com/mimblewimble/grin/issues/539 increase occurrence of this bug? When setting subtree root in chain, this is rollback after apply_block. Can rollback create position issue?

antiochp commented 6 years ago

@heunglee I think they may be related yes. Rollback is where a lot of the complexity comes from (but rollback is necessary).

antiochp commented 6 years ago

Block 1989 is pretty weird looking (may be of historical interest, not sure if this affects anything) - 

{ 'header': { 'difficulty': 441,
              'hash': '7adc099974fc897a776528abe779e87b1f3e93fa158d9550720707fdc0d640b0',
              'height': 1989,
              'kernel_root': '33b5bdf5a3f1bc90ef21f5372b4b32b8f80c84c09eb2822c8034615b0041335e',
              'nonce': 16307556380322611416,
              'previous': '7dbdbbcdc02f0cc4c022408110fbf3f648e98971b532b4a20b80e0d4ae34488c',
              'range_proof_root': '2cda491446d682f033e25a2b7c793f4c020d31a18c9e3aebc98839ad6b2eca04',
              'timestamp': '2017-11-18T01:32:39Z',
              'total_difficulty': 6935232,
              'utxo_root': 'cda79758965532f68167201657ffa966d9ba7c134df935f6ecba8ca96d4ecda7',
              'version': 1},
  'inputs': [ '097b7763627a673183408fa82b210ca06f13729f75afd7b667112d75a621ff708d',
              '0888c6b64db818b71198a7eb90e5bf2afa5bed95024ced998be39334b97894b7c6'],
  'kernels': [ { 'excess': '0811e1515a6dfc6324c5c8221624845866b96ca187e5a40016f81af2a45e588871',
                 'excess_sig': '304402206938f04e11fcfb92f0137acdd29ffe78945c03166c91bcd7135416a47dedb8d202207e9fbb2be91929df7093cbf7cb693a85db39974daf626e387c1e6a6ad97fa636',
                 'features': 'COINBASE_KERNEL',
                 'fee': 0,
                 'lock_height': 0},
               { 'excess': '0833b64f9d1356802e9db7bd39c5bed572f8c36b3bb255bd6f3e16ede3ef39ff4e',
                 'excess_sig': '3045022100fd75a3e9ee5e51c86cf21b7d0d413c9dee6b9a9b42ae2bafd47c481514110d1e022014fdb701431b21cb9eb2a9e0e8cda6043a15745e3dcba8e0adf487a7929f62af',
                 'features': '',
                 'fee': 8000000,
                 'lock_height': 1987},
               { 'excess': '096a1e1694c47c39de6a7738c0929d0ec495c68ce218fb4531e63a23661b8c94a7',
                 'excess_sig': '3045022100fb35d53739581d379e11606dcf0655a5d92d136487493453ff4148c3099613fc02202ede44467f200b2bcd1dfbf360debe0a88703b1db3df0e8ac7a7d239855f9e34',
                 'features': '',
                 'fee': 8000000,
                 'lock_height': 1987}],
  'outputs': [ { 'commit': '09c32e615e791d79c45f832c5077c24e19c5af603700d564c9b1802e4e09780390',
                 'height': 1989,
                 'lock_height': 2989,
                 'output_type': 'Coinbase',
                 'proof_hash': '5bfb9d6a0558cf0424326b8e9b5bcee41db10a93b70d28aa17830f39ea5a3e94',
                 'spent': True,
                 'switch_commit_hash': 'be7cbe671012ed484774335d78fe1dd5a05cdaee'},
               { 'commit': '08e747174f4d0c95658d8aff2db4513f9365d33fb6af313393b771bf93fe344388',
                 'height': 1989,
                 'lock_height': 0,
                 'output_type': 'Transaction',
                 'proof_hash': 'cdafa7ee2ce79d839b097c4b685d0daecba017b07bce188bb05f3f8140695686',
                 'spent': True,
                 'switch_commit_hash': 'd1be3167d28e9ad4dda831f1e3c416d78a991bab'},
               { 'commit': '08e747174f4d0c95658d8aff2db4513f9365d33fb6af313393b771bf93fe344388',
                 'height': 1989,
                 'lock_height': 0,
                 'output_type': 'Transaction',
                 'proof_hash': 'cdafa7ee2ce79d839b097c4b685d0daecba017b07bce188bb05f3f8140695686',
                 'spent': True,
                 'switch_commit_hash': 'd1be3167d28e9ad4dda831f1e3c416d78a991bab'},
               { 'commit': '08999d1ac59fa66e5ec7f71034626ecce00226f92dfb5a8874cf9aead968e30766',
                 'height': 1989,
                 'lock_height': 0,
                 'output_type': 'Transaction',
                 'proof_hash': '3486ee2e02d5b63a5c1805c7d143f1d798313c509217387e8e34e362814daa71',
                 'spent': True,
                 'switch_commit_hash': 'b69f572ac143376b61b75f3307d86a92bc99cbf7'},
               { 'commit': '08999d1ac59fa66e5ec7f71034626ecce00226f92dfb5a8874cf9aead968e30766',
                 'height': 1989,
                 'lock_height': 0,
                 'output_type': 'Transaction',
                 'proof_hash': '3486ee2e02d5b63a5c1805c7d143f1d798313c509217387e8e34e362814daa71',
                 'spent': True,
                 'switch_commit_hash': 'b69f572ac143376b61b75f3307d86a92bc99cbf7'}]}

Two pairs of duplicate outputs (these are the only duplicate outputs picked up by my script).

08e747174f4d0c95658d8aff2db4513f9365d33fb6af313393b771bf93fe344388
08999d1ac59fa66e5ec7f71034626ecce00226f92dfb5a8874cf9aead968e30766
antiochp commented 6 years ago

These are all the duplicate inputs - 

**** Found duplicate input: 
[ { 'commit': '093d5e239aab86ef4b8a53a2d290673e3aff77f3dda2bb0c4c6db706e9c97ce753',
    'height': 26126},
  { 'commit': '093d5e239aab86ef4b8a53a2d290673e3aff77f3dda2bb0c4c6db706e9c97ce753',
    'height': 21883}]
**** Found duplicate input: 
[ { 'commit': '093d2e911bae6f79601fd7e73feb85ea5701245e47afafe1cfe93efde7af47c100',
    'height': 26127},
  { 'commit': '093d2e911bae6f79601fd7e73feb85ea5701245e47afafe1cfe93efde7af47c100',
    'height': 21914}]
**** Found duplicate input: 
[ { 'commit': '086600051e9fd3e9f20c2b08e664d0575b044f80abfb2cfc8fe0cbdbd2f8337e8d',
    'height': 22079},
  { 'commit': '086600051e9fd3e9f20c2b08e664d0575b044f80abfb2cfc8fe0cbdbd2f8337e8d',
    'height': 21831}]
**** Found duplicate input: 
[ { 'commit': '08dd7b151d4622efdc4d9ba3424683106d93596532e9f70e679de98c0e97c137ca',
    'height': 22511},
  { 'commit': '08dd7b151d4622efdc4d9ba3424683106d93596532e9f70e679de98c0e97c137ca',
    'height': 21784}]
**** Found duplicate input: 
[ { 'commit': '09c234b3eb0befb7ce18053b1b1ec606077472cea945043ad54bea7168774a651a',
    'height': 26126},
  { 'commit': '09c234b3eb0befb7ce18053b1b1ec606077472cea945043ad54bea7168774a651a',
    'height': 21837}]
**** Found duplicate input: 
[ { 'commit': '0837611b5439793dadbe83f5f3417f01fe182276482c4cd6e54452650762f55e36',
    'height': 13899},
  { 'commit': '0837611b5439793dadbe83f5f3417f01fe182276482c4cd6e54452650762f55e36',
    'height': 12908}]
**** Found duplicate input: 
[ { 'commit': '085889c7b1eb836b5da450990ac90fa3bb46efc7a845aa996ef1c3a3d6cc31ca00',
    'height': 26126},
  { 'commit': '085889c7b1eb836b5da450990ac90fa3bb46efc7a845aa996ef1c3a3d6cc31ca00',
    'height': 21882}]
**** Found duplicate input: 
[ { 'commit': '08d915f77b00b5f0baae24d8eb769f1ca5e4fa47860236e847409d8f471e578100',
    'height': 22290},
  { 'commit': '08d915f77b00b5f0baae24d8eb769f1ca5e4fa47860236e847409d8f471e578100',
    'height': 22282}]
**** Found duplicate input: 
[ { 'commit': '0946913aed491c666c3b61f641405db74cabb5c1f3d55b3aa09ef352ee0fed7708',
    'height': 26126},
  { 'commit': '0946913aed491c666c3b61f641405db74cabb5c1f3d55b3aa09ef352ee0fed7708',
    'height': 22006}]
**** Found duplicate input: 
[ { 'commit': '08c831a8fd051ea0183eb0403ee69691bbc76d4b5da98b88c6d35d3423c7210255',
    'height': 22079},
  { 'commit': '08c831a8fd051ea0183eb0403ee69691bbc76d4b5da98b88c6d35d3423c7210255',
    'height': 21931}]
**** Found duplicate input: 
[ { 'commit': '08e88de957e8c5065b5b1aea355b3d930138ebe29ae236b283443381d27ed18a53',
    'height': 22512},
  { 'commit': '08e88de957e8c5065b5b1aea355b3d930138ebe29ae236b283443381d27ed18a53',
    'height': 21996}]
**** Found duplicate input: 
[ { 'commit': '08822c722b337569331ac7d124111a399b8df22fa86f70c81dd210c31b7f4b82fc',
    'height': 26126},
  { 'commit': '08822c722b337569331ac7d124111a399b8df22fa86f70c81dd210c31b7f4b82fc',
    'height': 22984}]
**** Found duplicate input: 
[ { 'commit': '0963aa286ac0baf4bc2d7c66d0c6b04b8d9e7138b515a40d4b56451ca1d2bb69f3',
    'height': 13899},
  { 'commit': '0963aa286ac0baf4bc2d7c66d0c6b04b8d9e7138b515a40d4b56451ca1d2bb69f3',
    'height': 12923}]
**** Found duplicate input: 
[ { 'commit': '08c50eb154106a427a68c40b6a825e19b82659fa2502137e1818e077f6cdf42f83',
    'height': 26127},
  { 'commit': '08c50eb154106a427a68c40b6a825e19b82659fa2502137e1818e077f6cdf42f83',
    'height': 21691}]
**** Found duplicate input: 
[ { 'commit': '088a95ff6a2ed0e6600bd468483788d0186b39c454ed97562b3723fa96f5737e91',
    'height': 22079},
  { 'commit': '088a95ff6a2ed0e6600bd468483788d0186b39c454ed97562b3723fa96f5737e91',
    'height': 21922}]
**** Found duplicate input: 
[ { 'commit': '08fa2d8f35a266bdf05fc5cf585ed9135ce380b10037f80c7ea75243feba5471de',
    'height': 26126},
  { 'commit': '08fa2d8f35a266bdf05fc5cf585ed9135ce380b10037f80c7ea75243feba5471de',
    'height': 22915}]
antiochp commented 6 years ago

Just putting everything down here to take a better look later.

This is what .grin/sumtree/utxo looks like - 

ls -al .grin/sumtrees/utxo/
total 60792
drwxr-xr-x 2 root root   16384 Dec 24 03:21 .
drwxr-xr-x 5 root root    4096 Dec 17 22:36 ..
-rw-r--r-- 1 root root 6364735 Dec 24 04:00 pmmr_dat.bin
-rw-r--r-- 1 root root  260184 Dec 24 04:00 pmmr_rm_log.bin
-rw-r--r-- 1 root root  260184 Dec 24 01:24 pmmr_rm_log.bin.0
-rw-r--r-- 1 root root  260016 Dec 21 09:34 pmmr_rm_log.bin.11738
-rw-r--r-- 1 root root  186504 Dec 19 18:59 pmmr_rm_log.bin.12349
-rw-r--r-- 1 root root  185736 Dec 19 18:34 pmmr_rm_log.bin.13110
-rw-r--r-- 1 root root  186888 Dec 19 19:13 pmmr_rm_log.bin.13113
-rw-r--r-- 1 root root  185844 Dec 19 18:39 pmmr_rm_log.bin.13291
-rw-r--r-- 1 root root  260184 Dec 23 14:01 pmmr_rm_log.bin.13512
-rw-r--r-- 1 root root  186384 Dec 19 18:55 pmmr_rm_log.bin.13944
-rw-r--r-- 1 root root  249048 Dec 21 02:41 pmmr_rm_log.bin.14108
-rw-r--r-- 1 root root  260028 Dec 21 09:57 pmmr_rm_log.bin.14445
-rw-r--r-- 1 root root  187632 Dec 19 19:37 pmmr_rm_log.bin.14575
-rw-r--r-- 1 root root  260028 Dec 21 09:59 pmmr_rm_log.bin.14653
-rw-r--r-- 1 root root  249540 Dec 21 03:01 pmmr_rm_log.bin.15229
-rw-r--r-- 1 root root  249540 Dec 21 03:05 pmmr_rm_log.bin.15251
-rw-r--r-- 1 root root  260184 Dec 22 21:52 pmmr_rm_log.bin.15486
-rw-r--r-- 1 root root  188496 Dec 19 20:05 pmmr_rm_log.bin.15496
-rw-r--r-- 1 root root  188232 Dec 19 19:59 pmmr_rm_log.bin.15508
-rw-r--r-- 1 root root  188532 Dec 19 20:07 pmmr_rm_log.bin.15545
-rw-r--r-- 1 root root  260184 Dec 22 20:46 pmmr_rm_log.bin.15729
-rw-r--r-- 1 root root  188604 Dec 19 20:11 pmmr_rm_log.bin.15741
-rw-r--r-- 1 root root  260184 Dec 22 20:46 pmmr_rm_log.bin.15779
-rw-r--r-- 1 root root  260028 Dec 21 10:07 pmmr_rm_log.bin.16072
-rw-r--r-- 1 root root  189288 Dec 19 20:36 pmmr_rm_log.bin.16602
-rw-r--r-- 1 root root  189072 Dec 19 20:27 pmmr_rm_log.bin.16682
-rw-r--r-- 1 root root  168816 Dec 19 12:40 pmmr_rm_log.bin.16693
-rw-r--r-- 1 root root  189456 Dec 19 20:41 pmmr_rm_log.bin.16732
-rw-r--r-- 1 root root  260028 Dec 21 10:08 pmmr_rm_log.bin.16742
-rw-r--r-- 1 root root  189156 Dec 19 20:32 pmmr_rm_log.bin.16929
-rw-r--r-- 1 root root  190080 Dec 19 21:00 pmmr_rm_log.bin.17422
-rw-r--r-- 1 root root  260028 Dec 21 10:13 pmmr_rm_log.bin.17741
-rw-r--r-- 1 root root  189624 Dec 19 20:45 pmmr_rm_log.bin.17819
-rw-r--r-- 1 root root  189660 Dec 19 20:46 pmmr_rm_log.bin.17887
-rw-r--r-- 1 root root  188712 Dec 19 20:16 pmmr_rm_log.bin.17896
-rw-r--r-- 1 root root   55356 Dec 18 12:29 pmmr_rm_log.bin.18707
-rw-r--r-- 1 root root  260184 Dec 22 20:58 pmmr_rm_log.bin.18751
-rw-r--r-- 1 root root  189156 Dec 19 20:31 pmmr_rm_log.bin.18902
-rw-r--r-- 1 root root  251520 Dec 21 04:35 pmmr_rm_log.bin.19074
-rw-r--r-- 1 root root  260028 Dec 21 10:24 pmmr_rm_log.bin.19142
-rw-r--r-- 1 root root  211152 Dec 20 01:18 pmmr_rm_log.bin.19218
-rw-r--r-- 1 root root   56316 Dec 18 12:37 pmmr_rm_log.bin.19246
-rw-r--r-- 1 root root  211536 Dec 20 01:33 pmmr_rm_log.bin.19827
-rw-r--r-- 1 root root  212064 Dec 20 01:49 pmmr_rm_log.bin.20454
-rw-r--r-- 1 root root  212520 Dec 20 02:06 pmmr_rm_log.bin.21047
-rw-r--r-- 1 root root  212532 Dec 20 02:07 pmmr_rm_log.bin.21128
-rw-r--r-- 1 root root  260184 Dec 22 21:23 pmmr_rm_log.bin.21367
-rw-r--r-- 1 root root   53724 Dec 18 00:41 pmmr_rm_log.bin.21454
-rw-r--r-- 1 root root   72888 Dec 18 02:50 pmmr_rm_log.bin.21517
-rw-r--r-- 1 root root  138912 Dec 18 11:25 pmmr_rm_log.bin.21533
-rw-r--r-- 1 root root  212940 Dec 20 02:20 pmmr_rm_log.bin.21535
-rw-r--r-- 1 root root  161628 Dec 19 09:47 pmmr_rm_log.bin.21554
-rw-r--r-- 1 root root  191544 Dec 19 21:48 pmmr_rm_log.bin.21568
-rw-r--r-- 1 root root  191592 Dec 19 21:50 pmmr_rm_log.bin.21604
-rw-r--r-- 1 root root  213036 Dec 20 02:23 pmmr_rm_log.bin.21638
-rw-r--r-- 1 root root   57972 Dec 18 12:40 pmmr_rm_log.bin.21657
-rw-r--r-- 1 root root  192492 Dec 19 22:21 pmmr_rm_log.bin.21684
-rw-r--r-- 1 root root   60420 Dec 18 13:24 pmmr_rm_log.bin.21694
-rw-r--r-- 1 root root  260184 Dec 22 21:36 pmmr_rm_log.bin.21705
-rw-r--r-- 1 root root  226176 Dec 20 10:25 pmmr_rm_log.bin.21727
-rw-r--r-- 1 root root   81444 Dec 18 14:55 pmmr_rm_log.bin.21804
-rw-r--r-- 1 root root  213348 Dec 20 02:34 pmmr_rm_log.bin.21826
-rw-r--r-- 1 root root  226044 Dec 20 10:18 pmmr_rm_log.bin.21840
-rw-r--r-- 1 root root   96948 Dec 18 16:13 pmmr_rm_log.bin.21847
-rw-r--r-- 1 root root   92064 Dec 18 15:45 pmmr_rm_log.bin.21857
-rw-r--r-- 1 root root  260184 Dec 23 16:30 pmmr_rm_log.bin.21881
-rw-r--r-- 1 root root   98628 Dec 18 16:18 pmmr_rm_log.bin.21882
-rw-r--r-- 1 root root  260184 Dec 21 19:37 pmmr_rm_log.bin.21918
-rw-r--r-- 1 root root  260184 Dec 21 17:48 pmmr_rm_log.bin.21928
-rw-r--r-- 1 root root  188604 Dec 19 20:11 pmmr_rm_log.bin.21931
-rw-r--r-- 1 root root  260184 Dec 23 09:25 pmmr_rm_log.bin.21932
-rw-r--r-- 1 root root  162000 Dec 19 09:56 pmmr_rm_log.bin.21970
-rw-r--r-- 1 root root  135924 Dec 18 19:06 pmmr_rm_log.bin.21974
-rw-r--r-- 1 root root   73488 Dec 18 02:52 pmmr_rm_log.bin.22010
-rw-r--r-- 1 root root  137724 Dec 18 19:26 pmmr_rm_log.bin.22019
-rw-r--r-- 1 root root  137808 Dec 18 20:25 pmmr_rm_log.bin.22145
-rw-r--r-- 1 root root  223104 Dec 20 08:15 pmmr_rm_log.bin.22185
-rw-r--r-- 1 root root  142392 Dec 19 02:12 pmmr_rm_log.bin.22511
-rw-r--r-- 1 root root  260184 Dec 23 17:18 pmmr_rm_log.bin.22550
-rw-r--r-- 1 root root  144252 Dec 19 02:55 pmmr_rm_log.bin.22563
-rw-r--r-- 1 root root  260184 Dec 23 17:20 pmmr_rm_log.bin.22600
-rw-r--r-- 1 root root  150804 Dec 19 05:22 pmmr_rm_log.bin.22667
-rw-r--r-- 1 root root  153408 Dec 19 06:21 pmmr_rm_log.bin.22737
-rw-r--r-- 1 root root  162900 Dec 19 10:19 pmmr_rm_log.bin.22789
-rw-r--r-- 1 root root  246984 Dec 21 01:02 pmmr_rm_log.bin.22875
-rw-r--r-- 1 root root  223440 Dec 20 08:27 pmmr_rm_log.bin.22885
-rw-r--r-- 1 root root  159768 Dec 19 08:59 pmmr_rm_log.bin.22915
-rw-r--r-- 1 root root  260184 Dec 24 01:08 pmmr_rm_log.bin.23010
-rw-r--r-- 1 root root  260184 Dec 23 07:05 pmmr_rm_log.bin.23023
-rw-r--r-- 1 root root  260184 Dec 24 01:09 pmmr_rm_log.bin.23105
-rw-r--r-- 1 root root  167808 Dec 19 12:31 pmmr_rm_log.bin.23110
-rw-r--r-- 1 root root  168624 Dec 19 12:35 pmmr_rm_log.bin.23121
-rw-r--r-- 1 root root  174804 Dec 19 15:38 pmmr_rm_log.bin.23132
-rw-r--r-- 1 root root  169668 Dec 19 13:06 pmmr_rm_log.bin.23182
-rw-r--r-- 1 root root  170616 Dec 19 13:32 pmmr_rm_log.bin.23211
-rw-r--r-- 1 root root  170712 Dec 19 13:36 pmmr_rm_log.bin.23215
-rw-r--r-- 1 root root  260184 Dec 24 01:14 pmmr_rm_log.bin.23285
-rw-r--r-- 1 root root  175008 Dec 19 15:44 pmmr_rm_log.bin.23319
-rw-r--r-- 1 root root  223788 Dec 20 08:40 pmmr_rm_log.bin.23349
-rw-r--r-- 1 root root  181800 Dec 19 17:40 pmmr_rm_log.bin.23473
-rw-r--r-- 1 root root  260184 Dec 24 01:20 pmmr_rm_log.bin.23476
-rw-r--r-- 1 root root  181848 Dec 19 17:46 pmmr_rm_log.bin.23477
-rw-r--r-- 1 root root  185352 Dec 19 18:21 pmmr_rm_log.bin.23500
-rw-r--r-- 1 root root  247440 Dec 21 01:21 pmmr_rm_log.bin.23518
-rw-r--r-- 1 root root  186648 Dec 19 19:03 pmmr_rm_log.bin.23550
-rw-r--r-- 1 root root  222060 Dec 20 08:03 pmmr_rm_log.bin.23558
-rw-r--r-- 1 root root  188532 Dec 19 20:08 pmmr_rm_log.bin.23611
-rw-r--r-- 1 root root  245916 Dec 21 00:10 pmmr_rm_log.bin.23769
-rw-r--r-- 1 root root  260184 Dec 23 07:26 pmmr_rm_log.bin.23779
-rw-r--r-- 1 root root  206016 Dec 19 23:33 pmmr_rm_log.bin.23796
-rw-r--r-- 1 root root  246048 Dec 21 00:16 pmmr_rm_log.bin.24070
-rw-r--r-- 1 root root  221328 Dec 20 07:34 pmmr_rm_log.bin.24300
-rw-r--r-- 1 root root  221760 Dec 20 07:50 pmmr_rm_log.bin.24338
-rw-r--r-- 1 root root  223080 Dec 20 08:12 pmmr_rm_log.bin.24366
-rw-r--r-- 1 root root  224052 Dec 20 08:50 pmmr_rm_log.bin.24413
-rw-r--r-- 1 root root  227892 Dec 20 11:31 pmmr_rm_log.bin.24572
-rw-r--r-- 1 root root  229032 Dec 20 12:17 pmmr_rm_log.bin.24622
-rw-r--r-- 1 root root  229476 Dec 20 12:39 pmmr_rm_log.bin.24641
-rw-r--r-- 1 root root  229512 Dec 20 13:00 pmmr_rm_log.bin.24676
-rw-r--r-- 1 root root  229512 Dec 20 15:26 pmmr_rm_log.bin.24816
-rw-r--r-- 1 root root  254280 Dec 21 06:44 pmmr_rm_log.bin.24946
-rw-r--r-- 1 root root  229512 Dec 20 20:17 pmmr_rm_log.bin.25103
-rw-r--r-- 1 root root  229512 Dec 20 20:56 pmmr_rm_log.bin.25139
-rw-r--r-- 1 root root  229512 Dec 20 21:42 pmmr_rm_log.bin.25199
-rw-r--r-- 1 root root  243888 Dec 20 22:16 pmmr_rm_log.bin.25231
-rw-r--r-- 1 root root  244440 Dec 20 23:00 pmmr_rm_log.bin.25278
-rw-r--r-- 1 root root  245340 Dec 20 23:41 pmmr_rm_log.bin.25315
-rw-r--r-- 1 root root  255096 Dec 21 07:23 pmmr_rm_log.bin.25327
-rw-r--r-- 1 root root  246432 Dec 21 00:33 pmmr_rm_log.bin.25364
-rw-r--r-- 1 root root  246816 Dec 21 00:53 pmmr_rm_log.bin.25387
-rw-r--r-- 1 root root  250932 Dec 21 04:05 pmmr_rm_log.bin.25489
-rw-r--r-- 1 root root  255264 Dec 21 07:30 pmmr_rm_log.bin.25595
-rw-r--r-- 1 root root  255264 Dec 21 07:30 pmmr_rm_log.bin.25613
-rw-r--r-- 1 root root  256392 Dec 21 08:25 pmmr_rm_log.bin.25769
-rw-r--r-- 1 root root  259956 Dec 21 09:29 pmmr_rm_log.bin.25827
-rw-r--r-- 1 root root  260028 Dec 21 10:12 pmmr_rm_log.bin.25885
-rw-r--r-- 1 root root  260028 Dec 21 10:34 pmmr_rm_log.bin.25910
-rw-r--r-- 1 root root  260028 Dec 21 10:52 pmmr_rm_log.bin.25933
-rw-r--r-- 1 root root  260184 Dec 22 20:37 pmmr_rm_log.bin.25943
-rw-r--r-- 1 root root  260028 Dec 21 12:07 pmmr_rm_log.bin.26027
-rw-r--r-- 1 root root  260028 Dec 21 12:09 pmmr_rm_log.bin.26029
-rw-r--r-- 1 root root  260076 Dec 21 13:08 pmmr_rm_log.bin.26103
-rw-r--r-- 1 root root  260184 Dec 21 16:06 pmmr_rm_log.bin.26126
-rw-r--r-- 1 root root  260028 Dec 21 12:06 pmmr_rm_log.bin.2617
-rw-r--r-- 1 root root  283476 Dec 21 15:47 pmmr_rm_log.bin.26232
-rw-r--r-- 1 root root  260184 Dec 21 20:52 pmmr_rm_log.bin.26549
-rw-r--r-- 1 root root  260184 Dec 21 20:57 pmmr_rm_log.bin.26556
-rw-r--r-- 1 root root  260184 Dec 21 21:14 pmmr_rm_log.bin.26571
-rw-r--r-- 1 root root  260184 Dec 21 23:38 pmmr_rm_log.bin.26718
-rw-r--r-- 1 root root  260184 Dec 22 13:50 pmmr_rm_log.bin.26956
-rw-r--r-- 1 root root  260184 Dec 22 03:41 pmmr_rm_log.bin.26973
-rw-r--r-- 1 root root  260184 Dec 22 03:45 pmmr_rm_log.bin.26979
-rw-r--r-- 1 root root  260184 Dec 22 04:44 pmmr_rm_log.bin.27038
-rw-r--r-- 1 root root  229512 Dec 20 16:44 pmmr_rm_log.bin.2720
-rw-r--r-- 1 root root  260184 Dec 22 12:10 pmmr_rm_log.bin.27494
-rw-r--r-- 1 root root  260184 Dec 22 14:13 pmmr_rm_log.bin.27612
-rw-r--r-- 1 root root  260184 Dec 22 14:53 pmmr_rm_log.bin.27654
-rw-r--r-- 1 root root  260184 Dec 22 15:02 pmmr_rm_log.bin.27665
-rw-r--r-- 1 root root  260184 Dec 22 15:42 pmmr_rm_log.bin.27704
-rw-r--r-- 1 root root  260184 Dec 22 15:52 pmmr_rm_log.bin.27712
-rw-r--r-- 1 root root  260184 Dec 22 15:58 pmmr_rm_log.bin.27719
-rw-r--r-- 1 root root  260184 Dec 22 16:32 pmmr_rm_log.bin.27755
-rw-r--r-- 1 root root  260184 Dec 22 16:44 pmmr_rm_log.bin.27767
-rw-r--r-- 1 root root  260184 Dec 22 17:09 pmmr_rm_log.bin.27794
-rw-r--r-- 1 root root  260184 Dec 22 17:34 pmmr_rm_log.bin.27818
-rw-r--r-- 1 root root  260184 Dec 22 17:37 pmmr_rm_log.bin.27823
-rw-r--r-- 1 root root  260184 Dec 22 18:12 pmmr_rm_log.bin.27854
-rw-r--r-- 1 root root  260184 Dec 22 18:43 pmmr_rm_log.bin.27883
-rw-r--r-- 1 root root  260184 Dec 22 18:51 pmmr_rm_log.bin.27893
-rw-r--r-- 1 root root  260184 Dec 22 19:29 pmmr_rm_log.bin.27935
-rw-r--r-- 1 root root  260184 Dec 22 19:31 pmmr_rm_log.bin.27938
-rw-r--r-- 1 root root  260184 Dec 22 20:04 pmmr_rm_log.bin.27974
-rw-r--r-- 1 root root  260184 Dec 22 20:07 pmmr_rm_log.bin.27976
-rw-r--r-- 1 root root  260184 Dec 22 20:44 pmmr_rm_log.bin.28014
-rw-r--r-- 1 root root  260184 Dec 22 21:29 pmmr_rm_log.bin.28065
-rw-r--r-- 1 root root  260184 Dec 22 21:40 pmmr_rm_log.bin.28073
-rw-r--r-- 1 root root  260184 Dec 22 21:49 pmmr_rm_log.bin.28081
-rw-r--r-- 1 root root  260184 Dec 23 00:05 pmmr_rm_log.bin.28219
-rw-r--r-- 1 root root  260184 Dec 23 00:11 pmmr_rm_log.bin.28224
-rw-r--r-- 1 root root  260184 Dec 23 00:31 pmmr_rm_log.bin.28243
-rw-r--r-- 1 root root  260184 Dec 23 01:31 pmmr_rm_log.bin.28303
-rw-r--r-- 1 root root  260184 Dec 23 01:38 pmmr_rm_log.bin.28310
-rw-r--r-- 1 root root  260184 Dec 23 02:02 pmmr_rm_log.bin.28335
-rw-r--r-- 1 root root  260184 Dec 23 04:54 pmmr_rm_log.bin.28518
-rw-r--r-- 1 root root  260184 Dec 23 05:58 pmmr_rm_log.bin.28573
-rw-r--r-- 1 root root  260184 Dec 23 06:21 pmmr_rm_log.bin.28603
-rw-r--r-- 1 root root  260184 Dec 23 06:40 pmmr_rm_log.bin.28617
-rw-r--r-- 1 root root  260184 Dec 23 06:46 pmmr_rm_log.bin.28623
-rw-r--r-- 1 root root  260184 Dec 23 07:50 pmmr_rm_log.bin.28687
-rw-r--r-- 1 root root  260184 Dec 23 08:17 pmmr_rm_log.bin.28714
-rw-r--r-- 1 root root  260184 Dec 23 08:27 pmmr_rm_log.bin.28724
-rw-r--r-- 1 root root  260184 Dec 23 08:54 pmmr_rm_log.bin.28759
-rw-r--r-- 1 root root  260184 Dec 23 10:14 pmmr_rm_log.bin.28844
-rw-r--r-- 1 root root  260184 Dec 23 10:19 pmmr_rm_log.bin.28849
-rw-r--r-- 1 root root  260184 Dec 23 10:36 pmmr_rm_log.bin.28864
-rw-r--r-- 1 root root  260184 Dec 23 10:43 pmmr_rm_log.bin.28872
-rw-r--r-- 1 root root  260184 Dec 23 10:56 pmmr_rm_log.bin.28880
-rw-r--r-- 1 root root  260184 Dec 23 10:56 pmmr_rm_log.bin.28881
-rw-r--r-- 1 root root  260184 Dec 23 11:05 pmmr_rm_log.bin.28882
-rw-r--r-- 1 root root  260184 Dec 23 11:40 pmmr_rm_log.bin.28910
-rw-r--r-- 1 root root  260184 Dec 23 11:53 pmmr_rm_log.bin.28928
-rw-r--r-- 1 root root  260184 Dec 23 12:04 pmmr_rm_log.bin.28941
-rw-r--r-- 1 root root  260184 Dec 23 12:33 pmmr_rm_log.bin.28968
-rw-r--r-- 1 root root  260184 Dec 23 12:45 pmmr_rm_log.bin.28987
-rw-r--r-- 1 root root  260184 Dec 23 14:23 pmmr_rm_log.bin.29095
-rw-r--r-- 1 root root  260184 Dec 23 15:05 pmmr_rm_log.bin.29133
-rw-r--r-- 1 root root  260184 Dec 23 16:05 pmmr_rm_log.bin.29191
-rw-r--r-- 1 root root  260184 Dec 23 16:24 pmmr_rm_log.bin.29207
-rw-r--r-- 1 root root  260184 Dec 23 19:15 pmmr_rm_log.bin.29375
-rw-r--r-- 1 root root  260184 Dec 23 19:37 pmmr_rm_log.bin.29411
-rw-r--r-- 1 root root  260184 Dec 23 20:45 pmmr_rm_log.bin.29490
-rw-r--r-- 1 root root  260184 Dec 23 22:29 pmmr_rm_log.bin.29595
-rw-r--r-- 1 root root  260184 Dec 23 23:25 pmmr_rm_log.bin.29647
-rw-r--r-- 1 root root  260184 Dec 24 00:30 pmmr_rm_log.bin.29719
-rw-r--r-- 1 root root  260184 Dec 24 01:40 pmmr_rm_log.bin.29769
-rw-r--r-- 1 root root  260184 Dec 24 01:43 pmmr_rm_log.bin.29774
-rw-r--r-- 1 root root  260184 Dec 24 01:50 pmmr_rm_log.bin.29785
-rw-r--r-- 1 root root  260184 Dec 24 03:28 pmmr_rm_log.bin.29877
-rw-r--r-- 1 root root  260184 Dec 24 01:16 pmmr_rm_log.bin.360
-rw-r--r-- 1 root root  260028 Dec 21 12:41 pmmr_rm_log.bin.4258
-rw-r--r-- 1 root root  229512 Dec 20 18:15 pmmr_rm_log.bin.4525
-rw-r--r-- 1 root root  210444 Dec 20 00:54 pmmr_rm_log.bin.4751
-rw-r--r-- 1 root root  260184 Dec 22 13:56 pmmr_rm_log.bin.4810
-rw-r--r-- 1 root root  260028 Dec 21 12:55 pmmr_rm_log.bin.4873
-rw-r--r-- 1 root root  171792 Dec 19 14:08 pmmr_rm_log.bin.4935
-rw-r--r-- 1 root root  229512 Dec 20 18:34 pmmr_rm_log.bin.4959
-rw-r--r-- 1 root root  229512 Dec 20 15:21 pmmr_rm_log.bin.4988
-rw-r--r-- 1 root root  229512 Dec 20 15:24 pmmr_rm_log.bin.5068
-rw-r--r-- 1 root root  260184 Dec 22 13:59 pmmr_rm_log.bin.5069
-rw-r--r-- 1 root root  210540 Dec 20 00:58 pmmr_rm_log.bin.5075
-rw-r--r-- 1 root root  210576 Dec 20 01:00 pmmr_rm_log.bin.5127
-rw-r--r-- 1 root root  260028 Dec 21 13:02 pmmr_rm_log.bin.5138
-rw-r--r-- 1 root root  210672 Dec 20 01:02 pmmr_rm_log.bin.5150
-rw-r--r-- 1 root root  229512 Dec 20 18:40 pmmr_rm_log.bin.5155
-rw-r--r-- 1 root root  229512 Dec 20 15:25 pmmr_rm_log.bin.5168
-rw-r--r-- 1 root root  229512 Dec 20 15:25 pmmr_rm_log.bin.5238
-rw-r--r-- 1 root root  229512 Dec 20 18:46 pmmr_rm_log.bin.5409
-rw-r--r-- 1 root root  260028 Dec 21 12:39 pmmr_rm_log.bin.6884
-rw-r--r-- 1 root root  173952 Dec 19 15:11 pmmr_rm_log.bin.7622
-rw-r--r-- 1 root root  260184 Dec 22 14:21 pmmr_rm_log.bin.7632
-rw-r--r-- 1 root root  229512 Dec 20 20:51 pmmr_rm_log.bin.8094
-rw-r--r-- 1 root root  229512 Dec 20 20:51 pmmr_rm_log.bin.8118
-rw-r--r-- 1 root root  174588 Dec 19 15:31 pmmr_rm_log.bin.8649
-rw-r--r-- 1 root root  260028 Dec 21 13:00 pmmr_rm_log.bin.9775
-rw-r--r-- 1 root root  260028 Dec 21 13:02 pmmr_rm_log.bin.9926
antiochp commented 6 years ago

Same thing sorted chronologically - 

ls -altr .grin/sumtrees/utxo/
total 60796
drwxr-xr-x 5 root root    4096 Dec 17 22:36 ..
-rw-r--r-- 1 root root   53724 Dec 18 00:41 pmmr_rm_log.bin.21454
-rw-r--r-- 1 root root   72888 Dec 18 02:50 pmmr_rm_log.bin.21517
-rw-r--r-- 1 root root   73488 Dec 18 02:52 pmmr_rm_log.bin.22010
-rw-r--r-- 1 root root  138912 Dec 18 11:25 pmmr_rm_log.bin.21533
-rw-r--r-- 1 root root   55356 Dec 18 12:29 pmmr_rm_log.bin.18707
-rw-r--r-- 1 root root   56316 Dec 18 12:37 pmmr_rm_log.bin.19246
-rw-r--r-- 1 root root   57972 Dec 18 12:40 pmmr_rm_log.bin.21657
-rw-r--r-- 1 root root   60420 Dec 18 13:24 pmmr_rm_log.bin.21694
-rw-r--r-- 1 root root   81444 Dec 18 14:55 pmmr_rm_log.bin.21804
-rw-r--r-- 1 root root   92064 Dec 18 15:45 pmmr_rm_log.bin.21857
-rw-r--r-- 1 root root   96948 Dec 18 16:13 pmmr_rm_log.bin.21847
-rw-r--r-- 1 root root   98628 Dec 18 16:18 pmmr_rm_log.bin.21882
-rw-r--r-- 1 root root  135924 Dec 18 19:06 pmmr_rm_log.bin.21974
-rw-r--r-- 1 root root  137724 Dec 18 19:26 pmmr_rm_log.bin.22019
-rw-r--r-- 1 root root  137808 Dec 18 20:25 pmmr_rm_log.bin.22145
-rw-r--r-- 1 root root  142392 Dec 19 02:12 pmmr_rm_log.bin.22511
-rw-r--r-- 1 root root  144252 Dec 19 02:55 pmmr_rm_log.bin.22563
-rw-r--r-- 1 root root  150804 Dec 19 05:22 pmmr_rm_log.bin.22667
-rw-r--r-- 1 root root  153408 Dec 19 06:21 pmmr_rm_log.bin.22737
-rw-r--r-- 1 root root  159768 Dec 19 08:59 pmmr_rm_log.bin.22915
-rw-r--r-- 1 root root  161628 Dec 19 09:47 pmmr_rm_log.bin.21554
-rw-r--r-- 1 root root  162000 Dec 19 09:56 pmmr_rm_log.bin.21970
-rw-r--r-- 1 root root  162900 Dec 19 10:19 pmmr_rm_log.bin.22789
-rw-r--r-- 1 root root  167808 Dec 19 12:31 pmmr_rm_log.bin.23110
-rw-r--r-- 1 root root  168624 Dec 19 12:35 pmmr_rm_log.bin.23121
-rw-r--r-- 1 root root  168816 Dec 19 12:40 pmmr_rm_log.bin.16693
-rw-r--r-- 1 root root  169668 Dec 19 13:06 pmmr_rm_log.bin.23182
-rw-r--r-- 1 root root  170616 Dec 19 13:32 pmmr_rm_log.bin.23211
-rw-r--r-- 1 root root  170712 Dec 19 13:36 pmmr_rm_log.bin.23215
-rw-r--r-- 1 root root  171792 Dec 19 14:08 pmmr_rm_log.bin.4935
-rw-r--r-- 1 root root  173952 Dec 19 15:11 pmmr_rm_log.bin.7622
-rw-r--r-- 1 root root  174588 Dec 19 15:31 pmmr_rm_log.bin.8649
-rw-r--r-- 1 root root  174804 Dec 19 15:38 pmmr_rm_log.bin.23132
-rw-r--r-- 1 root root  175008 Dec 19 15:44 pmmr_rm_log.bin.23319
-rw-r--r-- 1 root root  181800 Dec 19 17:40 pmmr_rm_log.bin.23473
-rw-r--r-- 1 root root  181848 Dec 19 17:46 pmmr_rm_log.bin.23477
-rw-r--r-- 1 root root  185352 Dec 19 18:21 pmmr_rm_log.bin.23500
-rw-r--r-- 1 root root  185736 Dec 19 18:34 pmmr_rm_log.bin.13110
-rw-r--r-- 1 root root  185844 Dec 19 18:39 pmmr_rm_log.bin.13291
-rw-r--r-- 1 root root  186384 Dec 19 18:55 pmmr_rm_log.bin.13944
-rw-r--r-- 1 root root  186504 Dec 19 18:59 pmmr_rm_log.bin.12349
-rw-r--r-- 1 root root  186648 Dec 19 19:03 pmmr_rm_log.bin.23550
-rw-r--r-- 1 root root  186888 Dec 19 19:13 pmmr_rm_log.bin.13113
-rw-r--r-- 1 root root  187632 Dec 19 19:37 pmmr_rm_log.bin.14575
-rw-r--r-- 1 root root  188232 Dec 19 19:59 pmmr_rm_log.bin.15508
-rw-r--r-- 1 root root  188496 Dec 19 20:05 pmmr_rm_log.bin.15496
-rw-r--r-- 1 root root  188532 Dec 19 20:07 pmmr_rm_log.bin.15545
-rw-r--r-- 1 root root  188532 Dec 19 20:08 pmmr_rm_log.bin.23611
-rw-r--r-- 1 root root  188604 Dec 19 20:11 pmmr_rm_log.bin.21931
-rw-r--r-- 1 root root  188604 Dec 19 20:11 pmmr_rm_log.bin.15741
-rw-r--r-- 1 root root  188712 Dec 19 20:16 pmmr_rm_log.bin.17896
-rw-r--r-- 1 root root  189072 Dec 19 20:27 pmmr_rm_log.bin.16682
-rw-r--r-- 1 root root  189156 Dec 19 20:31 pmmr_rm_log.bin.18902
-rw-r--r-- 1 root root  189156 Dec 19 20:32 pmmr_rm_log.bin.16929
-rw-r--r-- 1 root root  189288 Dec 19 20:36 pmmr_rm_log.bin.16602
-rw-r--r-- 1 root root  189456 Dec 19 20:41 pmmr_rm_log.bin.16732
-rw-r--r-- 1 root root  189624 Dec 19 20:45 pmmr_rm_log.bin.17819
-rw-r--r-- 1 root root  189660 Dec 19 20:46 pmmr_rm_log.bin.17887
-rw-r--r-- 1 root root  190080 Dec 19 21:00 pmmr_rm_log.bin.17422
-rw-r--r-- 1 root root  191544 Dec 19 21:48 pmmr_rm_log.bin.21568
-rw-r--r-- 1 root root  191592 Dec 19 21:50 pmmr_rm_log.bin.21604
-rw-r--r-- 1 root root  192492 Dec 19 22:21 pmmr_rm_log.bin.21684
-rw-r--r-- 1 root root  206016 Dec 19 23:33 pmmr_rm_log.bin.23796
-rw-r--r-- 1 root root  210444 Dec 20 00:54 pmmr_rm_log.bin.4751
-rw-r--r-- 1 root root  210540 Dec 20 00:58 pmmr_rm_log.bin.5075
-rw-r--r-- 1 root root  210576 Dec 20 01:00 pmmr_rm_log.bin.5127
-rw-r--r-- 1 root root  210672 Dec 20 01:02 pmmr_rm_log.bin.5150
-rw-r--r-- 1 root root  211152 Dec 20 01:18 pmmr_rm_log.bin.19218
-rw-r--r-- 1 root root  211536 Dec 20 01:33 pmmr_rm_log.bin.19827
-rw-r--r-- 1 root root  212064 Dec 20 01:49 pmmr_rm_log.bin.20454
-rw-r--r-- 1 root root  212520 Dec 20 02:06 pmmr_rm_log.bin.21047
-rw-r--r-- 1 root root  212532 Dec 20 02:07 pmmr_rm_log.bin.21128
-rw-r--r-- 1 root root  212940 Dec 20 02:20 pmmr_rm_log.bin.21535
-rw-r--r-- 1 root root  213036 Dec 20 02:23 pmmr_rm_log.bin.21638
-rw-r--r-- 1 root root  213348 Dec 20 02:34 pmmr_rm_log.bin.21826
-rw-r--r-- 1 root root  221328 Dec 20 07:34 pmmr_rm_log.bin.24300
-rw-r--r-- 1 root root  221760 Dec 20 07:50 pmmr_rm_log.bin.24338
-rw-r--r-- 1 root root  222060 Dec 20 08:03 pmmr_rm_log.bin.23558
-rw-r--r-- 1 root root  223080 Dec 20 08:12 pmmr_rm_log.bin.24366
-rw-r--r-- 1 root root  223104 Dec 20 08:15 pmmr_rm_log.bin.22185
-rw-r--r-- 1 root root  223440 Dec 20 08:27 pmmr_rm_log.bin.22885
-rw-r--r-- 1 root root  223788 Dec 20 08:40 pmmr_rm_log.bin.23349
-rw-r--r-- 1 root root  224052 Dec 20 08:50 pmmr_rm_log.bin.24413
-rw-r--r-- 1 root root  226044 Dec 20 10:18 pmmr_rm_log.bin.21840
-rw-r--r-- 1 root root  226176 Dec 20 10:25 pmmr_rm_log.bin.21727
-rw-r--r-- 1 root root  227892 Dec 20 11:31 pmmr_rm_log.bin.24572
-rw-r--r-- 1 root root  229032 Dec 20 12:17 pmmr_rm_log.bin.24622
-rw-r--r-- 1 root root  229476 Dec 20 12:39 pmmr_rm_log.bin.24641
-rw-r--r-- 1 root root  229512 Dec 20 13:00 pmmr_rm_log.bin.24676
-rw-r--r-- 1 root root  229512 Dec 20 15:21 pmmr_rm_log.bin.4988
-rw-r--r-- 1 root root  229512 Dec 20 15:24 pmmr_rm_log.bin.5068
-rw-r--r-- 1 root root  229512 Dec 20 15:25 pmmr_rm_log.bin.5168
-rw-r--r-- 1 root root  229512 Dec 20 15:25 pmmr_rm_log.bin.5238
-rw-r--r-- 1 root root  229512 Dec 20 15:26 pmmr_rm_log.bin.24816
-rw-r--r-- 1 root root  229512 Dec 20 16:44 pmmr_rm_log.bin.2720
-rw-r--r-- 1 root root  229512 Dec 20 18:15 pmmr_rm_log.bin.4525
-rw-r--r-- 1 root root  229512 Dec 20 18:34 pmmr_rm_log.bin.4959
-rw-r--r-- 1 root root  229512 Dec 20 18:40 pmmr_rm_log.bin.5155
-rw-r--r-- 1 root root  229512 Dec 20 18:46 pmmr_rm_log.bin.5409
-rw-r--r-- 1 root root  229512 Dec 20 20:17 pmmr_rm_log.bin.25103
-rw-r--r-- 1 root root  229512 Dec 20 20:51 pmmr_rm_log.bin.8094
-rw-r--r-- 1 root root  229512 Dec 20 20:51 pmmr_rm_log.bin.8118
-rw-r--r-- 1 root root  229512 Dec 20 20:56 pmmr_rm_log.bin.25139
-rw-r--r-- 1 root root  229512 Dec 20 21:42 pmmr_rm_log.bin.25199
-rw-r--r-- 1 root root  243888 Dec 20 22:16 pmmr_rm_log.bin.25231
-rw-r--r-- 1 root root  244440 Dec 20 23:00 pmmr_rm_log.bin.25278
-rw-r--r-- 1 root root  245340 Dec 20 23:41 pmmr_rm_log.bin.25315
-rw-r--r-- 1 root root  245916 Dec 21 00:10 pmmr_rm_log.bin.23769
-rw-r--r-- 1 root root  246048 Dec 21 00:16 pmmr_rm_log.bin.24070
-rw-r--r-- 1 root root  246432 Dec 21 00:33 pmmr_rm_log.bin.25364
-rw-r--r-- 1 root root  246816 Dec 21 00:53 pmmr_rm_log.bin.25387
-rw-r--r-- 1 root root  246984 Dec 21 01:02 pmmr_rm_log.bin.22875
-rw-r--r-- 1 root root  247440 Dec 21 01:21 pmmr_rm_log.bin.23518
-rw-r--r-- 1 root root  249048 Dec 21 02:41 pmmr_rm_log.bin.14108
-rw-r--r-- 1 root root  249540 Dec 21 03:01 pmmr_rm_log.bin.15229
-rw-r--r-- 1 root root  249540 Dec 21 03:05 pmmr_rm_log.bin.15251
-rw-r--r-- 1 root root  250932 Dec 21 04:05 pmmr_rm_log.bin.25489
-rw-r--r-- 1 root root  251520 Dec 21 04:35 pmmr_rm_log.bin.19074
-rw-r--r-- 1 root root  254280 Dec 21 06:44 pmmr_rm_log.bin.24946
-rw-r--r-- 1 root root  255096 Dec 21 07:23 pmmr_rm_log.bin.25327
-rw-r--r-- 1 root root  255264 Dec 21 07:30 pmmr_rm_log.bin.25595
-rw-r--r-- 1 root root  255264 Dec 21 07:30 pmmr_rm_log.bin.25613
-rw-r--r-- 1 root root  256392 Dec 21 08:25 pmmr_rm_log.bin.25769
-rw-r--r-- 1 root root  259956 Dec 21 09:29 pmmr_rm_log.bin.25827
-rw-r--r-- 1 root root  260016 Dec 21 09:34 pmmr_rm_log.bin.11738
-rw-r--r-- 1 root root  260028 Dec 21 09:57 pmmr_rm_log.bin.14445
-rw-r--r-- 1 root root  260028 Dec 21 09:59 pmmr_rm_log.bin.14653
-rw-r--r-- 1 root root  260028 Dec 21 10:07 pmmr_rm_log.bin.16072
-rw-r--r-- 1 root root  260028 Dec 21 10:08 pmmr_rm_log.bin.16742
-rw-r--r-- 1 root root  260028 Dec 21 10:12 pmmr_rm_log.bin.25885
-rw-r--r-- 1 root root  260028 Dec 21 10:13 pmmr_rm_log.bin.17741
-rw-r--r-- 1 root root  260028 Dec 21 10:24 pmmr_rm_log.bin.19142
-rw-r--r-- 1 root root  260028 Dec 21 10:34 pmmr_rm_log.bin.25910
-rw-r--r-- 1 root root  260028 Dec 21 10:52 pmmr_rm_log.bin.25933
-rw-r--r-- 1 root root  260028 Dec 21 12:06 pmmr_rm_log.bin.2617
-rw-r--r-- 1 root root  260028 Dec 21 12:07 pmmr_rm_log.bin.26027
-rw-r--r-- 1 root root  260028 Dec 21 12:09 pmmr_rm_log.bin.26029
-rw-r--r-- 1 root root  260028 Dec 21 12:39 pmmr_rm_log.bin.6884
-rw-r--r-- 1 root root  260028 Dec 21 12:41 pmmr_rm_log.bin.4258
-rw-r--r-- 1 root root  260028 Dec 21 12:55 pmmr_rm_log.bin.4873
-rw-r--r-- 1 root root  260028 Dec 21 13:00 pmmr_rm_log.bin.9775
-rw-r--r-- 1 root root  260028 Dec 21 13:02 pmmr_rm_log.bin.5138
-rw-r--r-- 1 root root  260028 Dec 21 13:02 pmmr_rm_log.bin.9926
-rw-r--r-- 1 root root  260076 Dec 21 13:08 pmmr_rm_log.bin.26103
-rw-r--r-- 1 root root  283476 Dec 21 15:47 pmmr_rm_log.bin.26232
-rw-r--r-- 1 root root  260184 Dec 21 16:06 pmmr_rm_log.bin.26126
-rw-r--r-- 1 root root  260184 Dec 21 17:48 pmmr_rm_log.bin.21928
-rw-r--r-- 1 root root  260184 Dec 21 19:37 pmmr_rm_log.bin.21918
-rw-r--r-- 1 root root  260184 Dec 21 20:52 pmmr_rm_log.bin.26549
-rw-r--r-- 1 root root  260184 Dec 21 20:57 pmmr_rm_log.bin.26556
-rw-r--r-- 1 root root  260184 Dec 21 21:14 pmmr_rm_log.bin.26571
-rw-r--r-- 1 root root  260184 Dec 21 23:38 pmmr_rm_log.bin.26718
-rw-r--r-- 1 root root  260184 Dec 22 03:41 pmmr_rm_log.bin.26973
-rw-r--r-- 1 root root  260184 Dec 22 03:45 pmmr_rm_log.bin.26979
-rw-r--r-- 1 root root  260184 Dec 22 04:44 pmmr_rm_log.bin.27038
-rw-r--r-- 1 root root  260184 Dec 22 12:10 pmmr_rm_log.bin.27494
-rw-r--r-- 1 root root  260184 Dec 22 13:50 pmmr_rm_log.bin.26956
-rw-r--r-- 1 root root  260184 Dec 22 13:56 pmmr_rm_log.bin.4810
-rw-r--r-- 1 root root  260184 Dec 22 13:59 pmmr_rm_log.bin.5069
-rw-r--r-- 1 root root  260184 Dec 22 14:13 pmmr_rm_log.bin.27612
-rw-r--r-- 1 root root  260184 Dec 22 14:21 pmmr_rm_log.bin.7632
-rw-r--r-- 1 root root  260184 Dec 22 14:53 pmmr_rm_log.bin.27654
-rw-r--r-- 1 root root  260184 Dec 22 15:02 pmmr_rm_log.bin.27665
-rw-r--r-- 1 root root  260184 Dec 22 15:42 pmmr_rm_log.bin.27704
-rw-r--r-- 1 root root  260184 Dec 22 15:52 pmmr_rm_log.bin.27712
-rw-r--r-- 1 root root  260184 Dec 22 15:58 pmmr_rm_log.bin.27719
-rw-r--r-- 1 root root  260184 Dec 22 16:32 pmmr_rm_log.bin.27755
-rw-r--r-- 1 root root  260184 Dec 22 16:44 pmmr_rm_log.bin.27767
-rw-r--r-- 1 root root  260184 Dec 22 17:09 pmmr_rm_log.bin.27794
-rw-r--r-- 1 root root  260184 Dec 22 17:34 pmmr_rm_log.bin.27818
-rw-r--r-- 1 root root  260184 Dec 22 17:37 pmmr_rm_log.bin.27823
-rw-r--r-- 1 root root  260184 Dec 22 18:12 pmmr_rm_log.bin.27854
-rw-r--r-- 1 root root  260184 Dec 22 18:43 pmmr_rm_log.bin.27883
-rw-r--r-- 1 root root  260184 Dec 22 18:51 pmmr_rm_log.bin.27893
-rw-r--r-- 1 root root  260184 Dec 22 19:29 pmmr_rm_log.bin.27935
-rw-r--r-- 1 root root  260184 Dec 22 19:31 pmmr_rm_log.bin.27938
-rw-r--r-- 1 root root  260184 Dec 22 20:04 pmmr_rm_log.bin.27974
-rw-r--r-- 1 root root  260184 Dec 22 20:07 pmmr_rm_log.bin.27976
-rw-r--r-- 1 root root  260184 Dec 22 20:37 pmmr_rm_log.bin.25943
-rw-r--r-- 1 root root  260184 Dec 22 20:44 pmmr_rm_log.bin.28014
-rw-r--r-- 1 root root  260184 Dec 22 20:46 pmmr_rm_log.bin.15729
-rw-r--r-- 1 root root  260184 Dec 22 20:46 pmmr_rm_log.bin.15779
-rw-r--r-- 1 root root  260184 Dec 22 20:58 pmmr_rm_log.bin.18751
-rw-r--r-- 1 root root  260184 Dec 22 21:23 pmmr_rm_log.bin.21367
-rw-r--r-- 1 root root  260184 Dec 22 21:29 pmmr_rm_log.bin.28065
-rw-r--r-- 1 root root  260184 Dec 22 21:36 pmmr_rm_log.bin.21705
-rw-r--r-- 1 root root  260184 Dec 22 21:40 pmmr_rm_log.bin.28073
-rw-r--r-- 1 root root  260184 Dec 22 21:49 pmmr_rm_log.bin.28081
-rw-r--r-- 1 root root  260184 Dec 22 21:52 pmmr_rm_log.bin.15486
-rw-r--r-- 1 root root  260184 Dec 23 00:05 pmmr_rm_log.bin.28219
-rw-r--r-- 1 root root  260184 Dec 23 00:11 pmmr_rm_log.bin.28224
-rw-r--r-- 1 root root  260184 Dec 23 00:31 pmmr_rm_log.bin.28243
-rw-r--r-- 1 root root  260184 Dec 23 01:31 pmmr_rm_log.bin.28303
-rw-r--r-- 1 root root  260184 Dec 23 01:38 pmmr_rm_log.bin.28310
-rw-r--r-- 1 root root  260184 Dec 23 02:02 pmmr_rm_log.bin.28335
-rw-r--r-- 1 root root  260184 Dec 23 04:54 pmmr_rm_log.bin.28518
-rw-r--r-- 1 root root  260184 Dec 23 05:58 pmmr_rm_log.bin.28573
-rw-r--r-- 1 root root  260184 Dec 23 06:21 pmmr_rm_log.bin.28603
-rw-r--r-- 1 root root  260184 Dec 23 06:40 pmmr_rm_log.bin.28617
-rw-r--r-- 1 root root  260184 Dec 23 06:46 pmmr_rm_log.bin.28623
-rw-r--r-- 1 root root  260184 Dec 23 07:05 pmmr_rm_log.bin.23023
-rw-r--r-- 1 root root  260184 Dec 23 07:26 pmmr_rm_log.bin.23779
-rw-r--r-- 1 root root  260184 Dec 23 07:50 pmmr_rm_log.bin.28687
-rw-r--r-- 1 root root  260184 Dec 23 08:17 pmmr_rm_log.bin.28714
-rw-r--r-- 1 root root  260184 Dec 23 08:27 pmmr_rm_log.bin.28724
-rw-r--r-- 1 root root  260184 Dec 23 08:54 pmmr_rm_log.bin.28759
-rw-r--r-- 1 root root  260184 Dec 23 09:25 pmmr_rm_log.bin.21932
-rw-r--r-- 1 root root  260184 Dec 23 10:14 pmmr_rm_log.bin.28844
-rw-r--r-- 1 root root  260184 Dec 23 10:19 pmmr_rm_log.bin.28849
-rw-r--r-- 1 root root  260184 Dec 23 10:36 pmmr_rm_log.bin.28864
-rw-r--r-- 1 root root  260184 Dec 23 10:43 pmmr_rm_log.bin.28872
-rw-r--r-- 1 root root  260184 Dec 23 10:56 pmmr_rm_log.bin.28880
-rw-r--r-- 1 root root  260184 Dec 23 10:56 pmmr_rm_log.bin.28881
-rw-r--r-- 1 root root  260184 Dec 23 11:05 pmmr_rm_log.bin.28882
-rw-r--r-- 1 root root  260184 Dec 23 11:40 pmmr_rm_log.bin.28910
-rw-r--r-- 1 root root  260184 Dec 23 11:53 pmmr_rm_log.bin.28928
-rw-r--r-- 1 root root  260184 Dec 23 12:04 pmmr_rm_log.bin.28941
-rw-r--r-- 1 root root  260184 Dec 23 12:33 pmmr_rm_log.bin.28968
-rw-r--r-- 1 root root  260184 Dec 23 12:45 pmmr_rm_log.bin.28987
-rw-r--r-- 1 root root  260184 Dec 23 14:01 pmmr_rm_log.bin.13512
-rw-r--r-- 1 root root  260184 Dec 23 14:23 pmmr_rm_log.bin.29095
-rw-r--r-- 1 root root  260184 Dec 23 15:05 pmmr_rm_log.bin.29133
-rw-r--r-- 1 root root  260184 Dec 23 16:05 pmmr_rm_log.bin.29191
-rw-r--r-- 1 root root  260184 Dec 23 16:24 pmmr_rm_log.bin.29207
-rw-r--r-- 1 root root  260184 Dec 23 16:30 pmmr_rm_log.bin.21881
-rw-r--r-- 1 root root  260184 Dec 23 17:18 pmmr_rm_log.bin.22550
-rw-r--r-- 1 root root  260184 Dec 23 17:20 pmmr_rm_log.bin.22600
-rw-r--r-- 1 root root  260184 Dec 23 19:15 pmmr_rm_log.bin.29375
-rw-r--r-- 1 root root  260184 Dec 23 19:37 pmmr_rm_log.bin.29411
-rw-r--r-- 1 root root  260184 Dec 23 20:45 pmmr_rm_log.bin.29490
-rw-r--r-- 1 root root  260184 Dec 23 22:29 pmmr_rm_log.bin.29595
-rw-r--r-- 1 root root  260184 Dec 23 23:25 pmmr_rm_log.bin.29647
-rw-r--r-- 1 root root  260184 Dec 24 00:30 pmmr_rm_log.bin.29719
-rw-r--r-- 1 root root  260184 Dec 24 01:08 pmmr_rm_log.bin.23010
-rw-r--r-- 1 root root  260184 Dec 24 01:09 pmmr_rm_log.bin.23105
-rw-r--r-- 1 root root  260184 Dec 24 01:14 pmmr_rm_log.bin.23285
-rw-r--r-- 1 root root  260184 Dec 24 01:16 pmmr_rm_log.bin.360
-rw-r--r-- 1 root root  260184 Dec 24 01:20 pmmr_rm_log.bin.23476
-rw-r--r-- 1 root root  260184 Dec 24 01:24 pmmr_rm_log.bin.0
-rw-r--r-- 1 root root  260184 Dec 24 01:40 pmmr_rm_log.bin.29769
-rw-r--r-- 1 root root  260184 Dec 24 01:43 pmmr_rm_log.bin.29774
-rw-r--r-- 1 root root  260184 Dec 24 01:50 pmmr_rm_log.bin.29785
drwxr-xr-x 2 root root   16384 Dec 24 03:21 .
-rw-r--r-- 1 root root  260184 Dec 24 03:28 pmmr_rm_log.bin.29877
-rw-r--r-- 1 root root 6365320 Dec 24 04:03 pmmr_dat.bin
-rw-r--r-- 1 root root  260184 Dec 24 04:03 pmmr_rm_log.bin
antiochp commented 6 years ago

This one jumps out as weird (we rewound to pos 0?) - 

-rw-r--r-- 1 root root  260184 Dec 24 01:24 pmmr_rm_log.bin.0

Edit:

Looking through the logs, this is actually ok (I think) - we processed a forked block at height 1 by the looks of it -

Dec 24 01:12:28.477 DEBG Rewind sumtrees to header e17ea64f at 0
Dec 24 01:12:28.477 DEBG Rewind sumtrees to output pos: 0, kernel pos: 0
Dec 24 01:12:49.828 DEBG Rewind sumtrees to header e17ea64f at 0
Dec 24 01:12:49.828 DEBG Rewind sumtrees to output pos: 0, kernel pos: 0

Same for the one at height 360 which also looked suspicious - 

Dec 24 01:15:26.901 DEBG Rewind sumtrees to header 1f727461 at 360
Dec 24 01:15:26.903 DEBG Rewind sumtrees to output pos: 713, kernel pos: 713
Dec 24 01:16:12.830 DEBG Rewind sumtrees to header 1f727461 at 360
Dec 24 01:16:12.831 DEBG Rewind sumtrees to output pos: 713, kernel pos: 713
heunglee commented 6 years ago

If there are duplicate inputs, may the block get AlreadySpent error due to #544 and the server gets stuck? The corresponding UTXO at the position of first one of the duplicate input may be None when pruning the UTXO sumtree? Are they indexed to the same position?

heunglee commented 6 years ago

In the part of duplicate inputs, **** Found duplicate input: [ { 'commit': '093d5e239aab86ef4b8a53a2d290673e3aff77f3dda2bb0c4c6db706e9c97ce753', 'height': 26126}, { 'commit': '093d5e239aab86ef4b8a53a2d290673e3aff77f3dda2bb0c4c6db706e9c97ce753', 'height': 21883}]

Block 21883 was inserted on 12/18/17 and block 26126 on 12/21/17, based on explorer records. The position of input 093d5...: is 51724 for both blocks. For some reason, the input was not removed from UTXO when block 21883 was processed.

antiochp commented 6 years ago

The size of each file should be increasing as the block heights increase (each spent output position is in there).

1) This suggests we may not handle some rewind scenarios correctly? If we rewind to a point earlier than a given output was spent then this commitment should no longer be in the remove list (it effectively becomes un-spent again). I may be mistaken here - do we truncate the remove log somewhere/somehow on rewind?

2) Also wondering if the remove list assumes duplicates commitments do not exist - we may have spent the earlier commitment but not spent the later commitment. How is this handled in the remove list (if it is in there as removed, we may not handle the later unspent commitment correctly).

ignopeverell commented 6 years ago
  1. Yes we do truncate. That was a general rule but some rewinds can be exceptions as you indicated.
  2. I don't think it does, it's purely positional.

Looking at your remove logs files, there are quite a few there with size 260184, which seems a little suspicious...

ignopeverell commented 6 years ago

Digged a little more into this, here are a few more comments.

  1. Forget my remark about equal size of rm logs, there just haven't been a lot of transactions lately (I guess we're all too busy chasing forks...)
  2. Regarding your comment about picking a random block, checking whether one of its inputs is spent using the API and finding it's not (even though it should), turns out there's a bug in is_unspent in chain/src/sumtrees.rs`. Working on that now, fix hopefully coming soon. That explains why we're seeing frequent blocks with double-spends, as the transaction pool relies on this.
  3. In addition to #544 I think we have an issue with longer forks because we always query the position index from db. However, a fork will be entirely applied in memory, populating the new_output_commits map. So nodes can refuse an otherwise valid input under this scenario:
    • Node is on fork A.
    • Fork B comes in, first block creates a UTXO.
    • Fork B 2nd block comes in, the position of the UTXO won't be found (because it's not in store yet), causing a rejection with AlreadySpent (which we should maybe rename NonExistentOrSpent)

So 2 would explain some of the strange findings in your debugging. Have you found out whether that block should actually have been accepted or not? If it should have been accepted, then 3 will fix this. If not, there's yet another problem (which may still be related to #544).

ignopeverell commented 6 years ago

My 2nd point should be fixed with bfab272 and my 3rd by ba1219f. Do the fixes make sense to you @antiochp ?

antiochp commented 6 years ago

Yeah - thanks for digging deeper into this. I think I get how the rm log works now (at least better than before) - I was missing the positional aspect to it (obvious now) and was assuming we were just looking for inclusion/exclusion in the logs.

If I understand https://github.com/mimblewimble/grin/commit/bfab272f5e10f8e0bbae4cd568ea95f0250e70e5 correctly - we find the entry at the given pos _and check the hash actually matches what we expect (and we need the switch commit hash to do this).

And with https://github.com/mimblewimble/grin/commit/ba1219fc47ba71ce512df9d481631224548653d9 we're checking the in-memory data first before going back to the store. This makes sense to me - good catch!

And yeah - I think NonExistentOrSpent would be a more descriptive name - I think a lot of people are getting confused by the seemingly explicit AlreadySpent.

I'm going to take these latest changes and see what the results look like from rerunning my script, now that we believe is_unspent is returning the correct results.

antiochp commented 6 years ago

Running a full sync from scratch locally and its stuck at block 1988 and does not appear to be able to get a valid block 1989 from any peers...

Dec 26 15:02:21.324 DEBG Received block 7adc0999 at 1989 from network, going to process.
Dec 26 15:02:21.324 DEBG pipe: process_block 7adc0999 at 1989 with 2 inputs and 5 outputs.
Dec 26 15:02:21.324 DEBG pipe: validate_header cuckoo size 16
Dec 26 15:02:21.328 DEBG Starting new sumtree extension.
Dec 26 15:02:21.527 DEBG Error returned, discarding sumtree extension.
Dec 26 15:02:21.527 INFO Rejected block 7adc0999 at 1989: DuplicateCommitment(Commitment(08e747174f4d0c95658d8aff2db4513f9365d33fb6af313393b771bf93fe344388))
Dec 26 15:02:21.527 DEBG Block 7adc0999 refused by chain: DuplicateCommitment(Commitment(08e747174f4d0c95658d8aff2db4513f9365d33fb6af313393b771bf93fe344388))

And from my script output above we see duplicate outputs in block 1989 - 

**** Found duplicate output: 
[ { 'commit': '08999d1ac59fa66e5ec7f71034626ecce00226f92dfb5a8874cf9aead968e30766',
    'height': 1989,
    'output_type': 'Transaction'},
  { 'commit': '08999d1ac59fa66e5ec7f71034626ecce00226f92dfb5a8874cf9aead968e30766',
    'height': 1989,
    'output_type': 'Transaction'}]
**** Found duplicate output: 
[ { 'commit': '08e747174f4d0c95658d8aff2db4513f9365d33fb6af313393b771bf93fe344388',
    'height': 1989,
    'output_type': 'Transaction'},
  { 'commit': '08e747174f4d0c95658d8aff2db4513f9365d33fb6af313393b771bf93fe344388',
    'height': 1989,
    'output_type': 'Transaction'}]

I have a feeling that bfab272 has hard-forked testnet1 at height 1988 with the rest of the chain now (correctly) invalid...

For refererence, everything looks good up to and including block 1988 (except most of the unspent coinbase outputs are missing from the UTXO set) - 

--------------------
Inputs:  189
Outputs:  2042
*** Duplicate (sets) of inputs:  0
*** Duplicate (sets) of outputs:  0
*** Spent yet still in UTXO set:  0
*** Unspent but missing from UTXO set:  1853
--------------------
antiochp commented 6 years ago

Bug in my recon code...

This is what it looks like now up to block 1988 (what we expect to see) - 

--------------------
Inputs:  189
Outputs:  2042
*** Duplicate (sets) of inputs:  0
*** Duplicate (sets) of outputs:  0
*** Spent yet still in UTXO set:  0
*** Unspent but missing from UTXO set:  0
--------------------
antiochp commented 6 years ago

Results from seed (at height 33988) -

--------------------
Inputs:  15989
Outputs:  53039
*** Duplicate (sets) of inputs:  16
*** Duplicate (sets) of outputs:  2
*** Spent yet still in UTXO set:  210
*** Unspent but missing from UTXO set:  111
--------------------
antiochp commented 6 years ago

So block 26126 appears to be the perfect storm of "invalidity". There are 6 inputs included in this block and every one of these is a legitimate AlreadySpent that should have been flagged as invalid.

Each input is present in the block chain as an input to a previous block. Each one was spent in the block immediately following the one it in which it originally appeared (or a couple blocks later in one case). These are presumably all cases of forks being handled in-memory where we had known issues around handling spent/unspent.

First example for commitment 08822c72 -

The other inputs to block 26126 all follow the same or very similar pattern.

Inputs -
[ { 'commit': '08822c722b337569331ac7d124111a399b8df22fa86f70c81dd210c31b7f4b82fc',
    'height': 26126},
  { 'commit': '08822c722b337569331ac7d124111a399b8df22fa86f70c81dd210c31b7f4b82fc',
    'height': 22984}]
Outputs -
[ { 'commit': '08822c722b337569331ac7d124111a399b8df22fa86f70c81dd210c31b7f4b82fc',
    'height': 22983,
    'height_spent': 26126,
    'output_type': 'Transaction'}]
Utxos - 
[ { 'commit': '08822c722b337569331ac7d124111a399b8df22fa86f70c81dd210c31b7f4b82fc',
    'height': 22983}]

----------

Inputs -
[ { 'commit': '09c234b3eb0befb7ce18053b1b1ec606077472cea945043ad54bea7168774a651a',
    'height': 26126},
  { 'commit': '09c234b3eb0befb7ce18053b1b1ec606077472cea945043ad54bea7168774a651a',
    'height': 21837}]
Outputs -
[ { 'commit': '09c234b3eb0befb7ce18053b1b1ec606077472cea945043ad54bea7168774a651a',
    'height': 21836,
    'height_spent': 26126,
    'output_type': 'Transaction'}]
Utxos - 
[ { 'commit': '09c234b3eb0befb7ce18053b1b1ec606077472cea945043ad54bea7168774a651a',
    'height': 21836}]

----------

Inputs -
[ { 'commit': '093d5e239aab86ef4b8a53a2d290673e3aff77f3dda2bb0c4c6db706e9c97ce753',
    'height': 26126},
  { 'commit': '093d5e239aab86ef4b8a53a2d290673e3aff77f3dda2bb0c4c6db706e9c97ce753',
    'height': 21883}]
Outputs -
[ { 'commit': '093d5e239aab86ef4b8a53a2d290673e3aff77f3dda2bb0c4c6db706e9c97ce753',
    'height': 21882,
    'height_spent': 26126,
    'output_type': 'Transaction'}]
Utxos - 
[ { 'commit': '093d5e239aab86ef4b8a53a2d290673e3aff77f3dda2bb0c4c6db706e9c97ce753',
    'height': 21882}]

----------

Inputs -
[ { 'commit': '08fa2d8f35a266bdf05fc5cf585ed9135ce380b10037f80c7ea75243feba5471de',
    'height': 26126},
  { 'commit': '08fa2d8f35a266bdf05fc5cf585ed9135ce380b10037f80c7ea75243feba5471de',
    'height': 22915}]
Outputs -
[ { 'commit': '08fa2d8f35a266bdf05fc5cf585ed9135ce380b10037f80c7ea75243feba5471de',
    'height': 22912,
    'height_spent': 26126,
    'output_type': 'Transaction'}]
Utxos - 
[ { 'commit': '08fa2d8f35a266bdf05fc5cf585ed9135ce380b10037f80c7ea75243feba5471de',
    'height': 22912}]

----------

Inputs -
[ { 'commit': '085889c7b1eb836b5da450990ac90fa3bb46efc7a845aa996ef1c3a3d6cc31ca00',
    'height': 26126},
  { 'commit': '085889c7b1eb836b5da450990ac90fa3bb46efc7a845aa996ef1c3a3d6cc31ca00',
    'height': 21882}]
Outputs -
[ { 'commit': '085889c7b1eb836b5da450990ac90fa3bb46efc7a845aa996ef1c3a3d6cc31ca00',
    'height': 21881,
    'height_spent': 26126,
    'output_type': 'Transaction'}]
Utxos - 
[ { 'commit': '085889c7b1eb836b5da450990ac90fa3bb46efc7a845aa996ef1c3a3d6cc31ca00',
    'height': 21881}]

----------

Inputs -
[ { 'commit': '0946913aed491c666c3b61f641405db74cabb5c1f3d55b3aa09ef352ee0fed7708',
    'height': 26126},
  { 'commit': '0946913aed491c666c3b61f641405db74cabb5c1f3d55b3aa09ef352ee0fed7708',
    'height': 22006}]
Outputs -
[ { 'commit': '0946913aed491c666c3b61f641405db74cabb5c1f3d55b3aa09ef352ee0fed7708',
    'height': 22005,
    'height_spent': 26126,
    'output_type': 'Transaction'}]
Utxos - 
[ { 'commit': '0946913aed491c666c3b61f641405db74cabb5c1f3d55b3aa09ef352ee0fed7708',
    'height': 22005}]
antiochp commented 6 years ago

I think this explains why we have been seeing some blocks that are "hard" to get past during sync, where nodes seem to get stuck at for a long time. Nodes needed to follow a particular fork and then rewind and reprocess a different fork (in memory) to get their internal state into a state that allowed them to continue. In some cases (block 1989 and block 26126 for example) this is actually an invalid state that we had been incorrectly treating as valid.

antiochp commented 6 years ago

Bunch of nodes running latest testnet1 code rewound all the way to block height 1988 and then off on their own chain. Currently at 8225485 @ 2791

Dec 27 19:28:51.740 DEBG peer total_diff @ height (ping/pong): 138.68.178.20:13414: 8225485 @ 2791 vs us: 8225485 @ 2791
Dec 27 19:28:52.924 DEBG peer total_diff @ height (ping/pong): 198.57.85.137:13414: 3532486 @ 966 vs us: 8225485 @ 2791
Dec 27 19:28:52.955 DEBG peer total_diff @ height (ping/pong): 138.68.178.20:13414: 8225485 @ 2791 vs us: 8225485 @ 2791
Dec 27 19:28:52.959 DEBG peer total_diff @ height (ping/pong): 165.227.63.166:13414: 8225485 @ 2791 vs us: 8225485 @ 2791
Dec 27 19:28:52.961 DEBG peer total_diff @ height (ping/pong): 37.59.48.28:13414: 8225485 @ 2791 vs us: 8225485 @ 2791
Dec 27 19:28:52.975 DEBG peer total_diff @ height (ping/pong): 51.15.219.12:13414: 7024734 @ 2104 vs us: 8225485 @ 2791
Dec 27 19:28:52.997 DEBG peer total_diff @ height (ping/pong): 141.138.37.91:13414: 6965661 @ 2010 vs us: 8225485 @ 2791
Dec 27 19:28:53.172 DEBG peer total_diff @ height (ping/pong): 217.24.232.145:13414: 8225485 @ 2791 vs us: 8225485 @ 2791
Dec 27 19:28:55.995 DEBG peer total_diff @ height (ping/pong): 165.227.63.166:13414: 8225485 @ 2791 vs us: 8225485 @ 2791
Dec 27 19:28:56.689 DEBG peer total_diff @ height (ping/pong): 217.24.232.145:13414: 8225485 @ 2791 vs us: 8225485 @ 2791
Dec 27 19:29:02.912 DEBG peer total_diff @ height (ping/pong): 198.57.85.137:13414: 3671068 @ 981 vs us: 8225485 @ 2791
Dec 27 19:29:06.856 DEBG peer total_diff @ height (ping/pong): 37.59.48.28:13414: 8225485 @ 2791 vs us: 8225485 @ 2791

Recon script outputs the following - 

--------------------
Inputs:  195
Outputs:  2845
*** Duplicate (sets) of inputs:  0
*** Duplicate (sets) of outputs:  0
*** Spent yet still in UTXO set:  0
*** Unspent but missing from UTXO set:  0
--------------------
antiochp commented 6 years ago

I'm going to close this for now.

If anyone is still stuck at 26126 or another block that now appears to be on an invalid chain - 1) rebuild and redeploy latest testnet1 code 2) clear out .grin/peers dir (or the full .grin dir for a full resync) 3) wait 10 mins or so to identify and ban peers on invalid forks 4) restart to sync against known good peers