Closed Th3Van closed 5 years ago
From the farmer point of view the behavior is correct. However later you will get a problem with the cheater detection. I will talk to the devs and see what we can do against it.
Meanwhile I will write a script to detect these bad shards on my own farmer and share it as soon as it is working correct.
Confirmed on my farmer Invalid Shard size: Contract Hash 0408f8952f44b81a43413dabbcc0223372688511 Contract size 369021953024 size 360373001
The debug script
const storj = require('storj-lib');
const stream = require('readable-stream');
const crypto = require('crypto');
// insert your storage location here.
// on Windows you need double backslash. C:\\storjshare\\whatever
const persistence = new storj.EmbeddedStorageAdapter('/root/.storjshare/storjshare/shares/a84cc4410fcc091a793fd4f4f1089ae173a2337e');
const manager = new storj.StorageManager(persistence);
const StorageAdapter = require('storj-lib/lib/storage/adapter');
const StorageItem = require('storj-lib/lib/storage/item');
const rstream = manager._storage.createReadStream();
const startDate = Number(new Date('2018-01-18T00:00:00.000Z'));
rstream.on('data', function(item) {
rstream.pause();
for (const nodeID in item.contracts) {
const contract = item.contracts[nodeID];
if (contract.get('store_begin') > startDate ) {
//if (item.hash === 'c56d9a40dcaf2f3f54753e749803cbdcfcc96363' ) {
manager.load(item.hash, function(err, shard) {
if (!(shard.shard instanceof stream.Writable)) {
const hasher = crypto.createHash('sha256');
var size = 0;
shard.shard.on('data', function (chunk) {
hasher.update(chunk);
size += chunk.length;
}).on('end', function () {
const hash = crypto.createHash('rmd160').update(hasher.digest()).digest('hex');
if (item.hash !== hash) {
console.log('Invalid Shard: Contract Hash %s\tShard Hash %s\tsize %s', item.hash, hash, size);
} else if (size !== contract.get('data_size')) {
console.log('Invalid Shard size: Contract Hash %s\tContract size %s\tsize %s', item.hash, contract.get('data_size'), size);
} else {
console.log('Valid Shard: Contract Hash %s', item.hash);
}
rstream.resume();
});
} else {
rstream.resume();
}
});
} else {
rstream.resume();
}
}
});
rstream.on('end', function() {
process.exit();
});
I ran the script and pulled a list of shards that had an age of maximum three hours :
server007.storj.dk:4101 - Valid Shard: Contract Hash 311a563fbbfb74f411325635f75ff8d792b21d5a
server007.storj.dk:4105 - Invalid Shard size: Contract Hash 6cb0a1833642d7eb00a4ad1e7a4495797434aeaa Contract size 4294963200 size 4100
server007.storj.dk:4108 - Valid Shard: Contract Hash 808ffe29fe155028acf15bc8272293602354f630
server007.storj.dk:4108 - Valid Shard: Contract Hash e0cb9c39ce52ff1972d3f2c25745a4c19af13b75
server007.storj.dk:4110 - Valid Shard: Contract Hash 96f2e408f1b89ecb7998a19155b8eb8b5502790e
server008.storj.dk:4103 - Invalid Shard size: Contract Hash a5f3e59c0dd835633a91e485f2be4f6a724a0ac9 Contract size 4294963200 size 4100
server008.storj.dk:4108 - Invalid Shard size: Contract Hash 53b3d0d92605b76614b2e1e52fb87451bf58f93d Contract size 4294963200 size 4100
server008.storj.dk:4108 - Valid Shard: Contract Hash 8ffc983d98b9c9e24bf59a7d941044b4b4a364b2
server009.storj.dk:4106 - Invalid Shard size: Contract Hash a75d0a8565fb25ed436571e83e8b89a8aea67496 Contract size 4294963200 size 4100
server010.storj.dk:4108 - Valid Shard: Contract Hash 622e86220c0fd03a944b919f5bb4bfa8bc338648
server011.storj.dk:4106 - Valid Shard: Contract Hash 8427092e46014c9c36d40fd15d52a514322f574d
server011.storj.dk:4108 - Valid Shard: Contract Hash 8b1cc039b72113c5674ef41e662da5a7cf76ebad
server011.storj.dk:4108 - Invalid Shard size: Contract Hash bb1d690210ee386550c09ad86a0a079fdd36e36e Contract size 4294963200 size 4100
server011.storj.dk:4111 - Valid Shard: Contract Hash 40ad59e5c307ca42a5b7616a7782b33074df392d
server011.storj.dk:4111 - Valid Shard: Contract Hash d117dd5d5bb61b92d3dbe4f526539c8b1823003d
server012.storj.dk:4101 - Valid Shard: Contract Hash dcd3ff03ba7da0769403a96a0339588f23c857b5
server012.storj.dk:4105 - Valid Shard: Contract Hash 786e3f1fe61950c37290f07689b15b7684eaa768
server012.storj.dk:4106 - Invalid Shard size: Contract Hash d379cc0a7faa2f8438f7cb6bcca61ed3b8eb3c94 Contract size 4294963200 size 4100
server012.storj.dk:4107 - Valid Shard: Contract Hash 03efcbd2914c81bfdcec18dc22e8dd9c24684ba3
server013.storj.dk:4103 - Valid Shard: Contract Hash 5cab7e29c4c25639b0576adc428b489162c5cac5
server013.storj.dk:4104 - Valid Shard: Contract Hash 83615bf569de98ce3bebe62ddb7f968bfea6cadd
server013.storj.dk:4105 - Valid Shard: Contract Hash a27b8b9126f5b7cff2741239f622e20296494831
server013.storj.dk:4105 - Valid Shard: Contract Hash b72283245bac96a3009836c39cef59782a0b7051
server013.storj.dk:4106 - Valid Shard: Contract Hash d85944ec1f0da2168ff3e552d36b057e5eba4281
server013.storj.dk:4108 - Valid Shard: Contract Hash 0b0fa951a8ac873ed2c0d239fb0733bb8f193c46
server013.storj.dk:4111 - Valid Shard: Contract Hash 9ff6b74232c8045bef498a0b5107c5d8e466bd05
server013.storj.dk:4112 - Valid Shard: Contract Hash ae637ef8e7f69964e0d05b1cc760cb305464baae
server014.storj.dk:4110 - Valid Shard: Contract Hash 984220ffde99293ab8ba7d4780716d1f01fdc8a4
server014.storj.dk:4111 - Invalid Shard size: Contract Hash c632e715f346622ba29266098b37818e4220a1aa Contract size 4294963200 size 4100
server015.storj.dk:4101 - Valid Shard: Contract Hash 1288a5e699ee9f87da4ea51a47078c91c265774e
server015.storj.dk:4108 - Valid Shard: Contract Hash 0c4be9bef186560bbf7e85185e232c97d2003138
server017.storj.dk:4101 - Valid Shard: Contract Hash 6741ffcdbd243d7c44d69126efbc181bab893e32
server017.storj.dk:4111 - Valid Shard: Contract Hash 6bbde2f58ce6bf955c7f5bcd4c9f8443aa021de5
server018.storj.dk:4101 - Invalid Shard size: Contract Hash 2d6375edde9d872737ff1fa45e35aa13a32e905c Contract size 4294963200 size 4100
server018.storj.dk:4104 - Invalid Shard size: Contract Hash 1495274b319d34ad5dcebae077a29a94ceef017e Contract size 4294963200 size 4100
server018.storj.dk:4105 - Valid Shard: Contract Hash 539b1eb15575cd22ff24a185662315ef1f92c37d
server018.storj.dk:4105 - Valid Shard: Contract Hash 9a47a784a8a75bd700b685f7714152f93b1d3061
server018.storj.dk:4105 - Valid Shard: Contract Hash ce0f62b2e59c742de057f815ec62d3596e4be04e
server018.storj.dk:4108 - Valid Shard: Contract Hash 5e43700e84037539fbbcc71592d69f63fb3d390e
server019.storj.dk:4105 - Invalid Shard size: Contract Hash 8c35f9288cabe4914210478c11182818cdb1c0c4 Contract size 4294963200 size 4100
server020.storj.dk:4102 - Invalid Shard size: Contract Hash 714bd6c314916a033889874397ecf5f4f8bee12c Contract size 4294963200 size 4100
server020.storj.dk:4103 - Valid Shard: Contract Hash 7c681da9dc0ecc822f27e35b889ecbdab221990c
server020.storj.dk:4105 - Valid Shard: Contract Hash 2d7a009a5d8ef9ae2cea04137b62f781d7af04f6
server020.storj.dk:4107 - Valid Shard: Contract Hash 43c28539c5570e08f92e7d7645567fec8e30feb1
server021.storj.dk:4101 - Valid Shard: Contract Hash 7b97f243a4e40ddb15d7a990b0facda718376c84
server021.storj.dk:4102 - Valid Shard: Contract Hash 4f57195d3ad6dc3cdb1200e7582c5a792fa2abf1
server022.storj.dk:4103 - Valid Shard: Contract Hash ed79a4dd172248fe9a2474ceb9437e6749fb2a62
server022.storj.dk:4105 - Valid Shard: Contract Hash 4ea5da4846e69dcd83be7f0805bd4f4b3bb23ce6
server022.storj.dk:4110 - Valid Shard: Contract Hash 9dd27599ea03ce2fe6d655d726bb1fb8ae843bcf
server023.storj.dk:4108 - Valid Shard: Contract Hash 6955ec6e04bedafb71e1202a17f73b6bac056781
server024.storj.dk:4101 - Valid Shard: Contract Hash 39a2c733dcb5b0c2a1faabdc40cf510842e24232
server024.storj.dk:4103 - Valid Shard: Contract Hash efd4c0dd61e9605c59f1dfbf63d48594bf0f15c5
server024.storj.dk:4107 - Valid Shard: Contract Hash 74e7e817f3977be1e67611bc437d15aa0976c995
server025.storj.dk:4108 - Invalid Shard size: Contract Hash 4e4d6d6b16c8599fffbe351fd1352af82de764ec Contract size 4294963200 size 4100
server025.storj.dk:4110 - Valid Shard: Contract Hash 9ecdd5c0f273796764b32024ada9e21e910f88a3
server025.storj.dk:4112 - Invalid Shard size: Contract Hash 64eddfde3c6e0cfc655142437cc16da87861476f Contract size 4294963200 size 4100
server026.storj.dk:4101 - Invalid Shard size: Contract Hash 1c4bd69b684047e7fc9a002448fcee5711f20d06 Contract size 4294963200 size 4100
server026.storj.dk:4101 - Invalid Shard size: Contract Hash 78d37b50c23ac5a6b6c2fe1ad781b47082f8e480 Contract size 4294963200 size 4100
server026.storj.dk:4101 - Invalid Shard size: Contract Hash de593291bbadabbd05862b217f12bc1472ee9154 Contract size 4294963200 size 4100
server026.storj.dk:4108 - Invalid Shard size: Contract Hash 85fcae49805e48033e7cf4398a7492f81712e16e Contract size 4294963200 size 4100
server026.storj.dk:4111 - Valid Shard: Contract Hash 1d307fde91f314c010dd941aaf92af54b61341ad
server026.storj.dk:4111 - Valid Shard: Contract Hash 7e16d7544d9508bf0cde6897534ba56a70cbddb6
server027.storj.dk:4103 - Valid Shard: Contract Hash 3d31d72faf04a82e7c6fe3ea79b55f436f38097e
server027.storj.dk:4110 - Valid Shard: Contract Hash bf0c2bc1306b471bb102d87dba32a0debcf0a907
server027.storj.dk:4110 - Valid Shard: Contract Hash efa5472a9f73d32cf7c18d707bbc6d3016b8ea3f
server028.storj.dk:4101 - Valid Shard: Contract Hash 57d98a7ca79e5fea53ec0de2470fbe0e1a9a4fd7
server029.storj.dk:4105 - Valid Shard: Contract Hash 765ddbca912aec204c1241853e47d9a36b6193c8
server029.storj.dk:4108 - Invalid Shard size: Contract Hash 9ddbc50de7d2883e34535aafdad8d27aeccdf60a Contract size 4294963200 size 4100
server029.storj.dk:4110 - Valid Shard: Contract Hash 71238d70a0d3302c002e51ddb6782aea10ce713e
server030.storj.dk:4101 - Valid Shard: Contract Hash 1469fd779cf0962881bf4b65b8353bd121b2c375
server030.storj.dk:4105 - Valid Shard: Contract Hash 45dc514f01a9d0a6f0850aff56caaaa53203ef53
server030.storj.dk:4106 - Valid Shard: Contract Hash 0a854e474210e857f946e96bb2c6cfd953fc1b48
server030.storj.dk:4112 - Valid Shard: Contract Hash 33ceab260c052aca58f3dfe622553ff09fdfad34
server030.storj.dk:4114 - Valid Shard: Contract Hash 916fbe2e483f5c0cc9c951f6d822e9c9ef57bef5
server030.storj.dk:4115 - Valid Shard: Contract Hash c6cf9cb3cc00a97e694bdefa0fb35f4759b2d59d
server030.storj.dk:4117 - Valid Shard: Contract Hash 810ee69920ad054ec47865bb6a497b5e42882af0
server030.storj.dk:4120 - Valid Shard: Contract Hash 0d3b27e6c707299b2dcf71dc1649065b59f69b17
server030.storj.dk:4122 - Valid Shard: Contract Hash 2079314afc356b9bc506d8f5ca7f19c1542c50e2
server030.storj.dk:4129 - Valid Shard: Contract Hash 8674442d1d258ba41d8345b2ed2dd4705c191e64
server030.storj.dk:4131 - Invalid Shard size: Contract Hash cd41318b9f6117c98a2b751863fa544e8c87db2f Contract size 4294963200 size 4100
server030.storj.dk:4142 - Invalid Shard size: Contract Hash b591f7d8cf1a5eff4689819a60002c33ec5faaca Contract size 4294963200 size 4100
server030.storj.dk:4144 - Valid Shard: Contract Hash 70d8818c58d142a0a45926644955acf5fe16ad65
server030.storj.dk:4148 - Valid Shard: Contract Hash 21dbe419c8c0478b91443931b9e350578430c72f
server030.storj.dk:4149 - Valid Shard: Contract Hash b0453d2fd003408849db2eba44f4c4fda0d6222c
server030.storj.dk:4154 - Valid Shard: Contract Hash b145ed55f0c3fd00e25bf7533fedb799cb0a877b
server030.storj.dk:4154 - Invalid Shard size: Contract Hash d97aa5bed368489eabccbc87b2c85fbbfaa78630 Contract size 4294963200 size 4100
server030.storj.dk:4157 - Valid Shard: Contract Hash 189f5950c7d5144b1933afa89d5edd817cdd7cbc
server030.storj.dk:4166 - Invalid Shard size: Contract Hash 4ea1b79771bbb87efa687551586757e5e7feb4b6 Contract size 4294963200 size 4100
server030.storj.dk:4166 - Valid Shard: Contract Hash 5afc62e066368b491bf0dc710eb7f61c794b4e62
server030.storj.dk:4166 - Valid Shard: Contract Hash 8ba4346f801ad28506a8ab6c3683612a9345747d
server030.storj.dk:4176 - Valid Shard: Contract Hash 739ebb1a0d052ce2980031682a7bd93cfd2b640d
server030.storj.dk:4178 - Valid Shard: Contract Hash 443709ab0221f0b6d0b81683e44455b69c206e8f
server030.storj.dk:4178 - Valid Shard: Contract Hash af82fb36de5eab9841c2e31b39f2d1eb01b4eb1e
server030.storj.dk:4184 - Valid Shard: Contract Hash 3049be69d099e5a3815bfd95231110483aa4caf8
Valid Shard
: 72Invalid Shard size
: 22-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Sending alloc response Shard upload completed Shard download completed Successfully mirrored shard Mirror download completed
Date (UTC) # Bytes # Bytes # Bytes # Bytes # Bytes
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
2018-05-01 20,829 18,085,115,613,773 468 911,452,842 6 5,328,404 6,407 10,613,485,145 5,530 9,392,305,780
2018-05-02 8,319 19,180,152,585,326 137 937,201,889 5 5,242,880 2,067 12,060,843,657 1,733 10,204,215,790
2018-05-03 5,082 48,832,809,574 180 1,194,217,439 3 3,145,728 1,977 12,305,583,861 1,723 11,192,425,087
2018-05-04 9,953 15,337,341,765,092 169 1,452,166,304 6 8,388,608 2,443 1,030,217,807,300 2,079 830,690,771,579
2018-05-05 11,383 35,864,522,026,879 102 1,683,620,208 1 83,762 2,056 2,411,065,618,444 1,705 1,820,512,171,738
2018-05-06 4,895 16,788,766,972,263 30 253,759,070 4 2,097,197 917 1,173,524,803,754 768 949,283,573,439
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Totals : 60,461 105,304,731,772,907 1,086 6,432,417,752 25 24,286,579 15,867 4,649,788,142,161 13,538 3,631,275,463,413
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Mirrored result for today and the last two days are way off, since my script are based on the shards size from the log files, and not the proper contract.get('data_size')
from the db.
👋 Hey! Thanks for this contribution. Apologies for the delay in responding!
We've decided to rearchitect Storj, so that we can scale better. You can read more about this decision here. This means that we are entirely focused on v3 at the moment, in the storj/storj repository. Our white paper for v3 is coming very, very soon - follow along on the blog and in our Rocketchat.
As this repository is part of the v2 network, we're no longer maintaining this repository. I am going to close this for now. If you have any questions, I encourage you to jump on Rocketchat and ask them there. Thanks!
Package Versions
Replace the values below using the output from
npm list storj
. Usenpm list -g storj
if installed globally.Replace the values below using the output from
node --version
.Expected Behavior
Please describe the program's expected behavior. Include an example of your usage code in the back ticks below if applicable.
I would expect these mirrored 4GB shards not to be reported back as successful to the bridge.
Actual Behavior
Please describe the program's actual behavior. Please include any stack traces or log output in the back ticks below.
Someone is uploading a bunch of 4,294,963,200 bytes shards which they are not, since mirroring is done in less than a second but reported as a sucessful 4GB mirror to the bridge.
I believe that this leads to farmes getting a lot
GigaByteHours
, which they don't have.Looking at my test farms see that in the last two days, it went from a few GB of
successfully mirrored shard
andmirror download completed
data per day to over 1TB in just one day :Logfile containing 204 4GB shards from just today log files :
Storj - 204 shards - 4GB fake shards - 05-05-2018.txt
Steps to Reproduce
Please include the steps the reproduce the issue, numbered below. Include as much detail as possible.