search

ZuInnoTe / hadoopcryptoledger

Hadoop Crypto Ledger - Analyzing CryptoLedgers, such as Bitcoin Blockchain, on Big Data platforms, such as Hadoop/Spark/Flink/Hive
Apache License 2.0
141 stars 51 forks source link

Adding looped read on listHeader for ethereum reader #38

Closed liorregev closed 6 years ago

liorregev commented 6 years ago

The EthereumBlockReader is read the first 10 bytes in a block being the listHeader variable. (line 193) Since read operations may return less than the maximal amount allowed even if there is data available, when reading from a remote location (in my tests, running on EMR and S3) the call fails, thinking there is not enough data for a list header. Reading in a loop until reaching 10 bytes or recieving a -1 return code solves this issue

jornfranke commented 6 years ago

Will fix it this week and also check bitcoin .. for the reading of blocks itself it is already a loop.

Thank you a lot.

On 27. Nov 2017, at 10:57, Lior Regev notifications@github.com wrote:

The EthereumBlockReader is read the first 10 bytes in a block being the listHeader variable. (line 193) Since read operations may return less than the maximal amount allowed even if there is data available, when reading from a remote location (in my tests, running on EMR and S3) the call fails, thinking there is not enough data for a list header. Reading in a loop until reaching 10 bytes or recieving a -1 return code solves this issue

You can view, comment on, or merge this pull request online at:

https://github.com/ZuInnoTe/hadoopcryptoledger/pull/38

Commit Summary

Added looped read for listHeader in case the read operation returned less bytes than expected but more are still available File Changes

M inputformat/src/main/java/org/zuinnote/hadoop/ethereum/format/common/EthereumBlockReader.java (18) Patch Links:

https://github.com/ZuInnoTe/hadoopcryptoledger/pull/38.patch https://github.com/ZuInnoTe/hadoopcryptoledger/pull/38.diff — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

jornfranke commented 6 years ago

ok, since the unit tests are passing according to Travis and no new unit tests are required (no new functionality), i will add it now. However, before i publish i will check also the Bitcoin reader, because there is a similar routine to check for the first magics of the blocks. Nevertheless i expect to release this as 1.1.2 this week, since the normal block reading functionality did it already correctly.

thanks again for pointing this out.

jornfranke commented 6 years ago

just for the future, please add an issue before (i did it for you: https://github.com/ZuInnoTe/hadoopcryptoledger/issues/39) and see also contributing (https://github.com/ZuInnoTe/hadoopcryptoledger/blob/master/CONTRIBUTING.md), but for now what you did is fine. thank you again.

liorregev commented 6 years ago

Oh, right right, sorry. Will read the contribution guide before further contributions. Happy to help