search

ZuInnoTe / hadoopcryptoledger

Hadoop Crypto Ledger - Analyzing CryptoLedgers, such as Bitcoin Blockchain, on Big Data platforms, such as Hadoop/Spark/Flink/Hive
Apache License 2.0
141 stars 51 forks source link

Possible data loss when reading transaction values #51

Closed skattoju-zz closed 6 years ago

skattoju-zz commented 6 years ago

My understanding is that the value field of an ethereum transaction can by upto 32 bytes long, however we store this as a long after parsing the RLP encoded data. It seems that we store 0L in case the data is longer than 8 bytes and there is no warning or error thrown when transaction values are discarded / replaced by zeros. Is this the case ?

jornfranke commented 6 years ago

Thank you for raising your concern.

Can you please provide a concrete example when you have seen sucha . transaction? A transaction with a value > the maximum size of an unsigned long?

If the datatype is indeed not a number then it returns 0L; The values are read correct, but numbers larger than the maximum size of an unsigned long are not converted.

We do indeed a conversion to long (and you may need to interpret it as unsigned long using JDK8 functionality), but this has been the case in the past for Ethereum clients. I have seen that they changed recently the data type to a 256 Bit long in the C++ client and to BigInt in the Go Ethereum client (both have different sizes btw.).

One could adapt this indeed just for the case such a transaction may appear. I will look into this and could replace value, gasprice, gaslimit to BigDecimal (without scale), because this one is compatible with all Big Data platforms (Hive/Spark).

ghost commented 6 years ago

value is the value in the atomic unit of the Ethereum currency which is one Wei, 10^18 Wei make one Ether and one Ether is 6200$ USD as of today. The biggest positive number that can be stored in an signed Long is 2^63-1 ~= 9.22 x 10^18 So basically all transactions over 9.22 Ether or 57 000$ are overflowing in Spark... this is a catastrophic data loss, that field becomes meaningless.

ghost commented 6 years ago

@jornfranke

I am working on a school project with @skattoju , we are putting our code in this Github Gist before we make a proper script, but you can see the first table in the Gist, the value becomes negative because of Long overflow.

https://gist.github.com/auth/github/callback?return_to=https://gist.github.com/adriannadiaz/24bd31cd710834c5e5d3d65beee89bab&browser_session_id=ca13624ab35f36fcfc1046ac816c2576b757b95b&code=c5cf99ed39ee0a3bba07&state=b271a8e60719485c0ac4d796255e51d23ce3899287b6566541820612a43bc99d

jornfranke commented 6 years ago

I see I will look into this. It might take a few days to use BigDecimal as the preferred type (including some testing). I hope this aligns still with your timings for the school project.

btw. the negative value has other reasons, because Java/Scale interpret a long by default as signed. You can change this by using corresponding functionality to interpret it as unsigned.

Nevertheless, this issue will be solved when moving to BigDecimal.

skattoju-zz commented 6 years ago

Thanks for the quick response Jörn ! :)

I Initially tried to do it with BigInteger but couldnt get it to work. I tried a rough hack with BigDecimal. Seems to work.. Needs a bit of testing and clean up..

df.printSchema() root |-- ethereumBlockHeader: struct (nullable = true) | |-- parentHash: binary (nullable = true) | |-- uncleHash: binary (nullable = true) | |-- coinBase: binary (nullable = true) | |-- stateRoot: binary (nullable = true) | |-- txTrieRoot: binary (nullable = true) | |-- receiptTrieRoot: binary (nullable = true) | |-- logsBloom: binary (nullable = true) | |-- difficulty: binary (nullable = true) | |-- timestamp: long (nullable = false) | |-- number: long (nullable = false) | |-- gasLimit: long (nullable = false) | |-- gasUsed: long (nullable = false) | |-- mixHash: binary (nullable = true) | |-- extraData: binary (nullable = true) | |-- nonce: binary (nullable = true) |-- ethereumTransactions: array (nullable = true) | |-- element: struct (containsNull = true) | | |-- nonce: binary (nullable = true) | | |-- value: decimal(38,18) (nullable = true) | | |-- receiveAddress: binary (nullable = true) | | |-- gasPrice: decimal(38,18) (nullable = true) | | |-- gasLimit: decimal(38,18) (nullable = true) | | |-- data: binary (nullable = true) | | |-- sig_v: binary (nullable = true) | | |-- sig_r: binary (nullable = true) | | |-- sig_s: binary (nullable = true) |-- uncleHeaders: array (nullable = true) | |-- element: struct (containsNull = true) | | |-- parentHash: binary (nullable = true) | | |-- uncleHash: binary (nullable = true) | | |-- coinBase: binary (nullable = true) | | |-- stateRoot: binary (nullable = true) | | |-- txTrieRoot: binary (nullable = true) | | |-- receiptTrieRoot: binary (nullable = true) | | |-- logsBloom: binary (nullable = true) | | |-- difficulty: binary (nullable = true) | | |-- timestamp: long (nullable = false) | | |-- number: long (nullable = false) | | |-- gasLimit: long (nullable = false) | | |-- gasUsed: long (nullable = false) | | |-- mixHash: binary (nullable = true) | | |-- extraData: binary (nullable = true) | | |-- nonce: binary (nullable = true)

df.select("ethereumBlockHeader.number","ethereumBlockHeader.timestamp","ethereumTransactions").withColumn("elements",explode("ethereumTransactions")).withColumn("reciveAddress",hex(col("elements.receiveAddress"))).withColumn("value",col("elements.value")*10**-18).drop("ethereumTransactions","elements").show()

+-------+----------+--------------------+--------------------+ | number| timestamp| reciveAddress| value| +-------+----------+--------------------+--------------------+ |5032230|1517792040|DD48836498857065B...| 0.095| The value matches the one in https://etherscan.io/tx/0x409de5f6c62750cea79ea0f47ce0aba380d82e3152ff0e7a84a5e24c5731b2d6 |5032230|1517792040|87CECBD9501BC06D3...| 0.06686616000000001| |5032230|1517792040|2CB984EAFA699D19C...| 0.1551519| |5032230|1517792040|CC66EA7EF61A67DF0...| 0.114480012| |5032230|1517792040|F5BEC430576FF1B82...| 0.05316639| |5032230|1517792040|DFF88E49A9899D3BF...|0.037106110000000005| |5032230|1517792040|7B88B19FF11A9508E...| 2.0227570000000004| |5032230|1517792040|70FAA28A6B8D6829A...|0.019795590000000002| |5032230|1517792040|D0A6E6C54DBC68DB5...| 8.032231450000001| |5032230|1517792040|86FA049857E0209AA...| 0.0| |5032230|1517792040|516E5436BAFDC1108...| 0.0| |5032230|1517792040|3883F5E181FCCAF84...| 0.0| |5032230|1517792040|F30B0BDE94D899120...|0.001363199999990...| |5032230|1517792040|AB001E77FF341919F...| 0.49472212000000004| |5032230|1517792040|8D12A197CB00D4747...| 0.0| |5032230|1517792040|04A9B540AB97151C0...| 0.9975170000000001| |5032230|1517792040|5397B342A5333638C...| 0.32| |5032230|1517792040|FA52274DD61E1643D...| 0.6378297300000001| |5032230|1517792040|FA52274DD61E1643D...| 1.50197410294| |5032230|1517792040|A291B96D25B9416C5...| 0.096| +-------+----------+--------------------+--------------------+ only showing top 20 rows

df.select("ethereumBlockHeader.number","ethereumBlockHeader.timestamp","ethereumTransactions").withColumn("elements",explode("ethereumTransactions")).withColumn("reciveAddress",hex(col("elements.receiveAddress"))).withColumn("value",col("elements.value")*10**-18).drop("ethereumTransactions","elements"). orderBy("value",ascending=True).show() +-------+----------+--------------------+-----+ | number| timestamp| reciveAddress|value| +-------+----------+--------------------+-----+ |5032237|1517792166|D0A6E6C54DBC68DB5...| null| The null value transactions seem to be done by Smart Contracts: https://etherscan.io/address/0xD0A6E6C54DBC68DB5DB3A091B171A77407FF7CCF |5032254|1517792330|D0A6E6C54DBC68DB5...| null| |5032237|1517792166|D0A6E6C54DBC68DB5...| null| |5032237|1517792166|D0A6E6C54DBC68DB5...| null| |5032237|1517792166|D0A6E6C54DBC68DB5...| null| |5032235|1517792110|D0A6E6C54DBC68DB5...| null| |5032254|1517792330|D0A6E6C54DBC68DB5...| null| |5032238|1517792180|550C6DE28D89D876C...| null| |5032235|1517792110|D0DA70AA64290B5AF...| null| |5032235|1517792110|D0A6E6C54DBC68DB5...| null| |5032235|1517792110|D0A6E6C54DBC68DB5...| null| |5032235|1517792110|644881659CE654F4E...| null| |5032235|1517792110|8E600825F053981D2...| null| |5032235|1517792110|AB14E8269DE8C9FF9...| null| |5032235|1517792110|907534E881EFA2470...| null| |5032236|1517792132|D0A6E6C54DBC68DB5...| null| |5032239|1517792198|E2481DC24FF407FBC...| null| |5032236|1517792132|C57FFB2D78F6F3E50...| null| |5032236|1517792132|D0A6E6C54DBC68DB5...| null| |5032246|1517792246|380D179942CE82DC2...| null| +-------+----------+--------------------+-----+ only showing top 20 rows

df.select("ethereumBlockHeader.number","ethereumBlockHeader.timestamp","ethereumTransactions").withColumn("elements",explode("ethereumTransactions")).withColumn("reciveAddress",hex(col("elements.receiveAddress"))).withColumn("value",col("elements.value")*10**-18).drop("ethereumTransactions","elements"). orderBy("value",ascending=False).show() +-------+----------+--------------------+-----------------+ | number| timestamp| reciveAddress| value| +-------+----------+--------------------+-----------------+ |5033063|1517804067|CE85247B032F7528B...|99.99978391600001| The max value in our subsample seems to be ~100 It approximately matches whats in https://etherscan.io/tx/0x0f0f24e2f839aa6b58bf98dbf2479be92e355cd984be47ad8e468fb730705775 |5033539|1517810437|3F5CE5FBFE3E9AF39...|99.99958000000001| |5032577|1517796827|418F3A0A79579276D...|99.99958000000001| |5033742|1517813385|FA74AD0BD9D61A307...|99.99958000000001| |5034188|1517819415|3F5CE5FBFE3E9AF39...| 99.999559| |5033419|1517808651|3F5CE5FBFE3E9AF39...| 99.999517| |5033457|1517809221|D19141A9B74F7A02C...| 99.999412| |5032823|1517800676|DE7286BD7E0B25FB9...| 99.999412| |5033129|1517804850|F726DC178D1A4D929...| 99.999412| |5032425|1517794686|EB574CD5A407FEFA5...| 99.999412| |5032857|1517801077|F5613E4DA78CEE6A1...| 99.999412| |5032871|1517801376|589127BDEFB29BC3F...| 99.999412| |5034049|1517817923|FA52274DD61E1643D...|99.99940640000001| |5032945|1517802237|876EABF441B2EE5B5...|99.99936988000002| |5032878|1517801475|876EABF441B2EE5B5...|99.99936988000002| |5033478|1517809454|65D647B02250AF9DD...| 99.99916| |5032381|1517794150|0AFEF24D425242219...| 99.99916| |5033556|1517810672|311BA850A46306FE5...| 99.99916| |5032623|1517797518|EE3BEF63FB30DC1F2...| 99.99916| |5033904|1517815831|3F5CE5FBFE3E9AF39...| 99.999034| +-------+----------+--------------------+-----------------+ only showing top 20 rows

On Sun, Mar 11, 2018 at 2:35 PM, Jörn Franke notifications@github.com wrote:

I see I will look into this. It might take a few days to use BigDecimal as the preferred type (including some testing). I hope this aligns still with your timings for the school project.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ZuInnoTe/hadoopcryptoledger/issues/51#issuecomment-372137845, or mute the thread https://github.com/notifications/unsubscribe-auth/AAJWbu9Tq2a6K7pUODIwwIG2J66PvYzoks5tdW5kgaJpZM4SlxdP .

jornfranke commented 6 years ago

Well JDK8 has functionality to interpret a long as unsigned.

Nevertheless a full fix will come later, maybe this week.

Keep in mind that Spark SQL only supports maximum a decimal with 38 precision, ie it will never support larger numbers. This is somehow strange because BigDecimal supports much more.

I will make also the raw byte sequences representing the full value, ie not limited by this, available.

You may need to interpret those or switch to standard Spark if you have values don’t fitting into a decimal 38,0.

On 12. Mar 2018, at 04:13, Siddhartha Kattoju notifications@github.com wrote:

Thanks for the quick response Jörn ! :)

I Initially tried to do it with BigInteger but couldnt get it to work. I tried a rough hack with BigDecimal. Seems to work.. Needs a bit of testing and clean up..

df.printSchema() root |-- ethereumBlockHeader: struct (nullable = true) | |-- parentHash: binary (nullable = true) | |-- uncleHash: binary (nullable = true) | |-- coinBase: binary (nullable = true) | |-- stateRoot: binary (nullable = true) | |-- txTrieRoot: binary (nullable = true) | |-- receiptTrieRoot: binary (nullable = true) | |-- logsBloom: binary (nullable = true) | |-- difficulty: binary (nullable = true) | |-- timestamp: long (nullable = false) | |-- number: long (nullable = false) | |-- gasLimit: long (nullable = false) | |-- gasUsed: long (nullable = false) | |-- mixHash: binary (nullable = true) | |-- extraData: binary (nullable = true) | |-- nonce: binary (nullable = true) |-- ethereumTransactions: array (nullable = true) | |-- element: struct (containsNull = true) | | |-- nonce: binary (nullable = true) | | |-- value: decimal(38,18) (nullable = true) | | |-- receiveAddress: binary (nullable = true) | | |-- gasPrice: decimal(38,18) (nullable = true) | | |-- gasLimit: decimal(38,18) (nullable = true) | | |-- data: binary (nullable = true) | | |-- sig_v: binary (nullable = true) | | |-- sig_r: binary (nullable = true) | | |-- sig_s: binary (nullable = true) |-- uncleHeaders: array (nullable = true) | |-- element: struct (containsNull = true) | | |-- parentHash: binary (nullable = true) | | |-- uncleHash: binary (nullable = true) | | |-- coinBase: binary (nullable = true) | | |-- stateRoot: binary (nullable = true) | | |-- txTrieRoot: binary (nullable = true) | | |-- receiptTrieRoot: binary (nullable = true) | | |-- logsBloom: binary (nullable = true) | | |-- difficulty: binary (nullable = true) | | |-- timestamp: long (nullable = false) | | |-- number: long (nullable = false) | | |-- gasLimit: long (nullable = false) | | |-- gasUsed: long (nullable = false) | | |-- mixHash: binary (nullable = true) | | |-- extraData: binary (nullable = true) | | |-- nonce: binary (nullable = true)

df.select("ethereumBlockHeader.number","ethereumBlockHeader.timestamp","ethereumTransactions").withColumn("elements",explode("ethereumTransactions")).withColumn("reciveAddress",hex(col("elements.receiveAddress"))).withColumn("value",col("elements.value")*10**-18).drop("ethereumTransactions","elements").show()

+-------+----------+--------------------+--------------------+ | number| timestamp| reciveAddress| value| +-------+----------+--------------------+--------------------+ |5032230|1517792040|DD48836498857065B...| 0.095| The value matches the one in https://etherscan.io/tx/0x409de5f6c62750cea79ea0f47ce0aba380d82e3152ff0e7a84a5e24c5731b2d6 |5032230|1517792040|87CECBD9501BC06D3...| 0.06686616000000001| |5032230|1517792040|2CB984EAFA699D19C...| 0.1551519| |5032230|1517792040|CC66EA7EF61A67DF0...| 0.114480012| |5032230|1517792040|F5BEC430576FF1B82...| 0.05316639| |5032230|1517792040|DFF88E49A9899D3BF...|0.037106110000000005| |5032230|1517792040|7B88B19FF11A9508E...| 2.0227570000000004| |5032230|1517792040|70FAA28A6B8D6829A...|0.019795590000000002| |5032230|1517792040|D0A6E6C54DBC68DB5...| 8.032231450000001| |5032230|1517792040|86FA049857E0209AA...| 0.0| |5032230|1517792040|516E5436BAFDC1108...| 0.0| |5032230|1517792040|3883F5E181FCCAF84...| 0.0| |5032230|1517792040|F30B0BDE94D899120...|0.001363199999990...| |5032230|1517792040|AB001E77FF341919F...| 0.49472212000000004| |5032230|1517792040|8D12A197CB00D4747...| 0.0| |5032230|1517792040|04A9B540AB97151C0...| 0.9975170000000001| |5032230|1517792040|5397B342A5333638C...| 0.32| |5032230|1517792040|FA52274DD61E1643D...| 0.6378297300000001| |5032230|1517792040|FA52274DD61E1643D...| 1.50197410294| |5032230|1517792040|A291B96D25B9416C5...| 0.096| +-------+----------+--------------------+--------------------+ only showing top 20 rows

df.select("ethereumBlockHeader.number","ethereumBlockHeader.timestamp","ethereumTransactions").withColumn("elements",explode("ethereumTransactions")).withColumn("reciveAddress",hex(col("elements.receiveAddress"))).withColumn("value",col("elements.value")*10**-18).drop("ethereumTransactions","elements"). orderBy("value",ascending=True).show() +-------+----------+--------------------+-----+ | number| timestamp| reciveAddress|value| +-------+----------+--------------------+-----+ |5032237|1517792166|D0A6E6C54DBC68DB5...| null| The null value transactions seem to be done by Smart Contracts: https://etherscan.io/address/0xD0A6E6C54DBC68DB5DB3A091B171A77407FF7CCF |5032254|1517792330|D0A6E6C54DBC68DB5...| null| |5032237|1517792166|D0A6E6C54DBC68DB5...| null| |5032237|1517792166|D0A6E6C54DBC68DB5...| null| |5032237|1517792166|D0A6E6C54DBC68DB5...| null| |5032235|1517792110|D0A6E6C54DBC68DB5...| null| |5032254|1517792330|D0A6E6C54DBC68DB5...| null| |5032238|1517792180|550C6DE28D89D876C...| null| |5032235|1517792110|D0DA70AA64290B5AF...| null| |5032235|1517792110|D0A6E6C54DBC68DB5...| null| |5032235|1517792110|D0A6E6C54DBC68DB5...| null| |5032235|1517792110|644881659CE654F4E...| null| |5032235|1517792110|8E600825F053981D2...| null| |5032235|1517792110|AB14E8269DE8C9FF9...| null| |5032235|1517792110|907534E881EFA2470...| null| |5032236|1517792132|D0A6E6C54DBC68DB5...| null| |5032239|1517792198|E2481DC24FF407FBC...| null| |5032236|1517792132|C57FFB2D78F6F3E50...| null| |5032236|1517792132|D0A6E6C54DBC68DB5...| null| |5032246|1517792246|380D179942CE82DC2...| null| +-------+----------+--------------------+-----+ only showing top 20 rows

df.select("ethereumBlockHeader.number","ethereumBlockHeader.timestamp","ethereumTransactions").withColumn("elements",explode("ethereumTransactions")).withColumn("reciveAddress",hex(col("elements.receiveAddress"))).withColumn("value",col("elements.value")*10**-18).drop("ethereumTransactions","elements"). orderBy("value",ascending=False).show() +-------+----------+--------------------+-----------------+ | number| timestamp| reciveAddress| value| +-------+----------+--------------------+-----------------+ |5033063|1517804067|CE85247B032F7528B...|99.99978391600001| The max value in our subsample seems to be ~100 It approximately matches whats in https://etherscan.io/tx/0x0f0f24e2f839aa6b58bf98dbf2479be92e355cd984be47ad8e468fb730705775 |5033539|1517810437|3F5CE5FBFE3E9AF39...|99.99958000000001| |5032577|1517796827|418F3A0A79579276D...|99.99958000000001| |5033742|1517813385|FA74AD0BD9D61A307...|99.99958000000001| |5034188|1517819415|3F5CE5FBFE3E9AF39...| 99.999559| |5033419|1517808651|3F5CE5FBFE3E9AF39...| 99.999517| |5033457|1517809221|D19141A9B74F7A02C...| 99.999412| |5032823|1517800676|DE7286BD7E0B25FB9...| 99.999412| |5033129|1517804850|F726DC178D1A4D929...| 99.999412| |5032425|1517794686|EB574CD5A407FEFA5...| 99.999412| |5032857|1517801077|F5613E4DA78CEE6A1...| 99.999412| |5032871|1517801376|589127BDEFB29BC3F...| 99.999412| |5034049|1517817923|FA52274DD61E1643D...|99.99940640000001| |5032945|1517802237|876EABF441B2EE5B5...|99.99936988000002| |5032878|1517801475|876EABF441B2EE5B5...|99.99936988000002| |5033478|1517809454|65D647B02250AF9DD...| 99.99916| |5032381|1517794150|0AFEF24D425242219...| 99.99916| |5033556|1517810672|311BA850A46306FE5...| 99.99916| |5032623|1517797518|EE3BEF63FB30DC1F2...| 99.99916| |5033904|1517815831|3F5CE5FBFE3E9AF39...| 99.999034| +-------+----------+--------------------+-----------------+ only showing top 20 rows

On Sun, Mar 11, 2018 at 2:35 PM, Jörn Franke notifications@github.com wrote:

I see I will look into this. It might take a few days to use BigDecimal as the preferred type (including some testing). I hope this aligns still with your timings for the school project.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ZuInnoTe/hadoopcryptoledger/issues/51#issuecomment-372137845, or mute the thread https://github.com/notifications/unsubscribe-auth/AAJWbu9Tq2a6K7pUODIwwIG2J66PvYzoks5tdW5kgaJpZM4SlxdP .

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

jornfranke commented 6 years ago

I confirm that support for large numbers will be released this week. The core library, the Flink datasource and Spark RDDs support nearly unlimited size values. Spark SQL and Hive are limited by Decimal (38,0).

jornfranke commented 6 years ago

i iwll publish it later today, tests nearly finalized (Hive was pending)

skattoju-zz commented 6 years ago

Thanks for the quick update. Just tested with the latest code. Looks good so far, no more nulls.

df.select("ethereumBlockHeader.number","ethereumBlockHeader.timestamp","ethereumTransactions").withColumn("elements",explode("ethereumTransactions")).withColumn("reciveAddress",hex(col("elements.receiveAddress"))).withColumn("value",col("elements.value")*10**-18).drop("ethereumTransactions","elements").orderBy("value",ascending=False).show() +-------+----------+--------------------+------------------+ | number| timestamp| reciveAddress| value| +-------+----------+--------------------+------------------+ |5032960|1517802440|C3590C15CD253E93A...| 5901.279158350001| |5032901|1517801832|C3590C15CD253E93A...| 5000.090000000001| |5033266|1517806590|390DE26D772D2E200...| 4499.994000000001| |5033514|1517810129|9604B579F60EAAF54...| 4388.712249| |5033684|1517812458|71FB72D4432B11F85...| 4300.0| |5032721|1517799013|876EABF441B2EE5B5...|3611.1014198400003| |5034034|1517817754|9E5EDF5A297DDD664...|3260.0000000000005| |5033961|1517816698|6546A1FCE138B8AB2...| 3100.0| |5033526|1517810262|E1962B47127AED263...| 3000.0| |5032610|1517797352|22B84D5FFEA8B801C...|2999.9970000000003| |5032610|1517797352|22B84D5FFEA8B801C...|2999.9970000000003| |5032610|1517797352|22B84D5FFEA8B801C...|2999.9970000000003| |5032610|1517797352|22B84D5FFEA8B801C...|2999.9970000000003| |5032610|1517797352|22B84D5FFEA8B801C...|2999.9970000000003| |5032610|1517797352|22B84D5FFEA8B801C...|2999.9970000000003| |5032610|1517797352|22B84D5FFEA8B801C...|2999.9970000000003| |5032610|1517797352|22B84D5FFEA8B801C...|2999.9970000000003| |5032610|1517797352|22B84D5FFEA8B801C...|2999.9970000000003| |5032610|1517797352|22B84D5FFEA8B801C...|2999.9970000000003| |5033719|1517812986|AFBA1B04EC9DCBDEA...| 2898.0| +-------+----------+--------------------+------------------+ only showing top 20 rows

df.select("ethereumBlockHeader.number","ethereumBlockHeader.timestamp","ethereumTransactions").withColumn("elements",explode("ethereumTransactions")).withColumn("reciveAddress",hex(col("elements.receiveAddress"))).withColumn("value",col("elements.value")*10**-18).drop("ethereumTransactions","elements").orderBy("value",ascending=True).show() +-------+----------+--------------------+-----+ | number| timestamp| reciveAddress|value| +-------+----------+--------------------+-----+ |5032231|1517792054|85E076361CC813A90...| 0.0| |5032230|1517792040|81C9151DE0C8BAFCD...| 0.0| |5032230|1517792040|68D57C9A1C35F63E2...| 0.0| |5032230|1517792040|AE84013BB37EF96AB...| 0.0| |5032230|1517792040|E41D2489571D32218...| 0.0| |5032231|1517792054|0BB217E40F8A5CB79...| 0.0| |5032230|1517792040|13F25CD52B21650CA...| 0.0| |5032230|1517792040|3883F5E181FCCAF84...| 0.0| |5032230|1517792040|448A5065AEBB8E423...| 0.0| |5032230|1517792040|8D12A197CB00D4747...| 0.0| |5032230|1517792040|9992EC3CF6A55B009...| 0.0| |5032230|1517792040|58D0A58E4B165A27E...| 0.0| |5032230|1517792040|8D12A197CB00D4747...| 0.0| |5032230|1517792040|86FA049857E0209AA...| 0.0| |5032230|1517792040|41E5560054824EA6B...| 0.0| |5032230|1517792040|516E5436BAFDC1108...| 0.0| |5032231|1517792054|86FA049857E0209AA...| 0.0| |5032231|1517792054|744D70FDBE2BA4CF9...| 0.0| |5032230|1517792040|8D12A197CB00D4747...| 0.0| |5032230|1517792040|AE84013BB37EF96AB...| 0.0| +-------+----------+--------------------+-----+ only showing top 20 rows

On Tue, Mar 13, 2018 at 7:48 PM, Jörn Franke notifications@github.com wrote:

i iwll publish it later today, tests nearly finalized (Hive was pending)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ZuInnoTe/hadoopcryptoledger/issues/51#issuecomment-372857379, or mute the thread https://github.com/notifications/unsubscribe-auth/AAJWbuIJs1CqaR3PHqUr52jzQxTv3DPcks5teFqxgaJpZM4SlxdP .

jornfranke commented 6 years ago

should be solved with latest release. Late me know (or do it yourself) if this issue can be closed.

skattoju-zz commented 6 years ago

Issue resolved.