search

tevador / RandomJS

Cryptocurrency proof-of-work algorithm based on random javascript execution
GNU General Public License v3.0
41 stars 11 forks source link

Reproducibility across different platforms #3

Open tevador opened 6 years ago

tevador commented 6 years ago

I added a new test project which can be used for testing results on different platforms.

If you want to test it on your platform:

  1. Start the sandbox node sandbox.js
  2. Run command ./Tevador.RandomJS.Test.exe --threads 2 --seed 123 --count 1000

The number of threads can be adjusted according to the number of cores of your processor. The command generates 1000 random programs and calculates a cumulative hash of all the outputs.

So far I have tested:

In all cases the result is 3cc26d6be7f5f542e8037247deec69c2e3a3d70db67dac5371f86e06fbe3c8cc.

(Valid as of commit 79d3c4c952c282d58edc9acaab7ec97cc3656b57)

ghost commented 6 years ago

./Tevador.RandomJS.Test.exe --threads 2 --seed 123 --count 1000 Collecting statistics from 1000 random program executions (seed = 123) 5 %, 10 %, 15 %, 20 %, 25 %, 30 %, 35 %, 40 %, 45 %, 50 %, 55 %, 60 %, 65 %, 70 %, 75 %, 80 %, 85 %, 90 %, 95 %, Completed in 8.4801327 seconds Runtime histogram: Longest runtimes: Seed = 92117886952e3f3c5010956afba334430a20eaa983b4ee5f6c8647e5010cf6b4, Runtime = 0.05701 s Seed = f767b319d74bdb44329dfeb4072b558bb89bbcd2dadef61b863f06f4c0f84365, Runtime = 0.05645 s Seed = fb0140e24f67a86b8d5da25870f056264d210a1b8a8fa678f86edfe0d17904f4, Runtime = 0.05559 s Seed = 0e1fd5a6d18e5cc2d0fa9cfb781be7f33f4f11a00c0a7b1085142192291d9bfc, Runtime = 0.05529 s Seed = 3a465dcc93206126d4c68d24b8d0854e95fd4d1be6ce3fb18583792997fb0602, Runtime = 0.04514 s Seed = d6e3a869b2518a2e5076cd498b82bb76297779cd9f1bf3121e0f2a74a37fd90c, Runtime = 0.04164 s Seed = beed8f13cc2af4ab76ec0a73c9b286a6c499571d8f554983d5d6fd46bbb654a2, Runtime = 0.04049 s Seed = 39b432618eb3fe0e374033e5bd24e28abc78a6f55fa576fe4a01c03fff5a8ff5, Runtime = 0.03823 s Seed = 46509a1b7306a0929fbfed991ac472499a5fb1c569d4e3ced8863c23eee61eb2, Runtime = 0.03597 s Seed = 73b8ec5cd13343519cee1ef7662121c961f778ed925c8d37665e0b6b2a68f41a, Runtime = 0.03493 s Runtime [s] Min: 0.00167; Max: 0.05701; Avg: 0.00771; Stdev: 0.00640; Runtime [s] 99.99th percentile: 0.0570068010129035 Average entropy of program output (est.): 1242.39 bits Cyclomatic complexity Min: 0; Max: 0; Avg: 0; Stdev: 0; Halstead difficulty Min: 0; Max: 0; Avg: 0; Stdev: 0; 0.00167 17 0.00231 60 0.00295 101 0.00359 106 0.00423 97 0.00487 90 0.00551 70 0.00615 62 0.00679 52 0.00743 42 0.00807 44 0.00871 27 0.00935 27 0.00999 20 0.01063 22 0.01127 22 0.01191 15 0.01255 10 0.01319 8 0.01382 12 0.01446 10 0.01510 6 0.01574 3 0.01638 9 0.01702 8 0.01766 4 0.01830 2 0.01894 5 0.01958 5 0.02022 3 0.02086 3 0.02150 4 0.02214 5 0.02278 1 0.02342 1 0.02406 2 0.02470 0 0.02534 0 0.02598 2 0.02662 4 0.02726 0 0.02790 1 0.02854 1 0.02918 0 0.02981 0 0.03045 3 0.03109 1 0.03173 0 0.03237 1 0.03301 0 0.03365 2 0.03429 1 0.03493 0 0.03557 1 0.03621 0 0.03685 0 0.03749 0 0.03813 1 0.03877 0 0.03941 0 0.04005 1 0.04069 0 0.04133 1 0.04197 0 0.04261 0 0.04325 0 0.04389 0 0.04453 1 0.04517 0 0.04580 0 0.04644 0 0.04708 0 0.04772 0 0.04836 0 0.04900 0 0.04964 0 0.05028 0 0.05092 0 0.05156 0 0.05220 0 0.05284 0 0.05348 0 0.05412 0 0.05476 1 0.05540 1 0.05604 1 0.05668 1 Cumulative output hash: 3cc26d6be7f5f542e8037247deec69c2e3a3d70db67dac5371f86e06fbe3c8cc

tevador commented 6 years ago

@wowario Your first result was different. Any idea why?

ghost commented 6 years ago

Yes, it was. The first test ran on an older version of ProgramOptions.xml with different weight values, I re-cloned the repo and ran the test a second time to get the correct result.

jwinterm commented 6 years ago

Tested with Intel(R) Core(TM) i5-6600K CPU @ 3.50GHz Ubuntu 18.04 x86_64 commit 79d3c4c952c282d58edc9acaab7ec97cc3656b57

Cumulative output hash: 3cc26d6be7f5f542e8037247deec69c2e3a3d70db67dac5371f86e06fbe3c8cc

tevador commented 6 years ago

We should test an x86 (32bit) platform (not that I'm expecting anyone to mine on such machine). Also more AMD CPUs (they can give different results for transcendental math functions).

jwinterm commented 6 years ago

I have a couple oldish AMD boxes around, don't think I have any x86s though. Will try to test on AMDs in next couple days.

jwinterm commented 6 years ago

Tested on - AMD Athlon(tm) II X2 220 Processor Ubuntu 16.04 x86_64 commit: 8923b3b04362784828f55e9481a0111d1ce1d6cb

got: Cumulative output hash: 3cc26d6be7f5f542e8037247deec69c2e3a3d70db67dac5371f86e06fbe3c8cc

bomb-on commented 6 years ago

Tested on: OS X 10.13.5 Intel Core i5 2.7GHz commit: 8923b3b

Results:

$ mono Tevador.RandomJS.Test.exe --threads 2 --seed 123 --count 1000

Collecting statistics from 1000 random program executions (seed = 123)
5 %, 10 %, 15 %, 20 %, 25 %, 30 %, 35 %, 40 %, 45 %, 50 %, 55 %, 60 %, 65 %, 70 %, 75 %, 80 %, 85 %, 90 %, 95 %,
Completed in 25.1238964 seconds
Longest runtimes:
Seed = 0e1fd5a6d18e5cc2d0fa9cfb781be7f33f4f11a00c0a7b1085142192291d9bfc, Runtime = 0.38644 s
Seed = 46509a1b7306a0929fbfed991ac472499a5fb1c569d4e3ced8863c23eee61eb2, Runtime = 0.25880 s
Seed = 39b432618eb3fe0e374033e5bd24e28abc78a6f55fa576fe4a01c03fff5a8ff5, Runtime = 0.22791 s
Seed = 2822106c992d76c830b35009fa847da93520c507e5190a48f5357a11a607356b, Runtime = 0.19583 s
Seed = 92117886952e3f3c5010956afba334430a20eaa983b4ee5f6c8647e5010cf6b4, Runtime = 0.16995 s
Seed = d8fe9d7d7855c0d2c6a81564efe881b147cd17d35410adfc609f6ef81d4f9c57, Runtime = 0.16821 s
Seed = fb0140e24f67a86b8d5da25870f056264d210a1b8a8fa678f86edfe0d17904f4, Runtime = 0.15999 s
Seed = e53b00a5e8db6f7295a5de8883ea7ead3600067f8f729882c25bc3f53efa79ac, Runtime = 0.15155 s
Seed = 42f0b1623c4650d47f8348bbeee27edcf9ed5ab50f3d783b2cecc7ffa8be7d8d, Runtime = 0.14507 s
Seed = 7fb4dbf4c3b50b4317176ca78a4d1478084d7e3ff994229ccd373d8ab9e7c54a, Runtime = 0.13481 s
Runtime [s] Min: 0.00323; Max: 0.38644; Avg: 0.02417; Stdev: 0.02796;
Runtime [s] 99.99th percentile: 0.386439353023889
Average entropy of program output (est.): 1242.39 bits
Cyclomatic complexity Min: 0; Max: 0; Avg: 0; Stdev: 0;
Lines of code Min: 0; Max: 0; Avg: 0; Stdev: 0;
Halstead difficulty Min: 0; Max: 0; Avg: 0; Stdev: 0;
Cumulative output hash: 3cc26d6be7f5f542e8037247deec69c2e3a3d70db67dac5371f86e06fbe3c8cc
tevador commented 6 years ago

Note that as of commit d32fd7759baf56e897f8f406e4ef2ce17142516a, the cumulative hash is different because the generator has been slightly modified.

The new hash is:

$ ./Tevador.RandomJS.Test.exe --count 1000 --seed 123 --threads 2
Collecting statistics from 1000 random program executions (seed = 123)
...
Cumulative output hash: 80d77dae86b3b49bfb4cf53281f10cb5db1d1e857d9adb7429bf116a9d7880ba

If you do git pull, you have to manually update your ProgramOptions.xml (copy from src/Tevador.RandomJS/ProgramOptions.xml to bin).

tevador commented 6 years ago

I decided to remove all transcendental functions (exp, log, sin, cos, cbrt, atan) from the generator due to poor cross-platform compatibility (rounding is not enough).

With these functions enabled, the results don't match between Windows and Linux with the XS engine.

The reason why the results match for the V8 is because V8 has its own software implementation of all these functions using only IEEE-754 safe operations. We could copy this implementation to the XS, but I don't see the point in that because an ASIC could easily include a faster on-chip implementation.

The only function that stays is Math.sqrt which is portable and implemented in hardware in most CPUs.

For testing of floating point math compatibility, please use the following command:

Tevador.RandomJS.Test.exe --count 1000 --seed 123 --threads 2 --customOptions --XML_AssignmentOperators_Add 1 --XML_AssignmentOperators_Sub 1 --XML_AssignmentOperators_Mul 1 --XML_AssignmentOperators_Div 1 --XML_AssignmentOperators_Mod 1 --XML_AssignmentOperators_PreInc 1 --XML_AssignmentOperators_PostDec 1 --XML_AssignmentOperators_PreDec 1 --XML_AssignmentOperators_PostInc 1 --XML_UnaryOperators_Sqrt 1 --XML_BinaryOperators_Add 1 --XML_BinaryOperators_Sub 1 --XML_BinaryOperators_Mul 1 --XML_BinaryOperators_Div 1 --XML_BinaryOperators_Mod 1 --XML_Literals_Numeric 1 --XML_NumericLiterals_DecimalInteger 3 --XML_NumericLiterals_BinaryInteger 1 --XML_NumericLiterals_OctalInteger 3 --XML_NumericLiterals_SmallInteger 3 --XML_NumericLiterals_HexInteger 3 --XML_NumericLiterals_FixedFloat 3 --XML_NumericLiterals_ExpFloat 3 --XML_Expressions_AssignmentExpression 4 --XML_Expressions_BinaryExpression 5 --XML_Expressions_Literal 2 --XML_Expressions_UnaryExpression 5 --XML_Statements_AssignmentStatement 1 --XML_AssignmentInForLoop_Add 1 --XML_GlobalVariablesCountRange_Min 100 --XML_GlobalVariablesCountRange_Span 0 --XML_VariableInitializerDepth 4 --XML_MaxExpressionDepth 4 --XML_ConstVariableChance 0.1 --XML_MaxStringLengthRange_Min 35 --XML_MaxStringLengthRange_Span 0

It generates programs with only math expressions and doesn't depend on ProgramOptions.xml.

Output with commit e13675695ede93b38b706363a374d012af2bb250 should be:

Cumulative output hash: b83a20eaa1d1615b860dd7477dc8945d23b84cd9082a7240df600b347410441c

It is now possible to also test with the XS engine. You have to clone and compile it from here: https://github.com/tevador/moddable

To use this engine, append the following command line option: --runnerUri file://PATH-TO-XST-EXECUTABLE, for example:

--runnerUri file:///home/tevador/moddable/build/bin/lin/release/xst
--runnerUri file://C:/projects/moddable/build/bin/win/release/xst.exe

For the NodeJS sandbox, you can use (it's the default option):

--runnerUri http://localhost:18111
ghost commented 6 years ago 
mono Tevador.RandomJS.Test.exe --count 1000 --seed 123 --threads 2 --customOptions --XML_AssignmentOperators_Add 1 --XML_AssignmentOperators_Sub 1 ..

..

Cumulative output hash: b83a20eaa1d1615b860dd7477dc8945d23b84cd9082a7240df600b347410441c
tevador commented 6 years ago

@fuwa0529 b83a20eaa1d1615b860dd7477dc8945d23b84cd9082a7240df600b347410441c is the correct hash for that commit.

I have also tested the latest commit from the dev branch on Intel Kaby Lake (Windows), AMD Ryzen and Raspberry Pi 3 (Ubuntu) and all give the same results.

I think at this point we can conclude that reproducibility is not an issue.

SChernykh commented 6 years ago

I have also tested the latest commit from the dev branch on Intel Kaby Lake (Windows), AMD Ryzen and Raspberry Pi 3 (Ubuntu) and all give the same results.

I'm curious about performance numbers for all these processors. Can even Raspberry Pi 3 be used for mining?

bomb-on commented 6 years ago

Tested on: OS X 10.13.5 Intel Core i5 2.7GHz commit: e136756

Results:

$ mono Tevador.RandomJS.Test.exe --count 1000 --seed 123 --threads 2 --customOptions --XML_AssignmentOperators_Add 1 --XML_AssignmentOperators_Sub 1 --XML_AssignmentOperators_Mul 1 --XML_AssignmentOperators_Div 1 --XML_AssignmentOperators_Mod 1 --XML_AssignmentOperators_PreInc 1 --XML_AssignmentOperators_PostDec 1 --XML_AssignmentOperators_PreDec 1 --XML_AssignmentOperators_PostInc 1 --XML_UnaryOperators_Sqrt 1 --XML_BinaryOperators_Add 1 --XML_BinaryOperators_Sub 1 --XML_BinaryOperators_Mul 1 --XML_BinaryOperators_Div 1 --XML_BinaryOperators_Mod 1 --XML_Literals_Numeric 1 --XML_NumericLiterals_DecimalInteger 3 --XML_NumericLiterals_BinaryInteger 1 --XML_NumericLiterals_OctalInteger 3 --XML_NumericLiterals_SmallInteger 3 --XML_NumericLiterals_HexInteger 3 --XML_NumericLiterals_FixedFloat 3 --XML_NumericLiterals_ExpFloat 3 --XML_Expressions_AssignmentExpression 4 --XML_Expressions_BinaryExpression 5 --XML_Expressions_Literal 2 --XML_Expressions_UnaryExpression 5 --XML_Statements_AssignmentStatement 1 --XML_AssignmentInForLoop_Add 1 --XML_GlobalVariablesCountRange_Min 100 --XML_GlobalVariablesCountRange_Span 0 --XML_VariableInitializerDepth 4 --XML_MaxExpressionDepth 4 --XML_ConstVariableChance 0.1 --XML_MaxStringLengthRange_Min 35 --XML_MaxStringLengthRange_Span 0

Collecting statistics from 1000 random program executions (seed = 123)
5 %, 10 %, 15 %, 20 %, 25 %, 30 %, 35 %, 40 %, 45 %, 50 %, 55 %, 60 %, 65 %, 70 %, 75 %, 80 %, 85 %, 90 %, 95 %,
Completed in 7.6793444 seconds
Longest runtimes:
Seed = 86460416fe8be2c82375ee36c6b48b3c0356f2d9270ff5f856e7202706020a15, Runtime = 0.04320 s
Seed = f74ea47f971bdfe927a67e0eb8bcb0836fc21838dfb69e7ed179defc424e289b, Runtime = 0.03316 s
Seed = 6fa68338a1d4656b071b682aa7828b73dafb4ba8b90f7bc82f384042adb5ff2f, Runtime = 0.03298 s
Seed = fdb87aac7f31faee0aa653df81007761c0cee3f436b3cb0ecb4b2e19932041f5, Runtime = 0.03287 s
Seed = 1a9d352a9af1e6c08bceba71ffe4fbfb7f61c06c8777bdc344f8d010fc74f77f, Runtime = 0.02954 s
Seed = a894dbd527ebf7e1669316b0f7d9b028b33c72c0a341d586d4d32f2e22daad23, Runtime = 0.02395 s
Seed = 3f6524e301d20266e08aca271a4248f6ad025756539a637aaca6bf951a393d96, Runtime = 0.01987 s
Seed = 1fde2adc66eb993b77be43a0df7b313c5908eaa118b78a6edbaea223d47067c1, Runtime = 0.01925 s
Seed = 0021020d8391b1c4a9009b676f61245d7625f4c7fe2b64d06bd304c8f4bf9360, Runtime = 0.01836 s
Seed = c309c0a2c910a6f472259c73169125ccd33b111748ed96c1c75b13a17e21b562, Runtime = 0.01786 s
Runtime [s] Min: 0.00308; Max: 0.04320; Avg: 0.00710; Stdev: 0.00324;
Runtime [s] 99.99th percentile: 0.0431984979659319
Average entropy of program output (est.): 6739.48 bits
Cyclomatic complexity Min: 0; Max: 0; Avg: 0; Stdev: 0;
Lines of code Min: 0; Max: 0; Avg: 0; Stdev: 0;
Halstead difficulty Min: 0; Max: 0; Avg: 0; Stdev: 0;
Matches 'SyntaxError' optimization: 0.00 %
'SyntaxError' optimization runtime: 0.00 %
Matches 'XS': 0.00 %
Cumulative output hash: b83a20eaa1d1615b860dd7477dc8945d23b84cd9082a7240df600b347410441c
tevador commented 6 years ago

@SChernykh

I'm curious about performance numbers for all these processors. Can even Raspberry Pi 3 be used for mining?

Commit 4fc608c9b476f5a8e0d18dc46c190f5a6a5c7b94, 1 core, XS engine:

So the Raspberry is about 6-7 times slower per core. It consumes ~5 W when mining with all 4 cores, but you need some kind of cooling solution because it will start thermal throttling quickly. The performance per watt seems to be comparable to the most efficient laptop processors, so it's definitely usable for mining.

jwinterm commented 6 years ago

OK, tested again with: AMD Athlon(tm) II X2 220 Processor Ubuntu 16.04 x86_64 commit: e13675695ede93b38b706363a374d012af2bb250

Got the same result as others using the options suggested:

jw@couch:~/RandomJS/src/bin$ mono Tevador.RandomJS.Test.exe --count 1000 --seed 123 --threads 2 --customOptions --XML_AssignmentOperators_Add 1 --XML_AssignmentOperators_Sub 1 --XML_AssignmentOperators_Mul 1 --XML_AssignmentOperators_Div 1 --XML_AssignmentOperators_Mod 1 --XML_AssignmentOperators_PreInc 1 --XML_AssignmentOperators_PostDec 1 --XML_AssignmentOperators_PreDec 1 --XML_AssignmentOperators_PostInc 1 --XML_UnaryOperators_Sqrt 1 --XML_BinaryOperators_Add 1 --XML_BinaryOperators_Sub 1 --XML_BinaryOperators_Mul 1 --XML_BinaryOperators_Div 1 --XML_BinaryOperators_Mod 1 --XML_Literals_Numeric 1 --XML_NumericLiterals_DecimalInteger 3 --XML_NumericLiterals_BinaryInteger 1 --XML_NumericLiterals_OctalInteger 3 --XML_NumericLiterals_SmallInteger 3 --XML_NumericLiterals_HexInteger 3 --XML_NumericLiterals_FixedFloat 3 --XML_NumericLiterals_ExpFloat 3 --XML_Expressions_AssignmentExpression 4 --XML_Expressions_BinaryExpression 5 --XML_Expressions_Literal 2 --XML_Expressions_UnaryExpression 5 --XML_Statements_AssignmentStatement 1 --XML_AssignmentInForLoop_Add 1 --XML_GlobalVariablesCountRange_Min 100 --XML_GlobalVariablesCountRange_Span 0 --XML_VariableInitializerDepth 4 --XML_MaxExpressionDepth 4 --XML_ConstVariableChance 0.1 --XML_MaxStringLengthRange_Min 35 --XML_MaxStringLengthRange_Span 0
Collecting statistics from 1000 random program executions (seed = 123)
5 %, 10 %, 15 %, 20 %, 20 %, 30 %, 35 %, 40 %, 45 %, 50 %, 55 %, 60 %, 65 %, 70 %, 75 %, 80 %, 85 %, 90 %, 95 %, 
Completed in 10.1174143 seconds
Longest runtimes:
Seed = 69338483f2db38253138ce5df46054706efa4ef560f1c5ff0613988e171b5160, Runtime = 0.02479 s
Seed = 1bee5af3511ebdbe3a67ace3450099f7017a2250ad4ff7f04c1b0a6bd41bcd9b, Runtime = 0.02241 s
Seed = 8debbf497bce946b21d31cb72b3ce444f73c68192896c058520e5fbc8dd1323a, Runtime = 0.02213 s
Seed = c8c7da935221e7b9adf811d532418ad349cea172574975fade635c0bde95718f, Runtime = 0.02132 s
Seed = 3d569937c30d61296a232bc174f24078e9ad4fa5010d9df89714750c2f0d2e2f, Runtime = 0.01967 s
Seed = 05257e9f3a45ba51156bc26681dd628e99ecc990486de7b6926946e43acb3915, Runtime = 0.01910 s
Seed = e7a7b232a38def90fa1abd08d2839ab5640894fd5b78ae8d441c63383c5a488e, Runtime = 0.01903 s
Seed = 6cf3be599f81d43ca2e8cce3c5a0707bad7890048eb8e03864712249f332b073, Runtime = 0.01891 s
Seed = 595a12c78615370757ad264311450d181f1089869cb59d897eaeed082046b481, Runtime = 0.01854 s
Seed = d561beeae65df624e12294d63e7ac4a227a1fc5dfb928c07cbfd37edc9d318af, Runtime = 0.01706 s
Runtime [s] Min: 0.00437; Max: 0.02479; Avg: 0.00749; Stdev: 0.00288;
Runtime [s] 99.99th percentile: 0.0247909612953663
Average entropy of program output (est.): 6739.48 bits
Cyclomatic complexity Min: 0; Max: 0; Avg: 0; Stdev: 0;
Lines of code Min: 0; Max: 0; Avg: 0; Stdev: 0;
Halstead difficulty Min: 0; Max: 0; Avg: 0; Stdev: 0;
Matches 'SyntaxError' optimization: 0.00 %
'SyntaxError' optimization runtime: 0.00 %
Matches 'XS': 0.00 %
Cumulative output hash: b83a20eaa1d1615b860dd7477dc8945d23b84cd9082a7240df600b347410441c
SChernykh commented 6 years ago

So the Raspberry is about 6-7 times slower per core. It consumes ~5 W when mining with all 4 cores, but you need some kind of cooling solution because it will start thermal throttling quickly. The performance per watt seems to be comparable to the most efficient laptop processors, so it's definitely usable for mining.

Huh, and considering 40$ price it will be the most profitable option: 8x Raspberry Pi 3 will cost 320$ and will be faster than Kaby Lake. Future mining rigs will look like this: https://www.youtube.com/watch?v=i_r3z1jYHAc

stoffu commented 6 years ago 
RandomJS/src/bin$ mono Tevador.RandomJS.Test.exe --count 1000 --seed 123 --threads 2 --customOptions --XML_AssignmentOperators_Add 1 --XML_AssignmentOperators_Sub 1 --XML_AssignmentOperators_Mul 1 --XML_AssignmentOperators_Div 1 --XML_AssignmentOperators_Mod 1 --XML_AssignmentOperators_PreInc 1 --XML_AssignmentOperators_PostDec 1 --XML_AssignmentOperators_PreDec 1 --XML_AssignmentOperators_PostInc 1 --XML_UnaryOperators_Sqrt 1 --XML_BinaryOperators_Add 1 --XML_BinaryOperators_Sub 1 --XML_BinaryOperators_Mul 1 --XML_BinaryOperators_Div 1 --XML_BinaryOperators_Mod 1 --XML_Literals_Numeric 1 --XML_NumericLiterals_DecimalInteger 3 --XML_NumericLiterals_BinaryInteger 1 --XML_NumericLiterals_OctalInteger 3 --XML_NumericLiterals_SmallInteger 3 --XML_NumericLiterals_HexInteger 3 --XML_NumericLiterals_FixedFloat 3 --XML_NumericLiterals_ExpFloat 3 --XML_Expressions_AssignmentExpression 4 --XML_Expressions_BinaryExpression 5 --XML_Expressions_Literal 2 --XML_Expressions_UnaryExpression 5 --XML_Statements_AssignmentStatement 1 --XML_AssignmentInForLoop_Add 1 --XML_GlobalVariablesCountRange_Min 100 --XML_GlobalVariablesCountRange_Span 0 --XML_VariableInitializerDepth 4 --XML_MaxExpressionDepth 4 --XML_ConstVariableChance 0.1 --XML_MaxStringLengthRange_Min 35 --XML_MaxStringLengthRange_Span 0
Collecting statistics from 1000 random program executions (seed = 123)
5 %, 10 %, 15 %, 20 %, 25 %, 30 %, 30 %, 40 %, 45 %, 50 %, 55 %, 60 %, 65 %, 70 %, 75 %, 80 %, 85 %, 90 %, 95 %, 
Completed in 8.1185582 seconds
Longest runtimes:
Seed = 21caf72218e8d1a19652a2679ae35970f84da1cf4eb437cf5bc0c65af08001a3, Runtime = 0.00976 s
Seed = 56d56c8f1a814f493d9f1ee5433de59a18971b2eb59f3a69f90679c866a98749, Runtime = 0.00871 s
Seed = 0b5499ef493be6e4b4851d0c49f203429b9614803f113c9919619c3e619052f2, Runtime = 0.00869 s
Seed = 39f3adaba09054c00e082b417b67930636c7f2b98f83e3481ccd8e27157ec5cb, Runtime = 0.00773 s
Seed = 20b1d24cda3e4257e3d9409e6f55fd983ca277e2d9a3994b48057b88e228c2bc, Runtime = 0.00750 s
Seed = c6b4bcd6199322b38f3dec1bd57a00e149183a283b92fd1a80ec93e7e3059959, Runtime = 0.00745 s
Seed = c645cd0948f1ff0f416ce886d8be805b90b774d4d6304ebd3201d4f986ab9058, Runtime = 0.00735 s
Seed = 816151bf8790a0a75a63de4e9b5e9fe64df33e658215721a6689ea0ca8d0682d, Runtime = 0.00719 s
Seed = e22a901ab382e93965cdc190720a9bd4fedad09d603968f962709cbbe8c1a8e0, Runtime = 0.00718 s
Seed = 0395d8b573caa0ad21a549292cc2e56b0531e530c73d3f975f9e2b2346cff08d, Runtime = 0.00712 s
Runtime [s] Min: 0.00268; Max: 0.00976; Avg: 0.00418; Stdev: 0.00096;
Runtime [s] 99.99th percentile: 0.0097556000109762
Average entropy of program output (est.): 6739.48 bits
Cyclomatic complexity Min: 0; Max: 0; Avg: 0; Stdev: 0;
Lines of code Min: 0; Max: 0; Avg: 0; Stdev: 0;
Halstead difficulty Min: 0; Max: 0; Avg: 0; Stdev: 0;
Matches 'SyntaxError' optimization: 0.00 %
'SyntaxError' optimization runtime: 0.00 %
Matches 'XS': 0.00 %
Cumulative output hash: b83a20eaa1d1615b860dd7477dc8945d23b84cd9082a7240df600b347410441c
antanst commented 6 years ago 
$ mono Tevador.RandomJS.Test.exe --count 1000 --seed 123 --threads 2 --customOptions --XML_AssignmentOperators_Add 1 --XML_AssignmentOperators_Sub 1 --XML_AssignmentOperators_Mul 1 --XML_AssignmentOperators_Div 1 --XML_AssignmentOperators_Mod 1 --XML_AssignmentOperators_PreInc 1 --XML_AssignmentOperators_PostDec 1 --XML_AssignmentOperators_PreDec 1 --XML_AssignmentOperators_PostInc 1 --XML_UnaryOperators_Sqrt 1 --XML_BinaryOperators_Add 1 --XML_BinaryOperators_Sub 1 --XML_BinaryOperators_Mul 1 --XML_BinaryOperators_Div 1 --XML_BinaryOperators_Mod 1 --XML_Literals_Numeric 1 --XML_NumericLiterals_DecimalInteger 3 --XML_NumericLiterals_BinaryInteger 1 --XML_NumericLiterals_OctalInteger 3 --XML_NumericLiterals_SmallInteger 3 --XML_NumericLiterals_HexInteger 3 --XML_NumericLiterals_FixedFloat 3 --XML_NumericLiterals_ExpFloat 3 --XML_Expressions_AssignmentExpression 4 --XML_Expressions_BinaryExpression 5 --XML_Expressions_Literal 2 --XML_Expressions_UnaryExpression 5 --XML_Statements_AssignmentStatement 1 --XML_AssignmentInForLoop_Add 1 --XML_GlobalVariablesCountRange_Min 100 --XML_GlobalVariablesCountRange_Span 0 --XML_VariableInitializerDepth 4 --XML_MaxExpressionDepth 4 --XML_ConstVariableChance 0.1 --XML_MaxStringLengthRange_Min 35 --XML_MaxStringLengthRange_Span 0
Collecting statistics from 1000 random program executions (seed = 123)
5 %, 10 %, 15 %, 20 %, 25 %, 30 %, 35 %, 40 %, 45 %, 50 %, 55 %, 60 %, 65 %, 70 %, 75 %, 80 %, 85 %, 90 %, 95 %,
Completed in 4.7085095 seconds
Longest runtimes:
Seed = 10324c0a8b1c464df6b2cb01dae9bed9d9bf6b5259bf878e699aceb4abf137b7, Runtime = 0.01711 s
Seed = 1ea883a9d1907c1df7e7a34138b0d85077047cb1461739e0e3d8dc283992609c, Runtime = 0.01609 s
Seed = 0e2a96206861eff978d52453296e9e8d7f6bd82e33e1331d1787d407a00e623d, Runtime = 0.01343 s
Seed = 06758e059356767db38bc96bee562e940278fef61aa2270858db9f0a425a232d, Runtime = 0.01181 s
Seed = baec1d3a2b533c4fa2011cdaf4e7089eec3f4877fe59efd093240b0be1cd0d40, Runtime = 0.01073 s
Seed = ab92adcefe0b3c6523bffb5251bd8adc3856c1404dc206ea44611ee806174f0f, Runtime = 0.00992 s
Seed = e22a901ab382e93965cdc190720a9bd4fedad09d603968f962709cbbe8c1a8e0, Runtime = 0.00990 s
Seed = 9709bbcd7b1c158b165bc4e43bdab9a3f0f53a07586fa0b1cceab4c70c19f1d9, Runtime = 0.00896 s
Seed = dcd137ef89f0e5b6813068076884ff6cbaf3ea8b05bda014d1587409981b97ff, Runtime = 0.00855 s
Seed = af9980ddbdee7addcea010a75c55d3d0e221fff6eb6775e816fac3f1818292ea, Runtime = 0.00824 s
Runtime [s] Min: 0.00229; Max: 0.01711; Avg: 0.00385; Stdev: 0.00125;
Runtime [s] 99.99th percentile: 0.0171129430236761
Average entropy of program output (est.): 6739.48 bits
Cyclomatic complexity Min: 0; Max: 0; Avg: 0; Stdev: 0;
Lines of code Min: 0; Max: 0; Avg: 0; Stdev: 0;
Halstead difficulty Min: 0; Max: 0; Avg: 0; Stdev: 0;
Matches 'SyntaxError' optimization: 0.00 %
'SyntaxError' optimization runtime: 0.00 %
Matches 'XS': 0.00 %
Cumulative output hash: b83a20eaa1d1615b860dd7477dc8945d23b84cd9082a7240df600b347410441c
antanst commented 6 years ago

Also tested with Node-Chakracore v10.1.0 official release binaries. Same system as previous post. Results are different.

$ mono Tevador.RandomJS.Test.exe --count 1000 --seed 123 --threads 2 --customOptions --XML_AssignmentOperators_Add 1 --XML_AssignmentOperators_Sub 1 --XML_AssignmentOperators_Mul 1 --XML_AssignmentOperators_Div 1 --XML_AssignmentOperators_Mod 1 --XML_AssignmentOperators_PreInc 1 --XML_AssignmentOperators_PostDec 1 --XML_AssignmentOperators_PreDec 1 --XML_AssignmentOperators_PostInc 1 --XML_UnaryOperators_Sqrt 1 --XML_BinaryOperators_Add 1 --XML_BinaryOperators_Sub 1 --XML_BinaryOperators_Mul 1 --XML_BinaryOperators_Div 1 --XML_BinaryOperators_Mod 1 --XML_Literals_Numeric 1 --XML_NumericLiterals_DecimalInteger 3 --XML_NumericLiterals_BinaryInteger 1 --XML_NumericLiterals_OctalInteger 3 --XML_NumericLiterals_SmallInteger 3 --XML_NumericLiterals_HexInteger 3 --XML_NumericLiterals_FixedFloat 3 --XML_NumericLiterals_ExpFloat 3 --XML_Expressions_AssignmentExpression 4 --XML_Expressions_BinaryExpression 5 --XML_Expressions_Literal 2 --XML_Expressions_UnaryExpression 5 --XML_Statements_AssignmentStatement 1 --XML_AssignmentInForLoop_Add 1 --XML_GlobalVariablesCountRange_Min 100 --XML_GlobalVariablesCountRange_Span 0 --XML_VariableInitializerDepth 4 --XML_MaxExpressionDepth 4 --XML_ConstVariableChance 0.1 --XML_MaxStringLengthRange_Min 35 --XML_MaxStringLengthRange_Span 0
Collecting statistics from 1000 random program executions (seed = 123)
5 %, 10 %, 15 %, 20 %, 25 %, 30 %, 35 %, 40 %, 45 %, 50 %, 55 %, 60 %, 65 %, 70 %, 75 %, 80 %, 85 %, 90 %, 95 %,
Completed in 8.3624088 seconds
Longest runtimes:
Seed = c98987a1924460048751a7ed8826dedccf35127302ccf677f7d31863d1824547, Runtime = 0.03301 s
Seed = d561beeae65df624e12294d63e7ac4a227a1fc5dfb928c07cbfd37edc9d318af, Runtime = 0.02909 s
Seed = f0347fedcd64695e23d126d4101bb36504979ca52d935c472bcfff348b107da2, Runtime = 0.02673 s
Seed = 4079a249630fe6a1425fa1e04d8d03fcf90001749878c54f5486359888ac554b, Runtime = 0.02255 s
Seed = ccce6bbfd69adaa61606abe6d3a15c8b2ea1dfb77cf9c31a40edbb823befb189, Runtime = 0.01903 s
Seed = 4c193b1c61cb20c405dc687e3854149ac78440e748ea90dfac9a37c143ac7aa9, Runtime = 0.01871 s
Seed = 44474a04ffd6aa03fc4f0bd71d45124276742460fc5a9504a084a9a68cdb3d2d, Runtime = 0.01860 s
Seed = a2c85adc16d97ff5f4bf356034d9a7749c8df41a7f60e57fe4e4ecf69ef15eb7, Runtime = 0.01771 s
Seed = 6d100db791b0bc891a5bf9dd0645b34007b1910828ad1db5da4bf043212fd3ee, Runtime = 0.01537 s
Seed = daf7cf653790ee9a2c74567a5da93d2d89ea4f35aa48bc18edcb87f1fcd26d80, Runtime = 0.01471 s
Runtime [s] Min: 0.00523; Max: 0.03301; Avg: 0.00816; Stdev: 0.00209;
Runtime [s] 99.99th percentile: 0.0330056299862918
Average entropy of program output (est.): 6739.20 bits
Cyclomatic complexity Min: 0; Max: 0; Avg: 0; Stdev: 0;
Lines of code Min: 0; Max: 0; Avg: 0; Stdev: 0;
Halstead difficulty Min: 0; Max: 0; Avg: 0; Stdev: 0;
Matches 'SyntaxError' optimization: 0.00 %
'SyntaxError' optimization runtime: 0.00 %
Matches 'XS': 0.00 %
Cumulative output hash: b87b2edee0e6a496f1ec2fcb3dd3f56e0db0d37b25acb92772eb9e0d8109ae36
tevador commented 6 years ago

@antanst We are no longer aiming for binary compatilibity with all JS engines. The XS engine will be used as reference and some results will not match with other engines. The most important thing is that results match across platforms with the same engine.

Nevertheless, you can try to find what causes the different outputs like this:

  1. Use just one thread --threads 1 (this is needed to make sure programs are always processed in the same order).
  2. Append --debug option to the command line. This will output the hashes of all programs.
  3. Redirect the output to a file: mono Tevador.RandomJS.Test.exe ..... > output1.txt
  4. Run the same command with the other engine and redirect to output2.txt.
  5. Use diff output1.txt output2.txt to find which seeds produce different hashes. Seed is the value before colon :.
  6. Run mono Tevador.RandomJS.exe *seed* > program.js to generate the exact program that causes the mismatch.
  7. Then you can run node fast-eval.js < program.js to compare the outputs. You can also use diff to see which lines differ. From here it's quite easy to find the mathematical operation that causes the mismatch.
antanst commented 6 years ago

Ok. Apologies for spamming the thread. Since you asked, I've followed your steps and traced the difference to this code output:

Code:

Function.prototype.toString = function() {
    return '[Function]' + this.name;
};
g = 1
g += function() {}
console.log(g)

Outputs of vanilla Node 8 and 10:

$ node fast-eval.js < ./test.js
1[Function]

Output of Chackra Node v10:

$ ./node-v10.1.0-darwin-x64/bin/node fast-eval.js < ./test.js
1[Function]g

Looks like a vanilla Node bug.

tevador commented 6 years ago

Interesting find. To me it looks like Chackra Node v10 bug, because the anonymous function doesn't have name g (should be empty).

JohannesLau commented 6 years ago

Ryzen 1700x Windows 10.0.17134 Build 17134 C:\Users\Johannes\Downloads\RandomJS-master\RandomJS-master\src\Tevador.RandomJS.Test\bin\Debug>Tevador.RandomJS.Test.exe --count 1000 --seed 123 --threads 2 --customOptions --XML_AssignmentOperators_Add 1 --XML_AssignmentOperators_Sub 1 --XML_AssignmentOperators_Mul 1 --XML_AssignmentOperators_Div 1 --XML_AssignmentOperators_Mod 1 --XML_AssignmentOperators_PreInc 1 --XML_AssignmentOperators_PostDec 1 --XML_AssignmentOperators_PreDec 1 --XML_AssignmentOperators_PostInc 1 --XML_UnaryOperators_Sqrt 1 --XML_BinaryOperators_Add 1 --XML_BinaryOperators_Sub 1 --XML_BinaryOperators_Mul 1 --XML_BinaryOperators_Div 1 --XML_BinaryOperators_Mod 1 --XML_Literals_Numeric 1 --XML_NumericLiterals_DecimalInteger 3 --XML_NumericLiterals_BinaryInteger 1 --XML_NumericLiterals_OctalInteger 3 --XML_NumericLiterals_SmallInteger 3 --XML_NumericLiterals_HexInteger 3 --XML_NumericLiterals_FixedFloat 3 --XML_NumericLiterals_ExpFloat 3 --XML_Expressions_AssignmentExpression 4 --XML_Expressions_BinaryExpression 5 --XML_Expressions_Literal 2 --XML_Expressions_UnaryExpression 5 --XML_Statements_AssignmentStatement 1 --XML_AssignmentInForLoop_Add 1 --XML_GlobalVariablesCountRange_Min 100 --XML_GlobalVariablesCountRange_Span 0 --XML_VariableInitializerDepth 4 --XML_MaxExpressionDepth 4 --XML_ConstVariableChance 0.1 --XML_MaxStringLengthRange_Min 35 --XML_MaxStringLengthRange_Span 0 Collecting statistics from 1000 random program executions (seed = 123) 5 %, 10 %, 15 %, 20 %, 25 %, 30 %, 35 %, 40 %, 45 %, 50 %, 55 %, 60 %, 65 %, 70 %, 75 %, 80 %, 85 %, 90 %, 95 %, Completed in 5.3005916 seconds Longest runtimes: Seed = 6eda4048d38469bb1a003a1fa77905ef394e57eb23d030b2e417440382adb38b, Runtime = 0.02108 s Seed = 1a1badef09d9f7ac65a6c798369b9afdb3bac23de7f7cbd98c532340d27c834f, Runtime = 0.02046 s Seed = 46509a1b7306a0929fbfed991ac472499a5fb1c569d4e3ced8863c23eee61eb2, Runtime = 0.02037 s Seed = e635da2a2215b8a98377d1057b9fa17fcfe7238a9de3722a59d80e8bf16aac8f, Runtime = 0.02011 s Seed = f525016ba483981a4faf26f2892f0033c0ca2bb4d1929eabc90aadbd021c9ed1, Runtime = 0.01887 s Seed = 8416a772f22270ab74c50934ca568d1cafed1fafe35d542494633c713ca1bb2a, Runtime = 0.01855 s Seed = 5b58f2d9d91e0638bbb522be955e3700decb7f281736329c0c76cc0a329718a8, Runtime = 0.01846 s Seed = f69690e35ed0063aac349ac68fccceeb0d9937e1cc6ea29e089823b2bbd4c9f2, Runtime = 0.01839 s Seed = 787102c377b98a079e9e522cd453f233fc8919fee7b512e5f1a2863b5701c492, Runtime = 0.01740 s Seed = dedc8ea24911abbed6e14398ad677fda754f679815e894bda3cdc5935d26841e, Runtime = 0.00834 s Runtime [s] Min: 0.00245; Max: 0.02108; Avg: 0.00408; Stdev: 0.00186; Runtime [s] 99.99th percentile: 0.0210800130000735 Average entropy of program output (est.): 6739.48 bits Cyclomatic complexity Min: 0; Max: 0; Avg: 0; Stdev: 0; Lines of code Min: 0; Max: 0; Avg: 0; Stdev: 0; Halstead difficulty Min: 0; Max: 0; Avg: 0; Stdev: 0; Matches 'SyntaxError' optimization: 0.00 % 'SyntaxError' optimization runtime: 0.00 % Matches 'XS': 0.00 % Cumulative output hash: b83a20eaa1d1615b860dd7477dc8945d23b84cd9082a7240df600b347410441c

timolson commented 6 years ago

The fact that Rasberry PI is an efficient option should be a huge warning flag that this is not an ASIC resistant PoW. These numbers indicate that the additional complexity of Intel’s branch prediction, cache lookahead, etc is not worth the gates. An ASIC designer will be writing a very simple, small-gate CPU-type thing which runs direct execution from the generator algo with no parsing, interpretation, bytecode, object code, or anything else needed in general purpose CPU’s. An ASIC will put lots of slow but small cores into a chip, and the design will not take long because they can exclude all the fancy stuff Intel and AMD do. ARM cores are easily licensable and customizable for particular applications. I’ve tried to say this in other ways in other threads, but these numbers back it up.

SChernykh commented 6 years ago

Specialized ARM core with proper core-to-cache ratio will be ~2 times more power efficient. I don't see a big problem with it. Many manufacturers can produce ARM cores because you said yourself they're easily licensable. This ARM-based hardware will be just like tuned Raspberry Pi, it can/will be used for other stuff apart mining.

timolson commented 6 years ago

2x is huge in mining. As we have seen, hashrate will increase to the point where it’s barely profitable vs electricity. A 2x ASIC means home mining on CPU is not profitable. Everyone will need the ASIC.

SChernykh commented 6 years ago

CPU mining is a different world. First, having just single CPU + motherboard + RAM + power supply is already very inefficient. Buying specialized ARM boards and connecting them to the same power bus will be the way to go: https://www.youtube.com/watch?v=i_r3z1jYHAc&t=284s These ARM boards will be easy to buy like RPi now.

SChernykh commented 6 years ago

There will also be PCI-e boards with many small ARM cores like Xeon Phi. And we'll be back to making rigs on PCI-e risers :100:

timolson commented 6 years ago

It’s true. That world would be ok. But I’m not convinced I can’t make an ASIC that’s faster than the ARM for running RandomJS programs only.