search

trufflesuite / truffle

:warning: The Truffle Suite is being sunset. For information on ongoing support, migration options and FAQs, visit the Consensys blog. Thank you for all the support over the years.
https://consensys.io/blog/consensys-announces-the-sunset-of-truffle-and-ganache-and-new-hardhat?utm_source=github&utm_medium=referral&utm_campaign=2023_Sep_truffle-sunset-2023_announcement_
MIT License
14.02k stars 2.32k forks source link

Tests randomly crashing at ProviderError.ExtendableError on Ubuntu (Linux) #729

Closed dwalintukan closed 5 years ago

dwalintukan commented 6 years ago

Issue

On a Ubuntu Linux environment (Trusty), tests randomly fail with this ExtendableError:

1) Contract: TopicEvent "after each" hook for "throws on an invalid result index":
Error: Could not connect to your Ethereum client. Please check that your Ethereum client:
    - is running
    - is accepting RPC connections (i.e., "--rpc" option is used in geth)
    - is accessible over the network
    - is properly configured in your Truffle configuration file (truffle.js)

      at ProviderError.ExtendableError (/home/travis/.nvm/versions/node/v8.9.3/lib/node_modules/truffle/build/cli.bundled.js:9401:17)
      at new ProviderError (/home/travis/.nvm/versions/node/v8.9.3/lib/node_modules/truffle/build/cli.bundled.js:325054:24)
      at /home/travis/.nvm/versions/node/v8.9.3/lib/node_modules/truffle/build/cli.bundled.js:325137:17
      at /home/travis/.nvm/versions/node/v8.9.3/lib/node_modules/truffle/build/cli.bundled.js:325195:24
      at XMLHttpRequest.request.onreadystatechange (/home/travis/.nvm/versions/node/v8.9.3/lib/node_modules/truffle/build/cli.bundled.js:328229:7)
      at XMLHttpRequestEventTarget.dispatchEvent (/home/travis/.nvm/versions/node/v8.9.3/lib/node_modules/truffle/build/cli.bundled.js:176415:18)
      at XMLHttpRequest._setReadyState (/home/travis/.nvm/versions/node/v8.9.3/lib/node_modules/truffle/build/cli.bundled.js:176705:12)
      at XMLHttpRequest._onHttpRequestError (/home/travis/.nvm/versions/node/v8.9.3/lib/node_modules/truffle/build/cli.bundled.js:176895:12)
      at ClientRequest.<anonymous> (/home/travis/.nvm/versions/node/v8.9.3/lib/node_modules/truffle/build/cli.bundled.js:176765:24)
      at Socket.socketOnEnd (_http_client.js:423:9)
      at endReadableNT (_stream_readable.js:1056:12)
      at _combinedTickCallback (internal/process/next_tick.js:138:11)

Specifically, I have a Travis-CI (continuous integration) setup and this is where the tests are failing. My local Mac OSX environment passes these tests with no problem. Every once and a while, they will fail with the same error, but I just run the tests again and they pass.

I'd say it happens like 10-15% of the time on Mac OSX, but it happens like 60-80% of the time on the Travis-CI linux env.

It feels like it used to have this error less on earlier Truffle versions. I just updated to 4.0.4 and it seems way more often now.

Steps to Reproduce

  1. Clone this repo: https://github.com/bodhiproject/bodhi-core
  2. Use Truffle v4.0.4 on a Ubuntu Linux environment (see environment details)
  3. truffle test

Expected Behavior

Tests should pass like they do on Mac OSX env.

Actual Results

I test this on my local machine (mac osx), when all tests pass which they do, I push up to Github. Then it fires off a Travis-CI test on the linux env and fails pretty much every time.

Environment

Travis-CI Env (fails)

Mac OSX Env (passes)

$ gcc --version:

Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 9.0.0 (clang-900.0.39.2)
Target: x86_64-apple-darwin16.7.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
Alonski commented 6 years ago

This happens to me as well:

PS D:\Projects\Fun Stuff\daostack> truffle develop
Truffle Develop started at http://localhost:9545/

Accounts:
(0) 0x627306090abab3a6e1400e9345bc60c78a8bef57
(1) 0xf17f52151ebef6c7334fad080c5704d77216b732
(2) 0xc5fdf4076b8f3a5357c5e395ab970b5b54098fef
(3) 0x821aea9a577a9b44299b9c15c88cf3087f3b5544
(4) 0x0d1d4e623d10f9fba5db95830f7d3839406c6af2
(5) 0x2932b7a2355d6fecc4b5c0b6bd44cc31df247a2e
(6) 0x2191ef87e392377ec08e7c08eb105ef5448eced5
(7) 0x0f4f2ac550a1b4e2280d04c21cea7ebd822934b5
(8) 0x6330a553fc93768f612722bb8c2ec78ac90b3bbc
(9) 0x5aeda56215b167893e80b4fe645ba6d5bab767de

Private Keys:
(0) c87509a1c067bbde78beb793e6fa76530b6382a4c0241e5e4a9ec0a0f44dc0d3
(1) ae6ae8e5ccbfb04590405997ee2d52d2b330726137b875053c36d94e974d162f
(2) 0dbbe8e4ae425a6d2687f1a7e3ba17bc98c673636790f1b8ad91193c05875ef1
(3) c88b703fb08cbea894b6aeff5a544fb92e78a18e19814cd85da83b71f772aa6c
(4) 388c684f0ba1ef5017716adb5d21a053ea8e90277d0868337519f97bede61418
(5) 659cbb0e2411a44db63778987b1e22153c086a95eb6b18bdf89de078917abc63
(6) 82d052c865f5763aad42add438569276c00d3d88a2d062d36b2bae914d58b8c8
(7) aa3680d5d48a8283413f7a108367c7299ca73f553735860a87b08f39395618b7
(8) 0f62d96d6675f32685bbdb8ac13cda7c23436f63efbb9d07700d8669ff12b7c4
(9) 8d5366123cb560bb606379f90a0bfd4769eecc0557f1b362dcae9012b548b1e5

Mnemonic: candy maple cake sugar pudding cream honey rich smooth crumble sweet treat

truffle(develop)> test
Compiling .\contracts\Migrations.sol...
Compiling .\contracts\VotingMachines\AbsoluteVote.sol...
Compiling .\contracts\VotingMachines\EmergentVoteScheme.sol...
Compiling .\contracts\VotingMachines\IntVoteInterface.sol...
Compiling .\contracts\VotingMachines\QuorumVote.sol...
Compiling .\contracts\controller\Avatar.sol...
Compiling .\contracts\controller\Controller.sol...
Compiling .\contracts\controller\DAOToken.sol...
Compiling .\contracts\controller\Reputation.sol...
Compiling .\contracts\globalConstraints\GlobalConstraintInterface.sol...
Compiling .\contracts\globalConstraints\TokenCapGC.sol...
Compiling .\contracts\test\ActionMock.sol...
Compiling .\contracts\test\Debug.sol...
Compiling .\contracts\test\ExecutableTest.sol...
Compiling .\contracts\test\GlobalConstraintMock.sol...
Compiling .\contracts\universalSchemes\ContributionReward.sol...
Compiling .\contracts\universalSchemes\ExecutableInterface.sol...
Compiling .\contracts\universalSchemes\GenesisScheme.sol...
Compiling .\contracts\universalSchemes\GlobalConstraintRegistrar.sol...
Compiling .\contracts\universalSchemes\OrganizationRegister.sol...
Compiling .\contracts\universalSchemes\SchemeRegistrar.sol...
Compiling .\contracts\universalSchemes\SimpleICO.sol...
Compiling .\contracts\universalSchemes\UniversalScheme.sol...
Compiling .\contracts\universalSchemes\UniversalSchemeInterface.sol...
Compiling .\contracts\universalSchemes\UpgradeScheme.sol...
Compiling .\contracts\universalSchemes\VestingScheme.sol...
Compiling .\contracts\universalSchemes\VoteInOrganizationScheme.sol...
Compiling zeppelin-solidity/contracts/lifecycle/Destructible.sol...
Compiling zeppelin-solidity/contracts/math/SafeMath.sol...
Compiling zeppelin-solidity/contracts/mocks/StandardTokenMock.sol...
Compiling zeppelin-solidity/contracts/ownership/Ownable.sol...
Compiling zeppelin-solidity/contracts/token/BurnableToken.sol...
Compiling zeppelin-solidity/contracts/token/MintableToken.sol...
Compiling zeppelin-solidity/contracts/token/StandardToken.sol...
Compiling zeppelin-solidity\contracts\math\SafeMath.sol...
Compiling zeppelin-solidity\contracts\ownership\Ownable.sol...
Compiling zeppelin-solidity\contracts\token\BasicToken.sol...
Compiling zeppelin-solidity\contracts\token\ERC20.sol...
Compiling zeppelin-solidity\contracts\token\ERC20Basic.sol...
Compiling zeppelin-solidity\contracts\token\StandardToken.sol...

  Contract: AbsoluteVote
    √ Sanity checks (3485ms)
    √ log the LogNewProposal event on proposing new proposal (1565ms)
    √ should log the LogCancelProposal event on canceling a proposal (982ms)
    √ should log the LogVoteProposal and LogCancelVoting events on voting and canceling the vote (1029ms)
    √ should log the LogExecuteProposal event (1395ms)
    √ All options can be voted (0-9) (2639ms)
    √ Double vote shouldn't double proposal's 'Option 2' count (1205ms)
    √ Vote cancellation should revert proposal's counters (1117ms)
    √ As allowOwner is set to true, Vote on the behalf of someone else should work (1014ms)
    √ As allowOwner is set to false, Vote on the behalf of someone else should NOT work (938ms)
    √ if the voter is not the proposal's owner, he shouldn't be able to vote on the behalf of someone else (892ms)
    √ Non-existent parameters hash shouldn't work (1085ms)
    √ Invalid percentage required( < 0 || > 100) shouldn't work (2029ms)
    √ Proposal voting or cancelling shouldn't be able after proposal has been executed (1180ms)
    √ the vote function should behave as expected (1976ms)
    √ cannot vote for another user (866ms)
    √ Should behave sensibly when voting with an empty reputation system (333ms)
    √ Should behave sensibly without an executable [TODO] execution isn't implemented yet (483ms)
    √ Proposal with wrong num of options (876ms)
    √ Test voteWithSpecifiedAmounts - More reputation than I own, negative reputation, etc.. (1139ms)
    √ Internal functions can not be called externally (963ms)
    √ Try to send wrong proposal id to the voting/cancel functions (1063ms)
    √ 2 proposals, 1 Reputation system (1703ms)
    as _not_ proposal owner - vote for myself
      √ vote "Option 1" then vote "Option 2" should register "Option 2" (1139ms)
      √ vote "Option 3" then vote "Option 4" should register "Option 4" (1206ms)
    as proposal owner - vote for another user
      √ vote "Option 1" then vote "Option 2" should register "Option 2" (1163ms)
      √ vote "Option 3" then vote "Option 4" should register "Option 4" (1015ms)

  Contract: Avatar
    √ genericAction no owner (466ms)
    √ generic call (553ms)
    √ pay ether to avatar (1282ms)
    √ sendEther from  (2278ms)
    √ externalTokenTransfer   (592ms)
    √ externalTokenTransferFrom & ExternalTokenIncreaseApproval (648ms)
    √ externalTokenTransferFrom & externalTokenDecreaseApproval (784ms)

  Contract: ContributionReward
    √ constructor (138ms)
    √ setParameters (1422ms)
    √ registerOrganization - check fee payment  (2066ms)
    √ submitContribution log (1745ms)
    √ submitContribution fees (1779ms)
    √ submitContribution without registration -should fail (1371ms)
    √ submitContribution check owner vote (1457ms)
    √ submitContribution check beneficiary==0 (1716ms)
    √ execute submitContribution  yes  (1778ms)
    √ execute submitContribution  mint reputation  (1759ms)
    √ execute submitContribution  mint tokens  (1519ms)
    √ execute submitContribution  send ethers  (2403ms)
    √ execute submitContribution  send externalToken  (2165ms)
    √ execute submitContribution proposal decision=='no' send externalToken   (1932ms)

  Contract: Controller
    √ mint reputation via controller (667ms)
    √ mint tokens via controller (633ms)
    √ register schemes (606ms)
    √ register schemes - check permissions for register new scheme (24778ms)
    √ register schemes - check permissions for updating existing scheme (1408ms)
    √ unregister schemes (646ms)
    √ unregister none registered scheme (590ms)
    √ unregister schemes - check permissions unregister scheme (20398ms)
    √ unregister self (753ms)
    √ isSchemeRegistered  (1787ms)
    √ addGlobalConstraint  (1804ms)
    √ removeGlobalConstraint  (2442ms)
    √ upgrade controller  (945ms)
    √ upgrade controller check permission (873ms)
    √ generic call (887ms)
    √ sendEther (2555ms)
    √ externalTokenTransfer (1030ms)
    √ externalTokenTransferFrom & ExternalTokenIncreaseApproval (1196ms)
    √ externalTokenTransferFrom & externalTokenDecreaseApproval (1799ms)
    √ globalConstraints mintReputation add & remove (1035ms)
    √ globalConstraints mintTokens add & remove (1211ms)
    √ globalConstraints register schemes add & remove (1071ms)
    √ globalConstraints unregister schemes add & remove (1001ms)
    √ globalConstraints generic call  add & remove (1551ms)
    √ globalConstraints sendEther  add & remove (3490ms)
    √ globalConstraints externalTokenTransfer  add & remove (1294ms)
    √ globalConstraints externalTokenTransferFrom , externalTokenIncreaseApproval , externalTokenDecreaseApproval (2016ms)

  Contract: DAOToken
    √ should put 0 Coins in the first account (70ms)
    √ should be owned by its creator (83ms)
    √ should be destructible (87ms)
    √ should mint tokens to owner account (7614ms)
    √ should allow minting tokens only by owner (6067ms)
    √ log the Mint event on mint (101ms)
    √ mint should be reflected in totalSupply (220ms)
    √ mint should be reflected in balances (110ms)
    √ totalSupply is 0 on init (77ms)
    √ burn (190ms)
    onlyOwner
      √ mint by owner (96ms)
      √ mint by not owner (71ms)

  Contract: GenesisScheme
    √ founders should get their share in reputation and tokens (2426ms)
    √ forgeOrg check avatar (383ms)
    √ forgeOrg check reputations and tokens to founders (328ms)
    √ forgeOrg check transfer ownership (405ms)
    √ setSchemes log (391ms)
    √ setSchemes from account that does not hold the lock (306ms)
    √ setSchemes increase approval for scheme  (413ms)
    √ setSchemes increase approval for scheme without fee (346ms)
    √ setSchemes check register (359ms)
    √ setSchemes check unregisterSelf (466ms)
    √ setSchemes delete lock (373ms)

  Contract: GlobalConstraintRegistrar
    √ constructor (142ms)
    √ setParameters (777ms)
    √ registerOrganization - check fee payment  (1457ms)
    √ proposeGlobalConstraint log (1580ms)
    √ proposeGlobalConstraint without registration -should fail (1292ms)
    √ proposeGlobalConstraint check owner vote (1269ms)
    √ execute proposeGlobalConstraint  (1612ms)
    √ proposeToRemoveGC log (1717ms)
    √ proposeToRemoveGC without registration -should fail (1189ms)
    √ proposeToRemoveGC check owner vote (1829ms)
    √ execute proposeToRemoveGC  (2184ms)
    √ execute proposeToRemoveGC (same as proposeGlobalConstraint) vote=NO  (1693ms)

  Contract: Migrations
    √ should have deployed entire DAOStack

  Contract: OrganizationRegister
    √ constructor (150ms)
    √ setParameters (192ms)
    √ registerOrganization - check fee payment  (1804ms)
    √ addOrPromoteAddress add and promote  (2300ms)
    √ addOrPromoteAddress add without enough fee should fail  (1163ms)
    √ addOrPromoteAddress add  without regisration -should fail  (1266ms)

  Contract: QuorumVote
    √ Sanity checks (1451ms)
    √ Quorum proposals should be executed when reaching the percentage required (1332ms)
    √ Invalid inputs shouldn't work (precReq, vote) (6588ms)
    √ All options can be voted (0-9) (2502ms)
    √ Double vote shouldn't double proposal's 'Option 2' count (1068ms)
    √ Vote cancellation should revert proposal's counters (1067ms)
    √ As allowOwner is set to true, Vote on the behalf of someone else should work (2654ms)
    √ As allowOwner is set to false, Vote on the behalf of someone else should NOT work (1128ms)
    √ if the voter is not the proposal's owner, he shouldn't be able to vote on the behalf of someone else (1200ms)
    √ Should not able to vote / cancel vote / proposal after proposal has been executed (1289ms)
    √ Only the owner of the proposal can cancel it (1463ms)
    √ log the LogNewProposal event on proposing new proposal (1231ms)
    √ Should log the LogCancelProposal event on canceling a proposal (857ms)
    √ Should log the LogVoteProposal and LogCancelVoting events on voting and canceling the vote (999ms)
    √ Should log the LogExecuteProposal event on executing quorum proposal with 'no' decision (925ms)
    √ cannot vote for another user (998ms)
    √ Should behave sensibly without an executable [TODO] execution isn't implemented yet (458ms)
    √ Test voteWithSpecifiedAmounts - More reputation than I own, negative reputation, etc.. (982ms)
    √ Internal functions can not be called externally (2591ms)
    √ Try to send wrong proposal id to the voting/cancel functions (1798ms)

  Contract: Reputation
    1) "before all" hook: prepare suite

  Contract: SchemeRegistrar
    2) "before all" hook: prepare suite

  Contract: SimpleICO
    3) "before all" hook: prepare suite

  Contract: TokenCapGC
    4) "before all" hook: prepare suite

  Contract: UpgradeScheme
    5) "before all" hook: prepare suite

  Contract: VestingScheme
    6) "before all" hook: prepare suite

  Contract: VoteInOrganizationScheme
    7) "before all" hook: prepare suite

  137 passing (4m)
  7 failing

  1) Contract: Reputation "before all" hook: prepare suite:
     Error: Could not connect to your Ethereum client. Please check that your Ethereum client:
    - is running
    - is accepting RPC connections (i.e., "--rpc" option is used in geth)
    - is accessible over the network
    - is properly configured in your Truffle configuration file (truffle.js)

      at ProviderError.ExtendableError (C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:9401:17)
      at new ProviderError (C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:325054:24)
      at C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:325137:17
      at C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:325195:24
      at XMLHttpRequest.request.onreadystatechange (C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:328229:7)
      at XMLHttpRequestEventTarget.dispatchEvent (C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:176415:18)
      at XMLHttpRequest._setReadyState (C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:176705:12)
      at XMLHttpRequest._onHttpRequestError (C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:176895:12)
      at ClientRequest.<anonymous> (C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:176765:24)

  2) Contract: SchemeRegistrar "before all" hook: prepare suite:
     Error: Could not connect to your Ethereum client. Please check that your Ethereum client:
    - is running
    - is accepting RPC connections (i.e., "--rpc" option is used in geth)
    - is accessible over the network
    - is properly configured in your Truffle configuration file (truffle.js)

      at ProviderError.ExtendableError (C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:9401:17)
      at new ProviderError (C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:325054:24)
      at C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:325137:17
      at C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:325195:24
      at XMLHttpRequest.request.onreadystatechange (C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:328229:7)
      at XMLHttpRequestEventTarget.dispatchEvent (C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:176415:18)
      at XMLHttpRequest._setReadyState (C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:176705:12)
      at XMLHttpRequest._onHttpRequestError (C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:176895:12)
      at ClientRequest.<anonymous> (C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:176765:24)

  3) Contract: SimpleICO "before all" hook: prepare suite:
     Error: Could not connect to your Ethereum client. Please check that your Ethereum client:
    - is running
    - is accepting RPC connections (i.e., "--rpc" option is used in geth)
    - is accessible over the network
    - is properly configured in your Truffle configuration file (truffle.js)

      at ProviderError.ExtendableError (C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:9401:17)
      at new ProviderError (C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:325054:24)
      at C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:325137:17
      at C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:325195:24
      at XMLHttpRequest.request.onreadystatechange (C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:328229:7)
      at XMLHttpRequestEventTarget.dispatchEvent (C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:176415:18)
      at XMLHttpRequest._setReadyState (C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:176705:12)
      at XMLHttpRequest._onHttpRequestError (C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:176895:12)
      at ClientRequest.<anonymous> (C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:176765:24)

  4) Contract: TokenCapGC "before all" hook: prepare suite:
     Error: Could not connect to your Ethereum client. Please check that your Ethereum client:
    - is running
    - is accepting RPC connections (i.e., "--rpc" option is used in geth)
    - is accessible over the network
    - is properly configured in your Truffle configuration file (truffle.js)

      at ProviderError.ExtendableError (C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:9401:17)
      at new ProviderError (C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:325054:24)
      at C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:325137:17
      at C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:325195:24
      at XMLHttpRequest.request.onreadystatechange (C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:328229:7)
      at XMLHttpRequestEventTarget.dispatchEvent (C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:176415:18)
      at XMLHttpRequest._setReadyState (C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:176705:12)
      at XMLHttpRequest._onHttpRequestError (C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:176895:12)
      at ClientRequest.<anonymous> (C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:176765:24)

  5) Contract: UpgradeScheme "before all" hook: prepare suite:
     Error: Could not connect to your Ethereum client. Please check that your Ethereum client:
    - is running
    - is accepting RPC connections (i.e., "--rpc" option is used in geth)
    - is accessible over the network
    - is properly configured in your Truffle configuration file (truffle.js)

      at ProviderError.ExtendableError (C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:9401:17)
      at new ProviderError (C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:325054:24)
      at C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:325137:17
      at C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:325195:24
      at XMLHttpRequest.request.onreadystatechange (C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:328229:7)
      at XMLHttpRequestEventTarget.dispatchEvent (C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:176415:18)
      at XMLHttpRequest._setReadyState (C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:176705:12)
      at XMLHttpRequest._onHttpRequestError (C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:176895:12)
      at ClientRequest.<anonymous> (C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:176765:24)

  6) Contract: VestingScheme "before all" hook: prepare suite:
     Error: Could not connect to your Ethereum client. Please check that your Ethereum client:
    - is running
    - is accepting RPC connections (i.e., "--rpc" option is used in geth)
    - is accessible over the network
    - is properly configured in your Truffle configuration file (truffle.js)

      at ProviderError.ExtendableError (C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:9401:17)
      at new ProviderError (C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:325054:24)
      at C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:325137:17
      at C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:325195:24
      at XMLHttpRequest.request.onreadystatechange (C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:328229:7)
      at XMLHttpRequestEventTarget.dispatchEvent (C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:176415:18)
      at XMLHttpRequest._setReadyState (C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:176705:12)
      at XMLHttpRequest._onHttpRequestError (C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:176895:12)
      at ClientRequest.<anonymous> (C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:176765:24)

  7) Contract: VoteInOrganizationScheme "before all" hook: prepare suite:
     Error: Could not connect to your Ethereum client. Please check that your Ethereum client:
    - is running
    - is accepting RPC connections (i.e., "--rpc" option is used in geth)
    - is accessible over the network
    - is properly configured in your Truffle configuration file (truffle.js)

      at ProviderError.ExtendableError (C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:9401:17)
      at new ProviderError (C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:325054:24)
      at C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:325137:17
      at C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:325195:24
      at XMLHttpRequest.request.onreadystatechange (C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:328229:7)
      at XMLHttpRequestEventTarget.dispatchEvent (C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:176415:18)
      at XMLHttpRequest._setReadyState (C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:176705:12)
      at XMLHttpRequest._onHttpRequestError (C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:176895:12)
      at ClientRequest.<anonymous> (C:\Users\Alon\AppData\Roaming\nvm\v8.9.3\node_modules\truffle\build\cli.bundled.js:176765:24)

Repo: https://github.com/daostack/daostack

I am running Windows 10 so this doesn't seem related to OS

For me it happens around 50% of the time locally and also happens during Travis build.

cgewecke commented 6 years ago

This issue has been open for a while and looks a lot like the non-deterministic ganache bug which @benjamincburns fixed recently. Downloading the latest ganache-cli (read the release notes!) should resolve this problem.

Closing but if anyone continues to see this error please re-open or comment. Thanks for reporting.

nickjm commented 6 years ago

I'm getting this error message when connecting to Rinkeby with Infura, truffle-wallet-provider, and ethereumjs-wallet. I'm not convinced this is Ganache, but maybe my error is caused my something else.

cgewecke commented 6 years ago

@nickjm Could you provide context (what are you doing in your code) or a reproduction path? A stacktrace might also be helpful.

hickscorp commented 6 years ago

I have this issue around 10% of my builds, with 128 tests active, randomly. The tests are running on the default dummy node (Ganache?), with the latest version of Truffle. Interestingly enough, it only fails when built in our CI (Drone, dockerized), not locally. Our environments are based on Ubuntu and use standard docker NodeJS:Carbon images.

barakman commented 6 years ago

Same problem on Windows 10:

  1) Contract: ...
     AssertionError: Could not connect to your Ethereum client. Please check that your Ethereum client:
    - is running
    - is accepting RPC connections (i.e., "--rpc" option is used in geth)
    - is accessible over the network
    - is properly configured in your Truffle configuration file (truffle.js)

      at Context.it (...)
      at <anonymous>
      at process._tickCallback (internal/process/next_tick.js:188:7)

  2) Contract: ... "after each" hook: after test for "...":
     Error: Could not connect to your Ethereum client. Please check that your Ethereum client:
    - is running
    - is accepting RPC connections (i.e., "--rpc" option is used in geth)
    - is accessible over the network
    - is properly configured in your Truffle configuration file (truffle.js)

      at ProviderError.ExtendableError (C:\Users\...\AppData\Roaming\npm\node_modules\truffle\build\webpack:\~\truffle-error\index.js:10:1)
      at new ProviderError (C:\Users\...\AppData\Roaming\npm\node_modules\truffle\build\webpack:\~\truffle-provider\error.js:17:1)
      at C:\Users\...\AppData\Roaming\npm\node_modules\truffle\build\webpack:\~\truffle-provider\wrapper.js:71:1
      at C:\Users\...\AppData\Roaming\npm\node_modules\truffle\build\webpack:\~\truffle-provider\wrapper.js:129:1
      at XMLHttpRequest.request.onreadystatechange (C:\Users\...\AppData\Roaming\npm\node_modules\truffle\build\webpack:\~\web3\lib\web3\httpprovider.js:128:1)
      at XMLHttpRequestEventTarget.dispatchEvent (C:\Users\...\AppData\Roaming\npm\node_modules\truffle\build\webpack:\~\xhr2\lib\xhr2.js:64:1)
      at XMLHttpRequest._setReadyState (C:\Users\...\AppData\Roaming\npm\node_modules\truffle\build\webpack:\~\xhr2\lib\xhr2.js:354:1)
      at XMLHttpRequest._onHttpRequestError (C:\Users\...\AppData\Roaming\npm\node_modules\truffle\build\webpack:\~\xhr2\lib\xhr2.js:544:1)
      at ClientRequest.<anonymous> (C:\Users\...\AppData\Roaming\npm\node_modules\truffle\build\webpack:\~\xhr2\lib\xhr2.js:414:1)
      at Socket.socketErrorListener (_http_client.js:387:9)
      at emitErrorNT (internal/streams/destroy.js:64:8)
      at _combinedTickCallback (internal/process/next_tick.js:138:11)
      at process._tickCallback (internal/process/next_tick.js:180:9)
barakman commented 6 years ago

@cgewecke: Any news on this?

cgewecke commented 6 years ago

@barakman No, everyone's seeing it intermittently, AFAIK it predates recent work to stabilize the test client and it's possible that it's related to issue 453 at ganache-cli which is under bounty and currently being worked on. There's more detail over there if you're interested.

Unfortunately this looks like (as @benjamincburns would say) a heisenbug. If anyone finds a consistent way of reproducing it they will be greeted with delight.

barakman commented 6 years ago

@cgewecke: Thank you for the info. P.S.: To me, heisenbug is kind of instinctively associated with Heisenberg (Uncertainty Principle's Heisenberg, not Breaking Bad's Heisenberg). More specifically, something like - "if you touch it, you change it", or in other words - this bug is unsolvable by definition, which I sincerely hope is not the case here... Or perhaps you were referring to the bug being uncertain, not the actual solution (and perhaps I went slightly off track with my "associativity")...

In my case, I see this occurring at random indeed, but - and this is actually very deterministic - in only 1 out of 23 scripts which truffle test executes. I've been banging my head for a while, trying to figure out in what sense this script is different from all the others, which could explain this. But I have failed to find any notable difference, so I cannot think of any sustainable workaround.

Thanks

cgewecke commented 6 years ago

@barakman Ahhh!! That's pretty good isolation of the problem. It could be something in the script you identified. Could also be related to the test execution that precedes it.

If there's any possibility of having another set of eyes look at the codebase, we'd definitely be interested.

barakman commented 6 years ago

@cgewecke: Most likely not related to the preceding script, as I also tried to truffle test this script alone, and the problem persisted. I'm not sure that I'm allowed to post the code publicly at this point, so I will try to minimize both the Solidity contract and the Javascript test, and if I am still able to reproduce the problem then I will post it here. Another interesting point of attention, is the fact that I haven't seen it happening when using testrpc-sc instead of ganache-cli. So the bug can probably be reduced to the differences between those two (and also, we can probably eliminate the chances of it being related to the truffle suite itself or to mocha).

cgewecke commented 6 years ago

@barakman Ok great, thank you. One thing about testrpc-sc is that it's much slower. If the underlying issue is a race condition, testrpc-sc could be introducing delays that mask it. Another important difference is that it uses an older version of ethereumjs-vm (from the fall). Possible clue there too, although have been seeing these disconnections all year.

barakman commented 6 years ago

@cgewecke: I can consistently reproduce the problem with the code below.

MyContract.sol:

pragma solidity ^0.4.18;

contract MyContract {
    uint256 public constant ONE = 1000000000;
    uint256 public constant GAMMA = 179437500000000000000000000000000000000000;
    uint256 public constant DELTA = 29437500;

    function buy(uint256 x, uint256 y, uint256 alpha, uint256 beta) external pure returns (uint256) {
        uint256 temp = alpha - beta * y;
        return x * (temp * ONE) / (temp * (ONE - DELTA) + GAMMA);
    }

    function sell(uint256 x, uint256 y, uint256 alpha, uint256 beta) external pure returns (uint256) {
        uint256 temp = alpha - beta * y;
        return x * (temp * (ONE + DELTA) - GAMMA) / (temp * ONE);
    }
}

MyContractUnitTest.js:

contract("MyContractUnitTest", () => {
    let interval = ["0", "20000000000000000000000000", "10000000000000000000000000000000000", "0"];
    let NUM_OF_TESTS_PER_INTERVAL = 10;

    function buyFunc(x, y, alpha, beta) {
        let temp = alpha.minus(beta.times(y));
        return x.times(temp.times(ONE)).dividedBy(temp.times(ONE.minus(DELTA)).plus(GAMMA));
    }

    function sellFunc(x, y, alpha, beta) {
        let temp = alpha.minus(beta.times(y));
        return x.times(temp.times(ONE.plus(DELTA)).minus(GAMMA)).dividedBy(temp.times(ONE));
    }

    async function buy(x, y, alpha, beta) {
        let fixedPoint = await myContract.buy(x, y, alpha, beta);
        let floatPoint = buyFunc(x, y, alpha, beta);
        return [fixedPoint, floatPoint];
    }

    async function sell(x, y, alpha, beta) {
        let fixedPoint = await myContract.sell(x, y, alpha, beta);
        let floatPoint = sellFunc(x, y, alpha, beta);
        return [fixedPoint, floatPoint];
    }

    let myContract;
    let ONE;
    let GAMMA;
    let DELTA;
    before(async () => {
        myContract = await artifacts.require("MyContract.sol").new();
        ONE = await myContract.ONE();
        GAMMA = await myContract.GAMMA();
        DELTA = await myContract.DELTA();
    });

    let AMOUNT = web3.toBigNumber(1000000);

    for (let func of [buy, sell]) {
        describe(`${func.name}:`, async () => {
            for (let row = 0; row < 100; row++) {
                for (let col = 0; col < 10; col++) {
                    let [minN, maxN, alpha, beta] = interval.map(x => web3.toBigNumber(x));
                    let incN = maxN.minus(minN).dividedBy(NUM_OF_TESTS_PER_INTERVAL - 1);
                    for (let i = 0; i < NUM_OF_TESTS_PER_INTERVAL; i++) {
                        let y = minN.plus(incN.times(i)).truncated();
                        it(`interval ${row} ${col}, test ${i}`, async () => {
                            try {
                                let [fixedPoint, floatPoint] = await func(AMOUNT, y, alpha, beta);
                                let ratio = fixedPoint.dividedBy(floatPoint);
                                assert(ratio.greaterThanOrEqualTo("0.99999") && ratio.lessThanOrEqualTo("1"), `ratio = ${ratio.toFixed()}`);
                            }
                            catch (error) {
                                assert(false, error.message);
                            }
                        });
                    }
                }
            }
        });
    }
});

My setup (as mentioned in a previous comment on this thread) is:

Thanks

cgewecke commented 6 years ago

@barakman Great!! Thank you. Going to open a companion issue over at ganache-cli and will talk to @benjamincburns about this and get his input.

barakman commented 6 years ago

@cgewecke: Also happens on testrpc-sc if you try "hard enough"...

barakman commented 6 years ago

@cgewecke : I am circulating around the conclusion that this problem stems from improper usage of the Mocha framework. More precisely, improper usage of the before, beforeEach, afterEach and after hooks.

The typical error messages, although poorly phrased, imply this conjecture as well:

You can read the relevant information here, namely:

You may also pick any file and add “root”-level hooks. For example, add beforeEach() outside of all describe() blocks. This will cause the callback to beforeEach() to run before any test case, regardless of the file it lives in.

So using these hooks on the root-level might be a bad idea in this case, since due to the nature of the tested system (communication with a TestRPC or Ganache process), they typically execute asynchronous code.

cgewecke commented 6 years ago

@barakman Agree it seems like disconnections happen at the 'seams' of the suites where the hooks are. The code at truffle-core here that sets up the 'contract' suite looks to me like it safely binds hooks within a describe and executes them asynchronously via Mocha's done callback. Do you see another place where they might be invoked?

barakman commented 6 years ago

@cgewecke: I have removed all my root-level hooks, and i'm still encountering disconnections. However, in opposed to before, the errors that I am getting are always "after each" hook.... And to be absolutely clear on that, I don't even have an afterEach hook in any of my tests! Maybe Mocha adds implicit calls to afterEach, when an explicit call to beforeEach exists in the code. However, so long as these implicit calls are not added at the root-level, it does not support my conjecture of hooked code running in the wrong scope to begin with. In short, I'm at a loss here...

By the way, in the code that you linked, there doesn't seem to be support for the after hook, and I am using this hook in my tests.

cgewecke commented 6 years ago

@barakman Hmmm....that's a nice observation about after - I wonder if that's causing problems.

barakman commented 6 years ago

@cgewecke: Thanks. I removed all the after hooks in my code (replaced them with its), and now I'm getting a "before each" hook... error. So I'm suspecting that one way or another - these hooks are not incorporated properly into Truffle. In order to bring an empirical evidence for this conjecture, I am now trying the following workaround:

If all tests pass without disconnections, then I'm pretty sure that we can stamp this as the cause of the problem. And even if not, I think that it could still be related to hooks which are added implicitly by the Mocha framework.

barakman commented 6 years ago

@cgewecke:

I take back my previous conjecture of this error occurring on a given test as a result of something which has executed on a previous test. This is because the error occurs when I execute truffle test separately for each test file (i.e., running truffle test test/SomeFile.js sequentially for each JS file). Moreover, in between calls to truffle test, I close and reopen the ganache-cli process. So this error cannot be related to any previous state stored by either truffle test or ganache-cli (unless one of them saves some "global information" in the operating-system's temp folder or something like that, which I sincerely doubt).

In addition to that, I have recently tested the new ganache-cli beta version (7.0.0-beta.0), where the problem persists.

I have posted my findings on a similar GIT thread which is closed by now, but I am hoping will reopen.

Thanks

barakman commented 6 years ago

@cgewecke:

I have conducted a more extensive research, by modifying file /node_modules/truffle/build/cli.bundled.js.

I started off by checking which path leads to the NOT_CONNECTED_MESSAGE error message ('Invalid JSON RPC response: ""'):

As expected, this error occurs only in the asynchronous path.

Second, I added some logging in this path, just before invoking callback(error, result):

if (payload.params == undefined)
    console.log(result.id, payload.method, 'no params');
else
    console.log(result.id, payload.method, payload.params.length);

Here is the consistency that I have observed:

  1. During normal execution:
    • result.id starts from a very large value, and grows on each evm operation, for example:
      • id = 1525799814567, method = evm_revert
      • id = 1525799814727, method = evm_snapshot
    • result.id starts from 1 and increments by 1 on each net or eth operation, for example:
      • id = 1, method = net_version
      • id = 2, method = eth_sendTransaction
      • id = 3, method = eth_getTransactionReceipt
      • id = 4, method = eth_call
      • id = 5, method = eth_newBlockFilter
      • id = 6, method = eth_getCode
      • id = 7, method = eth_blockNumber
      • id = 8, method = eth_uninstallFilter
    • In some cases, both result.id and payload.method are undefined. When that happens, result.id increments by 2 on the next net or eth operation, for example:
      • id = 12, method = eth_blockNumber
      • id = undefined, method = undefined
      • id = 14, method = eth_uninstallFilter
  2. Right before the error occurs:
    • In the last one or two cases, result.id is undefined but payload.method is valid.
    • In the last case, payload.method is sometimes eth_getLogs for the first time in the entire test.
    • Here are several examples for that:
      • Test 1 ended with:
      • id = undefined, method = net_version
      • id = 1525791759176, method = eth_getLogs
      • Test 2 ended with:
      • id = undefined, method = eth_call
      • id = undefined, method = eth_getLogs
      • Test 3 ended with:
      • id = undefined, method = eth_blockNumber
      • Test 4 ended with:
      • id = undefined, method = eth_call
      • id = undefined, method = eth_getLogs
      • Test 5 ended with:
      • id = undefined, method = eth_blockNumber

I am hopeful that the above information will provide some clues towards the source of this problem. And since I am able to reproduce it repeatedly, I will be happy to generate more logging in case you have any specific requests.

Thanks

barakman commented 6 years ago

@cgewecke:

Update to the above:

I later realized that:

I therefore changed the logging as follows:

if (result.id != undefined)
    console.log(`Normal: id = ${result.id}, method = ${payload.method}`);
else if (result[0] != undefined)
    console.log(`In [0]: id = ${result[0].id}, method = ${payload[0].method}`);
else
    console.log(`Problem: result = ${result}, payload = ${JSON.stringify(payload, null, 4)}`);

The new logging has improved my previous observation from this:

result.id = undefined
payload.method = undefined

To this:

result[0].id = <the previous id + 1>
payload[0].method = eth_getFilterChanges

All of that, during normal execution of course.

What happens right before the error can be described as follows: In the last one or two cases, the result is empty. The contents which the result normally holds:

Is it possible that somewhere in ganache-cli code, the payload is initialized but the result is not?

That could most certainly be classified as an "Invalid JSON RPC response" (which you guys can easily resolve).

Thanks

barakman commented 6 years ago

Some more observations:

When the last two result objects are empty:

  1. The last payload object always contains:
    • method = eth_getLogs
    • id = <some very large number>
  2. The error reported by truffle test is always "after each" hook: after test for...

When only the last result object are empty:

  1. The last payload object always contains:
    • method != eth_getLogs
    • id = <the previous id + 1>
  2. The error reported by truffle test is either "before each" hook: before test for... or No events were emitted
cgewecke commented 6 years ago

@barakman Thank you. . . this analysis is really helpful.

Is it possible that somewhere in ganache-cli code, the payload is initialized but the result is not?

@benjamincburns Does anything jump out at you as a possibility in the preceding three comments?

barakman commented 6 years ago

@cgewecke:

I've been doing my research only on truffle side, thinking I wasn't able to do much on ganache side because all the code inside /node_modules/ganache appeared "compacted".

But then I realized that I could go directly to the source code on GitHub, so I'm now looking there too trying to connect the dots...

I'm doing it without logging this time, because I would essentially need to reinstall it (npm) after every change, so too much hustle...

Thank you!

barakman commented 6 years ago

@cgewecke:

BTW, preliminary observation in ganache code:

It seems that the jsonrpc field is mostly set to "2.0" (string) and rarely set to 2.0 (double). Perhaps it doesn't make any difference when sent from the ganache process to the truffle process, but I would nevertheless make it consistent across the project.

barakman commented 6 years ago

@cgewecke:

A question about line 100 in file /ganache-core-develop/lib/provider.js:

callback(response.error ? err : null, response);

Looking a few lines above it, the variable response is either an object or an array of objects.

When it is an array of objects, I believe that response.error is undefined, hence the expression response.error ? err : null evaluates to null.

Is this really the desired behavior in this case?

Thanks

cgewecke commented 6 years ago

@barakman Just talked to @benjamincburns about this and he thinks it's ok. Have published an experimental truffle build which removes the eth_getLogs attempt in truffle test. If you have a chance could try running your suite with it and seeing if it makes any difference? Maybe we can isolate this problem to that call.

npm install -g darq-truffle@barakman
darq-truffle test # Example command
barakman commented 6 years ago

@cgewecke: I will do so, thank you very much!!! But as I mentioned above, I must also remind you that in some failure cases, the eth_getLogs method was not present in the erroneous payload (i.e, the payload which came along with an empty result did not contain this method).

cgewecke commented 6 years ago

@barakman Yes, I'm hoping that that's because the calls are getting queued over there or something. This is kind of a shot in the dark with a low likely-hood of success sorry.

barakman commented 6 years ago

@cgewecke:

My contracts are at Solidity 0.4.18, and my Truffle is subsequently at v4.13.

Can I use my local Truffle for compile and your proposed Truffle (which I have installed globally as you suggested) for test, or is it not likely to work?

cgewecke commented 6 years ago

@barakman Oh good question - the compiler is 0.4.23. If your pragmas use a caret you should have no problem. But if there's an issue let me know and I will republish with a build that lets you use any compiler you want.

barakman commented 6 years ago

@cgewecke: In any case, the problem persists. Here's the error message from (your) truffle:


[0m[31m     Error: Could not connect to your Ethereum client. Please check that your Ethereum client:
    - is running
    - is accepting RPC connections (i.e., "--rpc" option is used in geth)
    - is accessible over the network
    - is properly configured in your Truffle configuration file (truffle.js)
[0m[90m
      at ProviderError.ExtendableError (C:\Users\...\webpack:\dependencies\truffle-error\index.js:10:1)
      at new ProviderError (C:\Users\...\webpack:\dependencies\truffle-provider\error.js:17:1)
      at C:\Users\...\webpack:\dependencies\truffle-provider\wrapper.js:71:1
      at C:\Users\...\webpack:\dependencies\truffle-provider\wrapper.js:129:1
      at XMLHttpRequest.request.onreadystatechange (C:\Users\...\webpack:\dependencies\truffle-provider\~\web3\lib\web3\httpprovider.js:128:1)
      at XMLHttpRequestEventTarget.dispatchEvent (C:\Users\...\webpack:\dependencies\truffle-provider\~\xhr2\lib\xhr2.js:64:1)
      at XMLHttpRequest._setReadyState (C:\Users\...\webpack:\dependencies\truffle-provider\~\xhr2\lib\xhr2.js:354:1)
      at XMLHttpRequest._onHttpRequestError (C:\Users\...\webpack:\dependencies\truffle-provider\~\xhr2\lib\xhr2.js:544:1)
      at ClientRequest.<anonymous> (C:\Users\...\webpack:\dependencies\truffle-provider\~\xhr2\lib\xhr2.js:414:1)
      at Socket.socketErrorListener (_http_client.js:387:9)
      at emitErrorNT (internal/streams/destroy.js:64:8)
      at _combinedTickCallback (internal/process/next_tick.js:138:11)
      at process._tickCallback (internal/process/next_tick.js:180:9)
[0m

Note that there is no "after each" hook: after test for... in this case, which from previous empiric experience tells us that the last payload's method was not eth_getLogs.

But then again, it couldn't have been eth_getLogs anyway, since you've removed it, right?

I could run a few more tests to make sure that the error message never contains the "after each" hook: after test for... part. It wouldn't really provide a great deal of insight, but perhaps it might indicate (if indeed the case, of course) that one path out of several paths which lead to this error has been resolved.

Thanks

cgewecke commented 6 years ago

@barakman Was just looking at your original reproduction case and noticed that the describe is async. Mocha doesn't support async for describe unfortunately. If you have a chance could you check and see if that occurs anywhere in the suite you're running? You can just change those to regular function to make sure they're running/resolving in strict sequence.

barakman commented 6 years ago

@cgewecke: I have already changed that in all of my tests a while ago. I have also posted a simplified version of this reproduction case quite recently, in which the describe takes a non-async function. I posted it in order to inform benjamincburns that the problem persists with ganache-cli@7.0.0-beta.0. You can read it at the bottom of this thread.

barakman commented 6 years ago

@cgewecke:

I have added logging to your darq-truffle version:

sendAsync: function(originalSendAsync, preHook, postHook) {
    ...
    if (result.id == undefined && result[0] != undefined)
        console.log(`Problem: result = ${result}, payload = ${JSON.stringify(payload, null, 4)}`);
    callback(error, result);
    ...
}

Now, upon Could not connect to your Ethereum client error, the log output seems to be either one of the following cases:

  1. payload.method = eth_blockNumber, and the error message contains "before each" hook....
  2. payload.method = eth_call, and the error message does not contain any additional information.
  3. payload.method = net_version, and the error message does not contain any additional information.

As expected, there seems to be a slight reduction in the number of different failure scenarios, with the payload.method no longer being eth_getLogs, and the error message no longer containing "after each" hook....

So perhaps the next place to look into is the other 3 methods mentioned above - eth_blockNumber, eth_call and net_version.

I would always feel better finding and fixing a single source of a given problem, which I'm pretty sure is also the case here, but perhaps we should start off by eliminating each path that leads to the problem, and eventually trace the exact point of failure.

cgewecke commented 6 years ago

@barakman Agree - I'll look into how to disable those for debugging . . .

barakman commented 6 years ago

@cgewecke: The problem reproduces when I use GETH instead of Ganache.

I have also removed compiler optimization (see below) in order to rule out this factor as well:

solc: {
    optimizer: {
        enabled: false,
        runs: 5000000,
    }
}

Now, on the one hand, this implies that the problem is in Truffle. But on the other hand, I'm guessing that Ganache and GETH are possibly using the same code-base. So I'm not entirely confident that it is indeed Truffle to blame for.

barakman commented 6 years ago

@cgewecke:

A very important observation IMO:

  request.onreadystatechange = function () {
    if (request.readyState === 4 && request.timeout !== 1) {
      var result = request.responseText;
      var error = null;

      try {
        result = JSON.parse(result);
      } catch (e) {
        error = errors.InvalidResponse(request.responseText);
      }

      callback(error, result);
    }
  };

As long as request.status == 200, the responseText is a valid JSON RPC.

Once request.status == 0, the responseText is empty (hence an invalid JSON RPC).

Most likely, you need to ensure both request.readyState === 4 and request.status === 200.

I think that you may as well get rid of the request.timeout !== 1 assertion while you're at it, but I'm not an HTTP expert so I'll leave that decision to you.

I have not been able to resolve all my problems by adding this condition, as truffle simply terminates without any errors at all.

However, I'm pretty sure that the solution is tightly related with this issue.

It also explains the non-deterministic behavior (Heisenbug in your words) that we've witnessed, as HTTP responses tend to behave this way (in particularly with the regards to the partially-duplicated information embedded in the readyState and status fields).

cgewecke commented 6 years ago

@barakman Locating the problem here is a huge discovery, thank you. Ganache and Geth are very far apart from a code-base standpoint so I think this means the issue is very likely on this side. The code you've referenced is at web3. . .

Link for future reference

barakman commented 6 years ago

@cgewecke:

But I see it in /node_modules/truffle/build/cli.bundled.js.

Are you "linking" Web3 code as part of Truffle's "build" process?

If so, how can we wrap things up on this issue?

i.e., should I post this on Web3's GitHub?

Thanks!

cgewecke commented 6 years ago

@barakman We bundle all of our dependencies together and one is web3 - I'm just noting the location so we know where to investigate. I'm not sure about what to do... I'd guess I'd like to look at the values coming through that block and understand this better. What states trigger this crash?

As long as request.status == 200, the responseText is a valid JSON RPC.

Once request.status == 0, the responseText is empty (hence an invalid JSON RPC).

Most likely, you need to ensure both request.readyState === 4 and request.status === 200.

barakman commented 6 years ago

@cgewecke:

If I understand your question of 'What states trigger this crash?' correctly, then the code that I've used in order to trigger this crash is given below (please let me know if that's not what you meant).

Using Truffle + GETH, the problem is actually reproduced very consistently, at around test 540 (though it may be affected by GETH verbosity since it is highly timing-dependent).

On-Chain:

pragma solidity ^0.4.18;

contract MyContract {
    uint256 public constant ONE = 1000000000;
    uint256 public constant GAMMA = 179437500000000000000000000000000000000000;
    uint256 public constant DELTA = 29437500;
    uint256 public constant AMOUNT = 1000000;

    function buy(uint256 x, uint256 alpha, uint256 beta) external pure returns (uint256) {
        uint256 temp = alpha - beta * x;
        return AMOUNT * (temp * ONE) / (temp * (ONE - DELTA) + GAMMA);
    }

    function sell(uint256 x, uint256 alpha, uint256 beta) external pure returns (uint256) {
        uint256 temp = alpha - beta * x;
        return AMOUNT * (temp * (ONE + DELTA) - GAMMA) / (temp * ONE);
    }
}

Off-Chain:

contract("MyContractTest", function() {
    let hMyContract;
    let ONE;
    let GAMMA;
    let DELTA;
    let AMOUNT;

    let NUM_OF_TESTS = 10;

    let minN = web3.toBigNumber("0");
    let maxN = web3.toBigNumber("20000000000000000000000000");
    let alpha = web3.toBigNumber("10000000000000000000000000000000000");
    let beta = web3.toBigNumber("0");
    let incN = maxN.minus(minN).dividedBy(NUM_OF_TESTS - 1);

    describe("accuracy assertion:", function() {
        before(async function() {
            hMyContract = await artifacts.require("MyContract.sol").new();
            ONE = await hMyContract.ONE();
            GAMMA = await hMyContract.GAMMA();
            DELTA = await hMyContract.DELTA();
            AMOUNT = await hMyContract.AMOUNT();
        });
        for (let func of [buy, sell]) {
            for (let n = 0; n < 1000; n++) {
                for (let i = 0; i < NUM_OF_TESTS; i++) {
                    it(`${func.name} test ${n} ${i}`, async function() {
                        let x = minN.plus(incN.times(i)).truncated();
                        let [fixedPoint, floatPoint] = await func(x, alpha, beta);
                        let ratio = fixedPoint.dividedBy(floatPoint);
                        assert(ratio.greaterThanOrEqualTo("0.99999"), `ratio = ${ratio.toFixed()}`);
                        assert(ratio.lessThanOrEqualTo("1"), `ratio = ${ratio.toFixed()}`);
                    });
                }
            }
        }
    });

    function buyFunc(x, alpha, beta) {
        let temp = alpha.minus(beta.times(x));
        return AMOUNT.times(temp.times(ONE)).dividedBy(temp.times(ONE.minus(DELTA)).plus(GAMMA));
    }

    function sellFunc(x, alpha, beta) {
        let temp = alpha.minus(beta.times(x));
        return AMOUNT.times(temp.times(ONE.plus(DELTA)).minus(GAMMA)).dividedBy(temp.times(ONE));
    }

    async function buy(x, alpha, beta) {
        let fixedPoint = await hMyContract.buy(x, alpha, beta);
        let floatPoint = buyFunc(x, alpha, beta);
        return [fixedPoint, floatPoint];
    }

    async function sell(x, alpha, beta) {
        let fixedPoint = await hMyContract.sell(x, alpha, beta);
        let floatPoint = sellFunc(x, alpha, beta);
        return [fixedPoint, floatPoint];
    }
});

Thanks

cgewecke commented 6 years ago

@barakman Excellent, thank you!

barakman commented 6 years ago

@cgewecke:

Side note:

I'm not sure how your bundling procedure works and how exactly you integrate Web3 into Truffle, but the HttpProvider.prototype.sendAsync function appears 4 times, out of which, 3 cases implement timeout handling (asserting request.timeout !== 1 and setting request.ontimeout), and 1 case doesn't.

I don't think that this is directly related to the problem at hand, but it might be implying that you are somehow bundling two different versions of Web3.

barakman commented 6 years ago

@cgewecke:

Some more observations, and a partial understanding of the problem:

I conducted a comparison between a successful HTTP request (valid responseText) and an unsuccessful HTTP request (empty responseText).

Now, the request object is rather large (let alone circular), so it is quite difficult to do a thorough comparison, but I did notice that the "bad" request contained the following (which the "good" request didn't):

{
    Error: connect EADDRINUSE 127.0.0.1:8545
        at Object._errnoException (util.js:1024:11)
        at _exceptionWithHostPort (util.js:1046:20)
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1182:14)
            [stack]: 'Error: connect EADDRINUSE 127.0.0.1:8545\n    at Object._errnoException (util.js:1024:11)\n    at _exceptionWithHostPort (util.js:1046:20)\n    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1182:14)',
            [message]: 'connect EADDRINUSE 127.0.0.1:8545',
            code: 'EADDRINUSE',
            errno: 'EADDRINUSE',
            syscall: 'connect',
            address: '127.0.0.1',
            port: 8545
}

The EADDRINUSE error means that the ip port (127.0.0.1:8545 in this case) is currently busy, which made me think that perhaps this is not exactly a coding bug on either side.

I then checked the status of this port via (Windows command) netstat -aon | find "8545" and found something interesting:

As the test continues, more and more ports on 127.0.0.1 seem to be in use, waiting for something.

I believe that each one of them is waiting for a response on an HTTP request sent (asynchronously) to 127.0.0.1:8545.

When an unsuccessful HTTP request finally takes place, it seems that the highest port (65535) on 127.0.0.1 is waiting.

Here is what it looks like on my system:

First this:

  TCP    127.0.0.1:8545         0.0.0.0:0              LISTENING       1820
  TCP    127.0.0.1:49262        127.0.0.1:8545         TIME_WAIT       0
  TCP    127.0.0.1:49263        127.0.0.1:8545         TIME_WAIT       0
  TCP    127.0.0.1:49264        127.0.0.1:8545         TIME_WAIT       0
  ...
  TCP    127.0.0.1:49275        127.0.0.1:8545         TIME_WAIT       0

Then this:

  TCP    127.0.0.1:8545         0.0.0.0:0              LISTENING       1820
  TCP    127.0.0.1:8545         127.0.0.1:49513        CLOSE_WAIT      1820
  TCP    127.0.0.1:8545         127.0.0.1:49514        ESTABLISHED     1820
  TCP    127.0.0.1:49262        127.0.0.1:8545         TIME_WAIT       0
  TCP    127.0.0.1:49263        127.0.0.1:8545         TIME_WAIT       0
  TCP    127.0.0.1:49264        127.0.0.1:8545         TIME_WAIT       0
  ... // mostly consecutive but not always
  TCP    127.0.0.1:49512        127.0.0.1:8545         TIME_WAIT       0
  TCP    127.0.0.1:49513        127.0.0.1:8545         FIN_WAIT_2      15044
  TCP    127.0.0.1:49514        127.0.0.1:8545         ESTABLISHED     15044

Then this:

  TCP    127.0.0.1:8545         0.0.0.0:0              LISTENING       1820
  TCP    127.0.0.1:8545         127.0.0.1:58487        CLOSE_WAIT      1820
  TCP    127.0.0.1:8545         127.0.0.1:58488        ESTABLISHED     1820
  TCP    127.0.0.1:49262        127.0.0.1:8545         TIME_WAIT       0
  TCP    127.0.0.1:49263        127.0.0.1:8545         TIME_WAIT       0
  TCP    127.0.0.1:49264        127.0.0.1:8545         TIME_WAIT       0
  ... // mostly consecutive but not always
  TCP    127.0.0.1:58486        127.0.0.1:8545         TIME_WAIT       0
  TCP    127.0.0.1:58487        127.0.0.1:8545         FIN_WAIT_2      15044
  TCP    127.0.0.1:58488        127.0.0.1:8545         ESTABLISHED     15044

Then this:

  TCP    127.0.0.1:8545         0.0.0.0:0              LISTENING       1820
  TCP    127.0.0.1:49262        127.0.0.1:8545         TIME_WAIT       0
  TCP    127.0.0.1:49263        127.0.0.1:8545         TIME_WAIT       0
  TCP    127.0.0.1:49264        127.0.0.1:8545         TIME_WAIT       0
  ... // mostly consecutive but not always
  TCP    127.0.0.1:61586        127.0.0.1:8545         TIME_WAIT       0
  TCP    127.0.0.1:61587        127.0.0.1:8545         TIME_WAIT       0
  TCP    127.0.0.1:61588        127.0.0.1:8545         ESTABLISHED     15044

And finally this:

  TCP    127.0.0.1:8545         0.0.0.0:0              LISTENING       1820
  TCP    127.0.0.1:49152        127.0.0.1:8545         TIME_WAIT       0
  TCP    127.0.0.1:49153        127.0.0.1:8545         TIME_WAIT       0
  TCP    127.0.0.1:49156        127.0.0.1:8545         TIME_WAIT       0
  ... // mostly consecutive but not always
  TCP    127.0.0.1:65533        127.0.0.1:8545         TIME_WAIT       0
  TCP    127.0.0.1:65534        127.0.0.1:8545         TIME_WAIT       0
  TCP    127.0.0.1:65535        127.0.0.1:8545         TIME_WAIT       0

Funny enough, even at this point, there still seem to be plenty of available ports between 0 and 65535 (more notably between 0 and 49000), but they don't seem to be used for some reason.

In any case, once the truffle test process terminates, those waiting ports seem to "clean up" gradually, until only the listening port (8545) remains.

That is when I can execute truffle test once again (without shutting down the Ethereum client, GETH in this case).

So perhaps the solution to this problem is in Truffle - maintain a counter of the open requests, and whenever it reaches a certain threshold, delay before submitting the next request.

Of course, I could also do this in my tests (i.e., Truffle input) but I don't think that's an appropriate solution.

Thanks!

cgewecke commented 6 years ago

@barakman RE: duplicate bundling - yes, that's likely happening. Web3 is widespread and not consistently versioned - have fixed this for the work we're doing on Truffle V5.

Can you tell approximately what the cap on the number of requests is? ~15,000?

barakman commented 6 years ago

@cgewecke:

This specific test starts off with 1 before call, which executes 5 asynchronous web3 calls. It then proceeds with 20,000 it calls, each of which executing 1 asynchronous web3 call. AFAIK (and you can probably verify this in Truffle), each web3 call yields 1 HTTP request to the Ethereum client (Ganache or GETH in this case).

On average, the test fails after 5,400 it calls or so. Hence if my assumption above is correct, then we're not talking about more than 5,000 pending requests.

Of course, the info that I provided on the previous comment does suggest ~15,000 requests, so I will run this again to make sure. I suppose that there were a lot of gaps (idle ports) in between 49152 and 65535.

Is it possible that you're not closing requests properly on Truffle. AFAIK there's no need to do that, but I'm not an HTTP expert so just bringing it up...

Thanks.

barakman commented 6 years ago

@cgewecke: The exact number of busy ports immediately upon failure is 16353. BTW, I just noticed that between 49152 and 65535 there are exactly 16384 ports, which also happens to be equal to 2^14. Not sure if it provides any additional insight though...

In any case, this means that there are HTTP requests issued by Truffle, which are not directly related to the test. Alternatively, my assumption of 1 request per web3 call is incorrect, and there are something like 3 or more requests per web3 call.