holiman / goevmlab

Evm laboratory
GNU Lesser General Public License v3.0
245 stars 45 forks source link

neth (batch mode) OOM after a long time #173

Open holiman opened 6 days ago

holiman commented 6 days ago

Ran into this crash with the fuzzer:

INFO [10-26|06:48:54.470] Executing                                tests=95,043,214 time=359h0m40.000s    test/s=73.5 "avg steps"=2944.6 global=95,043,214
INFO [10-26|06:48:54.471] Stats gethbatch-0                        execSpeed=27.4ms  longest=25.518789124s    count=35,736,104
INFO [10-26|06:48:54.471] Stats eelsbatch-0                        execSpeed=196.1ms longest=14m21.197711271s count=3,158,778
INFO [10-26|06:48:54.471] Stats nethbatch-0                        execSpeed=27.6ms  longest=22.246265736s    count=42,054,730
INFO [10-26|06:48:54.471] Stats besubatch-0                        execSpeed=26.1ms  longest=15.834618767s    count=44,859,438
INFO [10-26|06:48:54.471] Stats erigonbatch-0                      execSpeed=59.8ms  longest=22.961329371s    count=16,689,147
INFO [10-26|06:48:54.471] Stats nimbus-0                           execSpeed=2.4228s longest=1m1.913436361s   count=536,270
INFO [10-26|06:48:54.471] Stats evmone-0                           execSpeed=21ms    longest=20.040833565s    count=42,968,246
INFO [10-26|06:48:54.471] Stats revm-0                             execSpeed=520ms   longest=5m45.291557981s  count=4,083,719
neth error: invalid character 'O' looking for beginning of value
  | Out of memory.
WARN [10-26|07:18:15.538] Slow test found                          evm=nethbatch-0   time=1m47.587881202s  cmd="/neth/nethtest -x --trace -m" file=/fuzztmp/10573375-mixed-8.json

Somewhere after 42M invocations, the nethermind instance went OOM (the testcase was nothing special). Possibly nethermind in batch mode has some memory leak.

TODO investigate.

file which potentially triggers the leak, 10573375-mixed-8.json:

{
  "10573375-mixed-8": {
    "env": {
      "currentCoinbase": "b94f5374fce5edbc8e2a8697c15331677e6ebf0b",
      "currentDifficulty": "0x200000",
      "currentRandom": "0x0000000000000000000000000000000000000000000000000000000000200000",
      "currentGasLimit": "0x26e1f476fe1e22",
      "currentNumber": "0x1",
      "currentTimestamp": "0x3e8",
      "previousHash": "0x044852b2a670ade5407e78fb2863c51de9fcb96542a07186fe3aeda6bb8a116d",
      "currentBaseFee": "0x10"
    },
    "pre": {
      "0x0000000000000000000000000000000000000000": {
        "code": "0x",
        "storage": {},
        "balance": "0x1",
        "nonce": "0x0"
      },
      "0x0000000000000000000000000000000000000001": {
        "code": "0x",
        "storage": {},
        "balance": "0x1",
        "nonce": "0x0"
      },
      "0x0000000000000000000000000000000000000002": {
        "code": "0x",
        "storage": {},
        "balance": "0x1",
        "nonce": "0x0"
      },
      "0x0000000000000000000000000000000000000003": {
        "code": "0x",
        "storage": {},
        "balance": "0x1",
        "nonce": "0x0"
      },
      "0x0000000000000000000000000000000000000004": {
        "code": "0x",
        "storage": {},
        "balance": "0x1",
        "nonce": "0x0"
      },
      "0x0000000000000000000000000000000000000005": {
        "code": "0x",
        "storage": {},
        "balance": "0x1",
        "nonce": "0x0"
      },
      "0x0000000000000000000000000000000000000006": {
        "code": "0x",
        "storage": {},
        "balance": "0x1",
        "nonce": "0x0"
      },
      "0x0000000000000000000000000000000000000007": {
        "code": "0x",
        "storage": {},
        "balance": "0x1",
        "nonce": "0x0"
      },
      "0x0000000000000000000000000000000000000008": {
        "code": "0x",
        "storage": {},
        "balance": "0x1",
        "nonce": "0x0"
      },
      "0x0000000000000000000000000000000000000009": {
        "code": "0x",
        "storage": {},
        "balance": "0x1",
        "nonce": "0x0"
      },
      "0x000000000000000000000000000000000000000a": {
        "code": "0x",
        "storage": {},
        "balance": "0x1",
        "nonce": "0x0"
      },
      "0x000000000000000000000000000000000000000b": {
        "code": "0x",
        "storage": {},
        "balance": "0x1",
        "nonce": "0x0"
      },
      "0x000000000000000000000000000000000000000c": {
        "code": "0x",
        "storage": {},
        "balance": "0x1",
        "nonce": "0x0"
      },
      "0x000000000000000000000000000000000000000d": {
        "code": "0x",
        "storage": {},
        "balance": "0x1",
        "nonce": "0x0"
      },
      "0x000000000000000000000000000000000000000e": {
        "code": "0x",
        "storage": {},
        "balance": "0x1",
        "nonce": "0x0"
      },
      "0x00000000000000000000000000000000000000f1": {
        "code": "0x600254507f60005450600954506006545060026001556002600055600260025560016001556000527f600954506001600355600354507f60006000600060006000600b5af250726a186020527f3d69008864038633623a669b3a8e6000527f0631655d0451f2433060006000606040527e6000600060f15af15020660261333673706020527f10ff60006000600060006060527f600060025af150fa9b746b60006000600060006000606040527f0b5af15060006080527f600355600060006000600060045af450735e34890115a16d77206060527f059260a0527f6000600060006000600060095af1506000600060006000600060085af250608060c0527f527f6008ff7f60026003557f7f7f60f7ff7f600060015560016000556000600160e0527f557f60a0527ffa3a1d3b6000527f95987f6009546000527f506000527f7f6000610100527f527f5c6436f160c0527f6000527f073260006020527f600060006000600060f4610120527f6020527f5af26000526060e0527f20527f7f506000606020527f6040527f0455610140527f606000527e606020527e60006040610100527f527f6000527f6000600e5a6040610160527f527ff46060527f507f6001600355606040527f610120527f20527f6004546060610180527f527f507f7c3a136040527f956080527f3960206060527f52610140527f7f6ba36101a0527f7f600854506009602060606080527f527f527f545060a0527f600260606101606101c0527f527f40527e556060006080527f6060527f527f016001556060a0527f026060c06101e0527f527f610180527f6040527f6080527f0255606000527e5450600860a0527f5450610200527f606060527f60c06101a0527f60e0527f527f406080527f527f606000527f0754610220527f60a0527f50600654506020606101c0527fc0527f52610100527f606060e0527f610240527f527f7f7f60006002556002546060a0527f6101e0527f80527f50bd60c0527f61610260527f0120527f853b6060e052610100527f7f20526060527f610200527f7f9014986d610280527f62696f5c10fe608052610140527f7ff25e6060c0527f2060e06101610220527f6102a0527f20527f52610100527f7f526060a0527f40527f610160527f7f466000600060006102c0527f610240527f606000526080527f610140527f7e60015af460610120527f6101806102e0527f527f606101610260527e527fe0527f4060a0527f527f60c0527f50610160527f610300527f5f5d143f118d6101a0610280527f527f7b62f09960610140527f60527f456d06610320527f610120527f095e606101610180526102a0527f7e526101c0527f7f405260a052610340527f7f7f406060c060e0610160527f527f527e60006102c0527f6000600060610161610360527f01e0527fa0527f610140527e606060527f2052610120527f6102e0527f7f6005610380527f5a610180527ff16080610200527f527f506101c0527f60006060c061016103006103a0527f527e52610160527f7f5260e0527f7f025560610220527e606101a0527e6161016103c0527fe0610320527f527f0140527f6000606060527e600060f65af15061610240527f6103e0527f0180527f6060610340527f80527f04616101610200527fc0527f0120527f5450610400527f60a0526101610260527f61610360527f0160527e527f7f60e0527f7f606101a0610420527f52610220527f7f01606101e0527f4061610380527f0280527f527f5450600154610440527f5060026000610140527f557f61018052610240527f6103a0527f7f3260806102610460527fa0527f527f6101610200527fc0527ffa1a03610120527f7277606103c0527fa0610480527f52610100610260526102c0527f7f527f60c0527f7f60046101610161022052616104a0527f03e0527f7fa0527f6101e0527f60527f54506102e0527f60610280527e6002556104c0527f7f600260610400527e606052610140527f7f610240527f527f6001610300527f6104e0527f55600654616102a0610420527f527f0200527f6101c0527f506101610180527f610500527f20527f60a0610320527f526102610440527f60527f7f60e0526102c0527f7f60610520527f60c0527f606000610220527f536101610340610460527f527f60526101e0527f610540527f7f602060610280526102e0527f7e527f606101a0527f01610480527f53606103610560527f60527f606002536061610240527f0140527e6003608052610300527f6104a052610580527f7f6102a0527f7f61610380527f0200527f53610100527f610180527f606061016105a0527fc06104c0527f5261026052610320527f7f7f6103a0527f2052606102c0527f606105c0527fe0527fc0527f6104e0527f7ff36060610220527e52606101605261036103c0526105e0527f7f40527f7f046020536061610500527f0280526102e0527f7f53602161016101610600527fe0527fa0527f6103e0527f5360606103610520527f60527f606061610240527f610620527f0120527f20527f610300527f226102a0610400527f610540527f527f53600560610640527fa061610380527f0100527f610180527f610200527f527f606161610560527f61610660527f0420527f02610320527f60527f01c052616103a0527f02c0527f7f60e0527f61610680527f0580527f2353606060610440527f6040527f2453610140527f610340527f60006106a0527f6103c0526105a0527f7f602561022052610280610460527f6102e0527f527f7f6106c0527f5360f360266101a0616105c0527f01e0527f61036103e0527f60527f526104806106e0527f527f7f5360006027610120527f606105e0527e6000610300527ff5606102a052610700527f7f60610400526104a0527f7f610240610380610600527f527f527f610100527f610720527fc0610160527f527f610200527f6061036104c0527f2061610620527f0420527f610740527f527f406101c0527f6103a0527f526102c0527f7e600060006061610461064052610760527f7fe0527f0260527e6060610440527f527f845af46101610340527f406103c052610780527f7f610660527f527f50610500527f50600260610220526102e0610460527f527f6107a0527f7f0455646101610680527f8052610161028052610520527f7f616103e0527f036107c0527f60527fe0527f7f876104806106a0527f527f836c78610120527f3d79136105406107e0527f527f60610300527fe0527f019c6061046106c0527e527f610240526104a0527f610800527f610380527f7f610560527f206000f36102a0527f6106e0527f60610160527f60610820527f526102005261610420526104c0527f7f610580527f0320527f610700527f7f60610840527f6101a06103a0527f527f80600060805260f360610260526102616105a0526107610860527f20527f7f04e0527fc0610440527f527f7fa05360a16061610340527f6103c052610880527f7f0140610740527f526105c0527f7e6000f0610500527f6000606161026104606108a0527f527f20527f010052610760527f6101806101c06105e0527f6102e0527f5261616108c0527f03610520527fe0527f61036052610780527f7f028052610480527f7f7f6106006108e0527f527f527f7e600060006000855af2505061056107a0527f40527f6006ff6a15a0610900527f5761610400527f610620527f0240526104a0527f6103616107c0527f0380527e610920527f527f7f18610560527f926101606102a052610640527f7f527f7c9f606107e052610940527f7e6101e06104206104c0527f527f527f6002610580527f6101a0610660527f52610960527f610800527f7f6103a0527f557f7f610320527fbd606101205261026104e0527f610980527f606105a061610820527f0680527f527f52610440527f7f6102c0527f7e6000606109a0527e6000606103c0527e60610840527ff15af1506106a0527f60616105c0527f056109c0527e527f04610340527f61610460527f610860527f0200527f54610180526106c06109e0527f527f7f506101c052616105e0527f02e061036105610880527f20527fe0527f52610a00527f7f610280527f7f6106e0527f60610480527f026000556103616108a0527f0600610a20527f527f60527f7f6004545060610540527f60610700527e52610140527f61046108610a40527fc0527e527f7f61022061610620527f04a0527f527f05610300610720527f527f610a60527f60006108e0527f610560527f527f5461610380527f02a0527f610640527f5060610a80527f01545061074052610900527f7f610420526104c0527f7f616101e0610580527f610aa0527f527f01a0527f7f1361066052610920527f7f610760527f600060006061032052610ac0527f7e606103a0527e6061026104e0527f6105610940527fa0527f40610461078052610ae0527f7f610680527f40527f527f0b6102c0527f5af4506000610960527f6000606101610b00527f60527f6060206107a0527f61616105c06106a0527f527f61050052610980527f610b20527f7f036103c0527f610460527f40527f026107c0527e527f527e52606101c06106610b40527f6109a0527fc0527f527e6105e0527f60616102e0610520527f6107e0527f527f610b60527f0260527f606109c0527f20527f20610480526106e0527f7f6103e0527f536106610b80527e527f610800527f60606109e0527f610360527f602153610540527f60006022610ba0527f53610700527f606060235360616108610a00527f20527f0220527e6161062052610bc0527f7f04a0527f6161030052610400527f7f61072052610a20527f7f610561084052610be0527f7f60527f0180527f610280610380527f610640527f527f6024610a40527f6101610c00527fe0527f616107610860527f40527f04c0527f53606040527f610580527f60610a610c20527f60527f602553610420610660527f527f610880527f600d60610760527f265360610c40527f610320610a80527f527f40526103a0526104e0527f7f6105a0526108a0527f7f610c60527f7f61610680527f61610aa0527f0780527f0240527f606102a0527f5a60610440610c80527f527f27536108c0527f60f16028610ac0527f5360610161026161056107a0527f610ca0527f6106a0527fc0527f0500527e527f6108e052610ae0527f7fa061036103c0527f610cc0527f40527f527f50602953616107c0527f0460527f606106c0610b00527f52610900610ce0527f527f7f60602a6105e0527f606052610520527f616102c0527f6107e0610b2052610d00527f7f527f0260527f610920527f7f536020602b6106e0527f6103e0527f53606061610d20527f06610b40527e527f6161048061080052610940527f7f527f0360610540527f52610d40527f7f602c5360610b60527f610700527f606060527f610220527f60610960527f2d610d60527f610820527f5361062052610b80527f7f61016102e0527f610400526104616107610d80527f20527f05610980527f60527fa0527f610ba0527f610840527f7fc0527f606102610da0527f80526103610640527f80527f7e606109a0527f2e610bc0527f53605261074052610dc0527f7f602f610860527f5360606080527f610580527f6030536161610be0527f09c0610de0527f527f04c0527f60610660527f610420526107610880527f60527f7e603153610c610e00527e527f616103006109e0527f527f0240526103a0527f7f6105a0527f60606032610e20527f616108610c20527fa0527f068052610780610a00527f527f7f6102a0526104e0610e40527f527f7f5360206061610c40527f0440527f6101e0526108c0527f7f610a20527f610e60527f336080526105c06107a0527f52610c60527f6106a0527f7f7f53605360346103610e80527f206103c061610a40527f08e0527f52610500610c80527f527f7f527f53606060610ea0527f6107c0527f3553606104606106c052610a60527f7f5261610ca0527f05610900610ec0527f527fe0527f7fa052610260526102c0527f7f7f606107e0527f610a80610cc052610ee0527f7f527ff36036610520527f53610920527f60606106e0527f6037536103e06106610f00527e610ce0527f527f610aa0527f527f602161610800527f03610480610940527f610f20527f527f40527f61610d00527f0200527f603861610ac0527f0700527f5360610540610f40527f527f536039536108205261610d20527f0960527f7f610620527f6060610ae052610f60527f7fa0527f6102e0527f60616104005261610d40527f0720527f6104a0527f6109610f80527f80527f7f6108610b00527f40527f0280527f603a61610d60527f056052610640610fa0527f527f7f53610360527f602260606109a0610b20527f527f610740610d80527f52610fc0527f7fc0610860527f527f3b536060603c5360610220526104c0527f610b40527f61610fe0527f0da0527f7e616109c0527f0660527f61610580527f61610880527f0760527f04611000527f20527f60610dc0527f610b60527f3d53610300527f6109e0527f60f3603e6103611020527f80527f53606102a061610de0527f066108a052610b80527f7f80527f52610780611040527f527f7e60610a00527f3f61046105610e00527fa0527fe0527f60006000610ba0611060527f527f60c0610440526108c0527f7f527ff56000610e20527f610a20527f60e061611080527f6107a0527f06a0610bc0527f527f527f6000600061032052610e40527f6161056110a0527fc06108e0527f52610a40527f7f03a05261610be0527f0500527f7f7f61610e606110c0527f527f026107c0527f40527f606106c0527e6000610461610a60527f610c00527f6110e0527f0900610e80527f527f60527f856102c0527f5af26105e0527f5050606107e052611100527f7e60006000610c610ea0527f20527f610a80527f61052061610920527f06e052611120527f7f527f60006000600c6103c0610ec0527f527f5a610c40527f6103405261610a611140527fa0527f610800527f04806106610940527e610ee0527f527f527f7ff15058610c611160527f60527f610700527f60e0526101610ac0527e52610540610f00527f527f610261611180527f082052610960527f610c80527f7f60527f6102e0527f7f60ff6161610f20527f6111a0527f610ae0527f0620527f01610720527e6103e0610ca0527f527f6104a0610980526111c0527f610f40527f7f52610840527f7f5360610b00527f1261010161056052610cc0526111e0527f7f7f610360610f60527f527f5360406161610740526109a0527f7f064052610b611200527f20527f7f0161610ce052610f80527f7f0860527f0253606a6101035360926101611220527f045360616104c0527f6109c052610b610fa0527f40610d00527f527f7f616105611240527f80527f610760527f610880527f0400527f610660610fc0527f527f0300527f61611260527f0d20527f6102610b60527f806109e0527f527f566161038052610fe0527f7f01611280527f20527f6101056108a0610d40527f527f610780527f610b80527f5360606161106112a0527e527f6161610a00527f05a052610680527f7f610d60527f04e0527f010653606112c0527f206104611020527f20610ba0527f6108c0527f527f61010761610a2052610d806112e0527f527f7f07a0527f53611040527f6060610108610320527f53610bc0527f606161611300527f06a0527f036105610da0527f61611060527f08e0527fc0610a40527f527fa052611320527f7e616105006107610be0527fc0527f527f61611080527f0dc0527f016102a052611340527f7f095360f3610440527f61610a60527f0900527f6101616110a0527f0c00527f611360527f610de0527f6106c0527f61014052600a61056107e0527fe0527f61016110c052611380527f7f605360610a80527f53610e00527f610c20527f610161610920527f610520526113a0527f7f6110e0527f5361036103c0526106e0527f7f40610e20527f5261080061610c6113c0527f40527f0aa052611100527f7f527f7f606161016104610940527f60610600610e6113e0527f40527f527f527f62536001611120527f6101610c60527f6102c052610ac0527f611400527f7f63610561070061610e60527f082052611140527f7f527f610960527f40527f611420527f53610c80527f600b61016453606061610ae0610e80611160527f527f527f0165611440527f53610620527f6103e0527f6000610461610ca0527f0861098052611180527f7f611460527f40610ea0527f527f8052610720610b00527f527f7f61610360527f0161056061611480527f11a0527f610cc0527f527f610ec0527f665360f361016753616109a0527f06406114a0527f52610b206111c0527f527f610860527f7f6101610c610ee0527fe0527f6860006114c0527f60610740527f6102e06111e0527f5260006103006104616109c0610b40527f616114e0527f0f00527f527f04610d00527fa052611200527f7e527f6105610880527f80527f611500527f5360f0610660527f61610f20527f610760527f611220527f610b60610d20527f611520527f527f03016109e0527f5360610380527f606103610f40527f611240527f025360611540527f6108a0527e6103035360610d40527f6061610b80527f03046104c06105611260611560527f527f610f60527f610a00527fa061610780527f0680527f527f610d60527f527f611580527f5360611280527f6108c0610b610f80527fa0527f527e610420527f61030553606115a0527f610a20527f60616112a0527f03610d80527f06536000610fa0527f616103a0616115c0527f07a052610bc0527f7f527f036112c0527f07616108e0527f06a0527f610da0526115e0527f610fc0527f7f5360610a40527f6105c0526112e0527f7f60616104e0610be052611600527f7f527f03085360006103610fe0527f09610dc0527f61611300527f6107c0527f611620527f610900527f04610a60527f40527f5360610c0052611000527f7f60611320527f611640527f61036106610de0527fc0527f0a53600061030b6105e0527f53608561030c6110611660527f611340527f20527f61610a80527f610c20527f610e00527f0920527f6107e052611680527f7f61610500611360527f527f03611040527fc05260536103e06106e0527f53606116a0527f6061610e20527f610c40611380527f527f610aa0527f03611060527f610460616116c0527f0940527f527fe15361610800527f066113a0527e52610e40527f7f605a6103616116e0527f611080527f0c60527fe253606161610ac0527f6113c0527f03e3536105206107611700527e610960610e60527f526110a0527f7f527f527f600361036113e0527f610c80611720527f527fe453610820527f600d61610ae0527f03e5616110c0527f0e80527f611400611740527f527f53605361610620527f03e653610980610ca0527f527f610480527f606110611760527fe052611420527f7f60610720610ea0527f527f61610b00527f03e7610840527f611780527f5360f16161610c611440527fc0611100527f527f0540527f03e853610ec0527f6117a0527f6109a0527f60616103e95360611460527f61610b20527f611120527f0640527f6117c0527f036103610ce0527fea53610ee0527f6107611480527f40610860527f527f600e6117e0527f61611140527f03eb6109c0527f536053610b40527f616114a0527f04610d610f611800527e527e527fa0527f6103ec611160527f53610560527f60606103ed6114c0527f611820527f610660610880527f527f6109610f20527fe0610b61611180527f0d20527f6052611840527f6114e0527f7f527f610760527f5360506103ee5360616103ef5360610f405261611860527f11a0527f7f611500527f036103f053600f61610d40527f036104610b80527f61611880527f08610a00527fa0527f61611520527f11c0527fc0610f60527f52610580527f616118a0527f0780527f6106610d60527f80527f7f611540527ff15360536111e0527f610ba06118c0527f527f61610f80527f03f253606061610a20527f03611560527ff3536108c052616118e0527f0d80611200527f527f7f60506103f4536061610fa0527f6103611580527ff553611900527f610bc0527f6107a0527f6003611220527f6103f66106610a610da0527f406115611920527fa0527f527fa0610fc0527f527f6105a0527f6108e052611240527f7f5361610b611940527fe0527f6115c0527f04e0527f60106103f7610dc052610fe0527f7f5360536103611960527f611260527ff861076115e0527f610a60527fc0527f5360616103f953610c0052611980527f7f600361611000527f61611280611600527f527f610de0527f0900527f6106c06119a0527f527f03fa5360116105c052610a80527f7f61611620527f03616112a0527f10206119c0527f527ffb53610c2052610e00527f7f6060616107e0527f05611640527e527f61616119e0527f09206112c0527f527f03fc611040527f5360006103fd53610aa052611660527f611a00527f610e20527f7f610c40527f606112e0527f6106e0527ff36103fe611060527f53611a20527f611680527f6103ff60006105610800527f610940610e611300527f40527f527f611a40527fe0527f60616116a0527f0c60527f611080527f610ac0527ef060006000600060611a60527f611320527e61052052606116c0527f6061610e60527f0700527f61056110a052611a80527f7f4053600061610c80611340527f526116e0527f7f61096052610ae0527f7f61611aa0527f0820527f05610e80527f416110c0527f53608561611700527f1360527f610542611ac0527f610600526053610620610ca0527f536060610621536061610b611720527f6161611ae0527f10e0611380527f527f0ea0527e527f07610980527f20527f5a61610840526117611b00527f40527f7f0622610cc0527f536113a0527f6061611100527f610623610ec0527f611b20527f536005611760527f6106245360610b20527f436106256113c0527f53606109a0611b40527f527f611120527f53611780527f61610ce0527f06610ee0527f26536060616107611b60527f6113e0527f610860527f40527f6117a0527f06275361611140527f0b40527f60611b80527ff161062853606161610f611400527e527f0d6117c0527e527f6106296109c052611ba0527f7f5360611160527f0561062a53604461062b611420526117e0527f7f53605361611bc0527f610b6052610f20527f7f088052610d20527f611180527f7f61062c611800527f611be0527f61611440527f0760527f536060616109e0527f062d536050610f40527f61062e611c00527f611820527f536111a0527f611460527f6061610b80610d40527f527f61062f53611c20527f6005610630611840527f6108a0527f5360610f6052611480527f6111c0527f7f611c40527f4561063153610a00527f611860527f60610780610d60527f527f610ba0527f61611c60527f14a0527f53610632536111e0527f60611880527f610f80527f60610633536050611c80527f6106345360616108c06114c0527f527f61610d806118a0527f527f0661611200611ca0527f527f0a20527f3561610fa0527f0bc0527f53606114e0527f056118c0527f6106611cc0527f36536046616107a0527f0637611220527f536053610da0527f61063861616118611ce0527fe0527f1500527f0fc0527f536061610639610861610be0527f0a40611240527f611d00527f527fe0611900527f527f5360611520527f0561063a5360610dc0610fe0527f52611d20527f7f4761063b536060611920527f6106611260527f3c61611540527f07c0527f53611d40527f60610c00527e61063d610a6052611940527f7f611000527f5360610de0526112611d60527f611560527f80527f7ff3610900527f61063e611960527f5361063f60006000f0611d80527f600060610c2052611020611580527f527f7e6112a0527f611980527f60006000611da0527f6000610e00527f8561610a80527f07e052605a616115a0527f0800616119a052611dc0527f7f09205260616112c0527f1040527f5361094053610c40527f6060610e206115611de0527fc06119c0527f527f527f6109415360f161094253606112e0527f61610aa06110611e00527f60527f527f616119e0527f09436115e0527f53600861094453600161610c610e611e20527f40527f60611300527f527f611a00527f09455360536109611600527f61108052611e40527f7f465360606109475360506109610ac0611a20527f527f48611320527f536061611e60527f61611620527f610e60527f094953610c6110a0527f611a40527f80527f600861611e80527f094a536002610961134052611640527f7f4b53605361094c5360611a60527f60611ea0527f61094d53610e80527f6110c0527f610ae0527f6050611660527f616113605261611ec0527f1a80527f7f610ca0527f094e53606161094f5360086109505360036109611061611ee0527f1680527f611aa0527fe0527f610ea0611380527f527f51536053610952536061611f00527f610b00610cc0527f52611ac0527f6116a0527f7f610953536008616111006113611f20527fa0527f527f0954536004610ec052611ae0527f7f610955536116c0527f606061611f40527f09565360006109575360f3610c6113c0527fe0611b00527f52611120527f7f61611f60527f09616116e0527f0b2052605861610ee0527f0b4053605361611b20527f0b4153611f80527f606113e0527f61610b425360611700527f09611140527f610b43536059611b40611fa0527f527f610d00527f610b4453610f00527f611400527f60611720527f60610b4553611fc0527f6000611b60527f610b4653611160527f6060610b47536000610b485360f06161611fe0527f611740527f1420611b80527f527f0b610d2052610f20527f7f49536060610b61612000527f1180527f4a536000610b6117611ba0527f60527f4b536060611440527f610b4c612020527f536000610b4d536060610b4e610f40527f611bc0527f536161611780527f11a0612040527f527f0d40527f60611460527e610b4f536060610b5053611be0527f6000610b51612060527f5360846117a0527f610b5253605a610f6111c0527f611480527f60611c00527f612080527f527f610b535360f4610d6052606117c0527f61610d8053600b610d81536054616120a0527f611c20527f0d8253606114a0527f6111e0527f53610d616117e0527f0f80527f6120c0527f8353606061611c40527f0d84536050610d85536061610d866114c0527f53600b6120e0527f61611800527f0d611200611c60527f527f87536055610d8853610fa0527f6053612100527f610d89536060616114e05261182052611c80527f7f7f0d8a536050610d8b5360612120527f611220527f61610d8c53600b610d8d536056610f611ca0527fc0611840527f52612140527f611500527f7f610d8e536053610d8f536061610d611240527f611cc0527f9053612160527f600b610d611860527f91536057610d611520527f92536060610d610fe052611c612180527fe0527f7f93536000610d94536061611880527f1260527ff3610d9553600061156121a0527f40527f611d00527f610d9660006000f560006000600060006118a0527f6110006121c0527f5260606110206112611d20527f80527f5360611560527e6110215360856110226121e0527f53606118c0527f5a6110235360611d40527ff161102453605061102553606112612200527f611580527fa05260506112c06118e0527f53611d60527f60616112c153601061612220527f12c25360266112c35360536112c4536115a0527f606161611d80527f1900527f612240527f6112c55360106112c65360276112c75360606112c85360006112c953611da052612260527f7f60f36115611920527fc05260616115e05360126115e15360ca6115e2536053612280527f61611dc0527f15e35360616115e453611940527f60126115e55360cb6115e6536122a0527f60606115e753611de0527f60006115e85360606115e9536000611960527f61156122c0527fea5360f06115eb53606061611e00527f15ec5360006115ed5360606115ee53606122e0527e6115611980527fef5360606115f053611e20527f60006115f15360606115f2612300527f5360006115f35360606115f4536119a0527f600061611e40527f15f553608561612320527f15f653605a6115f75360f16115f85360506115f95360506119c0611e60527f52612340527f60616119e05360156119e15360fa6119e25360536119e35360616119e4536061612360527f1e80527f156119e55360fb6119e65360606119e75360006119e85360f36119e9612380527f53600061611ea0526019611ec05360ea611ec1536060611ec2536000611ec3536123a0527f6060611ec4536000611ec55360f5611ec6536060611ec7536000611ec85360606123c0527f611ec9536000611eca536060611ecb536000611ecc536060611ecd536000611e6123e0527fce536060611ecf536000611ed0536085611ed153605a611ed25360f1611ed35361240052606061242053605061242153606161242253601e6124235360d461242453605361242553606061242653605061242753606161242853601e6124295360d561242a53605361242b53606161242c53601e61242d5360d661242e53606061242f5360006124305360f36124315361243260006000f06000600060006000845af45050",
        "storage": {
          "0x0000000000000000000000000000000000000000000000000000000000000000": "0x0000000000000000000000000000000000000000000000000000000000000010"
        },
        "balance": "0x0",
        "nonce": "0x0"
      },
      "0x00000000000000000000000000000000000000f2": {
        "code": "0x3da2986b3dfa600254503a99783383f1a09a5b98f0376000600060006000600060fa5af250a21350ff8034fa0360206000fd",
        "storage": {
          "0x0000000000000000000000000000000000000000000000000000000000000000": "0x0000000000000000000000000000000000000000000000000000000000000000",
          "0x0000000000000000000000000000000000000000000000000000000000000001": "0x0000000000000000000000000000000000000000000000000000000000000013",
          "0x0000000000000000000000000000000000000000000000000000000000000002": "0x000000000000000000000000000000000000000000000000000000000000000b",
          "0x0000000000000000000000000000000000000000000000000000000000000004": "0x0000000000000000000000000000000000000000000000000000000000000006"
        },
        "balance": "0x0",
        "nonce": "0x0"
      },
      "0x00000000000000000000000000000000000000f3": {
        "code": "0x6000600060006000600060015af2507203587b954a0a60006000557f7f8804309e2000767f7f600060006000600060065af4506000600060006000606000527ff96000527f5af4505d8b3b35356000527f606000527ff7ff60015450bc8545786020527f5c159163ff6020527f600060006000600060f15af46020527f507f60016060206040527f527e556007545060066040527f545060095450600060035560075450606040526060527f7f60600053606000526040527f6060527f6020602053606060215360016022536080527f60536023536060527f60606024536060606080527f255360606060527f60265360a0527f600260275360536028536060606080527f2953600060a0527f602a536060602b60c0527f536003602c6080527f536053602d536060602e536060a0527f60c0527ffd602f60e0527f536060603053600460315360536032536060a0527f606033536005603460e052610100527f7f60c0527f536060603553600060365360f3603753600060386000600060c052610120527f7f610100527ff560006060e0527e600060006000855af1505060045450600060610140527e6000600060610120527e60f560e0527f610100527f5af25060045450600060610160527e60006000600060025a610140527ff250600260035560206000fd6101205260610180527f616101405360016101415360006101610160527f4253605261014353606161016101a0527f4453600161014553602061014653606061014753610180527f600061014853606101c0527ff361014953600061014a60006000f5600060006000600060006101a0527f855a6101e0527ff150506000600060006000600060f15af2506f160585895c6000600060006101610200527fc05260606101e05360006101e15360606101e25360f46101e353605a6101e453610220527f60f46101e55360506101e65360176101e75360606101e85360046101e9536054610240527f6101ea5360506101eb5360066101ec53605f6101ed53606c6101ee5360426101610260527fef53608a6101f05360876101f153603c6101f25360096101f353603c6101f4536102805260606102a053604a6102a15360616102a25360016102a35360f56102a45360536102a55360606102a65360f16102a75360616102a85360016102a95360f66102aa5360536102ab5360616102ac5360016102ad5360f76102ae5360606102af5360006102b05360f36102b1536102b260006000f060006000600060006000855af15050",
        "storage": {
          "0x0000000000000000000000000000000000000000000000000000000000000000": "0x0000000000000000000000000000000000000000000000000000000000000005",
          "0x0000000000000000000000000000000000000000000000000000000000000001": "0x0000000000000000000000000000000000000000000000000000000000000011",
          "0x0000000000000000000000000000000000000000000000000000000000000002": "0x000000000000000000000000000000000000000000000000000000000000000d"
        },
        "balance": "0x0",
        "nonce": "0x0"
      },
      "0x00000000000000000000000000000000000000f4": {
        "code": "0x60206000f3",
        "storage": {
          "0x0000000000000000000000000000000000000000000000000000000000000004": "0x000000000000000000000000000000000000000000000000000000000000000d",
          "0x0000000000000000000000000000000000000000000000000000000000000007": "0x000000000000000000000000000000000000000000000000000000000000000d",
          "0x0000000000000000000000000000000000000000000000000000000000000008": "0x0000000000000000000000000000000000000000000000000000000000000013",
          "0x0000000000000000000000000000000000000000000000000000000000000009": "0x000000000000000000000000000000000000000000000000000000000000000b",
          "0x000000000000000000000000000000000000000000000000000000000000000a": "0x0000000000000000000000000000000000000000000000000000000000000001",
          "0x000000000000000000000000000000000000000000000000000000000000000b": "0x000000000000000000000000000000000000000000000000000000000000000d"
        },
        "balance": "0x0",
        "nonce": "0x0"
      },
      "0x00000000000000000000000000000000000000f5": {
        "code": "0x3d61934533fb9e458141438a197064474800471592cb6000600060006000600060f95af25060025450013497fd8f5645146317025d02f460206000fd",
        "storage": {
          "0x0000000000000000000000000000000000000000000000000000000000000000": "0x000000000000000000000000000000000000000000000000000000000000000e",
          "0x0000000000000000000000000000000000000000000000000000000000000002": "0x0000000000000000000000000000000000000000000000000000000000000009",
          "0x0000000000000000000000000000000000000000000000000000000000000003": "0x0000000000000000000000000000000000000000000000000000000000000000",
          "0x0000000000000000000000000000000000000000000000000000000000000004": "0x000000000000000000000000000000000000000000000000000000000000000c",
          "0x0000000000000000000000000000000000000000000000000000000000000005": "0x0000000000000000000000000000000000000000000000000000000000000008",
          "0x0000000000000000000000000000000000000000000000000000000000000008": "0x0000000000000000000000000000000000000000000000000000000000000010",
          "0x000000000000000000000000000000000000000000000000000000000000000a": "0x000000000000000000000000000000000000000000000000000000000000000a",
          "0x000000000000000000000000000000000000000000000000000000000000000b": "0x0000000000000000000000000000000000000000000000000000000000000012"
        },
        "balance": "0x0",
        "nonce": "0x0"
      },
      "0x00000000000000000000000000000000000000f6": {
        "code": "0x479a78345cf18ff59f1011f1600454505f7e5a6d7f86a46000600060006000600060015af1507f60026002556002600355600160015560006000556006545060035450600554506000527f6002600255600060005560095450600654507f600060006000600060025af4506020527fd63c11956e96468060f3ff7f600154506001606000527e5560026000556002606040527f0255600260015560045450600754506000527f6000606020527f0255600160046060527f55600260025560016003556006545060606000536020606020526040527f60016080527f604053605360415360606042536060604353606060445360026045536053606060a0527f527f6046536060604753600060485360606049536003604a536053604b53606060c0527f604c6080527f5360f3604d536060604e536004604f536053605053606060515360e0527f60056052536060a0527f60605353600060545360f36055536000605660006000610100527ff560006000600060006060c0527e855af150509fe72b60016001554560006000610120527f600060006000600a5af150318560e0527f121c7982fdf9951739778f60006002610140527f556a7f6001600355600060025560015450610100527f7f600160025560006000610160527f60006000600060046000527f5af25060006000600060610120527e6000600c60610180527e527f5af150537f600554507f9914866020527f946040600060610140527e606101a0527e6000600060f55af150606020527e600060006000527f606040527e60fa61016101c0527f60527f5a6000527ff4504410675a60006000600060006040527f60035af450526101e0527f606052610180527f7f18883b6020527f209209333a6001600160205260556040610200527f53600a6060527f606101a0527f416080527f5360836042536039604353604052610220527f7f6090604453607460455360676101c0527f604653608060a0527f527f608c60610240527f47536041604853607360495360196060527f6101e0527f604a53605f604b5360610260527f60c0527f606060a0527f4c536020604d536060604e5360610200527e604f5360610280527ffd6050608052605360e0527f60a05360606060c0527fa153605160610220527f6102a0527fa253606060a353600060a45360f360a553610100527f60a660006000f06000606102c0527f610240527f60e0527e600060006000855af150505d600060005579610120527f6102e0527f1580970a45610260527f695e57f274600060610100527e60006000600060035a610300527ff2507f6002610140527f610280527f6001557f3a60faff600154506000600060610320527f610120527e6000600060055af150606102a0527f610160527e606000527e6000610340527f60006060005260006020536060610140527f60216102c0527f5360f760610180610360527f527f2253605a60235360f1606020527f2453605060255360606102e0527f6026610380527f53610160527f606101a0527f206027536060602853600060295360fd602a61036103a0527e527f536040526060606053602b6061616101c0527f0180527f5360606062536103c0527f600060610320527f635360f3606453606560006000f060006000606101e0527e6103e0527f6000606101a0527e610340527f855af25050600254508c829789578760006003610400527f55600060610200527e60006000610360527f6000606101c0527ff65af2507f60610420527f055450600054507f60f5ff6000610220527f610380527f600060006000600060610440527ff95af1506101e0527f60006000606000527e60006000606103a0527f61024052610460527f7f045af26000527f50600454507f6009545060610200527f016004556103c052610480527f7f6001600260610260527f20527f557f3a49051660faff7d6000606020527e606104a0527e6103e0527f60610220527e60600052610280527f7e60075af2503b6040527f6104c0527f023a60006000610400527f60006000527f60006000610240527f6102a0527f606104e0527f40527f60f25af150600354610420527f506020526060527f7f659455476bfe81610500527f3d809a7f6102c0527f610260527f6002610440527f6001556006546060527f60610520527f20527f506000606080527f0455606102e0527f0260610460527f604052610280610540527f527f7f0155600754506009545060015450606080527f60006103610480527e52610560527f7f527f60a0527f085450606102a0527f6040527f01600455606060527e546104610580527fa0527f506005610320527f5450600854506060a0527f60c0527f6102c0527f056105a0527f5450606104c0527f0654506001600060610340527f20527f6060527f556080526105c0527f7f600260025560056104e0527f546102e0527f60e0527f60c052610360527f7f6105e0527f50600054506003545060055450610500527f6002600255600454506060610300610600527f527f60a0610380527f527f610100527f8052610520527f7f6060e0527f40527f610620527f0454507f756003545060025461036103a0527f20527f50610540527f60f4ff2a610640527fa499ff73610120527f6215376060c0527f610100527f01606103c05261056052610660527f7f7f0360a0610340527f527f551371466060527f88069e3d610140527f600052610680527f7f610580527f316103e0527f5805638cf28bf372610360527f610120527f60e06106a0527f527f374719726105a0527f60015460c052610400527f610160527f7f506000606106c0527e6000610380527f6060806105c0527f527e6000600a5a61014052610420527f6106e0527f7ff2506020610100527f610180527f526105e0527f7ff36103a0527f6e203ef1610700527f6000600160610440527fe0527f5560006000600060610600527f610160527e60610720527e60a06101a06103c0527f527f527f610460527f600b61012052610620527f7f610740527f5af15060f2ff44933e996040527ff26101006101806103e052610480527f7f61610760527f0640527f527f6101c0527f527f846005ff7f60606000536002610140527f6001610780527f5360c061610660527f04a0527f52610400527f7f6060600253606101e0527f616107a0527f01a0527f0260035360610680527f556061016104c0527f20527f0453606104206107c0527f527f6060527f610160527f7060056106a0527f610200527f536074606104e0526107e0527f7e6101c0527f527f600660610440527fe0527f6106c0527f53605f6007536051610800527f6061014052610500527f7f610220527f0853610180527f606106e0527f996061610820527f0460527f6101e0527f0953603c600a610520527f6080527f53601a600b610700610840527f527f5360610240527f9b61010052610480527f7f600c602061610540527f0161610860527f01a0610720527f610200527f527f60527f527f536087600d610260527f6104a0610880527f527f5361056052610740527f7f6012600e536049600f53606060a0527f6010536108a0527f610220527f60616101c0527f610760527f61610580527f04c0527f610280527f6108c0527f012052610180527f7f2060115360606012610780527f5360604052606105a0526108e0527f7e6061026104e0527f40527f60536102a0527f6060606107a0527f61536101e0610900527f527f6013606105c0527f625360c0526101a052610500527f7f7f616107c0527f610920527f0140527f6061026102c0527f60527f6105e0527f53606353606060645360f360610940527f6107e0527f6102610520527e527f6553606060665360146067610600527f5361610960527f02e0527f60610800527f6101610280527fc0527f53610540527f606853606061610980527f016052610620527f7f60610820527fe052610220527f7f60610300527f6953606109a0527f15606a53610560527f606102a06106610840527f40527f527f60606b536000606109c0527f6c6101e0527f5360f3606d53610320527f606e61610860527f058052610660526109e0527f7f7f60610240527e60610180526102c0527f7ef06000610100610880527f5260610a00527f606101205361610680527f036105a0527f40527f600061610200527f01216108610a20527fa0527f5360610260527f606102e0527f6106a0527f6101225360006105c0527f610a40527f6101236108c0527f610360527f536101a0527f606061012453606106c0527e61610a60527f01255361022052616108e0527f616105e0527f0300527f0280527f610380527f610a80527f7f60846106e0527f6101265360610900527f5a6101275360f461012853610600610aa0527f527f60506101c0527f61016103610700527f610920527f20526103a0527f7f29610ac0527f5360506102a0527f61610240610620527f527f012a5360610940527f61072052610ae0527f7f6061012b53602061012c53606103c0527f60610340527f61012d6161096052610b00527f7f0640527f53610740527f60006101616102c0527f01e052602e61610260527f610b20527f02610980527f6103e0527e5360536161610760527f0660527f02610360527f01610b40527f5360606102026109a0527f5360f3610203536102e0527f606161610780527f04610b60527e527f610680527f6102046109c0527f5360016102056102610380527f80527f610b80527f53602f616107a0527f020653605361026109e0527f07536106a0527f61042052610ba0527f7f606161610300527f02085360016107c0527f6102610a00527f096103a0527f610bc0527f53603061020a6106c0527f53606102a0610440527f527f606107610a20527fe0610be0527f527f61020b53600061020c53610320527f60f361616106e0527f03c0527f0261610c00527f0a40527f0d5360610800527e61610460527f020e60006000f560006000606102610c20527fc0526061610a60527f0700527e6102e0610820527f53606103406103e0527f52610c40527f610480527f7f606102610a80527fe15360006102e25361072052610840527f7f610c60527f60606102e35360006102e4536085610aa0527f6102e55360616104a0527f0400610c80527f527f5a61610860527f61610740527f03605260610ac0527f026103805360e661610ca0527f0381536053610382536060610361610880527f04c0527f83610ae0527f536107610cc0527f60527f60f2610420527f61038453606161038553600261036108a0527f610b00610ce0527f527f865360e761038753606104610780527fe0527f5361038853606061036104610d00527f4052610b20527f6108c0527f7f8953605061038a53606161038b53606107a052610d20527f7f026103610500610b40527f527f8c53606108e0527fe861038d53605361038e610d40527f53610460527f606061038f61610b60527f07c0527f536050610390610900527f610d60527f5360610520527f61610391536002610392610b80527f5360e961039353605361610d80527f07e0527f61610920527f0480527f6103945360616103610ba0527f610540527f610da0527f955360026103965360ea6103975361610940527f0800527f606061610bc0527f610dc0527f039853600061036104a05260996104610560527fc053605361610960527f04c1610de0527f610be0527f5360610820527f606104c25360f36104c35360616104c453600361610e00527f04c5536109610c00527f80527f610580527f609a6104610840527fc653605361610e20527f04c75360616104c85360610c20527f0361046109a0527fc953609b6104ca5360610e40527f606105a052610860527f7f6104cb53610c40527f60006104cc5360606109c052610e60527f7f6104cd5360006104ce5360f06104cf53606061610c60527f610880527f0461610e80527f05c0527fd0536109e0527f60006104d15360606104d2536000610c80527f6104610ea0527fd35360606104d4536108a0527f600061610a00527f04d5536105e0527f60610c610ec0527fa0527f606104d65360006104d75360846104d853605a616108c0610a20527f52610ee0527f7f04d9610cc0527f5360f46104da536050610600526061610620536004610621610f00527f5360db61610a4052610ce0527f7f06226108e0527f5360536106235360606106610f20527f24536050610625536061610626610d00527f53610a60527f6004610627536061610f40527f0900527fdc61062853605361062953606161610d20527f062a53600461610a80610f60527f527f062b5360dd61062c53606061610920527f062d5360610d40527e61062e53610f80527f60f361062f53610aa0527f61063060006000f06000600060006000610d60527f610fa0527f8461094052605a6109605360f4610961610ac0526053610ae0536060610ae153610fc0527f610d80527f6050610ae2536061610ae3536009610ae4536062610ae553605361610fe0527f0ae6536060610da0527f610ae7536050610ae8536061610ae9536009610aea53611000527f6063610aeb536053610a610dc0527fec536061610aed536009610aee53606461611020527f0aef536060610af0536000610af153610de0526060610e005360f3610e015360611040527f61610e0253600a610e035360f2610e04536053610e05536060610e0653600061611060527f0e07536061610e0853600a610e095360f3610e0a536060610e0b536000610e0c611080527f536060610e0d536000610e0e5360f5610e0f536060610e10536000610e1153606110a0527f60610e12536000610e13536060610e14536000610e15536060610e16536000616110c0527f0e17536060610e18536000610e19536085610e1a53605a610e1b5360f2610e1c6110e052605361110053606061110153605061110253606161110353600e61110453601d61110553605361110653606061110753605061110853606161110953600e61110a53601e61110b53605361110c53606161110d53600e61110e53601f61110f5360606111105360006111115360f361111253600061111360006000f560006000600060006000855af15050",
        "storage": {
          "0x0000000000000000000000000000000000000000000000000000000000000000": "0x0000000000000000000000000000000000000000000000000000000000000006",
          "0x0000000000000000000000000000000000000000000000000000000000000001": "0x000000000000000000000000000000000000000000000000000000000000000e",
          "0x0000000000000000000000000000000000000000000000000000000000000002": "0x0000000000000000000000000000000000000000000000000000000000000004",
          "0x0000000000000000000000000000000000000000000000000000000000000003": "0x0000000000000000000000000000000000000000000000000000000000000011",
          "0x0000000000000000000000000000000000000000000000000000000000000004": "0x000000000000000000000000000000000000000000000000000000000000000b",
          "0x0000000000000000000000000000000000000000000000000000000000000005": "0x0000000000000000000000000000000000000000000000000000000000000000",
          "0x0000000000000000000000000000000000000000000000000000000000000007": "0x0000000000000000000000000000000000000000000000000000000000000010",
          "0x0000000000000000000000000000000000000000000000000000000000000008": "0x0000000000000000000000000000000000000000000000000000000000000005",
          "0x0000000000000000000000000000000000000000000000000000000000000009": "0x0000000000000000000000000000000000000000000000000000000000000008",
          "0x000000000000000000000000000000000000000000000000000000000000000a": "0x0000000000000000000000000000000000000000000000000000000000000000"
        },
        "balance": "0x0",
        "nonce": "0x0"
      },
      "0x00000000000000000000000000000000000000f7": {
        "code": "0x60206000f3",
        "storage": {
          "0x0000000000000000000000000000000000000000000000000000000000000001": "0x000000000000000000000000000000000000000000000000000000000000000e",
          "0x0000000000000000000000000000000000000000000000000000000000000002": "0x0000000000000000000000000000000000000000000000000000000000000008"
        },
        "balance": "0x0",
        "nonce": "0x0"
      },
      "0x00000000000000000000000000000000000000f8": {
        "code": "0x7a32350a86064395955830fe6000600060006000600060f15af2505293723d06f50a446632f26171377b424244600060006000600060055af4506000600055fd1130495b16708060206000fd",
        "storage": {
          "0x0000000000000000000000000000000000000000000000000000000000000003": "0x000000000000000000000000000000000000000000000000000000000000000e",
          "0x0000000000000000000000000000000000000000000000000000000000000004": "0x0000000000000000000000000000000000000000000000000000000000000005",
          "0x0000000000000000000000000000000000000000000000000000000000000005": "0x0000000000000000000000000000000000000000000000000000000000000013",
          "0x0000000000000000000000000000000000000000000000000000000000000006": "0x000000000000000000000000000000000000000000000000000000000000000d",
          "0x0000000000000000000000000000000000000000000000000000000000000007": "0x000000000000000000000000000000000000000000000000000000000000000a"
        },
        "balance": "0x0",
        "nonce": "0x0"
      },
      "0x00000000000000000000000000000000000000f9": {
        "code": "0xf30010158ef57e386000600060006000600060025af250181764576e7f6001545060026002557f8f1d311a805169538012635c7f7f6001ff60006004556000527f8fff1b97ff67600060006000527f60006000600060f35af1505e9c6000527f036020527f6000527fa2597f7fb1bb600eff7f6020527f60006000556002545060026001556040527f6002606020527f04556006546020527f50606040527f056000527f546000527f6060527f5060006004556000527f606040527f026003556001606060527f02556040527f6080527f600454507f60006020527f60006000606020526060527f7e60006080527f600460a0527f5af150545e6020526060527f7f863c69a16202600154506040527f60805260a060c0527f527f7f60006003556000526040527f7f9f9f34376080527ffd9440743582600260e0527f546060c0527f4060a0527f527f50606060527e60006000600060095af4506060610100527f60a0527f527f60e0527fd16020600060c0527f60205260f36040536041606080610120527f527ef360605260806000610100527f600060c0527ff0600060e0527f60006080610140527f527f600060006000855af250506060610120527fa0527ffaff2102a4906060e0610160527f52610100527f7e6002557f5699329d516b600160610140527fa0527f600055eb610180527f6001545060c0527fdf5579610120527f610100527f9b57a060610160527f03546101a0527f50455c60045450096a1580a47b7f6060c0527f0354610140527f506060e061016101c0527f80527f52610120527f7f075450600654506000600355606860005360f46001616101e0527f0160526101a0527f7f53607e6002536060e052610140527f7e610100527f527f610200527f6067600353609b606101c0527f04610180527f53606060055360206006536060610220527f6007610160527f5360006008616101e0527f610120527f016101a0527e527f53610240527f60fd60205260606040536009604153606101610200527f80527f536042536060610260527f606101c0527f4353600a610140527f6044536061012052610220527f7f606045610280527f536000606101a0527f46536101e0527f60f3604753604860006000f0610240526102a0527f7f6000610160527f600060006000845af461016101610200527fc0527f40527f6102c0527f50610260527f5060006000600060006000600c5af2509b610180527f5a4158616102e0527f0220527f5833610280527f1b6101e0527f194415101a3e92ff610160527f4a7b610300527f8d45fa30f26c16041761026102a0527f40527f5f7b6101a0527f0a610200527f610320527f879537651d406f36445cf379638a662e6102c0527f602061610260527f018052610340527f60606101a05360006161610220527f01c0526001616102e0527f01e05360a161610360527f01e1610280527f5360536101e25360606101e35360f36101e461610300527f02610380527f40527f5360616101e55360016102a0527f6101e65360a26101e75360536101616103a0527f0320527fe85360616101e95360610260527f016101ea6102c0527f5360a361016103c0527feb536060610340527f6101ec5360006101ed5360f36101ee536101ef610280526103e0527f6102e0527f60606102610360527fa05360006102a15360606102a25360006102610400527fa35360f06102a453606061030052610380527f7f6102a55360006102a6536060610420527f6102a75360006102a85360606102a9536000616103a0527f02610320527faa53610440527f60606102ab5360006102ac5360606102ad5360006102ae536103c0527f608561610460527f02af5361034052606061036053605a61036153606161036253600261036103e0610480527f527f635360b06103645360536103655360606103665360f261036753606161036104a0527f6853610400527f60026103695360b161036a53605361036b53606061036c53606104c0527f5061036d536061610420527f61036e53600261036f5360b261037053605361036104e0527f715360606103725360506103610440527f735360616103745360026103755360610500527fb361037653605361037753606161037853610460527f60026103795360b46103610520527f7a53606061037b53600061037c5360f361037d53610361048052607e6104a053610540527f60606104a15360006104a25360606104a35360006104a45360f06104a5536060610560527f6104a65360006104a75360606104a85360006104a95360606104aa5360006104610580527fab5360606104ac5360006104ad5360606104ae5360006104af5360856104b0536105a05260606105c053605a6105c15360616105c25360046105c35360b16105c45360536105c55360606105c65360f26105c75360616105c85360046105c95360b26105ca5360536105cb5360606105cc5360506105cd5360616105ce5360046105cf5360b36105d05360536105d15360606105d25360506105d35360616105d45360046105d55360b46105d65360536105d75360616105d85360046105d95360b56105da5360606105db5360006105dc5360f36105dd5360006105de60006000f56000600060006000845af45050",
        "storage": {
          "0x0000000000000000000000000000000000000000000000000000000000000000": "0x0000000000000000000000000000000000000000000000000000000000000001"
        },
        "balance": "0x0",
        "nonce": "0x0"
      },
      "0x00000000000000000000000000000000000000fa": {
        "code": "0x600060006000600060095af4501c1b8300465c7f6005545060006002557f7f600454507f7f7f5f3d3fa17409494993586b9f04636000527f087a5ba31860026002556000527f5f6000527f5d7877816a6000527f966000526020527f7f326000526060602053600060216020527f53606060226020527f53600060236040527f536060606020527f24536000606020527f256040527f536060602653600060606060527f40527f275360606028536000602953606040527f60606060527f2a53600e602b6080527f536040527f605a6060527f602c5360f2602d536050602e5360076080527f602f60a0527f6060527f53609560305360846031536080527f60836060527f603253608160a060c0527f527f603353603060345360606080527f6035536020603660a0527f536060603760e0527f536060c0527e6038608052605360a053606060a15360f360a0527f60a25360c0610100527f527f60606060e0527fa353603960a453605360a553606060a653603a60a75360610120527f6060a860c05260e052610100527f7f7f53600060a95360f360aa5360ab600060610140527ef060006000600060006000855a610120527ff2610100527f5060e0527f5060610160527e60005539ec60035450600160035560f8ff7f610140527f6009545060036101610180527f20527f5450606060610100527e53600160015360ff600253610160527f6060606101a0527f0353600060046000610140527f527f53606060055360610120527e60066101806101c0527f527f53606060075360006008536060600953610160527f6000600a53606020526101e0527f7f606101a0527f600b610140527f536000600c536060600d5360086061018052610200527f7f0e53605a600f6101c0527f5360f1601053605060604052610160527f7f1153610220527f608a601253606101a0527f126101e0527f60135360f2601453604a6015536060610240527f60165360206017610180527f5360605261610200527f01c0527f606060805360610260527f60608153606060825360186083536053608453606060610220527f6101a05261610280527f01e0527f7f85536000608653606060875360196088536053608953610240527f6102a0527f6060608a5360f3608b610200527f536101c0527f6060608c53601a608d5360536102c0527f610260527f608e536060608f53601b60905360610220527f60609153600061016102e0527fe052606061610280527f02005360926102015360536102025360606102610240610300527f527f035360f3610204536102a0527f6060610205536093610206536053610207610320527f53606061020853610260527f6094616102c0527f020953606061020a53600061610340527f020b53606061020c53600061020d5360f06102806102e0527f527f61020e5360610360527f6061020f536000610210536060610211536000610212536060610300527f6102610380527f6102a0527f1353600061021453606061021553600061021653606061021761036103a0527f20527f536000610218536102c0527f608561021953605a61021a5360f261021b6103c0527f536050610340527f61021c53605061021d5361026102e052601e6103005360606103e0527f61030153600061036103605260026103805360536103815360606103825360f3610400527f6103835360616103845360036103855360036103865360536103875360616103610420527f8853600361038953600461038a53606061038b53600061038c53606061038d53610440527f600061038e5360f061038f536060610390536000610391536060610392536000610460527f6103935360606103945360006103955360606103965360006103975360606103610480527f9853600061039953608561039a53605a61039b5360f161039c53605061039d536104a05260606104c05360506104c15360616104c25360036104c353609e6104c45360536104c55360616104c65360036104c753609f6104c85360606104c95360006104ca5360f36104cb536104cc60006000f060006000600060006000855af15050",
        "storage": {
          "0x0000000000000000000000000000000000000000000000000000000000000000": "0x0000000000000000000000000000000000000000000000000000000000000011",
          "0x0000000000000000000000000000000000000000000000000000000000000002": "0x0000000000000000000000000000000000000000000000000000000000000008",
          "0x0000000000000000000000000000000000000000000000000000000000000003": "0x0000000000000000000000000000000000000000000000000000000000000005"
        },
        "balance": "0x0",
        "nonce": "0x0"
      },
      "0xa94f5374fce5edbc8e2a8697c15331677e6ebf0b": {
        "code": "0x",
        "storage": {},
        "balance": "0xffffffffff",
        "nonce": "0x0"
      }
    },
    "transaction": {
      "gasPrice": "0x10",
      "nonce": "0x0",
      "to": "0x00000000000000000000000000000000000000f1",
      "data": [
        "0x9f08000be94f5478327f274d3bfbbb8600ae77629851d8403c36c1750b21463f41a0910607b10aa821d6bef50f671e35f8b637414171c6bc587e59b2da9d20c6a4bf14430ef91ceb2b526abdf9e5"
      ],
      "gasLimit": [
        "0x7a1200"
      ],
      "value": [
        "0x"
      ],
      "sender": "0xa94f5374fce5edbc8e2a8697c15331677e6ebf0b",
      "secretKey": "0x45a915e4d060149eb4365960e6a7a45f334393093061116b197e3240065ff2d8"
    },
    "out": "0x",
    "post": {
      "Cancun": [
        {
          "hash": "0x0000000000000000000000000000000000000000000000000000000000000000",
          "logs": "0x0000000000000000000000000000000000000000000000000000000000000000",
          "indexes": {
            "data": 0,
            "gas": 0,
            "value": 0
          }
        }
      ]
    }
  }
}
holiman commented 6 days ago

Tried to repro it

root@a04949b121fe:/# yes "/fuzztmp/10573375-mixed-8.json" | /neth/nethtest --trace -m -x  2>/dev/null | grep "time" | xargs -L1 bash checkmem.sh nethtest
PID 456170 proc nethtest | Rss: 728460 kB | Pss: 1417619 kB | Shared Clean 7348 kB | Shared Dirty 40 kB | Private 732420 kB
PID 456170 proc nethtest | Rss: 751540 kB | Pss: 1464707 kB | Shared Clean 7348 kB | Shared Dirty 48 kB | Private 754788 kB
PID 456170 proc nethtest | Rss: 775672 kB | Pss: 1512089 kB | Shared Clean 7292 kB | Shared Dirty 32 kB | Private 778992 kB
PID 456170 proc nethtest | Rss: 799328 kB | Pss: 1559395 kB | Shared Clean 7360 kB | Shared Dirty 32 kB | Private 802492 kB
PID 456170 proc nethtest | Rss: 823460 kB | Pss: 1608071 kB | Shared Clean 7364 kB | Shared Dirty 32 kB | Private 826936 kB
PID 456170 proc nethtest | Rss: 847404 kB | Pss: 1658023 kB | Shared Clean 7356 kB | Shared Dirty 32 kB | Private 853024 kB
PID 456170 proc nethtest | Rss: 874820 kB | Pss: 1713117 kB | Shared Clean 7412 kB | Shared Dirty 28 kB | Private 880676 kB

After a couple of minutes:

PID 456170 proc nethtest | Rss: 4451112 kB | Pss: 8861466 kB | Shared Clean 7420 kB | Shared Dirty 28 kB | Private 4443600 kB
PID 456170 proc nethtest | Rss: 4451112 kB | Pss: 8861496 kB | Shared Clean 7436 kB | Shared Dirty 28 kB | Private 4443584 kB

A few minutes later it has gone down a bit again:

PID 456170 proc nethtest | Rss: 4329912 kB | Pss: 8619118 kB | Shared Clean 7540 kB | Shared Dirty 32 kB | Private 4322404 kB
PID 456170 proc nethtest | Rss: 4329912 kB | Pss: 8619137 kB | Shared Clean 7496 kB | Shared Dirty 32 kB | Private 4322384 kB

Seems to stabilize around here

PID 456170 proc nethtest | Rss: 4351724 kB | Pss: 8662759 kB | Shared Clean 7492 kB | Shared Dirty 36 kB | Private 4344196 kB

So yeah, no obvious easily repdocucible leak.

memchecker-script:

#!/bin/bash
if [[ -z "$2" ]]; then
  exit 0
fi
for pid in $(ps -ef | awk '{print $2}'); do
    if [[ -z "$pid" ]]; then
        continue
    fi
    if [[ $pid == "PID" ]]; then
        continue
    fi
    a=$(ps -p $pid -o comm=)
    if [[ $a != "nethtest" ]]; then
        continue
    fi
    if [ -f /proc/$pid/smaps ]; then
        rss=$(awk 'BEGIN {i=0} /^Rss/ {i = i + $2} END {print i}' /proc/$pid/smaps)
        pss=$(awk 'BEGIN {i=0} /^Pss/ {i = i + $2 + 0.5} END {print i}' /proc/$pid/smaps)
        sc=$(awk 'BEGIN {i=0} /^Shared_Clean/ {i = i + $2} END {print i}' /proc/$pid/smaps)
        sd=$(awk 'BEGIN {i=0} /^Shared_Dirty/ {i = i + $2} END {print i}' /proc/$pid/smaps)
        pc=$(awk 'BEGIN {i=0} /^Private_Clean/ {i = i + $2} END {print i}' /proc/$pid/smaps)
        pd=$(awk 'BEGIN {i=0} /^Private_Dirty/ {i = i + $2} END {print i}' /proc/$pid/smaps)
        echo "PID $pid proc $a | Rss: $rss kB | Pss: $pss kB | Shared Clean $sc kB | Shared Dirty $sd kB | Private $(($pd + $pc)) kB"
    fi
done
holiman commented 6 days ago

After running for a few hours on a server, nethtest is way larger than any of the other:

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                                                                                                               
2537959 root      20   0  537.8g  13.8g  64148 R  53.3   5.5 213:47.46 nethtest                                                                                                                                                                                                              
2537951 root      20   0   53.2g   1.6g  28348 S  54.6   0.6 113:57.21 java                                                                                                                                                                                                                  
2537955 root      20   0   11.0g   1.5g  41004 S 175.2   0.6      6,25 erigon_vm   

cc @MarekM25 @LukaszRozmej any ideas?

LukaszRozmej commented 5 days ago

After running for a few hours on a server, nethtest is way larger than any of the other:

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                                                                                                               
2537959 root      20   0  537.8g  13.8g  64148 R  53.3   5.5 213:47.46 nethtest                                                                                                                                                                                                              
2537951 root      20   0   53.2g   1.6g  28348 S  54.6   0.6 113:57.21 java                                                                                                                                                                                                                  
2537955 root      20   0   11.0g   1.5g  41004 S 175.2   0.6      6,25 erigon_vm   

cc @MarekM25 @LukaszRozmej any ideas?

We either have to reproduce, or if you have similar situation next time, you could grab us memory dump/snapshot? You can use https://www.jetbrains.com/help/dotmemory/Working_with_dotMemory_Command-Line_Profiler.html

holiman commented 5 days ago

A day later (same server), it has gone from 13.8 to 15.5:

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                                                                                                               
2537959 root      20   0  538.6g  15.5g  64164 R  50.0   6.2     28,36 nethtest                                                                                                                                                                                                              
2537951 root      20   0   55.0g   1.8g  28348 S  54.6   0.7     15,18 java            

I'll try to dump it.... But, it's going to be a pretty damn large dump..?

holiman commented 5 days ago

Meh getting that thing to work inside a docker on a remote server seems not trivial

root@fb2a64aee5d9:/# /fuzztmp/JetBrains.dotMemory.linux-x64.2024.2.7/linux-x64/dotMemory -h 
Unhandled exception. System.Exception: XOpenDisplay failed
   at Avalonia.X11.AvaloniaX11Platform.Initialize(X11PlatformOptions options) in Z:\BuildAgent\work\f216b9e13ea6fd05\Avalonia\src\Avalonia.X11\X11Platform.cs:line 55
   at Avalonia.AvaloniaX11PlatformExtensions.<>c.<UseX11>b__0_0() in Z:\BuildAgent\work\f216b9e13ea6fd05\Avalonia\src\Avalonia.X11\X11Platform.cs:line 354
   at Avalonia.AppBuilder.SetupUnsafe() in Z:\BuildAgent\work\f216b9e13ea6fd05\Avalonia\src\Avalonia.Controls\AppBuilder.cs:line 328
   at Avalonia.AppBuilder.Setup() in Z:\BuildAgent\work\f216b9e13ea6fd05\Avalonia\src\Avalonia.Controls\AppBuilder.cs:line 316
LukaszRozmej commented 5 days ago

Meh getting that thing to work inside a docker on a remote server seems not trivial

root@fb2a64aee5d9:/# /fuzztmp/JetBrains.dotMemory.linux-x64.2024.2.7/linux-x64/dotMemory -h 
Unhandled exception. System.Exception: XOpenDisplay failed
   at Avalonia.X11.AvaloniaX11Platform.Initialize(X11PlatformOptions options) in Z:\BuildAgent\work\f216b9e13ea6fd05\Avalonia\src\Avalonia.X11\X11Platform.cs:line 55
   at Avalonia.AvaloniaX11PlatformExtensions.<>c.<UseX11>b__0_0() in Z:\BuildAgent\work\f216b9e13ea6fd05\Avalonia\src\Avalonia.X11\X11Platform.cs:line 354
   at Avalonia.AppBuilder.SetupUnsafe() in Z:\BuildAgent\work\f216b9e13ea6fd05\Avalonia\src\Avalonia.Controls\AppBuilder.cs:line 328
   at Avalonia.AppBuilder.Setup() in Z:\BuildAgent\work\f216b9e13ea6fd05\Avalonia\src\Avalonia.Controls\AppBuilder.cs:line 316

We have a dockerfile with diagnostic tools: https://github.com/NethermindEth/nethermind/blob/master/Dockerfile.diag#L38