taikoxyz / raiko

Multi-proofs for Taiko. SNARKS, STARKS and Trusted Execution Enclave. Our previous ZK-EVM circuits are deprecated.
Apache License 2.0
96 stars 75 forks source link

SGX prove fails with: Can not serialize input for SGX io error #227

Open davaymne opened 1 month ago

davaymne commented 1 month ago

Describe the bug

BACKGROUND: Node successfully generated and submitted proof for block 179947

But when it tried to prove follow yo blocks, e.g. 180,466 it gets error:

ERROR[05-20|14:49:17.718] Failed to request proof                  height=180,466 error="Can not serialize input for SGX io error: Broken pipe (os error 32), output is Ok(Output { status: ExitStatus(unix_wait_status(256)), stdout: \"Starting one shot mode\\nGlobal options: GlobalOpts { secrets_dir: \\\"/root/.config/raiko/secrets\\\", config_dir: \\\"/root/.config/raiko/config\\\" }, OneShot options: OneShotArgs { sgx_instance_id: 5466 }\\nmemory allocation of 184 bytes failed\\n\", stderr: \"Gramine is starting. Parsing TOML manifest file, this may take some time...\\n-----------------------------------------------------------------------------------------------------------------------\\nGramine detected the following insecure configurations:\\n\\n  - loader.insecure__use_cmdline_argv = true   (forwarding command-line args from untrusted host to the app)\\n  - sys.insecure__allow_eventfd = true         (host-based eventfd is enabled)\\n  - sgx.allowed_files = [ ... ]                (some files are passed through from untrusted host without verification)\\n\\nGramine will continue application execution, but this configuration must not be used in production!\\n-----------------------------------------------------------------------------------------------------------------------\\n\\n[P1:T4:sgx-guest] error: Out-of-memory in library OS\\n\" })" endpoint=http://51.77.21.107:8080
INFO [05-20|14:49:55.410] Proof generated                          height=180,466 time=1m4.972853124s producer=SGXProofProducer

SGX server is configured with Processor memory reserved: 512 MB:

Screenshot 2024-05-20 at 17 01 15

Steps to reproduce

Steps to reproduce here.

Spam policy

davaymne commented 1 month ago

UPDATE:

SGX prover keeps throwing error messages, but at the same time keep generating proof:

INFO [05-20|22:54:26.674] Request proof from raiko-host service    blockID=182,180 coinbase=0x44c7dB3ac68d92398f88Cb2BD98C118925080e11 height=182,180 hash=4d1881..b6ea37
ERROR[05-20|22:54:28.793] Failed to request proof                  height=182,180 error="Can not serialize input for SGX io error: Broken pipe (os error 32), output is Ok(Output { status: ExitStatus(unix_wait_status(256)), stdout: \"Starting one shot mode\\nGlobal options: GlobalOpts { secrets_dir: \\\"/root/.config/raiko/secrets\\\", config_dir: \\\"/root/.config/raiko/config\\\" }, OneShot options: OneShotArgs { sgx_instance_id: 5466 }\\nmemory allocation of 184 bytes failed\\n\", stderr: \"Gramine is starting. Parsing TOML manifest file, this may take some time...\\n-----------------------------------------------------------------------------------------------------------------------\\nGramine detected the following insecure configurations:\\n\\n  - loader.insecure__use_cmdline_argv = true   (forwarding command-line args from untrusted host to the app)\\n  - sys.insecure__allow_eventfd = true         (host-based eventfd is enabled)\\n  - sgx.allowed_files = [ ... ]                (some files are passed through from untrusted host without verification)\\n\\nGramine will continue application execution, but this configuration must not be used in production!\\n-----------------------------------------------------------------------------------------------------------------------\\n\\n[P1:T15:sgx-guest] error: Out-of-memory in library OS\\n\" })" endpoint=http://51.77.21.107:8080
INFO [05-20|22:54:38.324] Check synced L1 snippet from anchor      blockID=182,180 l1Height=1,581,891
ERROR[05-20|22:54:40.855] Failed to request proof                  height=182,180 error="Can not serialize input for SGX io error: Broken pipe (os error 32), output is Ok(Output { status: ExitStatus(unix_wait_status(256)), stdout: \"Starting one shot mode\\nGlobal options: GlobalOpts { secrets_dir: \\\"/root/.config/raiko/secrets\\\", config_dir: \\\"/root/.config/raiko/config\\\" }, OneShot options: OneShotArgs { sgx_instance_id: 5466 }\\nmemory allocation of 184 bytes failed\\n\", stderr: \"Gramine is starting. Parsing TOML manifest file, this may take some time...\\n-----------------------------------------------------------------------------------------------------------------------\\nGramine detected the following insecure configurations:\\n\\n  - loader.insecure__use_cmdline_argv = true   (forwarding command-line args from untrusted host to the app)\\n  - sys.insecure__allow_eventfd = true         (host-based eventfd is enabled)\\n  - sgx.allowed_files = [ ... ]                (some files are passed through from untrusted host without verification)\\n\\nGramine will continue application execution, but this configuration must not be used in production!\\n-----------------------------------------------------------------------------------------------------------------------\\n\\n[P1:T13:sgx-guest] error: Out-of-memory in library OS\\n\" })" endpoint=http://51.77.21.107:8080
INFO [05-20|22:54:41.456] Check synced L1 snippet from anchor      blockID=182,180 l1Height=1,581,891
INFO [05-20|22:54:53.298] Proof generated                          height=182,180 time=26.623564692s producer=SGXProofProducer
INFO [05-20|22:54:53.298] NewProofSubmitter block proof            blockID=182,180 coinbase=0x44c7dB3ac68d92398f88Cb2BD98C118925080e11 parentHash=3ce44b..de1151 hash=4d1881..b6ea37 stateRoot=77c37b..e9d758 proof=0000155a717639a029c7e5db6eddd6561b101269a4a17ecabcd937853b3a5be3e8c0e0357f2f450fdf26da8fbf2505131cefea562f6e72f868081312ece07165295e92d4194f76b353e02c50f54c17e49db04877963ab4df1c tier=200
INFO [05-20|22:54:53.308] Build proof submission transaction       blockID=182,180 gasLimit=0 guardian=false
INFO [05-20|22:54:53.328] Publishing transaction                   service=prover tx=a7edc7..fde19a nonce=72 gasTipCap=1,000,000,000 gasFeeCap=12,598,439,738 gasLimit=298,584
INFO [05-20|22:54:53.331] Transaction successfully published       service=prover tx=a7edc7..fde19a nonce=72 gasTipCap=1,000,000,000 gasFeeCap=12,598,439,738 gasLimit=298,584
INFO [05-20|22:54:56.456] Check synced L1 snippet from anchor      blockID=182,180 l1Height=1,581,891
INFO [05-20|22:55:04.988] Proof assignment request body            feeToken=0x0000000000000000000000000000000000000000 expiry=1,716,247,504 tierFees="[{Tier:100 Fee:+1000000000} {Tier:200 Fee:+1000000000} {Tier:1000 Fee:+0}]" blobHash=012680..ba5bfc currentUsedCapacity=0
INFO [05-20|22:55:04.988] Prover's ETH balance                     balance=36.56251266 address=0xfbfd4F6993BC0D3481B9bf61AD0892f817a2e7aC
INFO [05-20|22:55:04.989] Prover's Taiko token balance             balance=0           address=0xfbfd4F6993BC0D3481B9bf61AD0892f817a2e7aC
WARN [05-20|22:55:04.989] Prover does not have required on-chain Taiko token balance providedProver=0xfbfd4F6993BC0D3481B9bf61AD0892f817a2e7aC taikoTokenBalance=0 minTaikoTokenBalance=0
INFO [05-20|22:55:04.990] Proof assignment request body            feeToken=0x0000000000000000000000000000000000000000 expiry=1,716,247,504 tierFees="[{Tier:100 Fee:+1100000000} {Tier:200 Fee:+1100000000} {Tier:1000 Fee:+0}]" blobHash=012680..ba5bfc currentUsedCapacity=0
INFO [05-20|22:55:04.991] Prover's ETH balance                     balance=36.56251266 address=0xfbfd4F6993BC0D3481B9bf61AD0892f817a2e7aC
INFO [05-20|22:55:04.992] Prover's Taiko token balance             balance=0           address=0xfbfd4F6993BC0D3481B9bf61AD0892f817a2e7aC
WARN [05-20|22:55:04.992] Prover does not have required on-chain Taiko token balance providedProver=0xfbfd4F6993BC0D3481B9bf61AD0892f817a2e7aC taikoTokenBalance=0 minTaikoTokenBalance=0
INFO [05-20|22:55:04.993] Proof assignment request body            feeToken=0x0000000000000000000000000000000000000000 expiry=1,716,247,504 tierFees="[{Tier:100 Fee:+1320000000} {Tier:200 Fee:+1320000000} {Tier:1000 Fee:+0}]" blobHash=012680..ba5bfc currentUsedCapacity=0
INFO [05-20|22:55:04.993] Prover's ETH balance                     balance=36.56251266 address=0xfbfd4F6993BC0D3481B9bf61AD0892f817a2e7aC
INFO [05-20|22:55:04.994] Prover's Taiko token balance             balance=0           address=0xfbfd4F6993BC0D3481B9bf61AD0892f817a2e7aC
WARN [05-20|22:55:04.994] Prover does not have required on-chain Taiko token balance providedProver=0xfbfd4F6993BC0D3481B9bf61AD0892f817a2e7aC taikoTokenBalance=0 minTaikoTokenBalance=0
INFO [05-20|22:55:05.340] Transaction confirmed                    service=prover tx=a7edc7..fde19a block=07b0e5..730636:1581901 effectiveGasPrice=7,418,930,979
INFO [05-20|22:55:05.341] "💰 Your block proof was accepted"     blockID=182,180 parentHash=3ce44b..de1151 hash=4d1881..b6ea37 stateRoot=77c37b..e9d758 txHash=a7edc7..fde19a tier=200 isContest=false
smtmfft commented 1 month ago

It's OOM, try increasing gramine memory related setting maybe helpful. Reference: https://gramine.readthedocs.io/en/latest/manifest-syntax.html#enclave-size