search

occlum / occlum

Occlum is a memory-safe, multi-process library OS for Intel SGX
https://occlum.io/
Other
1.41k stars 235 forks source link

[BUG]Fatal: Failed to retrieve file descriptor allowance: function not implemented #1044

Open prz23 opened 2 years ago

prz23 commented 2 years ago

Describe the bug

I'm having problems trying to run Geth[https://github.com/ethereum/go-ethereum] in Occlum. It is Fatal: Failed to retrieve file descriptor allowance: function not implemented.

To reproduce

Steps to reproduce the behavior:

  1. docker run -it --rm occlum/occlum:0.29.1-ubuntu20.04

  2. clone Geth git clone https://github.com/ethereum/go-ethereum && cd go-ethereum and git checkout v1.10.25

  3. replace go with occlum-go in Makefile: GORUN = env GO111MODULE=on occlum-go run

  4. build geth with make geth

  5. copy the binary to occlum directory & rename it to web_server: cp ./build/bin/geth ~/demos/golang/web_server/web_server

  6. configure the resource & command to launch Geth in run_golang_on_occlum.sh

    new_json="$(jq '.resource_limits.user_space_size = "5000MB" |
            .process.default_mmap_size = "4500MB" |
            .resource_limits.kernel_space_heap_size = "6400MB" |
            .resource_limits.kernel_space_stack_size = "60MB" |
            .process.default_stack_size = "250MB" |
            .process.default_heap_size = "640MB"' Occlum.json)" && \
echo "${new_json}" > Occlum.json

    occlum run /bin/web_server --syncmode light --http

  7. start SGX_MODE=SIM ./run_golang_on_occlum.sh

    root@e7853f7e50a2:~/demos/golang/web_server# SGX_MODE=SIM ./run_golang_on_occlum.sh
/root/demos/golang/web_server/occlum_instance initialized as an Occlum instance
Enclave sign-tool: /opt/occlum/sgxsdk-tools/bin/x64/sgx_sign
Enclave sign-key: /opt/occlum/etc/template/Enclave.pem
SGX mode: SIM
rm -rf /root/demos/golang/web_server/occlum_instance/build
Building new image...
[+] Home dir is /root
[+] Open token file success!
[+] Token file valid!
[+] Init Enclave Successful 41476499177474!
Generate the SEFS image successfully
Building the initfs...
[+] Home dir is /root
[+] Open token file success!
[+] Token file valid!
[+] Init Enclave Successful 41545218654210!
Generate the SEFS image successfully
Building libOS...
Signing the enclave...
<EnclaveConfiguration>
<ProdID>0</ProdID>
<ISVSVN>0</ISVSVN>
<StackMaxSize>62914560</StackMaxSize>
<StackMinSize>62914560</StackMinSize>
<HeapMaxSize>6710886400</HeapMaxSize>
<HeapMinSize>6710886400</HeapMinSize>
<TCSNum>32</TCSNum>
<TCSPolicy>1</TCSPolicy>
<DisableDebug>0</DisableDebug>
<MiscSelect>0</MiscSelect>
<MiscMask>0xFFFFFFFF</MiscMask>
<ReservedMemMaxSize>5242880000</ReservedMemMaxSize>
<ReservedMemMinSize>5242880000</ReservedMemMinSize>
<ReservedMemInitSize>5242880000</ReservedMemInitSize>
<ReservedMemExecutable>1</ReservedMemExecutable>
<EnableKSS>0</EnableKSS>
<ISVEXTPRODID_H>0</ISVEXTPRODID_H>
<ISVEXTPRODID_L>0</ISVEXTPRODID_L>
<ISVFAMILYID_H>0</ISVFAMILYID_H>
<ISVFAMILYID_L>0</ISVFAMILYID_L>
<PKRU>0</PKRU>
</EnclaveConfiguration>
tcs_num 32, tcs_max_num 32, tcs_min_pool 1
The required memory is 13971877888B.
The required memory is 0x340c9f000, 13644412 KB.
Succeed.
Built the Occlum image and enclave successfully
occlum run /bin/web_server
INFO [11-03|06:24:46.925] Starting Geth on Ethereum mainnet...
INFO [11-03|06:24:46.925] Dropping default light client cache      provided=1024 updated=128
INFO [11-03|06:24:46.927] Maximum peer count                       ETH=0 LES=10 total=50
INFO [11-03|06:24:46.928] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
Fatal: Failed to retrieve file descriptor allowance: function not implemented
root@e7853f7e50a2:~/demos/golang/web_server#

Expected behavior

Copying ./build/bin/geth elsewhere also reports this err, but it all works fine. Smartcard socket not found, disabling err="stat /run/pcscd/pcscd.comm: no such file or directory"

Environment

prz23 commented 2 years ago 
func Maximum() (int, error) {
    var limit syscall.Rlimit
    if err := syscall.Getrlimit(syscall.RLIMIT_NOFILE, &limit); err != nil {
        return 0, err
    }
    return int(limit.Max), nil
}
prz23 commented 2 years ago

the program will crush when syscalls about file descriptors is called.

func Raise(max uint64) (uint64, error) {
    // Get the current limit
    var limit syscall.Rlimit
    if err := syscall.Getrlimit(syscall.RLIMIT_NOFILE, &limit); err != nil {
        return 0, err
    }
    // Try to update the limit to the max allowance
    limit.Cur = limit.Max
    if limit.Cur > max {
        limit.Cur = max
    }
    if err := syscall.Setrlimit(syscall.RLIMIT_NOFILE, &limit); err != nil {
        return 0, err
    }
    // MacOS can silently apply further caps, so retrieve the actually set limit
    if err := syscall.Getrlimit(syscall.RLIMIT_NOFILE, &limit); err != nil {
        return 0, err
    }
    return limit.Cur, nil
}
JustinDrake commented 1 year ago

On the topic of running Geth in SGX, the Flashbots team recently released this post. They have Geth running in SGX with Gramine—the diff is tiny.

qzheng527 commented 1 year ago

@prz23 the two syscall gettrlimit and settrlimit are implemented by this PR. https://github.com/occlum/occlum/pull/1083 You can try apply the PR and rebuild/install the Occlum, then see if the Geth can run well in Occlum.