cms register-data-keys执行报错：CapsuleManager server error code: 13 Internal Error

vi0eros commented 7 months ago

跟着trustedflow文档：https://www.secretflow.org.cn/zh-CN/docs/trustedflow/main/quick_start/step2 文档，执行到alice上传密钥时报错，下面是命令和报错内容：

cms register-data-keys


lany@sgx:~# cms register-data-keys
Traceback (most recent call last):
File "/home/lany/.local/bin/cms", line 8, in <module>
sys.exit(cms())
File "/usr/lib/python3/dist-packages/click/core.py", line 764, in __call__
return self.main(*args, **kwargs)
File "/usr/lib/python3/dist-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/lib/python3/dist-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/lib/python3/dist-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/usr/lib/python3/dist-packages/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "/home/lany/.local/lib/python3.8/site-packages/cli/cms.py", line 146, in register_data_keys
ctx.obj.create_data_keys(
File "/home/lany/.local/lib/python3.8/site-packages/sdc/capsule_manager_frame.py", line 391, in create_data_keys
request, self.get_public_key(), private_key, cert_pems
File "/home/lany/.local/lib/python3.8/site-packages/sdc/capsule_manager_frame.py", line 176, in get_public_key
raise CapsuleManagerError(response.status.code, response.status.message)
sdc.error.CapsuleManagerError: CapsuleManager server error code: 13, error message: err code: Internal error; err detail: runified_attestation_generate_auth_report err: Error { code: UnifiedAttErr { errcode: 279248896 }, details: Some("runified_attestation_generate_auth_report: report generate failure"), location: Some(ErrorLocation { line: 112, file: "capsule-manager/src/remote_attestation/unified_attestation_wrapper.rs" }) }; location: [line = 69, file = capsule-manager/src/server/ra_impl.rs]

- 于此同时CapsuleManager容器内报错：

root@sgx:/home/admin/occlum_release# occlum run /bin/capsule_manager --config_path /host/config.yaml config Config { port: Some( 8888, ), log_config: LogConfig { log_dir: Some( "/host/logs", ), log_level: Some( "info", ), enable_console_logger: Some( true, ), }, scheme: Some( "RSA", ), storage_backend: Some( "inmemory", ), server_cert_path: Some( "/host/resources/cert/server.crt", ), server_cert_key_path: Some( "/host/resources/cert/server.key", ), client_ca_cert_path: Some( "/host/resources/ca.crt", ), enable_tls: Some( false, ), mode: Some( "production", ), } ST, HZ O, AntGroup CN, CapsuleManager C, CN OU, SecretFlow L, HZ [2024-03-05T19:31:10.077906042+08:00] [capsule_manager] [INFO] Server run at: 0.0.0.0:8888 mode Some("production")

[INFO][ual/utils/untrusted/untrusted_json.cpp:35] configurations_is_signed is missed or not string in config file [WARN][ual/utils/untrusted/untrusted_json.cpp:308] Please use signed configuration file in release mode [INFO][ual/utils/untrusted/untrusted_json.cpp:316] Load configuration file unified_attestation.json successfully [INFO][ual/common/uak.cpp:20] Initialize UAK ... [ERROR][ual/network/pccs/pccs_client.cpp:86] Canot find sgx-tcb-info in pccs response header [ERROR][ual/network/pccs/pccs_client.cpp:184] [Function] GetCollateral [ERROR][ual/generation/platforms/sgx2/untrusted/generator_sgx_dcap.cpp:295] [Function] CreatePassportReport [ERROR][ual/generation/core/generator.cpp:83] [Function] GenerateReport [ERROR][ual/generation/untrusted/untrusted_ua_generation.cpp:34] [Function] UaGenerateReport [ERROR][ual/generation/untrusted/untrusted_ua_generation.cpp:42] [Function] UaGenerateReportJson [ERROR][ual/generation/untrusted/untrusted_unified_attestation_generation.cpp:36] [Function] UnifiedAttestationGenerateReport [2024-03-05T19:31:20.938393325+08:00] [monitor] [INFO] |get_ra_cert|13|"err code: Internal error; err detail: runified_attestation_generate_auth_report err: Error { code: UnifiedAttErr { errcode: 279248896 }, details: Some(\"runified_attestation_generate_auth_report: report generate failure\"), location: Some(ErrorLocation { line: 112, file: \"capsule-manager/src/remote_attestation/unified_attestation_wrapper.rs\" }) }; location: [line = 69, file = capsule-manager/src/server/ra_impl.rs]"

yuki252111 commented 7 months ago

生产模式必须要在sgx的机器上跑。这个报错看起来不是在sgx机器上

vi0eros commented 7 months ago

生产模式必须要在sgx的机器上跑。这个报错看起来不是在sgx机器上

我是在sgx机器上跑，下面是我的配置：

 root@sgx:/opt/intel/sgx/SampleCode/SampleEnclave# cpuid | grep -i sgx
   SGX: Software Guard Extensions supported = true
      SGX_LC: SGX launch config supported      = true
   Software Guard Extensions (SGX) capability (0x12/0):
      SGX1 supported                         = true
      SGX2 supported                         = true
      SGX ENCLV E*VIRTCHILD, ESETCONTEXT     = true
      SGX ENCLS ETRACKC, ERDINFO, ELDBC, ELDUC = true
   SGX attributes (0x12/1):
   SGX EPC enumeration (0x12/n):
      SGX: Software Guard Extensions supported = true
      SGX_LC: SGX launch config supported      = true
   Software Guard Extensions (SGX) capability (0x12/0):
      SGX1 supported                         = true
      SGX2 supported                         = true
      SGX ENCLV E*VIRTCHILD, ESETCONTEXT     = true
      SGX ENCLS ETRACKC, ERDINFO, ELDBC, ELDUC = true
   SGX attributes (0x12/1):
   SGX EPC enumeration (0x12/n):
      SGX: Software Guard Extensions supported = true
      SGX_LC: SGX launch config supported      = true
   Software Guard Extensions (SGX) capability (0x12/0):
      SGX1 supported                         = true
      SGX2 supported                         = true
      SGX ENCLV E*VIRTCHILD, ESETCONTEXT     = true
      SGX ENCLS ETRACKC, ERDINFO, ELDBC, ELDUC = true
   SGX attributes (0x12/1):
   SGX EPC enumeration (0x12/n):
      SGX: Software Guard Extensions supported = true
      SGX_LC: SGX launch config supported      = true
   Software Guard Extensions (SGX) capability (0x12/0):
      SGX1 supported                         = true
      SGX2 supported                         = true
      SGX ENCLV E*VIRTCHILD, ESETCONTEXT     = true
      SGX ENCLS ETRACKC, ERDINFO, ELDBC, ELDUC = true
   SGX attributes (0x12/1):
   SGX EPC enumeration (0x12/n):
      SGX: Software Guard Extensions supported = true
      SGX_LC: SGX launch config supported      = true
   Software Guard Extensions (SGX) capability (0x12/0):
      SGX1 supported                         = true
      SGX2 supported                         = true
      SGX ENCLV E*VIRTCHILD, ESETCONTEXT     = true
      SGX ENCLS ETRACKC, ERDINFO, ELDBC, ELDUC = true
   SGX attributes (0x12/1):
   SGX EPC enumeration (0x12/n):
      SGX: Software Guard Extensions supported = true
      SGX_LC: SGX launch config supported      = true
   Software Guard Extensions (SGX) capability (0x12/0):
      SGX1 supported                         = true
      SGX2 supported                         = true
      SGX ENCLV E*VIRTCHILD, ESETCONTEXT     = true
      SGX ENCLS ETRACKC, ERDINFO, ELDBC, ELDUC = true
   SGX attributes (0x12/1):
   SGX EPC enumeration (0x12/n):
      SGX: Software Guard Extensions supported = true
      SGX_LC: SGX launch config supported      = true
   Software Guard Extensions (SGX) capability (0x12/0):
      SGX1 supported                         = true
      SGX2 supported                         = true
      SGX ENCLV E*VIRTCHILD, ESETCONTEXT     = true
      SGX ENCLS ETRACKC, ERDINFO, ELDBC, ELDUC = true
   SGX attributes (0x12/1):
   SGX EPC enumeration (0x12/n):
      SGX: Software Guard Extensions supported = true
      SGX_LC: SGX launch config supported      = true
   Software Guard Extensions (SGX) capability (0x12/0):
      SGX1 supported                         = true
      SGX2 supported                         = true
      SGX ENCLV E*VIRTCHILD, ESETCONTEXT     = true
      SGX ENCLS ETRACKC, ERDINFO, ELDBC, ELDUC = true
   SGX attributes (0x12/1):
   SGX EPC enumeration (0x12/n):
      SGX: Software Guard Extensions supported = true
      SGX_LC: SGX launch config supported      = true
   Software Guard Extensions (SGX) capability (0x12/0):
      SGX1 supported                         = true
      SGX2 supported                         = true
      SGX ENCLV E*VIRTCHILD, ESETCONTEXT     = true
      SGX ENCLS ETRACKC, ERDINFO, ELDBC, ELDUC = true
   SGX attributes (0x12/1):
   SGX EPC enumeration (0x12/n):
      SGX: Software Guard Extensions supported = true
      SGX_LC: SGX launch config supported      = true
   Software Guard Extensions (SGX) capability (0x12/0):
      SGX1 supported                         = true
      SGX2 supported                         = true
      SGX ENCLV E*VIRTCHILD, ESETCONTEXT     = true
      SGX ENCLS ETRACKC, ERDINFO, ELDBC, ELDUC = true
   SGX attributes (0x12/1):
   SGX EPC enumeration (0x12/n):
      SGX: Software Guard Extensions supported = true
      SGX_LC: SGX launch config supported      = true
   Software Guard Extensions (SGX) capability (0x12/0):
      SGX1 supported                         = true
      SGX2 supported                         = true
      SGX ENCLV E*VIRTCHILD, ESETCONTEXT     = true
      SGX ENCLS ETRACKC, ERDINFO, ELDBC, ELDUC = true
   SGX attributes (0x12/1):
   SGX EPC enumeration (0x12/n):
      SGX: Software Guard Extensions supported = true
      SGX_LC: SGX launch config supported      = true
   Software Guard Extensions (SGX) capability (0x12/0):
      SGX1 supported                         = true
      SGX2 supported                         = true
      SGX ENCLV E*VIRTCHILD, ESETCONTEXT     = true
      SGX ENCLS ETRACKC, ERDINFO, ELDBC, ELDUC = true
   SGX attributes (0x12/1):
   SGX EPC enumeration (0x12/n):

yuki252111 commented 7 months ago

[ERROR][ual/network/pccs/pccs_client.cpp:86] Canot find sgx-tcb-info in pccs response header 可以看一下这个，pccs url有正常的替换么

vi0eros commented 7 months ago

[ERROR][ual/network/pccs/pccs_client.cpp:86] 无法在 pccs 响应头中找到 sgx-tcb-info 可以看一下这个，pccs url 有正常的替换么

下面三个配置pccs的地方我都有替换：

CapsuleManager容器内：

root@sgx:/home/admin# cat occlum_release/image/etc/kubetee/unified_attestation.json 
{
"ua_ias_url": "",
"ua_ias_spid": "",
"ua_ias_apk_key": "",
"ua_dcap_lib_path": "",
"ua_dcap_pccs_url": "https://localhost:8081/sgx/certification/v4/",
"ua_uas_url": "",
"ua_uas_app_key": "",
"ua_uas_app_secret": ""
}

宿主机：

lany@sgx:~$ cat /etc/sgx_default_qcnl.conf 
{
// *** ATTENTION : This file is in JSON format so the keys are case sensitive. Don't change them.

//PCCS server address
"pccs_url": "https://localhost:8081/sgx/certification/v4/"

// To accept insecure HTTPS certificate, set this option to false
,"use_secure_cert": false
.......

lany@sgx:~$ sudo cat /opt/intel/sgx-dcap-pccs/config/default.json 
{
"HTTPS_PORT" : 8081,
"hosts" : "0.0.0.0",
"uri": "https://api.trustedservices.intel.com/sgx/certification/v4/",
"ApiKey": "xxxxx",
"proxy" : "",
"RefreshSchedule": "0 0 1 * * *",
......

yuki252111 commented 7 months ago

CapsuleManager容器里面的 occlum_release/image/etc/kubetee/unified_attestation.json 和 /etc/sgx_default_qcnl.conf 里面的 pccs_url 改成真实的

vi0eros commented 7 months ago

CapsuleManagerContainer 里面的 occlum_release/image/etc/kubetee/unified_attestation.json 和 /etc/sgx_default_qcnl.conf 里面的 pccs_url 改成真实的

我都更改为真实ip了，并重启了pccs服务和docker容器，但还是报同样的错误。

yuki252111 commented 7 months ago

改了之后用occlum重新签一下

yuki252111 commented 7 months ago

而且路径不应该是/etc/sgx_default_qcnl.conf么，我看你那个路径是opt下面

vi0eros commented 7 months ago

/opt/intel/sgx-dcap-pccs/config/default.json

重新签后也不行，路径/opt/intel/sgx-dcap-pccs/config/default.json是pccs的配置文件。occlum_release/image/etc/kubetee/unified_attestation.json 和 /etc/sgx_default_qcnl.conf都修改了。

附上pccs服务的日志：


lany@sgx:~$ cat /opt/intel/sgx-dcap-pccs/logs/pccs_server.log
......

2024-03-06 13:15:37.584 [info]: HTTPS Server is running on: https://localhost:8081 2024-03-06 13:15:51.084 [info]: Client Request-ID : 6e5c60e138d24aaf9a2ccdb682322e5b 2024-03-06 13:15:51.089 [info]: 127.0.0.1 - - [06/Mar/2024:05:15:51 +0000] "GET /sgx/certification/v4/pckcrl?ca=processor HTTP/1.1" 200 604 "-" "-"

2024-03-06 13:15:51.093 [info]: Client Request-ID : f23fefc5b4db42519589375288bb1e70 2024-03-06 13:15:51.095 [info]: 127.0.0.1 - - [06/Mar/2024:05:15:51 +0000] "GET /sgx/certification/v4/tcb?fmspc=00A067110000 HTTP/1.1" 200 4675 "-" "-"

2024-03-06 14:24:04.692 [info]: Client Request-ID : 4f07123e54be476189ee0c0603536436 2024-03-06 14:24:04.694 [info]: 10.170.58.170 - - [06/Mar/2024:06:24:04 +0000] "GET /sgx/certification/v4/pckcrl?ca=processor HTTP/1.1" 200 604 "-" "-"

2024-03-06 14:24:04.699 [info]: Client Request-ID : 914417fa5c5b4600acf60413d6b8cfed 2024-03-06 14:24:04.701 [info]: 10.170.58.170 - - [06/Mar/2024:06:24:04 +0000] "GET /sgx/certification/v4/tcb?fmspc=00A067110000 HTTP/1.1" 200 4675 "-" "-"

yuki252111 commented 7 months ago

主要贴一下容器里面 occlum_release/image/etc/kubetee/unified_attestation.json 和 /etc/sgx_default_qcnl.conf的配置，我看一下

vi0eros commented 7 months ago

主要贴一下容器里面 occlum_release/image/etc/kubetee/unified_attestation.json 和 /etc/sgx_default_qcnl.conf 的配置，我看一下

root@sgx:/home/admin# cat occlum_release/image/etc/kubetee/unified_attestation.json 
{
    "ua_ias_url": "",
    "ua_ias_spid": "",
    "ua_ias_apk_key": "",
    "ua_dcap_lib_path": "",
    "ua_dcap_pccs_url": "https://10.170.58.170:8081/sgx/certification/v4/",
    "ua_uas_url": "",
    "ua_uas_app_key": "",
    "ua_uas_app_secret": ""
}

root@sgx:/home/admin# cat /etc/sgx_default_qcnl.conf                                
{
  // *** ATTENTION : This file is in JSON format so the keys are case sensitive. Don't change them.

  //PCCS server address
  "pccs_url": "https://10.170.58.170:8081/sgx/certification/v4/"

  // To accept insecure HTTPS certificate, set this option to false
  ,"use_secure_cert": false

  // You can use the Intel PCS or another PCCS to get quote verification collateral.  Retrieval of PCK 
  // Certificates will always use the PCCS described in pccs_url.  When collateral_service is not defined, both 
  // PCK Certs and verification collateral will be retrieved using pccs_url  
  //,"collateral_service": "https://api.trustedservices.intel.com/sgx/certification/v4/"

  // If you use a PCCS service to get the quote verification collateral, you can specify which PCCS API version is to be used.
  // The legacy 3.0 API will return CRLs in HEX encoded DER format and the sgx_ql_qve_collateral_t.version will be set to 3.0, while
  // the new 3.1 API will return raw DER format and the sgx_ql_qve_collateral_t.version will be set to 3.1. The pccs_api_version 
  // setting is ignored if collateral_service is set to the Intel PCS. In this case, the pccs_api_version is forced to be 3.1 
  // internally.  Currently, only values of 3.0 and 3.1 are valid.  Note, if you set this to 3.1, the PCCS use to retrieve 
  // verification collateral must support the new 3.1 APIs.
  //,"pccs_api_version": "3.1"

  // Maximum retry times for QCNL. If RETRY is not defined or set to 0, no retry will be performed.
  // It will first wait one second and then for all forthcoming retries it will double the waiting time.
  // By using retry_delay you disable this exponential backoff algorithm
  ,"retry_times": 6

  // Sleep this amount of seconds before each retry when a transfer has failed with a transient error
  ,"retry_delay": 10

  // If local_pck_url is defined, the QCNL will try to retrieve PCK cert chain from local_pck_url first,
  // and failover to pccs_url as in legacy mode.
  //,"local_pck_url": "http://localhost:8081/sgx/certification/v4/"

  // If local_pck_url is not defined, set pck_cache_expire_hours to a none-zero value will enable local cache. 
  // The PCK certificates will be cached in memory and then to the disk drive. 
  // The local cache files will be sequentially searched in the following directories until located in one of them:
  // Linux : $AZDCAP_CACHE, $XDG_CACHE_HOME, $HOME, $TMPDIR, /tmp/
  // Windows : $AZDCAP_CACHE, $LOCALAPPDATA\..\..\LocalLow
  // Please be aware that the environment variable pertains to the account executing the process that loads QPL,
  // not the account used to log in. For instance, if QPL is loaded by QGS, then those environment variables relate to
  // the "qgsd" account, which is the account that runs the QGS daemon.
  // You can remove the local cache files either manually or by using the QPL API, sgx_qpl_clear_cache. If you opt to
  // delete them manually, navigate to the aforementioned caching directories, find the folder named .dcap-qcnl, and delete it.
  // Restart the service after all cache folders were deleted. The same method applies to "verify_collateral_cache_expire_hours"
  ,"pck_cache_expire_hours": 168

  // To set cache expire time for quote verification collateral in hours
  // See the above comment for pck_cache_expire_hours for more information on the local cache.
  ,"verify_collateral_cache_expire_hours": 168

  // When the "local_cache_only" parameter is set to true, the QPL/QCNL will exclusively use PCK certificates 
  // from local cache files and will not request any PCK certificates from service providers, whether local or remote. 
  // To ensure that the PCK cache is available for use, an administrator must pre-populate the cache folders with 
  // the appropriate cache files. To generate these cache files for specific platforms, the administrator can use 
  // the PCCS admin tool. Once the cache files are generated, the administrator must distribute them to each platform 
  // that requires provisioning.
  ,"local_cache_only": false

  // You can add custom request headers and parameters to the get certificate API.
  // But the default PCCS implementation just ignores them. 
  //,"custom_request_options" : {
  //  "get_cert" : {
  //    "headers": {
  //      "head1": "value1"
  //    },
  //    "params": {
  //      "param1": "value1",
  //      "param2": "value2"
  //    }
  //  }
  //}
}

yuki252111 commented 7 months ago

这个机器是阿里云上购买的么，如果是的话直接使用阿里云的pccs试试 https://help.aliyun.com/zh/ecs/user-guide/build-an-sgx-encrypted-computing-environment，这个报错就是pccs服务没有正确的返回tcb相关的信息

wsjswy commented 7 months ago

用v3版本的pccs: https://10.170.58.170:8081/sgx/certification/v3/ 或者直接使用aliyun的pccs

vi0eros commented 7 months ago

使用v3版本的pccs：https://10.170.58.170:8081/sgx /certification/v3/或者直接使用aliyun的pccs

使用v3版本的pccs解决了这个问题，但执行cms register-data-keys出现了新的错误，机器是自己购置的：

lany@sgx:~/Desktop$ cms register-data-keys
[ERROR][external/jinzhao_attest/ual/verification/core/verifier_interface.cpp:228] [VERIFY] MRTRUSTAPP: String not equal
[ERROR][external/jinzhao_attest/ual/verification/core/verifier_interface.cpp:33] [Function] Verify
[ERROR][external/jinzhao_attest/ual/verification/core/verifier.cpp:48] [Function] Verify
[ERROR][external/jinzhao_attest/ual/verification/core/ua_verification.cpp:36] [Function] UaVerifyReport
[ERROR][external/jinzhao_attest/ual/verification/core/ua_verification.cpp:61] [Function] UaVerifyReportJson
[ERROR][external/jinzhao_attest/ual/verification/core/unified_attestation_verification.cpp:42] [Function] UnifiedAttestationVerifyReport
/usr/lib/python3/dist-packages/requests/__init__.py:89: RequestsDependencyWarning: urllib3 (2.2.1) or chardet (5.2.0) doesn't match a supported version!
  warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
Traceback (most recent call last):
  File "/home/lany/.local/bin/cms", line 8, in <module>
    sys.exit(cms())
  File "/usr/lib/python3/dist-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3/dist-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3/dist-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/lany/.local/lib/python3.8/site-packages/cli/cms.py", line 146, in register_data_keys
    ctx.obj.create_data_keys(
  File "/home/lany/.local/lib/python3.8/site-packages/sdc/capsule_manager_frame.py", line 391, in create_data_keys
    request, self.get_public_key(), private_key, cert_pems
  File "/home/lany/.local/lib/python3.8/site-packages/sdc/capsule_manager_frame.py", line 192, in get_public_key
    ual.verify_report(response.attestation_report, policy)
  File "/home/lany/.local/lib/python3.8/site-packages/sdc/ual/ual.py", line 109, in verify_report
    raise CapsuleManagerError(err_code, "verify failed.")
sdc.error.CapsuleManagerError: CapsuleManager server error code: 286916608, error message: verify failed.

我的alice.crt是按照https://www.secretflow.org.cn/zh-CN/docs/trustedflow/main/quick_start 中步骤生成的自签发证书。

yuki252111 commented 7 months ago

mrenclave的值校验失败，应该是alice.crt中mrenclave字段与实际的不一致。打印一下已确认（在此之前，因为修改过pccs url，所以先重签一下occlum_release）

vi0eros commented 7 months ago

mrenclave的值校验失败，应该是alice.crt中mrenclave字段与实际的不一致。打印一下已确认（在此之前，因为修改过pccs url，所以先重签一下occlum_release）

重签后修改mrenclave的值解决了这个问题，感谢您的回复！

asterinas / trustflow-capsule-manager-sdk

cms register-data-keys执行报错：CapsuleManager server error code: 13 Internal Error #11