search

quic / ai-hub-models

The Qualcomm® AI Hub Models are a collection of state-of-the-art machine learning models optimized for performance (latency, memory etc.) and ready to deploy on Qualcomm® devices.
https://aihub.qualcomm.com
BSD 3-Clause "New" or "Revised" License
417 stars 58 forks source link

[BUG] fails to create context from binary for the 4 QNN Context Bin files (llama_v2_7b_chat_quantized_TokenGenerator_x_Quantized.bin) #85

Closed taeyeonlee closed 4 days ago

taeyeonlee commented 1 month ago

Hi, It fails to create context from binary for the 4 QNN Context Bin files (llama_v2_7b_chat_quantized_TokenGenerator_1_Quantized.bin, llama_v2_7b_chat_quantized_TokenGenerator_2_Quantized.bin, llama_v2_7b_chat_quantized_TokenGenerator_3_Quantized.bin, llama_v2_7b_chat_quantized_TokenGenerator_4_Quantized.bin), in the Android mobile S24 Ultra. even though it succeed to create context from binary One by One, it means that it succeed to create context from binary for one llama_v2_7b_chat_quantized_TokenGenerator_1_Quantized.bin, and executeGraphs it and freeContext, then create context from binary for one file llama_v2_7b_chat_quantized_TokenGenerator_2_Quantized.bin, and executeGraphs it and freeContext, and then until llama_v2_7b_chat_quantized_TokenGenerator_4_Quantized.bin, and executeGraphs it and freeContext.

How to create context from binary for the 4 QNN Context Bin files (llama_v2_7b_chat_quantized_TokenGenerator_1_Quantized.bin, llama_v2_7b_chat_quantized_TokenGenerator_2_Quantized.bin, llama_v2_7b_chat_quantized_TokenGenerator_3_Quantized.bin, llama_v2_7b_chat_quantized_TokenGenerator_4_Quantized.bin) ?

The fail log is following.

2024-08-07 17:34:36.233 9745-9772 LlamaNative com.test.llama I [taeyeon] QNN System function pointers are populated 2024-08-07 17:34:36.233 9745-9772 LlamaNative com.test.llama I [taeyeon] succeed to create system handle. m_qnnFunctionPointers.qnnSystemInterface.systemContextCreate 2024-08-07 17:34:36.233 9745-9772 LlamaNative com.test.llama I [taeyeon] succeed to allocate memory. bufferSize = 1083146928 2024-08-07 17:34:37.441 9745-9772 LlamaNative com.test.llama I [taeyeon] succeed to read binary data = /data/local/tmp/sample_app/llama_v2_7b_chat_quantized_TokenGenerator_1_Quantized.bin 2024-08-07 17:34:37.441 9745-9772 LlamaNative com.test.llama I [taeyeon] succeed to get context binary info. binaryInfoSize = 0 2024-08-07 17:34:37.441 9745-9772 LlamaNative com.test.llama I Extracting graphsInfo for graph Idx: 0 2024-08-07 17:34:37.441 9745-9772 LlamaNative com.test.llama I Extracting tensorInfo for tensor tensorsCount : 516 2024-08-07 17:34:37.442 9745-9772 LlamaNative com.test.llama I Extracting tensorInfo for tensor tensorsCount : 513 2024-08-07 17:34:37.442 9745-9772 LlamaNative com.test.llama I [taeyeon] succeed to copy metadata. m_graphsCount[4] = 1 2024-08-07 17:34:37.446 9745-9772 com.test.llama com.test.llama I vendor/qcom/proprietary/adsprpc/src/fastrpc_apps_user.c:2331: remote_handle_control_domain: requested QOS 1, latency 100 for domain 3 handle 0x7cfa118450 2024-08-07 17:34:37.447 9745-9772 com.test.llama com.test.llama I vendor/qcom/proprietary/adsprpc/src/fastrpc_apps_user.c:1872: manage_poll_qos: poll mode updated to 3 for domain 3, handle 0x7cfa118450 for timeout 9999 2024-08-07 17:34:37.447 9745-9772 com.test.llama com.test.llama I vendor/qcom/proprietary/adsprpc/src/fastrpc_apps_user.c:2331: remote_handle_control_domain: requested QOS 3, latency 9999 for domain 3 handle 0x7cfa118450 2024-08-07 17:34:37.757 9745-9772 LlamaNative com.test.llama I [taeyeon] succeed to create context from binary. 2024-08-07 17:34:37.757 9745-9772 LlamaNative com.test.llama I [taeyeon] graphRetrieve graphName: tmp0b0a0nm3 2024-08-07 17:34:37.757 9745-9772 LlamaNative com.test.llama I [taeyeon] graphRetrieve graphName: tmp0b0a0nm3 2024-08-07 17:34:37.832 9745-9772 LlamaNative com.test.llama I [taeyeon] succeed to allocate memory. bufferSize = 821006960 2024-08-07 17:34:38.711 9745-9772 LlamaNative com.test.llama I [taeyeon] succeed to read binary data = /data/local/tmp/sample_app/llama_v2_7b_chat_quantized_TokenGenerator_2_Quantized.bin 2024-08-07 17:34:38.711 9745-9772 LlamaNative com.test.llama I [taeyeon] succeed to get context binary info. binaryInfoSize = 0 2024-08-07 17:34:38.711 9745-9772 LlamaNative com.test.llama I Extracting graphsInfo for graph Idx: 0 2024-08-07 17:34:38.711 9745-9772 LlamaNative com.test.llama I Extracting tensorInfo for tensor tensorsCount : 516 2024-08-07 17:34:38.711 9745-9772 LlamaNative com.test.llama I Extracting tensorInfo for tensor tensorsCount : 513 2024-08-07 17:34:38.712 9745-9772 LlamaNative com.test.llama I [taeyeon] succeed to copy metadata. m_graphsCount[5] = 1 2024-08-07 17:34:38.716 9745-9772 com.test.llama com.test.llama I vendor/qcom/proprietary/adsprpc/src/fastrpc_apps_user.c:2331: remote_handle_control_domain: requested QOS 1, latency 100 for domain 3 handle 0x7cfa118450 2024-08-07 17:34:38.716 9745-9772 com.test.llama com.test.llama I vendor/qcom/proprietary/adsprpc/src/fastrpc_apps_user.c:1872: manage_poll_qos: poll mode updated to 3 for domain 3, handle 0x7cfa118450 for timeout 9999 2024-08-07 17:34:38.716 9745-9772 com.test.llama com.test.llama I vendor/qcom/proprietary/adsprpc/src/fastrpc_apps_user.c:2331: remote_handle_control_domain: requested QOS 3, latency 9999 for domain 3 handle 0x7cfa118450 2024-08-07 17:34:39.218 9745-9772 LlamaNative com.test.llama I [taeyeon] succeed to create context from binary. 2024-08-07 17:34:39.218 9745-9772 LlamaNative com.test.llama I [taeyeon] graphRetrieve graphName: tmp5a2tztgk 2024-08-07 17:34:39.218 9745-9772 LlamaNative com.test.llama I [taeyeon] graphRetrieve graphName: tmp5a2tztgk 2024-08-07 17:34:39.278 9745-9772 LlamaNative com.test.llama I [taeyeon] succeed to allocate memory. bufferSize = 821002864 2024-08-07 17:34:40.172 9745-9772 LlamaNative com.test.llama I [taeyeon] succeed to read binary data = /data/local/tmp/sample_app/llama_v2_7b_chat_quantized_TokenGenerator_3_Quantized.bin 2024-08-07 17:34:40.172 9745-9772 LlamaNative com.test.llama I [taeyeon] succeed to get context binary info. binaryInfoSize = 0 2024-08-07 17:34:40.172 9745-9772 LlamaNative com.test.llama I Extracting graphsInfo for graph Idx: 0 2024-08-07 17:34:40.173 9745-9772 LlamaNative com.test.llama I Extracting tensorInfo for tensor tensorsCount : 516 2024-08-07 17:34:40.173 9745-9772 LlamaNative com.test.llama I Extracting tensorInfo for tensor tensorsCount : 513 2024-08-07 17:34:40.173 9745-9772 LlamaNative com.test.llama I [taeyeon] succeed to copy metadata. m_graphsCount[6] = 1 2024-08-07 17:34:40.177 9745-9772 com.test.llama com.test.llama I vendor/qcom/proprietary/adsprpc/src/fastrpc_apps_user.c:2331: remote_handle_control_domain: requested QOS 1, latency 100 for domain 3 handle 0x7cfa118450 2024-08-07 17:34:40.178 9745-9772 com.test.llama com.test.llama I vendor/qcom/proprietary/adsprpc/src/fastrpc_apps_user.c:1872: manage_poll_qos: poll mode updated to 3 for domain 3, handle 0x7cfa118450 for timeout 9999 2024-08-07 17:34:40.178 9745-9772 com.test.llama com.test.llama I vendor/qcom/proprietary/adsprpc/src/fastrpc_apps_user.c:2331: remote_handle_control_domain: requested QOS 3, latency 9999 for domain 3 handle 0x7cfa118450 2024-08-07 17:34:40.722 9745-9772 LlamaNative com.test.llama I [taeyeon] succeed to create context from binary. 2024-08-07 17:34:40.722 9745-9772 LlamaNative com.test.llama I [taeyeon] graphRetrieve graphName: tmpxdxtl6kr 2024-08-07 17:34:40.722 9745-9772 LlamaNative com.test.llama I [taeyeon] graphRetrieve graphName: tmpxdxtl6kr 2024-08-07 17:34:40.783 9745-9772 LlamaNative com.test.llama I [taeyeon] succeed to allocate memory. bufferSize = 952636560 2024-08-07 17:34:41.809 9745-9772 LlamaNative com.test.llama I [taeyeon] succeed to read binary data = /data/local/tmp/sample_app/llama_v2_7b_chat_quantized_TokenGenerator_4_Quantized.bin 2024-08-07 17:34:41.810 9745-9772 LlamaNative com.test.llama I [taeyeon] succeed to get context binary info. binaryInfoSize = 0 2024-08-07 17:34:41.810 9745-9772 LlamaNative com.test.llama I Extracting graphsInfo for graph Idx: 0 2024-08-07 17:34:41.810 9745-9772 LlamaNative com.test.llama I Extracting tensorInfo for tensor tensorsCount : 516 2024-08-07 17:34:41.810 9745-9772 LlamaNative com.test.llama I Extracting tensorInfo for tensor tensorsCount : 513 2024-08-07 17:34:41.810 9745-9772 LlamaNative com.test.llama I [taeyeon] succeed to copy metadata. m_graphsCount[7] = 1 2024-08-07 17:34:41.814 9745-9772 com.test.llama com.test.llama I vendor/qcom/proprietary/adsprpc/src/fastrpc_apps_user.c:2331: remote_handle_control_domain: requested QOS 1, latency 100 for domain 3 handle 0x7cfa118450 2024-08-07 17:34:41.815 9745-9772 com.test.llama com.test.llama I vendor/qcom/proprietary/adsprpc/src/fastrpc_apps_user.c:1872: manage_poll_qos: poll mode updated to 3 for domain 3, handle 0x7cfa118450 for timeout 9999 2024-08-07 17:34:41.815 9745-9772 com.test.llama com.test.llama I vendor/qcom/proprietary/adsprpc/src/fastrpc_apps_user.c:2331: remote_handle_control_domain: requested QOS 3, latency 9999 for domain 3 handle 0x7cfa118450 2024-08-07 17:34:42.300 9745-9772 com.test.llama com.test.llama E vendor/qcom/proprietary/adsprpc/src/fastrpc_mem.c:511: Error 0x1: fastrpc_mmap failed to map buffer fd 178, addr 0x79851b2000, length 0x38a00000, domain 3, flags 0x3, ioctl ret 0xffffffff, errno Bad address 2024-08-07 17:34:42.304 9745-9772 com.test.llama com.test.llama I vendor/qcom/proprietary/adsprpc/src/fastrpc_apps_user.c:3407: open_device_node: no access to default device of domain 3, open thru HAL, (sess_id 2) 2024-08-07 17:34:42.305 9745-9772 dsp-client com.test.llama E DspClient.cpp (127): Error: open_hal_session: invalid argument(s): client instance 0x7ea9f89530, domain 11 2024-08-07 17:34:42.305 9745-9772 com.test.llama com.test.llama E vendor/qcom/proprietary/adsprpc/src/fastrpc_apps_user.c:3426: Error 0x0: open_device_node failed for domain ID 11, sess ID 2 (errno 13, Permission denied) 2024-08-07 17:34:42.305 9745-9772 com.test.llama com.test.llama E vendor/qcom/proprietary/adsprpc/src/fastrpc_apps_user.c:2088::Error: 0x200: 0 <= (dev = open_device_node((int)domain)) 2024-08-07 17:34:42.305 9745-9772 com.test.llama com.test.llama W vendor/qcom/proprietary/adsprpc/src/fastrpc_apps_user.c:2145:Warning 0x200: remote_get_info failed to get attribute 1 for domain 11 (errno Permission denied) 2024-08-07 17:34:42.305 9745-9772 com.test.llama com.test.llama E vendor/qcom/proprietary/adsprpc/src/fastrpc_apps_user.c:2371: Error 0x200: remote_handle_control_domain failed for request ID 2 on domain 3 (errno Permission denied) 2024-08-07 17:34:42.305 9745-9772 com.test.llama com.test.llama E vendor/qcom/proprietary/adsprpc/src/fastrpc_apps_user.c:2382: Error 0x200: remote_handle_control failed for request ID 2 (errno Permission denied) 2024-08-07 17:34:42.308 9745-9772 LlamaNative com.test.llama E Could not create context from binary. 2024-08-07 17:34:42.308 9745-9772 LlamaNative com.test.llama E Cleaning up graph Info structures. 2024-08-07 17:34:42.382 9745-9772 LlamaNative com.test.llama E ERROR Create From Binary failure

Best regards,

bhushan23 commented 1 month ago

@taeyeonlee What command/script are you running here?

taeyeonlee commented 1 month ago

@bhushan23 QnnSampleApp::createFromBinary() is called as like below.

 std::vector<size_t> models = {app->MODEL_TokenGenerator_1, app->MODEL_TokenGenerator_2, app->MODEL_TokenGenerator_3, app->MODEL_TokenGenerator_4};
            if (sample_app::StatusCode::SUCCESS != app->createFromBinary(models)) {

The QnnSampleApp.cpp is modified as below. 


sample_app::StatusCode sample_app::QnnSampleApp::createFromBinary(std::vector<size_t> models) {

  if (nullptr == m_qnnFunctionPointers.qnnSystemInterface.systemContextCreate ||
      nullptr == m_qnnFunctionPointers.qnnSystemInterface.systemContextGetBinaryInfo ||
      nullptr == m_qnnFunctionPointers.qnnSystemInterface.systemContextFree) {
    ALOGE("QNN System function pointers are not populated.");
    return StatusCode::FAILURE;
  }

  auto returnStatus = StatusCode::SUCCESS;
  QnnSystemContext_Handle_t sysCtxHandle{nullptr};
  if (QNN_SUCCESS != m_qnnFunctionPointers.qnnSystemInterface.systemContextCreate(&sysCtxHandle)) {
    ALOGE("Could not create system handle.");
    return StatusCode::FAILURE;
  }

  for (size_t i = 0; i < models.size(); i++) {
    size_t model_index = models[i];
    uint64_t bufferSize{0};
    std::shared_ptr<uint8_t> buffer{nullptr};
    // read serialized binary into a byte buffer
    tools::datautil::StatusCode status{tools::datautil::StatusCode::SUCCESS};
    std::tie(status, bufferSize) = tools::datautil::getFileSize(m_cachedBinaryPath[model_index]);
    if (0 == bufferSize) {
      ALOGE("Received path to an empty file. Nothing to deserialize.");
      return StatusCode::FAILURE;
    }

    buffer = std::shared_ptr<uint8_t>(new uint8_t[bufferSize], std::default_delete<uint8_t[]>());
    if (!buffer) {
      ALOGE("Failed to allocate memory.");
      return StatusCode::FAILURE;
    }

    status = tools::datautil::readBinaryFromFile(
        m_cachedBinaryPath[model_index], reinterpret_cast<uint8_t*>(buffer.get()), bufferSize);
    if (status != tools::datautil::StatusCode::SUCCESS) {
      ALOGE("Failed to read binary data.");
      return StatusCode::FAILURE;
    }

    const QnnSystemContext_BinaryInfo_t* binaryInfo{nullptr};
    Qnn_ContextBinarySize_t binaryInfoSize{0};
    if (StatusCode::SUCCESS == returnStatus &&
        QNN_SUCCESS != m_qnnFunctionPointers.qnnSystemInterface.systemContextGetBinaryInfo(
                          sysCtxHandle,
                          static_cast<void*>(buffer.get()),
                          bufferSize,
                          &binaryInfo,
                          &binaryInfoSize)) {
      ALOGE("Failed to get context binary info");
      returnStatus = StatusCode::FAILURE;
    }

    // fill GraphInfo_t based on binary info
    if (StatusCode::SUCCESS == returnStatus) {
      qnn_wrapper_api::GraphInfo_t ** graphsInfo;
      uint32_t graphsCount;
      if (!copyMetadataToGraphsInfo(binaryInfo, graphsInfo, graphsCount)) {
        ALOGE("Failed to copy metadata.");
        returnStatus = StatusCode::FAILURE;
      } else {
        m_graphsInfo[model_index] = graphsInfo;
        m_graphsCount[model_index] = graphsCount;
      }
    }

    if (StatusCode::SUCCESS == returnStatus &&
        nullptr == m_qnnFunctionPointers.qnnInterface.contextCreateFromBinary) {
      ALOGE("contextCreateFromBinaryFnHandle is nullptr.");
      returnStatus = StatusCode::FAILURE;
    }

    if (StatusCode::SUCCESS == returnStatus &&
        m_qnnFunctionPointers.qnnInterface.contextCreateFromBinary(
            m_backendHandle,
            m_deviceHandle,
            (const QnnContext_Config_t**)&m_contextConfig,
            static_cast<void*>(buffer.get()),
            bufferSize,
            &m_context,
            m_profileBackendHandle)) {
      ALOGE("Could not create context from binary.");
      returnStatus = StatusCode::FAILURE;
    }
    else {
        ALOGI("[taeyeon] succeed to create context from binary.");
    }

    if (ProfilingLevel::OFF != m_profilingLevel) {
      extractBackendProfilingInfo(m_profileBackendHandle);
    }

    if (StatusCode::SUCCESS == returnStatus) {
      for (size_t graphIdx = 0; graphIdx < m_graphsCount.at(model_index); graphIdx++) {
        if (nullptr == m_qnnFunctionPointers.qnnInterface.graphRetrieve) {
          ALOGE("graphRetrieveFnHandle is nullptr.");
          returnStatus = StatusCode::FAILURE;
          break;
        }
                ALOGI("[taeyeon] graphRetrieve graphName: %s", (*m_graphsInfo.at(model_index))[graphIdx].graphName);
        if (QNN_SUCCESS !=
            m_qnnFunctionPointers.qnnInterface.graphRetrieve(
                m_context, (*m_graphsInfo.at(model_index))[graphIdx].graphName, &((*m_graphsInfo.at(model_index))[graphIdx].graph))) {
          ALOGE("Unable to retrieve graph handle for graph Idx: %d", graphIdx);
          returnStatus = StatusCode::FAILURE;
        }
          ALOGI("[taeyeon] graphRetrieve graphName: %s", (*m_graphsInfo.at(model_index))[graphIdx].graphName);
      }
    }

    if (StatusCode::SUCCESS != returnStatus) {
      ALOGE("Cleaning up graph Info structures.");
      qnn_wrapper_api::freeGraphsInfo(&m_graphsInfo.at(model_index), m_graphsCount.at(model_index));
    }
  }

  m_qnnFunctionPointers.qnnSystemInterface.systemContextFree(sysCtxHandle);
  sysCtxHandle = nullptr;

  m_isContextCreated = true;

  return returnStatus;
}
bhushan23 commented 4 days ago

Hi @taeyeonlee please follow https://github.com/quic/ai-hub-models/tree/main/qai_hub_models/models/llama_v2_7b_chat_quantized/gen_ondevice_llama to run llama2 models on device with Genie.

We will soon be releasing C++ app using Genie C++ APIs