search

RenderKit / ospray

An Open, Scalable, Portable, Ray Tracing Based Rendering Engine for High-Fidelity Visualization
http://ospray.org
Apache License 2.0
1.01k stars 182 forks source link

Trick superbuild into using pre-download dependecies (ISPC, TBB, etc)? #479

Closed kevinsmia1939 closed 3 years ago

kevinsmia1939 commented 3 years ago

Hello,

Hello, I tried to package a software called F3D as Flatpak and I want to submit to Flathub. https://gitlab.kitware.com/f3d/f3d

F3D need OSPRay. Flatpak does not support downloading anything while it build, everything must be pre-download.

OSPRay superbuild need to download ISPC but cannot due to how Flatpak-builder work.

I attempt to pre-download ispc-v1.15.0-linux.tar.gz

  1. mkdir -p scripts/superbuild/build/
  2. cd scripts/superbuild/build/ && cmake .. (at this point, nothing is being attempt to download, works fine without network)
  3. Pre-download ispc-v1.15.0-linux.tar.gz and placed it in scripts/superbuild/build/ispc/src
  4. touch scripts/superbuild/build/ispc/stamp/ispc-download scripts/superbuild/build/ispc/stamp/ispc-done scripts/superbuild/build/ispc/stamp/ispc-update (create stamp file to tell superbuild that the file was already downloaded and up to date)
  5. cmake --build scripts/superbuild/build/

I got this error.

make: Entering directory '/run/build/OSPRay/scripts/superbuild/build'
make[1]: Entering directory '/run/build/OSPRay/scripts/superbuild/build'
make[2]: Entering directory '/run/build/OSPRay/scripts/superbuild/build'
Scanning dependencies of target ispc
make[2]: Leaving directory '/run/build/OSPRay/scripts/superbuild/build'
make[2]: Entering directory '/run/build/OSPRay/scripts/superbuild/build'
[  1%] Creating directories for 'ispc'
[  3%] Performing download step (download, verify and extract) for 'ispc'
-- File already exists but no hash specified (use URL_HASH):
  file='/run/build/OSPRay/scripts/superbuild/build/ispc/src/ispc-v1.15.0-linux.tar.gz'
Old file will be removed and new file downloaded from URL.
-- Downloading...
   dst='/run/build/OSPRay/scripts/superbuild/build/ispc/src/ispc-v1.15.0-linux.tar.gz'
   timeout='none'
   inactivity timeout='none'

It seems to check for file integrety(?), but I don't know what no hash specified mean, is is sha256, md5, etc? Where can I find this hash value? I have some experience tricking superbuild before, but archive verification simply use md5 sum.

Not sure how it work in this case. How can I skip this step immediately to extract and build ISPC?

Thank you.

johguenther commented 3 years ago

The check for the hash is done in generated ispc/stamp/download-ispc.cmake. However, in my version of the file actually no hash is used / selected ("no check" branch): 

function(check_file_hash has_hash hash_is_good)
  if("${has_hash}" STREQUAL "")
    message(FATAL_ERROR "has_hash Can't be empty")
  endif()

  if("${hash_is_good}" STREQUAL "")
    message(FATAL_ERROR "hash_is_good Can't be empty")
  endif()

  if("" STREQUAL "")
    # No check
    set("${has_hash}" FALSE PARENT_SCOPE)
    set("${hash_is_good}" FALSE PARENT_SCOPE)
    return()
  endif()

Now I'm not sure where this is configured, because this is not actively enforced by use. Which CMake version do you use?

kevinsmia1939 commented 3 years ago

cmake version 3.19.6 I also tried delete ispc/stamp/download-ispc.cmake just before cmake install, does not work either.

kevinsmia1939 commented 3 years ago

I got confuse by the empty hash as well, therefor, the hash must be empty for it to skip re-downloading? I need to fix this by adding actual hash value to it?

johguenther commented 3 years ago

Can you please post the first lines of your download-ispc.cmake (corresponding to above)? My understanding of the code is, that on my side the "hash" is empty, which will skip the test. If on the other side it is e.g. "SHA256", it will be checked (with the file command in later lines).

kevinsmia1939 commented 3 years ago

This is download-ispc.cmake, after it was generate by command cmake ..

# Distributed under the OSI-approved BSD 3-Clause License.  See accompanying
# file Copyright.txt or https://cmake.org/licensing for details.

cmake_minimum_required(VERSION 3.5)

function(check_file_hash has_hash hash_is_good)
  if("${has_hash}" STREQUAL "")
    message(FATAL_ERROR "has_hash Can't be empty")
  endif()

  if("${hash_is_good}" STREQUAL "")
    message(FATAL_ERROR "hash_is_good Can't be empty")
  endif()

  if("" STREQUAL "")
    # No check
    set("${has_hash}" FALSE PARENT_SCOPE)
    set("${hash_is_good}" FALSE PARENT_SCOPE)
    return()
  endif()

  set("${has_hash}" TRUE PARENT_SCOPE)

  message(STATUS "verifying file...
       file='/home/kev/Downloads/ospray-2.5.0/scripts/superbuild/build/ispc/src/ispc-v1.15.0-linux.tar.gz'")

  file("" "/home/kev/Downloads/ospray-2.5.0/scripts/superbuild/build/ispc/src/ispc-v1.15.0-linux.tar.gz" actual_value)

  if(NOT "${actual_value}" STREQUAL "")
    set("${hash_is_good}" FALSE PARENT_SCOPE)
    message(STATUS " hash of
    /home/kev/Downloads/ospray-2.5.0/scripts/superbuild/build/ispc/src/ispc-v1.15.0-linux.tar.gz
  does not match expected value
    expected: ''
      actual: '${actual_value}'")
  else()
    set("${hash_is_good}" TRUE PARENT_SCOPE)
  endif()
endfunction()

function(sleep_before_download attempt)
  if(attempt EQUAL 0)
    return()
  endif()

  if(attempt EQUAL 1)
    message(STATUS "Retrying...")
    return()
  endif()

  set(sleep_seconds 0)

  if(attempt EQUAL 2)
    set(sleep_seconds 5)
  elseif(attempt EQUAL 3)
    set(sleep_seconds 5)
  elseif(attempt EQUAL 4)
    set(sleep_seconds 15)
  elseif(attempt EQUAL 5)
    set(sleep_seconds 60)
  elseif(attempt EQUAL 6)
    set(sleep_seconds 90)
  elseif(attempt EQUAL 7)
    set(sleep_seconds 300)
  else()
    set(sleep_seconds 1200)
  endif()

  message(STATUS "Retry after ${sleep_seconds} seconds (attempt #${attempt}) ...")

  execute_process(COMMAND "${CMAKE_COMMAND}" -E sleep "${sleep_seconds}")
endfunction()

if("/home/kev/Downloads/ospray-2.5.0/scripts/superbuild/build/ispc/src/ispc-v1.15.0-linux.tar.gz" STREQUAL "")
  message(FATAL_ERROR "LOCAL can't be empty")
endif()

if("https://github.com/ispc/ispc/releases/download/v1.15.0/ispc-v1.15.0-linux.tar.gz" STREQUAL "")
  message(FATAL_ERROR "REMOTE can't be empty")
endif()

if(EXISTS "/home/kev/Downloads/ospray-2.5.0/scripts/superbuild/build/ispc/src/ispc-v1.15.0-linux.tar.gz")
  check_file_hash(has_hash hash_is_good)
  if(has_hash)
    if(hash_is_good)
      message(STATUS "File already exists and hash match (skip download):
  file='/home/kev/Downloads/ospray-2.5.0/scripts/superbuild/build/ispc/src/ispc-v1.15.0-linux.tar.gz'
  =''"
      )
      return()
    else()
      message(STATUS "File already exists but hash mismatch. Removing...")
      file(REMOVE "/home/kev/Downloads/ospray-2.5.0/scripts/superbuild/build/ispc/src/ispc-v1.15.0-linux.tar.gz")
    endif()
  else()
    message(STATUS "File already exists but no hash specified (use URL_HASH):
  file='/home/kev/Downloads/ospray-2.5.0/scripts/superbuild/build/ispc/src/ispc-v1.15.0-linux.tar.gz'
Old file will be removed and new file downloaded from URL."
    )
    file(REMOVE "/home/kev/Downloads/ospray-2.5.0/scripts/superbuild/build/ispc/src/ispc-v1.15.0-linux.tar.gz")
  endif()
endif()

set(retry_number 5)

message(STATUS "Downloading...
   dst='/home/kev/Downloads/ospray-2.5.0/scripts/superbuild/build/ispc/src/ispc-v1.15.0-linux.tar.gz'
   timeout='none'
   inactivity timeout='none'"
)
set(download_retry_codes 7 6 8 15)
set(skip_url_list)
set(status_code)
foreach(i RANGE ${retry_number})
  if(status_code IN_LIST download_retry_codes)
    sleep_before_download(${i})
  endif()
  foreach(url https://github.com/ispc/ispc/releases/download/v1.15.0/ispc-v1.15.0-linux.tar.gz)
    if(NOT url IN_LIST skip_url_list)
      message(STATUS "Using src='${url}'")

      file(
        DOWNLOAD
        "${url}" "/home/kev/Downloads/ospray-2.5.0/scripts/superbuild/build/ispc/src/ispc-v1.15.0-linux.tar.gz"
        SHOW_PROGRESS
        # no TIMEOUT
        # no INACTIVITY_TIMEOUT
        STATUS status
        LOG log

        )

      list(GET status 0 status_code)
      list(GET status 1 status_string)

      if(status_code EQUAL 0)
        check_file_hash(has_hash hash_is_good)
        if(has_hash AND NOT hash_is_good)
          message(STATUS "Hash mismatch, removing...")
          file(REMOVE "/home/kev/Downloads/ospray-2.5.0/scripts/superbuild/build/ispc/src/ispc-v1.15.0-linux.tar.gz")
        else()
          message(STATUS "Downloading... done")
          return()
        endif()
      else()
        string(APPEND logFailedURLs "error: downloading '${url}' failed
        status_code: ${status_code}
        status_string: ${status_string}
        log:
        --- LOG BEGIN ---
        ${log}
        --- LOG END ---
        "
        )
      if(NOT status_code IN_LIST download_retry_codes)
        list(APPEND skip_url_list "${url}")
        break()
      endif()
    endif()
  endif()
  endforeach()
endforeach()

message(FATAL_ERROR "Each download failed!
  ${logFailedURLs}
  "
)
johguenther commented 3 years ago

OK, so hash is empty as well, which should then actually work. According to CMake ExternalProject, the hash is only checked if URL_HASH is given, which we do not set in scripts/superbuild/dependencies/dep_ispc.cmake.

kevinsmia1939 commented 3 years ago

OK, so hash is empty as well, which should then actually work. According to CMake ExternalProject, the hash is only checked if URL_HASH is given, which we do not set in scripts/superbuild/dependencies/dep_ispc.cmake.

Did you mean it should use pre-download ispc because hash was not specify? In that case, something else cause it to re-download ispc. Maybe missing stamp file?

johguenther commented 3 years ago

I think I found the solution: To make CMake accept the pre-downloaded ispc.tar.gz you indeed need to specify a hash, like this:

diff --git a/scripts/superbuild/dependencies/dep_ispc.cmake b/scripts/superbuild/dependencies/dep_ispc.cmake
index 8640e25f2..5f1e1a56c 100644
--- a/scripts/superbuild/dependencies/dep_ispc.cmake
+++ b/scripts/superbuild/dependencies/dep_ispc.cmake
@@ -30,6 +30,7 @@ ExternalProject_Add(${COMPONENT_NAME}
   URL ${ISPC_URL}
   CONFIGURE_COMMAND ""
   BUILD_COMMAND ""
+  URL_HASH "SHA256=b67f50ab16b38d29e28b0a2dbb9733fd6fc1276cb5a5be0cac78e356941f881f"
   INSTALL_COMMAND "${CMAKE_COMMAND}" -E copy_if_different
     <SOURCE_DIR>/bin/ispc${CMAKE_EXECUTABLE_SUFFIX}
     ${COMPONENT_PATH}/bin/ispc${CMAKE_EXECUTABLE_SUFFIX}

CMake will then check the hash and does not try to download the package again.

kevinsmia1939 commented 3 years ago

@johguenther Hello,

That works, I however cannot do this for embree for some reason. I create this patch to add URL_HASH Embree however, does not acknowledge it.

diff -ruN a/scripts/superbuild/dependencies/dep_embree.cmake b/scripts/superbuild/dependencies/dep_embree.cmake
--- a/scripts/superbuild/dependencies/dep_embree.cmake  2021-02-09 00:19:59.000000000 +0700
+++ b/scripts/superbuild/dependencies/dep_embree.cmake  2021-04-19 22:39:13.790877268 +0700
@@ -61,6 +61,7 @@
     URL ${EMBREE_URL}
     CONFIGURE_COMMAND ""
     BUILD_COMMAND ""
+    URL_HASH "SHA256=83aa90b1526fed8db80e800bb5c080cce3f06a8729b6dc24e597bac7f931cf98"
     INSTALL_COMMAND "${CMAKE_COMMAND}" -E copy_directory
       <SOURCE_DIR>/
       ${COMPONENT_PATH}
diff -ruN a/scripts/superbuild/dependencies/dep_glfw.cmake b/scripts/superbuild/dependencies/dep_glfw.cmake
--- a/scripts/superbuild/dependencies/dep_glfw.cmake    2021-02-09 00:19:59.000000000 +0700
+++ b/scripts/superbuild/dependencies/dep_glfw.cmake    2021-04-19 22:31:25.062879436 +0700
@@ -27,6 +27,7 @@
     -DGLFW_BUILD_EXAMPLES=OFF
     -DGLFW_BUILD_TESTS=OFF
   BUILD_COMMAND ${DEFAULT_BUILD_COMMAND}
+  URL_HASH "SHA256=bbd2c42c660b725e9755eb417e40b373f0d4c03138c9b2e210d02cd308bd99cd"
   BUILD_ALWAYS ${ALWAYS_REBUILD}
 )

diff -ruN a/scripts/superbuild/dependencies/dep_glm.cmake b/scripts/superbuild/dependencies/dep_glm.cmake
--- a/scripts/superbuild/dependencies/dep_glm.cmake 2021-02-09 00:19:59.000000000 +0700
+++ b/scripts/superbuild/dependencies/dep_glm.cmake 2021-04-19 22:33:55.360160367 +0700
@@ -17,6 +17,7 @@
   URL "https://github.com/g-truc/glm/archive/master.zip"
   CONFIGURE_COMMAND ""
   BUILD_COMMAND ""
+  URL_HASH "SHA256=97198b71b24ad5087114c1fb64dc3111aee1c7976cb5ae8a7c4476f3eeab8d69"
   INSTALL_COMMAND "${CMAKE_COMMAND}" -E copy_directory
     <SOURCE_DIR>
     ${COMPONENT_PATH}
diff -ruN a/scripts/superbuild/dependencies/dep_ispc.cmake b/scripts/superbuild/dependencies/dep_ispc.cmake
--- a/scripts/superbuild/dependencies/dep_ispc.cmake    2021-02-09 00:19:59.000000000 +0700
+++ b/scripts/superbuild/dependencies/dep_ispc.cmake    2021-04-19 22:32:45.359563634 +0700
@@ -30,6 +30,7 @@
   URL ${ISPC_URL}
   CONFIGURE_COMMAND ""
   BUILD_COMMAND ""
+  URL_HASH "SHA256=b67f50ab16b38d29e28b0a2dbb9733fd6fc1276cb5a5be0cac78e356941f881f"
   INSTALL_COMMAND "${CMAKE_COMMAND}" -E copy_if_different
     <SOURCE_DIR>/bin/ispc${CMAKE_EXECUTABLE_SUFFIX}
     ${COMPONENT_PATH}/bin/ispc${CMAKE_EXECUTABLE_SUFFIX}
diff -ruN a/scripts/superbuild/dependencies/dep_tbb.cmake b/scripts/superbuild/dependencies/dep_tbb.cmake
--- a/scripts/superbuild/dependencies/dep_tbb.cmake 2021-02-09 00:19:59.000000000 +0700
+++ b/scripts/superbuild/dependencies/dep_tbb.cmake 2021-04-19 22:30:09.426235310 +0700
@@ -33,6 +33,7 @@
   URL ${TBB_URL}
   CONFIGURE_COMMAND ""
   BUILD_COMMAND ""
+  URL_HASH "SHA256=f89b7757ad97ca9f972072de86684b4486dd70ee8f2363e02ffbd275d34d34ac"
   INSTALL_COMMAND "${CMAKE_COMMAND}" -E copy_directory
     <SOURCE_DIR>/${TBB_FOLDER}
     ${COMPONENT_PATH}

Going fine until Embree.

Applying patch add_superbuild_sha256.patch
patching file scripts/superbuild/dependencies/dep_embree.cmake
patching file scripts/superbuild/dependencies/dep_glfw.cmake
patching file scripts/superbuild/dependencies/dep_glm.cmake
patching file scripts/superbuild/dependencies/dep_ispc.cmake
patching file scripts/superbuild/dependencies/dep_tbb.cmake
Running: mkdir -p scripts/superbuild/build/
Running: cd scripts/superbuild/build/ && cmake ..
-- The C compiler identification is GNU 10.2.0
-- The CXX compiler identification is GNU 10.2.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Proceeding with a 64-bit generator. (8)
-- Configuring done
-- Generating done
-- Build files have been written to: /run/build/OSPRay/scripts/superbuild/build
Running: touch scripts/superbuild/build/ispc/stamp/ispc-download
Running: cmake --build scripts/superbuild/build/
Scanning dependencies of target ispc
[  1%] Creating directories for 'ispc'
[  3%] Performing download step (download, verify and extract) for 'ispc'
-- verifying file...
       file='/run/build/OSPRay/scripts/superbuild/build/ispc/src/ispc-v1.15.0-linux.tar.gz'
-- File already exists and hash match (skip download):
  file='/run/build/OSPRay/scripts/superbuild/build/ispc/src/ispc-v1.15.0-linux.tar.gz'
  SHA256='b67f50ab16b38d29e28b0a2dbb9733fd6fc1276cb5a5be0cac78e356941f881f'
-- extracting...
     src='/run/build/OSPRay/scripts/superbuild/build/ispc/src/ispc-v1.15.0-linux.tar.gz'
     dst='/run/build/OSPRay/scripts/superbuild/build/ispc/src'
-- extracting... [tar xfz]
-- extracting... [analysis]
-- extracting... [rename]
-- extracting... [clean up]
-- extracting... done
[  4%] No update step for 'ispc'
[  6%] No patch step for 'ispc'
[  7%] No configure step for 'ispc'
[  9%] No build step for 'ispc'
[ 10%] Performing install step for 'ispc'
[ 12%] Completed 'ispc'
[ 12%] Built target ispc
Scanning dependencies of target glm
[ 14%] Creating directories for 'glm'
[ 15%] Performing download step (download, verify and extract) for 'glm'
-- verifying file...
       file='/run/build/OSPRay/scripts/superbuild/build/glm/master.zip'
-- File already exists and hash match (skip download):
  file='/run/build/OSPRay/scripts/superbuild/build/glm/master.zip'
  SHA256='97198b71b24ad5087114c1fb64dc3111aee1c7976cb5ae8a7c4476f3eeab8d69'
-- extracting...
     src='/run/build/OSPRay/scripts/superbuild/build/glm/master.zip'
     dst='/run/build/OSPRay/scripts/superbuild/build/glm/src'
-- extracting... [tar xfz]
-- extracting... [analysis]
-- extracting... [rename]
-- extracting... [clean up]
-- extracting... done
[ 17%] No update step for 'glm'
[ 18%] No patch step for 'glm'
[ 20%] No configure step for 'glm'
[ 21%] No build step for 'glm'
[ 23%] Performing install step for 'glm'
[ 25%] Completed 'glm'
[ 25%] Built target glm
Scanning dependencies of target tbb
[ 26%] Creating directories for 'tbb'
[ 28%] Performing download step (download, verify and extract) for 'tbb'
-- verifying file...
       file='/run/build/OSPRay/scripts/superbuild/build/tbb/oneapi-tbb-2021.1.1-lin.tgz'
-- File already exists and hash match (skip download):
  file='/run/build/OSPRay/scripts/superbuild/build/tbb/oneapi-tbb-2021.1.1-lin.tgz'
  SHA256='f89b7757ad97ca9f972072de86684b4486dd70ee8f2363e02ffbd275d34d34ac'
-- extracting...
     src='/run/build/OSPRay/scripts/superbuild/build/tbb/oneapi-tbb-2021.1.1-lin.tgz'
     dst='/run/build/OSPRay/scripts/superbuild/build/tbb/src'
-- extracting... [tar xfz]
-- extracting... [analysis]
-- extracting... [rename]
-- extracting... [clean up]
-- extracting... done
[ 29%] No update step for 'tbb'
[ 31%] No patch step for 'tbb'
[ 32%] No configure step for 'tbb'
[ 34%] No build step for 'tbb'
[ 35%] Performing install step for 'tbb'
[ 37%] Completed 'tbb'
[ 37%] Built target tbb
Scanning dependencies of target embree
[ 39%] Creating directories for 'embree'
[ 40%] Performing download step (download, verify and extract) for 'embree'
-- File already exists but no hash specified (use URL_HASH):
  file='/run/build/OSPRay/scripts/superbuild/build/embree/v3.12.2.zip'
Old file will be removed and new file downloaded from URL.
-- Downloading...
   dst='/run/build/OSPRay/scripts/superbuild/build/embree/v3.12.2.zip'
   timeout='none'
   inactivity timeout='none'
-- Using src='http://github.com/embree/embree/archive/v3.12.2.zip'
-- Retrying...
-- Using src='http://github.com/embree/embree/archive/v3.12.2.zip'
-- Retry after 5 seconds (attempt #2) ...
johguenther commented 3 years ago

Provide -DBUILD_EMBREE_FROM_SOURCE=OFF as cmake option when configuring the superbuild.

kevinsmia1939 commented 3 years ago

Hi,

I decided to build OSPRay the standard way. I use ISPC binary build instead.

johguenther commented 3 years ago

FYI, v2.6.0 now includes URL_HASH (via 1fd89f23fe7a46fb6f23ebaa7cc9c5f7ce4159d4).