search

KhronosGroup / OpenCL-SDK

OpenCL SDK
Apache License 2.0
577 stars 121 forks source link

Errors trying to build simple header+loader binary #58

Closed juliohm closed 7 months ago

juliohm commented 2 years ago

I am trying to cross-compile OpenCL for different applications in Julia, but the build is giving this error as if the CMAKE_CXX_STANDARD at the top level was ignored. Can you please take a look at the following build record?

https://pastebin.com/V5CCYxDP

You can see that the C99 standard is being used for some reason in the build.

MathiasMagnus commented 2 years ago

@juliohm Thanks for reporting this.

The issue isn't that the build is using C99 for some reason. The issue is that it isn't using C99, but uses the default C standard, which I guess is C89. It was a conscious decision that the OpenCL ecosystem going forward won't support 30 year old language standards. Unfortunately our OpenCL-SDK CI coverage isn't quite the same breadth yet as the rest of the ecosystem, and this failure is triggered in the utility libraries.

We'll discuss this issue, my educated guess is that the utility libraries will target C11 and onward, maybe C99. We'd need to improve our testing in CI to make sure such fiascos don't happen in the future.

I also saw that you build a full blows SDK, however my guess is that Julia is only using the OpenCL headers and libraries. If you can pass extra arguments to CMake, I'd highly suggest adding -D OPENCL_SDK_BUILD_SAMPLES=OFF to reduce build times significantly.

juliohm commented 2 years ago

Thank you @MathiasMagnus I will try to build it again with the proposed option. Will also try -D CMAKE_C_STANDARD=11 in the command line to see if the error goes away following suggestions from Julia community members.

MathiasMagnus commented 2 years ago

You beat me to, I was just about to amend my message with the same idea. Let me know how it turns out.

juliohm commented 2 years ago

Ok, the option CMAKE_C_STANDARD=11 fixed that error, but now I have another error:

In file included from /workspace/srcdir/OpenCL-SDK/lib/include/CL/Utils/Utils.hpp:6:0,
                 from /workspace/srcdir/OpenCL-SDK/lib/src/Utils/Utils.cpp:2:
/workspace/srcdir/OpenCL-SDK/lib/include/CL/Utils/Detail.hpp:25:41: error: ‘std::integer_sequence’ has not been declared
                                    std::integer_sequence<int, Is...>)
                                         ^
/workspace/srcdir/OpenCL-SDK/lib/include/CL/Utils/Detail.hpp:25:57: error: expected ‘,’ or ‘...’ before ‘<’ token
                                    std::integer_sequence<int, Is...>)
                                                         ^
/workspace/srcdir/OpenCL-SDK/lib/include/CL/Utils/Detail.hpp: In function ‘void cl::util::detail::for_each_in_tuple(const std::tuple<_Elements ...>&, F&&)’:
/workspace/srcdir/OpenCL-SDK/lib/include/CL/Utils/Detail.hpp:38:17: error: ‘make_integer_sequence’ is not a member of ‘std’
                 std::make_integer_sequence<int, sizeof...(Ts)>());
                 ^
/workspace/srcdir/OpenCL-SDK/lib/include/CL/Utils/Detail.hpp:38:44: error: expected primary-expression before ‘int’
                 std::make_integer_sequence<int, sizeof...(Ts)>());
                                            ^
/workspace/srcdir/OpenCL-SDK/lib/include/CL/Utils/Detail.hpp:38:64: error: expected primary-expression before ‘)’ token
                 std::make_integer_sequence<int, sizeof...(Ts)>());
                                                                ^
/workspace/srcdir/OpenCL-SDK/lib/include/CL/Utils/Detail.hpp: At global scope:
/workspace/srcdir/OpenCL-SDK/lib/include/CL/Utils/Detail.hpp:45:57: error: ‘std::index_sequence’ has not been declared
             auto transform_tuple(Tuple&& t, F&& f, std::index_sequence<Is...>)
                                                         ^
/workspace/srcdir/OpenCL-SDK/lib/include/CL/Utils/Detail.hpp:45:71: error: expected ‘,’ or ‘...’ before ‘<’ token
             auto transform_tuple(Tuple&& t, F&& f, std::index_sequence<Is...>)
                                                                       ^
/workspace/srcdir/OpenCL-SDK/lib/include/CL/Utils/Detail.hpp: In function ‘auto cl::util::detail::transform_tuple(const std::tuple<_Elements ...>&, F&&)’:
/workspace/srcdir/OpenCL-SDK/lib/include/CL/Utils/Detail.hpp:55:17: error: ‘make_index_sequence’ is not a member of ‘std’
                 std::make_index_sequence<sizeof...(Args)>{});
                 ^
/workspace/srcdir/OpenCL-SDK/lib/include/CL/Utils/Detail.hpp:55:58: error: expected primary-expression before ‘{’ token
                 std::make_index_sequence<sizeof...(Args)>{});
                                                          ^
/workspace/srcdir/OpenCL-SDK/lib/include/CL/Utils/Detail.hpp: At global scope:
/workspace/srcdir/OpenCL-SDK/lib/include/CL/Utils/Detail.hpp:64:50: error: ‘std::index_sequence’ has not been declared
             auto apply(F&& f, Tuple&& args, std::index_sequence<I...>)
                                                  ^
/workspace/srcdir/OpenCL-SDK/lib/include/CL/Utils/Detail.hpp:64:64: error: expected ‘,’ or ‘...’ before ‘<’ token
             auto apply(F&& f, Tuple&& args, std::index_sequence<I...>)
                                                                ^
/workspace/srcdir/OpenCL-SDK/lib/include/CL/Utils/Detail.hpp:73:38: error: expected type-specifier
                   typename Indices = std::make_index_sequence<
                                      ^
/workspace/srcdir/OpenCL-SDK/lib/include/CL/Utils/Detail.hpp:73:38: error: expected ‘>’
make[2]: *** [lib/CMakeFiles/OpenCLUtilsCpp.dir/build.make:76: lib/CMakeFiles/OpenCLUtilsCpp.dir/src/Utils/Utils.cpp.o] Error 1
make[2]: Leaving directory '/workspace/srcdir/OpenCL-SDK/build'
make[1]: *** [CMakeFiles/Makefile2:1067: lib/CMakeFiles/OpenCLUtilsCpp.dir/all] Error 2
make[1]: Leaving directory '/workspace/srcdir/OpenCL-SDK/build'
make: *** [Makefile:166: all] Error 2

Should I edit these hpp files to include some missing stdlib?

juliohm commented 2 years ago

It is been a while since the last time I touched C/C++, all my work nowadays is in Julia, I don't know the latest trends in standards, stdlibs, etc. Any help is appreciated. I think we are almost there 🙏🏽

MathiasMagnus commented 2 years ago

Yeah, the issue at hand is this:

-- The C compiler identification is GNU 4.8.5
-- The CXX compiler identification is GNU 4.8.5

I would ask what operating system this is, but GCC 4.8.5 shares the feature set of 4.8.0, which was released March 2013, so it's 9 years old snapshot of C++ and as such pre-dates C++14. I don't think we will back-port the utility libraries to C++11, as we would have to reimplement quite some metaprogramming utilities. Our baseline of testing on Linux are the compilers found in oldest supported Ubuntu LTS, which is GCC 7. We don't (currently) have the assets to have CI runners sporting older systems/toolchains. Support below that is on a best-effort basis.

I see two short-term solutions:

juliohm commented 2 years ago

Thank you @MathiasMagnus , I am following all your suggestions. I requested GCC 7 in the cross-compilation script and followed the instructions in the OpenCL-ICD-Loader. In the first compilation step I already get errors related to C++:

/workspace/srcdir/OpenCL-Headers/build/tests/lang_cpp/../../../tests/test_headers.c: In function ‘int test_long()’:
/workspace/srcdir/OpenCL-Headers/build/tests/lang_cpp/../../../tests/test_headers.c:400:21: error: expected ‘)’ before ‘PRId64’
     printf("b:   %" PRId64 "\n", b );
                     ^~~~~~
/workspace/srcdir/OpenCL-Headers/build/tests/lang_cpp/../../../tests/test_headers.c:401:21: error: expected ‘)’ before ‘PRId64’
     printf("b2:  %" PRId64 " %" PRId64 " \n", b2.s[0], b2.s[1] );
                     ^~~~~~
/workspace/srcdir/OpenCL-Headers/build/tests/lang_cpp/../../../tests/test_headers.c:402:21: error: expected ‘)’ before ‘PRId64’
     printf("b4:  %" PRId64 " %" PRId64 " %" PRId64 " %" PRId64 "\n", b4.s[0], b4.s[1], b4.s[2], b4.s[3] );
                     ^~~~~~
/workspace/srcdir/OpenCL-Headers/build/tests/lang_cpp/../../../tests/test_headers.c:403:21: error: expected ‘)’ before ‘PRId64’
     printf("b8:  %" PRId64 " %" PRId64 " %" PRId64 " %" PRId64 " %" PRId64 " %" PRId64 " %" PRId64 " %" PRId64 "\n", b8.s[0], b8.s[1], b8.s[2], b8.s[3], b8.s[4], b8.s[5], b8.s[6], b8.s[7] );
                     ^~~~~~
/workspace/srcdir/OpenCL-Headers/build/tests/lang_cpp/../../../tests/test_headers.c:404:21: error: expected ‘)’ before ‘PRId64’
     printf("b16: %" PRId64 " %" PRId64 " %" PRId64 " %" PRId64 " %" PRId64 " %" PRId64 " %" PRId64 " %" PRId64 " %" PRId64 " %" PRId64 " %" PRId64 " %" PRId64 " %" PRId64 " %" PRId64 " %" PRId64 " %" PRId64 "\n", b16.s[0], b16.s[1], b16.s[2], b16.s[3], b16.s[4], b16.s[5], b16.s[6], b16.s[7],
                     ^~~~~~
/workspace/srcdir/OpenCL-Headers/build/tests/lang_cpp/../../../tests/test_headers.c:411:29: error: expected ‘)’ before ‘PRId64’
     printf("__cl_long2:  %" PRId64 " %" PRId64 " \n", ((cl_long*)&v2)[0], ((cl_long*)&v2)[1] );
                             ^~~~~~
/workspace/srcdir/OpenCL-Headers/build/tests/lang_cpp/../../../tests/test_headers.c: In function ‘int test_ulong()’:
/workspace/srcdir/OpenCL-Headers/build/tests/lang_cpp/../../../tests/test_headers.c:460:21: error: expected ‘)’ before ‘PRIu64’
     printf("b:   %" PRIu64 "\n", b );
                     ^~~~~~
/workspace/srcdir/OpenCL-Headers/build/tests/lang_cpp/../../../tests/test_headers.c:461:21: error: expected ‘)’ before ‘PRIu64’
     printf("b2:  %" PRIu64 " %" PRIu64 " \n", b2.s[0], b2.s[1] );
                     ^~~~~~
/workspace/srcdir/OpenCL-Headers/build/tests/lang_cpp/../../../tests/test_headers.c:462:21: error: expected ‘)’ before ‘PRIu64’
     printf("b4:  %" PRIu64 " %" PRIu64 " %" PRIu64 " %" PRIu64 "\n", b4.s[0], b4.s[1], b4.s[2], b4.s[3] );
                     ^~~~~~
/workspace/srcdir/OpenCL-Headers/build/tests/lang_cpp/../../../tests/test_headers.c:463:21: error: expected ‘)’ before ‘PRIu64’
     printf("b8:  %" PRIu64 " %" PRIu64 " %" PRIu64 " %" PRIu64 " %" PRIu64 " %" PRIu64 " %" PRIu64 " %" PRIu64 "\n", b8.s[0], b8.s[1], b8.s[2], b8.s[3], b8.s[4], b8.s[5], b8.s[6], b8.s[7] );
                     ^~~~~~
/workspace/srcdir/OpenCL-Headers/build/tests/lang_cpp/../../../tests/test_headers.c:464:21: error: expected ‘)’ before ‘PRIu64’
     printf("b16: %" PRIu64 " %" PRIu64 " %" PRIu64 " %" PRIu64 " %" PRIu64 " %" PRIu64 " %" PRIu64 " %" PRIu64 " %" PRIu64 " %" PRIu64 " %" PRIu64 " %" PRIu64 " %" PRIu64 " %" PRIu64 " %" PRIu64 " %" PRIu64 "\n", b16.s[0], b16.s[1], b16.s[2], b16.s[3], b16.s[4], b16.s[5], b16.s[6], b16.s[7],
                     ^~~~~~
/workspace/srcdir/OpenCL-Headers/build/tests/lang_cpp/../../../tests/test_headers.c:471:30: error: expected ‘)’ before ‘PRIu64’
     printf("__cl_ulong2:  %" PRIu64 " %" PRIu64 " \n", ((cl_ulong*)&v2)[0], ((cl_ulong*)&v2)[1] );
                              ^~~~~~
make[2]: *** [tests/lang_cpp/CMakeFiles/headers_cpp_220.dir/build.make:76: tests/lang_cpp/CMakeFiles/headers_cpp_220.dir/test_headers.cpp.o] Error 1
make[2]: Leaving directory '/workspace/srcdir/OpenCL-Headers/build'
make[1]: *** [CMakeFiles/Makefile2:4453: tests/lang_cpp/CMakeFiles/headers_cpp_220.dir/all] Error 2
make[1]: Leaving directory '/workspace/srcdir/OpenCL-Headers/build'
make: *** [Makefile:146: all] Error 2

Are we required to clone OpenCL-CLHPP as well? Can you please share a set of updated instructions that I can reproduce in a fresh session with GCC 7? Appreciate your help.

bashbaug commented 2 years ago

I'm the one responsible for these changes: https://github.com/KhronosGroup/OpenCL-Headers/pull/140

I thought I might have messed something up but it looks like I'm including the right header (<inttypes.h>), which is required since C99.

I can try to dig up a gcc7 system to repro if needed. In the meantime, can you please check that the intended version if inttypes.h is being included and that it really does not have these format macro constants defined? Thanks!

juliohm commented 2 years ago

I'm happy with any build that works. If you have another preferred compiler, please document it and make sure that the CI build is up to date 🙏🏼 Also, it is very good practice to set CMAKE_C_STANDARD and CMAKE_CXX_STANDARD to avoid these build issues that are hard to fix by non-maintainers.

Em ter., 27 de set. de 2022 12:02, Ben Ashbaugh @.***> escreveu:

I'm the one responsible for these changes: KhronosGroup/OpenCL-Headers#140 https://github.com/KhronosGroup/OpenCL-Headers/pull/140

I thought I might have messed something up but it looks like I'm including the right header (), which is required since C99.

I can try to dig up a gcc7 system to repro if needed. In the meantime, can you please check that the intended version if inttypes.h is being included and that it really does not have these format macro constants defined? Thanks!

— Reply to this email directly, view it on GitHub https://github.com/KhronosGroup/OpenCL-SDK/issues/58#issuecomment-1259638994, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAZQW3PS3SNTZ447MHPSFT3WAMD6ZANCNFSM6AAAAAAQV65VSM . You are receiving this because you were mentioned.Message ID: @.***>

bashbaug commented 2 years ago

Also, it is very good practice to set CMAKE_C_STANDARD and CMAKE_CXX_STANDARD to avoid these build issues that are hard to fix by non-maintainers.

Yes, agreed. We've done this in some places (e.g. OpenCL-ICD-Loader) but not in the headers.

See https://github.com/KhronosGroup/OpenCL-Headers/pull/208, I'm not sure if it might help to solve this problem? On my system it adds -std=gnu99 to my gcc compiles. I'm able to compile fine without this too (gcc 9.4.0), but maybe it will help for older gcc versions?

juliohm commented 2 years ago

I've incorporated the diff but the error is still there with GCC 7. I will try with GCC 9 now.

bashbaug commented 2 years ago

Come to think of it, we have gcc 7 in our CI for OpenCL-Headers.

Example: https://github.com/KhronosGroup/OpenCL-Headers/actions/runs/3116481007/jobs/5054348256

I can't explain why this isn't working.... :confused:

juliohm commented 2 years ago

If you want to reproduce the exact environment where I am trying to cross-compile OpenCL, it should take a few simple steps:

  1. Download Julia v1.7.3 from the binaries page: https://julialang.org/downloads/oldreleases/
  2. Install the BinaryBuilder package in the Julia prompt: using Pkg; Pkg.add("BinaryBuilder")
  3. Launch the wizard with using BinaryBuilder; BinaryBuilder.run_wizard()
  4. Provide the link to the OpenCL-Headers GitHub repository and follow the instructions

When you are done selecting the GCC compiler and commit from the repository that you want to build, it will show a prompt where you can type the CMAKE commands in the README. You should get the same error message I am getting.

bashbaug commented 2 years ago

Thanks, I have been able to reproduce the error you are seeing. Stay tuned...

juliohm commented 2 years ago

Thank you for taking a look into it. For completeness, this is the full process recorded in a video in case you want to finish the cross-compilation yourself: https://binarybuilder.org

bashbaug commented 2 years ago

I did a visual diff of the inttypes.h when the error occurs and my inttypes.h and noticed a difference:

/* The ISO C99 standard specifies that these macros must only be
   defined if explicitly requested.  */
#if !defined __cplusplus || defined __STDC_FORMAT_MACROS

So it seems that these headers at least require __STDC_FORMAT_MACROS to be defined for C++ compiles or the format macro constants will not be defined.

There's some interesting information in a note on cppreference as well:

The C99 standard suggests that C++ implementations should not define the above limit, constant, or format macros unless the macros STDC_LIMIT_MACROS, STDC_CONSTANT_MACROS or STDC_FORMAT_MACROS (respectively) are defined before including the relevant C header (stdint.h or inttypes.h). This recommendation was not adopted by any C++ standard and was removed in C11. However, some implementations (such as glibc 2.17) try to apply this rule, and it may be necessary to define the STDC macros; C++ compilers may try to work around this by automatically defining them in some circumstances.

Regardless, we should fix this in the OpenCL-Header tests and check to see if a similar problem occurs in any of the OpenCL repos. Here are two additions to the test_headers.c file that seemed to fix the problem:

  1. Define __STDC_FORMAT_MACROS for C++ compiles before including inttypes.h:
#if defined(__cplusplus)
#define __STDC_FORMAT_MACROS
#endif
  1. Include cinttypes instead of inttypes.h for C++ compiles:
#if defined(__cplusplus)
#include <cinttypes>
#else
#include <inttypes.h>
#endif

Any preference? If not, I'll probably make a PR with (2). Thanks!

juliohm commented 2 years ago

Thank you @bashbaug , any option that works is fine.

As soon as the changes are merged into the main branch we can retry the cross-compilation again with the specific commit. The video I linked above shows how the binaries can be generated for all 13 platforms automatically after we define a single build script that works.

bashbaug commented 2 years ago

PR for (2) is here: https://github.com/KhronosGroup/OpenCL-Headers/pull/209

juliohm commented 2 years ago

Thank you @bashbaug , can you confirm that the build was successful for the OpenCL-ICD-Loader as well after the OpenCL-Headers? Should we wait for the PR to get merged into the main branch?

juliohm commented 2 years ago

@bashbaug @MathiasMagnus I've successfully built OpenCL for 11 different platforms with this script:

https://github.com/juliohm/cross-platform-opencl

Can you help adding support for the remaining platforms?

You can run the script for a specific platform by passing the name of the platform in the command line:

$ julia --project build_tarballs.jl x86_64-unknown-freebsd

This is an example build that is failing due to a compilation error in OpenCL-ICD-Loader on FreeBSD. If we can solve all these tiny issues, we will be able to provide a more robust experience in future releases.

juliohm commented 2 years ago

The missing platforms are:

 macOS x86_64
 macOS aarch64
 FreeBSD x86_64
 Windows i686
 Windows x86_64

with corresponding identifier:

 "x86_64-apple-darwin"
 "aarch64-apple-darwin"
 "x86_64-unknown-freebsd"
 "i686-w64-mingw32"
 "x86_64-w64-mingw32"

If we can fix at least the build on Windows platforms, we will be able to provide OpenCL for the majority of Julia users.

juliohm commented 2 years ago

Most errors I am getting trying to build and install OpenCL are actually errors that occur in the test folders. Ideally the CMake build instructions wouldn't attempt to test the code during installation targets to avoid additional dependencies that are only required at test time.

Can the OpenCL maintainers take a look at the current CMake files for OpenCL-Headers and OpenCL-ICD-Loader to make sure that nothing else is being done other than building the library and copying the files to the prefix? I could then try to help fixing possible compilation errors on FreeBSD, Windows and MacOS.

bashbaug commented 2 years ago

Can the OpenCL maintainers take a look at the current CMake files for OpenCL-Headers and OpenCL-ICD-Loader to make sure that nothing else is being done other than building the library and copying the files to the prefix?

If it helps, all of our CMake files check for the standard BUILD_TESTING variable, so to bypass testing you can simply set BUILD_TESTING=0.

juliohm commented 2 years ago

Thank you @bashbaug , I've updated the cross-compilation script with this option turned off and the build is now successful in MacOS, totaling 13 platforms. We are only missing Windows and FreeBSD now.

Do you think you can take a look at the error messages produced on Windows?

bashbaug commented 2 years ago

Excellent, that's encouraging.

I started looking at the Windows build issues yesterday. I didn't get too far but my hunch is that they are related to this issue: https://github.com/KhronosGroup/OpenCL-ICD-Loader/issues/130#issuecomment-732088977

Can you say what this package is being used for? We may have some different options if it's only providing an import library to link to vs. building a DLL for deployment, but let's see if we can get things building without special options first.

BTW - somebody else was trying to get FreeBSD working here, with some success: https://github.com/KhronosGroup/OpenCL-SDK/issues/55#issuecomment-1264817406

juliohm commented 2 years ago

Can you say what this package is being used for? We may have some different options if it's only providing an import library to link to vs. building a DLL for deployment, but let's see if we can get things building without special options first.

We are trying to revive the OpenCL.jl package, which enables opencl calls in familiar Julia syntax: https://github.com/JuliaGPU/OpenCL.jl

The package simply loads the binaries that were built by robots on GitHub for the 13 platforms and makes direct C calls to the opencl api with Julia's ccall function: https://docs.julialang.org/en/v1/manual/calling-c-and-fortran-code

At the moment the OpenCL.jl package is assuming that the user installed opencl beforehand in the target system. We are changing that to the BinaryBuilder.jl approach which downloads the appropriate precompiled binary depending on the target system.

juliohm commented 2 years ago

@bashbaug I am a bit lost regarding the organization of OpenCL libraries. Say I want to compile a library to load later in Julia applications and make direct C calls with this library. Is it correct to say that I only need to complie OpenCL-Headers and OpenCL-ICD-Loader to get the library of interest and load it somewhere? What is ICD? I am finding some material online saying that there are two ways of working with OpenCL, but I am not an expert in GPUs, so everything is very confusing to me.

Besides installing the library, do we need anything else on the target machine? Do we also need to install custom drivers depending on the hardware we have? I am following this page by Beignet for Intel GPUs and many questions started to pop:

https://www.freedesktop.org/wiki/Software/Beignet

I loaded the library on a Julia session, but any command is giving me the error:

CLError(code=-1001, CL_PLATFORM_NOT_FOUND_KHR)

I wonder what it means exactly. I wonder if the compiled library was loaded correctly because the error seems be something that was thrown by the library. Is the error occurring because I don't have a opencl-compatible driver installed? I have a NVIDA GPU with a proprietary driver installed.

juliohm commented 2 years ago

I understood that the ICD is a system that loads different OpenCL driver implementations. And that we still need to have at least one driver installed manually on the target system to be able to use the library functions. Is that correct?

bashbaug commented 2 years ago

I think I'm overdue to write a full page for the OpenCL-Guide to describe how this works, so consider this a short preview draft for now, and I'll try to write up something more comprehensive shortly.

The "two methods" you mentioned may have come from my article describing how OpenCL works on Linux. It's correct, but for the purposes of this discussion we'll assume that we're using the OpenCL ICD loader, which is by far the most common mechanism for client usages and the one provided by the OpenCL SDK.

Terminology

Here are a few useful bits of terminology - see also the cl_khr_icd specification:

  1. ICD: Stands for "installable client driver". This is a mechanism where multiple OpenCL implementations from multiple vendors can coexist and be installed on the same system. An OpenCL implementation from a vendor is an OpenCL installable client driver. If you want to run OpenCL on a device then you need an OpenCL installable client driver for that device.

  2. ICD Loader: This is a library that is responsible for discovering the ICDs that are installed on the system, loading them, and enumerating them to applications. The OpenCL ICD loader is not tied to any one OpenCL implementation, so many operating system vendors provide packages for the OpenCL ICD loader, though some vendors will distribute an OpenCL ICD loader as well to ease deployment.

Usage

OpenCL applications link with the OpenCL ICD loader when building to provide definitions for the OpenCL APIs. Because the OpenCL ICD loader is a shared library (either libOpenCL.so for Linux or other Unix-like OSes or OpenCL.dll on Windows), linking with the OpenCL ICD loader adds run-time dependencies that must be satisfied for the application to run. This means that you will need the OpenCL ICD loader (or the OpenCL ICD loader import library OpenCL.lib on Windows) when you build your application, and you will also need an OpenCL ICD loader when you run your application to satisfy the run-time dependencies.

You only need an OpenCL ICD installed to run an OpenCL application on an OpenCL device, and you do not need any OpenCL ICDs installed purely to build an OpenCL application (this is useful for testing, though!).

Troubleshooting

Let's look at a couple of potential OpenCL ICD loader-related problems:

Missing OpenCL ICD Loader

If your target system does not have an OpenCL ICD loader installed then your application will fail to start because the run-time dependencies are not satisfied. This is not OpenCL-specific, and similar behavior will also be seen if any other shared libraries are missing.

Example:

$ ldd ./a.out
    libOpenCL.so.1 => not found
   ...
$ ./a.out
./a.out: error while loading shared libraries: libOpenCL.so.1: cannot open shared object file: No such file or directory

To fix this problem, either ensure that an existing OpenCL ICD loader can be found (may involve setting environment variables like LD_LIBRARY_PATH), install an OpenCL ICD loader package, or build and install an OpenCL ICD loader.

Out-of-Date OpenCL ICD Loader

If your target system has an OpenCL ICD loader installed but it is too old then your application will still fail to start because some run-time dependencies will still be unsatisfied. This is also not OpenCL-specific, and similar behavior will also be seen if any other shared libraries are too old.

Example:

$ nm -u ./specconst 
                 ...
                 U clSetProgramSpecializationConstant@@OPENCL_2.2
$ ./specconst 
./specconst: /usr/lib/x86_64-linux-gnu/libOpenCL.so.1: version `OPENCL_2.2' not found (required by ./specconst)

To fix this problem, either install an updated OpenCL ICD loader package, or build and install a newer OpenCL ICD loader.

No OpenCL Platforms

If your application is able to start but is unable to find any OpenCL platforms - either clGetPlatformIDs returns an error or it returns that zero platforms were found - then it is either likely that no OpenCL ICDs are installed, or that the OpenCL ICD loader cannot find the installed ICDs, or that there is a problem with the installed OpenCL ICDs (maybe it was for a device that has since been removed from the system).

The fix for this problem is a bit more open-ended but the article linked previously describes some troubleshooting steps to check to ensure that an OpenCL ICD exists and that it can be found by the OpenCL ICD loader. Note that the Khronos OpenCL ICD loader added several environment variables that were not available when the article was written that can be helpful for debug, and the ocl-icd OpenCL ICD loader also has similar environment variables.

bashbaug commented 2 years ago

Since you mentioned Intel GPUs, if you have an Intel GPU that is Broadwell or newer I'd strongly recommend using the "NEO" compute runtime rather than Beignet. NEO is our open source production driver stack and it is much more up-to-date and well-maintained.

We also ship a CPU OpenCL implementation as part of our oneAPI basekit.

If you are looking for an open source CPU implementation I'd recommend having a look at PoCL, which provides an up-to-date CPU OpenCL device and more.

juliohm commented 2 years ago

Thank you @bashbaug for the very detailed explanation ❤️ It helped a lot. We did some experiments yesterday and learned about ICD in practice by installing and removing drivers from different vendors. I wonder if this explanation could go in the README of this repository as well, to educate outsiders about these terms. I myself had a lot of trouble finding OpenCL-ICD-Loader and understanding that it is the library needed as a shared library in OpenCL programs. I started my journey here in the SDK repository thinking it was the main project to compile as a OpenCL programmer.

On the topic of drivers, do you know if there exists a list of recommended drivers for each device out there? We are trying to automate this installation process for end-users in Julia. Currently we already install the ICD shared library automatically depending on the platform. So whenever users decide that they will use OpenCL.jl, the Julia package manager downloads the correct ICD library as an artifact:

using Pkg

# installs OpenCL.jl, which in turn installs the appropriate
# shared library for the platform, we call it OpenCL_jll
Pkg.add("OpenCL")

However, the user still needs to manually install a driver, which is a bit painful. If we had a compiled list of drivers somewhere, we could try to download them at installation time, to make sure that at least one "recommended" driver is available. People who know what they are doing can always choose their drivers at runtime I suppose, but beginners will get an error message and will not know what is happening.

juliohm commented 2 years ago

Coming back to the original topic of the issue, we are only missing Windows now in our cross-compilation 🎉

Do you know a fix for the following error?

$ julia --project build_tarballs.jl x86_64-w64-mingw32                                                               ✔ 
[ Info: Building for x86_64-w64-mingw32
[ Info: Cached file found in /home/juliohm/.julia/packages/BinaryBuilderBase/h1spZ/deps/downloads/c3afd4ad0a37f0b61c0b8656ca4914002ba7994bba05aa2c47fde59b652289c9-209.patch
[ Info: Cached file found in /home/juliohm/.julia/packages/BinaryBuilderBase/h1spZ/deps/downloads/82725e3ec4e9fe333aeb53f75e13b74cef22c7cb662ee2af51108f0d764e1985-185.patch
 Downloading artifact: GCCBootstrap-x86_64-w64-mingw32.v6.1.0.x86_64-linux-musl.unpacked
  Downloaded artifact: GCCBootstrap-x86_64-w64-mingw32.v6.1.0.x86_64-linux-musl.unpacked
make[1]: Leaving directory '/workspace/srcdir/OpenCL-Headers/build'
Install the project...
/usr/bin/cmake -P cmake_install.cmake
-- Install configuration: "Release"
-- Installing: /workspace/destdir/include/CL
-- Installing: /workspace/destdir/include/CL/cl_icd.h
-- Installing: /workspace/destdir/include/CL/cl_dx9_media_sharing_intel.h
-- Installing: /workspace/destdir/include/CL/cl_gl_ext.h
-- Installing: /workspace/destdir/include/CL/cl_va_api_media_sharing_intel.h
-- Installing: /workspace/destdir/include/CL/cl_layer.h
-- Installing: /workspace/destdir/include/CL/cl_ext_intel.h
-- Installing: /workspace/destdir/include/CL/cl_d3d10.h
-- Installing: /workspace/destdir/include/CL/cl_platform.h
-- Installing: /workspace/destdir/include/CL/cl_dx9_media_sharing.h
-- Installing: /workspace/destdir/include/CL/cl_version.h
-- Installing: /workspace/destdir/include/CL/opencl.h
-- Installing: /workspace/destdir/include/CL/cl_gl.h
-- Installing: /workspace/destdir/include/CL/cl_ext.h
-- Installing: /workspace/destdir/include/CL/cl_egl.h
-- Installing: /workspace/destdir/include/CL/cl_half.h
-- Installing: /workspace/destdir/include/CL/cl.h
-- Installing: /workspace/destdir/include/CL/cl_d3d11.h
-- Installing: /workspace/destdir/share/cmake/OpenCLHeaders/OpenCLHeadersTargets.cmake
-- Installing: /workspace/destdir/share/cmake/OpenCLHeaders/OpenCLHeadersConfig.cmake
-- Installing: /workspace/destdir/share/cmake/OpenCLHeaders/OpenCLHeadersConfigVersion.cmake
Re-run cmake no build system arguments
-- The C compiler identification is GNU 6.1.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /opt/bin/x86_64-w64-mingw32-libgfortran3-cxx11/x86_64-w64-mingw32-gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE  
-- Looking for secure_getenv
-- Looking for secure_getenv - not found
-- Looking for __secure_getenv
-- Looking for __secure_getenv - not found
-- Configuring done
-- Generating done
-- Build files have been written to: /workspace/srcdir/OpenCL-ICD-Loader/build
/usr/bin/cmake -S/workspace/srcdir/OpenCL-ICD-Loader -B/workspace/srcdir/OpenCL-ICD-Loader/build --check-build-system CMakeFiles/Makefile.cmake 0
/usr/bin/cmake -E cmake_progress_start /workspace/srcdir/OpenCL-ICD-Loader/build/CMakeFiles /workspace/srcdir/OpenCL-ICD-Loader/build//CMakeFiles/progress.marks
/usr/bin/make  -f CMakeFiles/Makefile2 all
make[1]: Entering directory '/workspace/srcdir/OpenCL-ICD-Loader/build'
/usr/bin/make  -f CMakeFiles/OpenCL.dir/build.make CMakeFiles/OpenCL.dir/depend
make[2]: Entering directory '/workspace/srcdir/OpenCL-ICD-Loader/build'
cd /workspace/srcdir/OpenCL-ICD-Loader/build && /usr/bin/cmake -E cmake_depends "Unix Makefiles" /workspace/srcdir/OpenCL-ICD-Loader /workspace/srcdir/OpenCL-ICD-Loader /workspace/srcdir/OpenCL-ICD-Loader/build /workspace/srcdir/OpenCL-ICD-Loader/build /workspace/srcdir/OpenCL-ICD-Loader/build/CMakeFiles/OpenCL.dir/DependInfo.cmake --color=
Dependee "/workspace/srcdir/OpenCL-ICD-Loader/build/CMakeFiles/OpenCL.dir/DependInfo.cmake" is newer than depender "/workspace/srcdir/OpenCL-ICD-Loader/build/CMakeFiles/OpenCL.dir/depend.internal".
Dependee "/workspace/srcdir/OpenCL-ICD-Loader/build/CMakeFiles/CMakeDirectoryInformation.cmake" is newer than depender "/workspace/srcdir/OpenCL-ICD-Loader/build/CMakeFiles/OpenCL.dir/depend.internal".
Scanning dependencies of target OpenCL
make[2]: Leaving directory '/workspace/srcdir/OpenCL-ICD-Loader/build'
/usr/bin/make  -f CMakeFiles/OpenCL.dir/build.make CMakeFiles/OpenCL.dir/build
make[2]: Entering directory '/workspace/srcdir/OpenCL-ICD-Loader/build'
[ 10%] Building C object CMakeFiles/OpenCL.dir/loader/icd.c.obj
[ 20%] Building C object CMakeFiles/OpenCL.dir/loader/icd_dispatch.c.obj
/opt/bin/x86_64-w64-mingw32-libgfortran3-cxx11/x86_64-w64-mingw32-gcc --sysroot=/opt/x86_64-w64-mingw32/x86_64-w64-mingw32/sys-root/ -DCL_ENABLE_LAYERS -DCL_TARGET_OPENCL_VERSION=300 -DOpenCL_EXPORTS @CMakeFiles/OpenCL.dir/includes_C.rsp -O3 -DNDEBUG -std=gnu99 -MD -MT CMakeFiles/OpenCL.dir/loader/icd.c.obj -MF CMakeFiles/OpenCL.dir/loader/icd.c.obj.d -o CMakeFiles/OpenCL.dir/loader/icd.c.obj -c /workspace/srcdir/OpenCL-ICD-Loader/loader/icd.c
[ 30%] Building C object CMakeFiles/OpenCL.dir/loader/icd_dispatch_generated.c.obj
/opt/bin/x86_64-w64-mingw32-libgfortran3-cxx11/x86_64-w64-mingw32-gcc --sysroot=/opt/x86_64-w64-mingw32/x86_64-w64-mingw32/sys-root/ -DCL_ENABLE_LAYERS -DCL_TARGET_OPENCL_VERSION=300 -DOpenCL_EXPORTS @CMakeFiles/OpenCL.dir/includes_C.rsp -O3 -DNDEBUG -std=gnu99 -MD -MT CMakeFiles/OpenCL.dir/loader/icd_dispatch.c.obj -MF CMakeFiles/OpenCL.dir/loader/icd_dispatch.c.obj.d -o CMakeFiles/OpenCL.dir/loader/icd_dispatch.c.obj -c /workspace/srcdir/OpenCL-ICD-Loader/loader/icd_dispatch.c
[ 40%] Building C object CMakeFiles/OpenCL.dir/loader/windows/icd_windows.c.obj
/opt/bin/x86_64-w64-mingw32-libgfortran3-cxx11/x86_64-w64-mingw32-gcc --sysroot=/opt/x86_64-w64-mingw32/x86_64-w64-mingw32/sys-root/ -DCL_ENABLE_LAYERS -DCL_TARGET_OPENCL_VERSION=300 -DOpenCL_EXPORTS @CMakeFiles/OpenCL.dir/includes_C.rsp -O3 -DNDEBUG -std=gnu99 -MD -MT CMakeFiles/OpenCL.dir/loader/icd_dispatch_generated.c.obj -MF CMakeFiles/OpenCL.dir/loader/icd_dispatch_generated.c.obj.d -o CMakeFiles/OpenCL.dir/loader/icd_dispatch_generated.c.obj -c /workspace/srcdir/OpenCL-ICD-Loader/loader/icd_dispatch_generated.c
[ 50%] Building C object CMakeFiles/OpenCL.dir/loader/windows/icd_windows_dxgk.c.obj
/opt/bin/x86_64-w64-mingw32-libgfortran3-cxx11/x86_64-w64-mingw32-gcc --sysroot=/opt/x86_64-w64-mingw32/x86_64-w64-mingw32/sys-root/ -DCL_ENABLE_LAYERS -DCL_TARGET_OPENCL_VERSION=300 -DOpenCL_EXPORTS @CMakeFiles/OpenCL.dir/includes_C.rsp -O3 -DNDEBUG -std=gnu99 -MD -MT CMakeFiles/OpenCL.dir/loader/windows/icd_windows.c.obj -MF CMakeFiles/OpenCL.dir/loader/windows/icd_windows.c.obj.d -o CMakeFiles/OpenCL.dir/loader/windows/icd_windows.c.obj -c /workspace/srcdir/OpenCL-ICD-Loader/loader/windows/icd_windows.c
[ 60%] Building C object CMakeFiles/OpenCL.dir/loader/windows/icd_windows_envvars.c.obj
/opt/bin/x86_64-w64-mingw32-libgfortran3-cxx11/x86_64-w64-mingw32-gcc --sysroot=/opt/x86_64-w64-mingw32/x86_64-w64-mingw32/sys-root/ -DCL_ENABLE_LAYERS -DCL_TARGET_OPENCL_VERSION=300 -DOpenCL_EXPORTS @CMakeFiles/OpenCL.dir/includes_C.rsp -O3 -DNDEBUG -std=gnu99 -MD -MT CMakeFiles/OpenCL.dir/loader/windows/icd_windows_dxgk.c.obj -MF CMakeFiles/OpenCL.dir/loader/windows/icd_windows_dxgk.c.obj.d -o CMakeFiles/OpenCL.dir/loader/windows/icd_windows_dxgk.c.obj -c /workspace/srcdir/OpenCL-ICD-Loader/loader/windows/icd_windows_dxgk.c
[ 70%] Building C object CMakeFiles/OpenCL.dir/loader/windows/icd_windows_hkr.c.obj
/opt/bin/x86_64-w64-mingw32-libgfortran3-cxx11/x86_64-w64-mingw32-gcc --sysroot=/opt/x86_64-w64-mingw32/x86_64-w64-mingw32/sys-root/ -DCL_ENABLE_LAYERS -DCL_TARGET_OPENCL_VERSION=300 -DOpenCL_EXPORTS @CMakeFiles/OpenCL.dir/includes_C.rsp -O3 -DNDEBUG -std=gnu99 -MD -MT CMakeFiles/OpenCL.dir/loader/windows/icd_windows_envvars.c.obj -MF CMakeFiles/OpenCL.dir/loader/windows/icd_windows_envvars.c.obj.d -o CMakeFiles/OpenCL.dir/loader/windows/icd_windows_envvars.c.obj -c /workspace/srcdir/OpenCL-ICD-Loader/loader/windows/icd_windows_envvars.c
/opt/bin/x86_64-w64-mingw32-libgfortran3-cxx11/x86_64-w64-mingw32-gcc --sysroot=/opt/x86_64-w64-mingw32/x86_64-w64-mingw32/sys-root/ -DCL_ENABLE_LAYERS -DCL_TARGET_OPENCL_VERSION=300 -DOpenCL_EXPORTS @CMakeFiles/OpenCL.dir/includes_C.rsp -O3 -DNDEBUG -std=gnu99 -MD -MT CMakeFiles/OpenCL.dir/loader/windows/icd_windows_hkr.c.obj -MF CMakeFiles/OpenCL.dir/loader/windows/icd_windows_hkr.c.obj.d -o CMakeFiles/OpenCL.dir/loader/windows/icd_windows_hkr.c.obj -c /workspace/srcdir/OpenCL-ICD-Loader/loader/windows/icd_windows_hkr.c
[ 80%] Building C object CMakeFiles/OpenCL.dir/loader/windows/icd_windows_apppackage.c.obj
/opt/bin/x86_64-w64-mingw32-libgfortran3-cxx11/x86_64-w64-mingw32-gcc --sysroot=/opt/x86_64-w64-mingw32/x86_64-w64-mingw32/sys-root/ -DCL_ENABLE_LAYERS -DCL_TARGET_OPENCL_VERSION=300 -DOpenCL_EXPORTS @CMakeFiles/OpenCL.dir/includes_C.rsp -O3 -DNDEBUG -std=gnu99 -MD -MT CMakeFiles/OpenCL.dir/loader/windows/icd_windows_apppackage.c.obj -MF CMakeFiles/OpenCL.dir/loader/windows/icd_windows_apppackage.c.obj.d -o CMakeFiles/OpenCL.dir/loader/windows/icd_windows_apppackage.c.obj -c /workspace/srcdir/OpenCL-ICD-Loader/loader/windows/icd_windows_apppackage.c
/workspace/srcdir/OpenCL-ICD-Loader/loader/windows/icd_windows_apppackage.c:32:22: fatal error: AppModel.h: No such file or directory
 #include <AppModel.h>
                      ^
compilation terminated.
make[2]: *** [CMakeFiles/OpenCL.dir/build.make:182: CMakeFiles/OpenCL.dir/loader/windows/icd_windows_apppackage.c.obj] Error 1
make[2]: *** Waiting for unfinished jobs....
/workspace/srcdir/OpenCL-ICD-Loader/loader/windows/icd_windows_hkr.c: In function ‘khrIcdOsVendorsEnumerateHKR’:
/workspace/srcdir/OpenCL-ICD-Loader/loader/windows/icd_windows_hkr.c:217:21: error: ‘CM_GETIDLIST_FILTER_CLASS’ undeclared (first use in this function)
     ULONG ulFlags = CM_GETIDLIST_FILTER_CLASS |
                     ^~~~~~~~~~~~~~~~~~~~~~~~~
/workspace/srcdir/OpenCL-ICD-Loader/loader/windows/icd_windows_hkr.c:217:21: note: each undeclared identifier is reported only once for each function it appears in
/workspace/srcdir/OpenCL-ICD-Loader/loader/windows/icd_windows_hkr.c:218:21: error: ‘CM_GETIDLIST_FILTER_PRESENT’ undeclared (first use in this function)
                     CM_GETIDLIST_FILTER_PRESENT;
                     ^~~~~~~~~~~~~~~~~~~~~~~~~~~
/workspace/srcdir/OpenCL-ICD-Loader/loader/windows/icd_windows_hkr.c:343:22: error: ‘DEVPKEY_Device_ClassGuid’ undeclared (first use in this function)
                     &DEVPKEY_Device_ClassGuid,
                      ^~~~~~~~~~~~~~~~~~~~~~~~
make[2]: *** [CMakeFiles/OpenCL.dir/build.make:167: CMakeFiles/OpenCL.dir/loader/windows/icd_windows_hkr.c.obj] Error 1
/workspace/srcdir/OpenCL-ICD-Loader/loader/windows/icd_windows.c: In function ‘khrIcdOsVendorsEnumerateOnce’:
/workspace/srcdir/OpenCL-ICD-Loader/loader/windows/icd_windows.c:407:5: warning: implicit declaration of function ‘InitOnceExecuteOnce’ [-Wimplicit-function-declaration]
     InitOnceExecuteOnce(&initialized, khrIcdOsVendorsEnumerate, NULL, NULL);
     ^~~~~~~~~~~~~~~~~~~
make[2]: Leaving directory '/workspace/srcdir/OpenCL-ICD-Loader/build'
make[1]: *** [CMakeFiles/Makefile2:83: CMakeFiles/OpenCL.dir/all] Error 2
make[1]: Leaving directory '/workspace/srcdir/OpenCL-ICD-Loader/build'
make: *** [Makefile:136: all] Error 2
juliohm commented 2 years ago

I opened #61 to continue the discussion of drivers.

juliohm commented 2 years ago

@bashbaug if you want to test the build in a different platform where an error occurs, you can pass the option --debug to the script and it will open the session in the emulated environment:

$ julia --project build_tarballs.jl --debug x86_64-w64-mingw32

In this attempt with Windows for example, you will get a compilation error. You can then edit the files normally and re-run the CMake and build instructions until everything works.

Kerilk commented 2 years ago

Disclaimer: I don't develop on Windows, so @jenatali or @MathiasMagnus would know more.

Just a couple of facts, and maybe a solution:

Maybe we could test for the presence of AppModel.h before enabling OpenCLOn12, as it cannot be provided on older platform, or crash during the CMake phase with guidance as potential solutions.

Maybe MSYS2 need to update their msys2-w32api-headers package to include AppModel.h. see: https://packages.msys2.org/package/msys2-w32api-headers?repo=msys&variant=x86_64

I hope this helps.

juliohm commented 2 years ago
  • warning: "Disable support for OpenCLOn12. Support for OpenCLOn12 should only be disabled when building an import lib to link with, and must be enabled when building an ICD loader for distribution!"

Thank you @Kerilk , that is very helpful. From the warning I understand that we should enable the option as we are distributing the ICD loader library in Julia programs, correct?

Maybe we could test for the presence of AppModel.h before enabling OpenCLOn12, as it cannot be provided on older platform, or crash during the CMake phase with guidance as potential solutions.

That would be wonderful. Do you think you could cook a small PR with the proposed fix? I can easily grab the diff from the PR to update the build instructions in the script.

juliohm commented 2 years ago

When I try to disable the option in the command line I get additional build errors. I really need some help from Windows developers 🙏🏽

bashbaug commented 2 years ago

I started looking at the Windows build issues and they do appear to be due to the issue I linked previously:

KhronosGroup/OpenCL-ICD-Loader#130 (comment)

In short, the included version of cfgmgr32.h is a little too old - it's missing the commit from July 2020 that adds the symbols the ICD loader requires - see the history for this file. Is there any chance we could use a slightly newer version of mingw? The one we have is awfully close (it has the previous commit, from April 2020) but not quite new enough.

If we can't move to a newer version, the next best option I can think of is to add a CMake control to disable enumerating drivers via HKR, similar to the control to disable OpenCLon12. This is REALLY something we shouldn't do if we're building an OpenCL ICD loader that will actually be used to enumerate drivers, but it would be OK if all we need is an OpenCL.lib import library (or in the case of mingw, a libOpenCL.dll.a import library) to link with.

If I bypass the HKR enumeration and disable OpenCLon12 I am able to build successfully:

Install the project...
/usr/bin/cmake -P cmake_install.cmake
-- Install configuration: "Release"
-- Installing: /workspace/destdir/lib/libOpenCL.dll.a
-- Installing: /workspace/destdir/bin/libOpenCL.dll
-- Installing: /workspace/destdir/share/cmake/OpenCLICDLoader/OpenCLICDLoaderTargets.cmake
-- Installing: /workspace/destdir/share/cmake/OpenCLICDLoader/OpenCLICDLoaderTargets-release.cmake
-- Installing: /workspace/destdir/share/cmake/OpenCLICDLoader/OpenCLICDLoaderConfig.cmake
-- Installing: /workspace/destdir/share/cmake/OpenCLICDLoader/OpenCLICDLoaderConfigVersion.cmake
-- Up-to-date: /workspace/destdir/lib/libOpenCL.dll.a
-- Up-to-date: /workspace/destdir/bin/libOpenCL.dll
sandbox:${WORKSPACE}/srcdir #
bashbaug commented 2 years ago

Another thought: we could just use magic numbers in the case that the headers are tool old and the newer symbols are not defined. This appears to be what the Vulkan loader is doing:

https://github.com/KhronosGroup/Vulkan-Loader/blob/823078c862f2d6deff673ca8d45a591f620f5712/loader/loader_windows.c#L232

bashbaug commented 2 years ago

PR https://github.com/KhronosGroup/OpenCL-ICD-Loader/pull/186 let me get farther and I can build x86_64-w64-mingw32 now if I disable OpenCLon12. :tada:.

I'm getting a different error for i686-w64-mingw32 though, unfortunately. I'll need to take a look at this tomorrow.

We'll also need to decide if disabling OpenCLon12 is OK in this use-case or if we need to find an alternate solution.

juliohm commented 2 years ago

That is awesome @bashbaug ! 🎉 Thank you for helping!

Is there any chance we could use a slightly newer version of mingw?

I will investigate this with the BinaryBuilder.jl maintainers. I am currently specifying GCC 6.1 in the build script as the oldest version for which the cross-compilation worked. Any version that is more recent should work fine, but I am not sure about the mingw story as I never developed on Windows myself.

Will keep you posted about any update 👍🏽

bashbaug commented 2 years ago

Good news, I got i686-w64-mingw32 working also. The fix for this one was pretty simple - we need to define a newer version of _WIN32_WINNT for MinGW. I added it to the same PR https://github.com/KhronosGroup/OpenCL-ICD-Loader/pull/186.

There are still a few linker warnings for this configuration that look a little scary though:

Warning: resolving _clEnqueueNDRangeKernel by linking to _clEnqueueNDRangeKernel@36
Warning: resolving _clEnqueueNativeKernel by linking to _clEnqueueNativeKernel@40
Warning: resolving _clEnqueueReadBuffer by linking to _clEnqueueReadBuffer@36
...

This appears to have something to do with __stdcall vs. __cdecl calling conventions. I've been building on Linux so far so I don't have an easy way of testing this configuration. Is there some way we can tell if this is actually a problem before digging deeper? Worst-case I suppose I could try installing 32-bit MinGW on my Windows system...

bashbaug commented 2 years ago

The linker warnings do seem to be a problem. This is a good description what is going on: https://www.willus.com/mingw/yongweiwu_stdcall.html

Specifically, for stdcall functions (the calling convention used by the 32-bit Windows ICD loader) mingw is generating decorated exports (with an @n suffix) whereas MSVC is not. I need to find a way to get the undecorated name exported instead. I have a few ideas to try...

bashbaug commented 2 years ago

Hi @juliohm, I think the last of the ICD loader fixes for mingw were merged today. Would you mind giving the latest code a try to see if any issues still remain? Thanks!

juliohm commented 2 years ago

Thank you @bashbaug for the update. I will try to rerun the build script today with Windows in it. I am assuming that I can just clone the master branch of the ICD loader and everything should work fine 🙏🏽

juliohm commented 2 years ago

@bashbaug I've updated the build script at https://github.com/juliohm/cross-platform-opencl with the latest commits from the Headers and ICD-Loader repositories. The build is successful on all platforms except on Windows as before.

Any additional patch that I should consider?

bashbaug commented 2 years ago

Hi, I grabbed the latest build script - thanks! Can you confirm this is the build error you are seeing on Windows? This is from x86_64, but I see a similar error with i686 also:

bashbaug@bashbaug-nuc:~/git/cross-platform-opencl$ julia --project build_tarballs.jl --debug x86_64-w64-mingw32
[ Info: Building for x86_64-w64-mingw32
┌ Warning: Linked library CFGMGR32.dll could not be resolved and could not be auto-mapped
└ @ BinaryBuilder.Auditor ~/.julia/packages/BinaryBuilder/wohhx/src/Auditor.jl:383
[ Info: Found a valid dl path OpenCL.dll while looking for libOpenCL
[ Info: Could not locate libOpenCL inside ["/home/bashbaug/git/cross-platform-opencl/build/x86_64-w64-mingw32/m7Tomo8T/x86_64-w64-mingw32-libgfortran3-cxx11/destdir/bin"]
┌ Error: Built OpenCL but libopencl still unsatisfied:
└ @ BinaryBuilder ~/.julia/packages/BinaryBuilder/wohhx/src/AutoBuild.jl:906
ERROR: LoadError: Cannot continue with unsatisfied build products!

If so, I think we're close, but maybe the build script is looking for the wrong files or in the wrong directory? It looks like everything was built correctly, and I see OpenCL.dll, libOpenCL.dll.a, and all of the right headers:

bashbaug@bashbaug-nuc:~/git/cross-platform-opencl$ ls -R /home/bashbaug/git/cross-platform-opencl/build/x86_64-w64-mingw32/m7Tomo8T/x86_64-w64-mingw32-libgfortran3-cxx11/destdir
/home/bashbaug/git/cross-platform-opencl/build/x86_64-w64-mingw32/m7Tomo8T/x86_64-w64-mingw32-libgfortran3-cxx11/destdir:
bin  include  lib  logs  share

/home/bashbaug/git/cross-platform-opencl/build/x86_64-w64-mingw32/m7Tomo8T/x86_64-w64-mingw32-libgfortran3-cxx11/destdir/bin:
OpenCL.dll

/home/bashbaug/git/cross-platform-opencl/build/x86_64-w64-mingw32/m7Tomo8T/x86_64-w64-mingw32-libgfortran3-cxx11/destdir/include:
CL

/home/bashbaug/git/cross-platform-opencl/build/x86_64-w64-mingw32/m7Tomo8T/x86_64-w64-mingw32-libgfortran3-cxx11/destdir/include/CL:
cl_d3d10.h  cl_dx9_media_sharing.h        cl_egl.h  cl_ext_intel.h  cl_gl.h  cl_half.h  cl_layer.h     cl_va_api_media_sharing_intel.h  opencl.h
cl_d3d11.h  cl_dx9_media_sharing_intel.h  cl_ext.h  cl_gl_ext.h     cl.h     cl_icd.h   cl_platform.h  cl_version.h

/home/bashbaug/git/cross-platform-opencl/build/x86_64-w64-mingw32/m7Tomo8T/x86_64-w64-mingw32-libgfortran3-cxx11/destdir/lib:
libOpenCL.dll.a
...

Any ideas?

juliohm commented 2 years ago

Can you confirm this is the build error you are seeing on Windows?

Yep, the exact same error. This cross-compilation environment is supposed to be reproducible 🙏🏽

If so, I think we're close, but maybe the build script is looking for the wrong files or in the wrong directory?

That is the same guess, but I am not familiar with Windows builds to hunt the source of the issue. It is probably specific to mingw compiler flags?

bashbaug commented 2 years ago

Yep, the exact same error. This cross-compilation environment is supposed to be reproducible 🙏🏽

OK great, just wanted to be 100% sure.

I think we need to understand what is going on here:

[ Info: Found a valid dl path OpenCL.dll while looking for libOpenCL [ Info: Could not locate libOpenCL inside ["/home/bashbaug/git/cross-platform-opencl/build/x86_64-w64-mingw32/m7Tomo8T/x86_64-w64-mingw32-libgfortran3-cxx11/destdir/bin"] ┌ Error: Built OpenCL but libopencl still unsatisfied:

From my end everything looks like it is working as expected:

We must still be doing something wrong though, or at least not what the BinaryBuilder is expecting.

I'll poke around at this but any insight you could provide would be much appreciated. Thank you again for all of your help so far!

juliohm commented 7 months ago

Thank you all for the help here 👍🏽 I believe that most issues were solved. Feel free to open a separate issue to track the Windows build if it is still necessary.