[Build] Pre-installed dependencies #20310

Open MustafaYounes1 opened 6 months ago

Describe the issue

Hello,

Due to some environmental restrictions, I had to build ORT 1.15.1 from source on Windows x64 with VS2019.

By looking into the build script, one can find that we can use preinstalled eigen, and that was necessary for me, cause I'm adding ORT as a new dependency to a dependencies project that has eigen 3.3.7 among other dependencies.

However, the build process crashes with the following error, seemingly, eigen 3.3.7 is not compatible with the eigen ORT gets from here

The pulled eigen version is 3.3.9, however, when I tried to compare it against the eigen 3.3.9 from the official repo on GitLab, they seemed not to be consistent, so is it a special eigen version that ORT needs to get compiled from source?

Additionally, I have in my dependencies project boost 1.75.0, and by checking the cmake/deps.txt, we can see that ORT needs mp11 which is part of boost 1.79.0, so is it possible to add a preinstalled mp11 as well?

BTW, boost 1.81.0 is shown as a dependency in the deps.txt, yet it's not built with ORT, so it's a bit confusing that it's there.

Urgency

Would appreciate fast replies :)

Target platform

Windows x64

Build script

python tools\ci_build\build.py --config Release --build_shared_lib --parallel --skip_tests --compile_no_warning_as_error --skip_submodule_sync --use_preinstalled_eigen --eigen_path EIGEN_SOURCE_PATH --cmake_extra_defines onnxruntime_BUILD_UNIT_TESTS=OFF CMAKE_INSTALL_PREFIX=install CMAKE_TLS_VERIFY=OFF

Error / output

Many build errors stemming from incompatible eigen version:

fatal error C1083: Cannot open include file: 'Eigen/src/Core/arch/Default/Half.h': No such file or directory
error C2039: 'emplace_back': is not a member of 'Eigen::MaxSizeVector<Eigen::MaxSizeVector<unsigned int>>
error C2039: 'ArrayX': is not a member of 'Eigen'
...

Visual Studio Version

2019

GCC / Compiler Version

N/A

The compiler says it cannot find the include file. I am inclined to think that your eigen path specification is either incorrect, or your Eigne does not have that header file that is required for ORT to build. It is true you can specify Eigen on a side, but the code would still need what it needs.

boost::mp11 is a header only dependency, lots of boost (and Eigen) modules are header only, so they are pulled and then used during the build.

The ONNX Runtime source code is compatible with the 3rd party libraries that are listed in cmake/deps.txt with the specific versions. For example, the latest code use mp11 from boost 1.82.0. The other versions, boost 1.75.0/1.79.0/1.81.0 may or may not work. For eigen, we are using the git commit id e7248b26a1ed53fa030c5c459f7ea095dfd276ac . You hit the error because you were using a different version of eigen.

thanks for the replies @yuslepukhin, @snnn

@yuslepukhin I can confirm that I'm providing the correct eigen path. @snnn I do agree with u, as I stated in the description, it seems that I have an incompatible version of eigen in my project dependencies.

The strange thing is the incompatibility between eigen releases and its master branch. For instance, I checked eigen 3.3.7 (the one I have), 3.3.8, 3.3.9, 3.4.0 and all of them missed this include Eigen/src/Core/arch/Default/Half.h, whereas it could be found in any commit on the master branch.

And I don't get the reason! but it seems that we have to have eigen from their master branch if we want to use a preinstalled eigen when building ORT from source, don't we?

But how about mp11? with ORT build script we have an option to provide a path to a preinstalled eigen, but we don't have such an option with boost, so assuming we have a compatible version of boost, how we can use its mp11 when building ORT?

Have you checked the eigen commit id e7248b26a1ed53fa030c5c459f7ea095dfd276ac ?

I haven't tried it yet, but after checking the source code there I think it has what it is missed. The mentioned commit belongs to the latest eigen version 3.4.0, and I have eigen 3.3.7 in my dependencies project.

Additionally, as I mentioned in my description, I have to use ORT 1.15.1, which depends on eigen 3.3.9.

In brief, what one can get from this dicussion is that it is recommended to use a pre-installed eigen of the exact required version mentioned in 'cmake/deps.txt' and we should pull it only from the eigen master branch, right?

You don't support eigen releases, do you? Only the eigen master branch?

Would you please provide hints to use 'mp11' from a pre-installed boost?

Eigen doesn't have 3.7.0. You might miss something.

The latest Eigen release doesn't work well with the latest ONNX Runtime code. Therefore we chose a different commit id to use.

Sorry about the typo, I meant 3.3.7 and not 3.7.0

It is not a matter of eigen version, it is a matter of from where you get eigen.

For instance, the target ORT 1.15.1 depends on eigen 3.3.9 (specifically, master branch commit 'd10b27fe37736d2944630ecd7557cefa95cf87c9'), if you checked the eigen source in the mentioned commit and compare it against the eigen 3.3.9 release, you will find that the mentioned commit has some source code that does not exist in the release (e.g. Eigen/src/Core/arch/Default/Half.h) (please check the mentioned links)

My pre-installed eigen is 3.3.7 and I download it from the eigen releases section.

Eigen 3.3.9 was released on December 4, 2020. Most fp16 hardware was release after that. So we cannot get the old Eigen work.

But the commit ORT 1.15.1 depends on was pushed to the master branch on August 14, 2020 (see) (before the eigen 3.3.9 release) I don't see the reason why eigen releases are not compatible with the eigen master branch.

After all, I see that ORT supports preinstalled eigen from their master branch, it's a bit strange that we can't use preinstalled eigen releases when building ORT.

Do you provide an option to use a preinstalled mp11?, and if yes, please elaborate a bit about it.

thanx.

Eigen does not release often, but we need features that appear in commits in between releases, as @snnn mentioned. You are welcome to produce a PR that upgrades it to something later if there is a need.

I do not think we provide something for mp11.

The only eigen release that worked with ORT 1.15.1 is eigen 3.4.0, so, it seems that I have to upgrade the eigen version in my dependencies project from 3.3.7 to 3.4.0.

The latest ORT release 1.17.3 depends on eigen 3.4.0, so there is no need to a PR for upgrading eigen, however, I managed to extend the build script so that pre-installed mp11 can be used, I successfully build ORT 1.15.1 with a pre-installed Boost::mp11 1.75.0.

So, if you think that the community would be interested in using a pre-installed Boost::mp11, I can open a PR for that, otherwise, we can close this issue.

Thanks for the support @snnn, @yuslepukhin

This issue has been automatically marked as stale due to inactivity and will be closed in 30 days if no further activity occurs. If further support is needed, please provide an update and/or more details.

microsoft / onnxruntime