Failed to build with Quantlib CMAKE_CXX_STANDARD set to 20

[xwzhu]

CXX_STANDARD 14 and 17 builds fine, however with 20 I got the following error on Windows with VS2022 17.9.6:

>------ Build started: Configuration: windows-xad-msvc-release -------
  [1/862] Building CXX object ql\CMakeFiles\ql_library.dir\experimental\catbonds\catrisk.cpp.obj
  FAILED: ql/CMakeFiles/ql_library.dir/experimental/catbonds/catrisk.cpp.obj 
  "C:\PROGRA~1\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\bin\Hostx64\x64\cl.exe"  /nologo /TP -DBOOST_ALL_NO_LIB -DNOMINMAX -DQL_COMPILATION -DQL_INCLUDE_FIRST=ql/qlrisks.hpp -D_CRT_SECURE_NO_WARNINGS -D_SILENCE_CXX17_ITERATOR_BASE_CLASS_DEPRECATION_WARNING -ID:\QuantLib\build\windows-xad-msvc-release -ID:\QuantLib -ID:\QuantLib-Risks-Cpp\ql\.. -ID:\XAD\src -ID:\QuantLib\build\windows-xad-msvc-release\xad\src -external:IC:\Users\Administrator\source\repos\boost_1_84_0 -external:W0 /DWIN32 /D_WINDOWS /GR /EHsc /O2 /Ob2 /DNDEBUG -std:c++20 -MT -W3 /wd4267 /wd4819 /wd26812 /bigobj /showIncludes /Foql\CMakeFiles\ql_library.dir\experimental\catbonds\catrisk.cpp.obj /Fdql\CMakeFiles\ql_library.dir\ql_library.pdb /FS -c D:\QuantLib\ql\experimental\catbonds\catrisk.cpp
C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\include\random(3348): error C2665: 'sqrt': no overloaded function could convert all the argument types
  C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\include\cmath(530): note: could be 'long double sqrt(long double) noexcept'
  C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\include\random(3348): note: 'long double sqrt(long double) noexcept': cannot convert argument 1 from 'xad::UnaryExpr<Scalar,xad::scalar_sub2_op<Scalar,int>,xad::UnaryExpr<Scalar,xad::scalar_prod_op<Scalar,QuantLib::Integer>,xad::ADVar<Scalar>>>' to 'long double'
          with
          [
              Scalar=double
          ]
  C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\include\random(3348): note: No user-defined-conversion operator available that can perform this conversion, or the operator cannot be called
  C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\include\cmath(264): note: or       'float sqrt(float) noexcept'
  C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\include\random(3348): note: 'float sqrt(float) noexcept': cannot convert argument 1 from 'xad::UnaryExpr<Scalar,xad::scalar_sub2_op<Scalar,int>,xad::UnaryExpr<Scalar,xad::scalar_prod_op<Scalar,QuantLib::Integer>,xad::ADVar<Scalar>>>' to 'float'
          with
          [
              Scalar=double
          ]
  C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\include\random(3348): note: No user-defined-conversion operator available that can perform this conversion, or the operator cannot be called
  C:\Program Files (x86)\Windows Kits\10\include\10.0.22621.0\ucrt\corecrt_math.h(489): note: or       'double sqrt(double)'
  C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\include\random(3348): note: 'double sqrt(double)': cannot convert argument 1 from 'xad::UnaryExpr<Scalar,xad::scalar_sub2_op<Scalar,int>,xad::UnaryExpr<Scalar,xad::scalar_prod_op<Scalar,QuantLib::Integer>,xad::ADVar<Scalar>>>' to 'double'
          with
          [
              Scalar=double
          ]
  C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\include\random(3348): note: No user-defined-conversion operator available that can perform this conversion, or the operator cannot be called
  C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\include\cmath(674): note: or       'double sqrt(_Ty) noexcept'
  C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\include\random(3348): note: 'double sqrt(_Ty) noexcept': could not deduce template argument for '__formal'
  C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\include\cstddef(36): note: 'std::enable_if_t<false,int>' : Failed to specialize alias template
  C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\include\random(3348): note: while trying to match the argument list '(xad::UnaryExpr<Scalar,xad::scalar_sub2_op<Scalar,int>,xad::UnaryExpr<Scalar,xad::scalar_prod_op<Scalar,QuantLib::Integer>,xad::ADVar<Scalar>>>)'
          with
          [
              Scalar=double
          ]
  C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\include\random(3348): note: the template instantiation context (the oldest one first) is
  D:\QuantLib\ql/experimental/catbonds/catrisk.hpp(110): note: see reference to class template instantiation 'std::gamma_distribution<QuantLib::Real>' being compiled
  C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\include\random(3311): note: while compiling class 'std::gamma_distribution<QuantLib::Real>::param_type'
  C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\include\random(3342): note: while compiling class template member function 'void std::gamma_distribution<QuantLib::Real>::param_type::_Init(_Ty,_Ty) noexcept'
          with
          [
              _Ty=QuantLib::Real
          ]
  C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\include\random(3319): note: see the first reference to 'std::gamma_distribution<QuantLib::Real>::param_type::_Init' in 'std::gamma_distribution<QuantLib::Real>::param_type::param_type'
  C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\include\random(3362): note: see the first reference to 'std::gamma_distribution<QuantLib::Real>::param_type::param_type' in 'std::gamma_distribution<QuantLib::Real>::gamma_distribution'
  D:\QuantLib\ql\experimental\catbonds\catrisk.cpp(84): note: see the first reference to 'std::gamma_distribution<QuantLib::Real>::gamma_distribution' in 'QuantLib::BetaRiskSimulation::BetaRiskSimulation'

[auto-differentiation-dev]

It looks like this comes from the line 3348 in the standard <random> header.

Probably the best way to tackle this problem is to add a test using the same above line to the XAD unit test suite and experiment if the problem can be reproduced there - and then fixed. There might be some implicit conversion rules or overload resolution rules from C++20 introduced that get into the way, so that XAD's overload of sqrt is not found.

Feel free to give it a try.

[xwzhu]

Hi thanks for the reply. I tried to reproduce with a smaller example but failed.

#include <random>
#include <XAD/XAD.hpp>

int main()
{
    std::gamma_distribution<xad::AReal<double>> gammaAlpha;
}

This is the minimal example which I tried to reproduce the error however it stumbles on a static assert at the begining of std::gamma_distribution, which seems has nothing to do with cpp 20, as I failed with other CMAKE_CXX_STANDARD version too.

_EXPORT_STD template <class _Ty = double>
class gamma_distribution { // gamma distribution
public:
    _RNG_REQUIRE_REALTYPE(gamma_distribution, _Ty);

    using result_type = _Ty;
    ...
}

which expands to

#define _RNG_REQUIRE_REALTYPE(_RandType, _CheckedType)                    \
    static_assert(_Is_any_of_v<_CheckedType, float, double, long double>, \
        "invalid template argument for " #_RandType                       \
        ": N4950 [rand.req.genl]/1.4 requires one of float, double, or long double")

However I am unsure why QL version builds just fine with cpp version under 20. As I can see in ql\experimental\catbonds\catrisk.hpp, the member variables are just declared as I do ...

        std::exponential_distribution<Real> exponential_;
        std::gamma_distribution<Real> gammaAlpha_;
        std::gamma_distribution<Real> gammaBeta_;

Would you be able to shed some light on this please?

[xwzhu]

Ok after adding the std compatibility header I managed to reproduce the compilation error without QL involved:

#include <XAD/StdCompatibility.hpp>
#include <XAD/XAD.hpp>
#include <random>

int main()
{
    std::gamma_distribution<xad::AReal<double>> gammaAlpha;
}

However with this minimal example, it always fails to build regardless of the cpp version I set to compiler. I am very confused now why the QL version ever works ...

[raneamri]

Hello @xwzhu.

I tried recreating this with no success. Could you provide the compilation error? We'll use that as a starting point.

[raneamri]

I believe I found the issue in the <numeric> library in the std::inner_product function:

template <class _InputIterator1, class _InputIterator2, class _Tp, class _BinaryOperation1, class _BinaryOperation2>
_LIBCPP_INLINE_VISIBILITY _LIBCPP_CONSTEXPR_SINCE_CXX20
_Tp
inner_product(_InputIterator1 __first1, _InputIterator1 __last1, _InputIterator2 __first2,
              _Tp __init, _BinaryOperation1 __binary_op1, _BinaryOperation2 __binary_op2)
{
    for (; __first1 != __last1; ++__first1, (void) ++__first2)
#if _LIBCPP_STD_VER >= 20
        __init = __binary_op1(_VSTD::move(__init), __binary_op2(*__first1, *__first2));
#else
        __init = __binary_op1(__init, __binary_op2(*__first1, *__first2));
#endif
    return __init;
}

The move function throws an error when __init doesn't match the type the iterators are pointing to, which is the case when called with XAD variables since __init is called as 0. at QuantLib/ql/experimental/credit/spotlosslatentmodel.hpp:238:16. The correct fix would be to replace 0. with Real() but since we can't change that, we'll instead overload the inner_product function to check if types match and receive __init as the correct type.

[auto-differentiation-dev]

Hi @raneamri - thanks for finding the reasons of this issue.

We in fact had similar issues before (which showed without changing the C++ standard), and we solved them by submitting a PR to QuantLib. Luigi is usually happy to accept these PRs to keep AAD compatibility.

Here is an example one which included many fixes for inner_product calls, and we did many smaller PRs afterward.

Would you mind preparing that PR to QuantLib? Just make sure you do a sanity check for similar calls to inner_product and that the build with C++20 succeeds with that.

[raneamri]

Yes of course. I'll fork QuantLib and prepare a PR. I'll link it here once it's ready for you in case you'd like to give it a look.

[xwzhu]

Hello @xwzhu.

I tried recreating this with no success. Could you provide the compilation error? We'll use that as a starting point.

Hi @raneamri, mine was with MSVC compiler on Windows, and looks like yours is not? I have't tried mine on Linux yet, but seems on linux my case built fine?

[raneamri]

Correct, the issue I fixed came up on MacOS Sonoma 14.4.1 (Apple Silicon) with CXX 20.

I believe I've diagnosed your issue and I'm working on a fix. With the nature of this repo and the fact I'm not working on a Windows machine it might take a bit longer, so we appreciate your patience. 😄

[raneamri]

I can also confirm this issue is AAD related. The issue doesn't occur when AAD is toggled off according to CI/CD.

[xwzhu]

Correct, the issue I fixed came up on MacOS Sonoma 14.4.1 (Apple Silicon) with CXX 20.

I believe I've diagnosed your issue and I'm working on a fix. With the nature of this repo and the fact I'm not working on a Windows machine it might take a bit longer, so we appreciate your patience. 😄

Great, cheers!