SunCC and Error "Overloading ambiguity between std::ceil(double) and std::ceil(float)"

[noloader]

Sorry about this report. I thought it would be quicker to report it with a patch/diff. I was also concerned about the type used in the static_cast for XMSS.

Sun Studio 12.5 and its SunCC reports the following error:

/opt/developerstudio12.5/bin/CC  -m64 -std=c++11 -KPIC +p -features=extensions -D__FUNCTION__=__func__ -xO2 +w -erroff=truncwarn,wnoretvalue -Ibuild/include -Ibuild/include/external -c ./src/lib/pubkey/xmss/xmss_wots_parameters.cpp -o build/obj/lib/pubkey_xmss_wots_parameters.o
"./src/lib/pubkey/xmss/xmss_wots_parameters.cpp", line 83: Error: Overloading ambiguity between "std::ceil(double)" and "std::ceil(float)".
1 Error(s) detected.

...

/opt/developerstudio12.5/bin/CC  -m64 -std=c++11 -KPIC +p -features=extensions -D__FUNCTION__=__func__ -xO2 +w -erroff=truncwarn,wnoretvalue -Ibuild/include -Ibuild/include/external -c ./src/lib/tls/tls_extensions.cpp -o build/obj/lib/tls_extensions.o
"./src/lib/tls/tls_extensions.cpp", line 410: Error: Overloading ambiguity between "built-in operator==(int, float)" and "built-in operator==(int, double)".
"./src/lib/tls/tls_extensions.cpp", line 416: Error: Overloading ambiguity between "built-in operator==(int, float)" and "built-in operator==(int, double)".
2 Error(s) detected.
*** Error code 2

[noloader]

This issue appears to be another case of SunCC being strict with the C++ standard: Overloading ambiguity between "std::pow(double, int)" and "std::pow(long double, int)". SunCC also provides all 3 overloads, while compilers like MSVC does not.

The fix looks fairly trivial on the surface - select either a double or a float:

$ git diff src/lib/pubkey/xmss/xmss_wots_parameters.cpp
diff --git a/src/lib/pubkey/xmss/xmss_wots_parameters.cpp b/src/lib/pubkey/xmss/xmss_wots_parameters.cpp
index 704bcab36..784cfa011 100644
--- a/src/lib/pubkey/xmss/xmss_wots_parameters.cpp
+++ b/src/lib/pubkey/xmss/xmss_wots_parameters.cpp
@@ -80,11 +80,12 @@ XMSS_WOTS_Parameters::XMSS_WOTS_Parameters(ots_algorithm_t oid)
       }

    m_w == 16 ? m_lg_w = 4 : m_lg_w = 2;
-   m_len_1 = static_cast<size_t>(ceil((8 * element_size()) / m_lg_w));
+   m_len_1 = static_cast<size_t>(
+      ceil(static_cast<float>(element_size()) * 8 / m_lg_w));
    m_len_2 = static_cast<size_t>(
-      floor(log2(m_len_1 * (wots_parameter() - 1)) / m_lg_w) + 1);
+      floor(log2(static_cast<float>(m_len_1) * (wots_parameter() - 1)) / m_lg_w) + 1);
    BOTAN_ASSERT(m_len == m_len_1 + m_len_2, "Invalid XMSS WOTS parameter "
-                "\"len\" detedted.");
+                "\"len\" detected.");
    }

 secure_vector<uint8_t>

float was selected above because it seems like if an ALU is going to be slower, then its going to be slower on a double. I'm sure there are plenty of coprocessors which make that supposition fall on its face :)

$ git diff src/lib/tls/tls_extensions.cpp
diff --git a/src/lib/tls/tls_extensions.cpp b/src/lib/tls/tls_extensions.cpp
index 7692b2fd3..dfd909c7c 100644
--- a/src/lib/tls/tls_extensions.cpp
+++ b/src/lib/tls/tls_extensions.cpp
@@ -407,13 +407,13 @@ Supported_Point_Formats::Supported_Point_Formats(TLS_Data_Reader& reader,
       {
       uint8_t format = reader.get_byte();

-      if(format == UNCOMPRESSED)
+      if(static_cast<ECPointFormat>(format) == UNCOMPRESSED)
          {
          m_prefers_compressed = false;
          reader.discard_next(len-i-1);
          return;
          }
-      else if(format == ANSIX962_COMPRESSED_PRIME)
+      else if(static_cast<ECPointFormat>(format) == ANSIX962_COMPRESSED_PRIME)
          {
          m_prefers_compressed = true;
          reader.discard_next(len-i-1);

[randombit]

Closing as stale