blsInit exception thrown when `BLS_ETH` is set in C#

[colibrishin]

I was planning to use this bls in C# with the binding you kindly made. If I set the SET_ETH, C# binding throws blsInit in Init().

Compiled with CMake (+ VS 2019) with SET_ETH=1 in windows x64 and uses a binding with C#, any other options stay the same.

Build started...
1>------ Build started: Project: ZERO_CHECK, Configuration: RelWithDebInfo x64 ------
1>Checking Build System
2>------ Build started: Project: gen_bint-x64-win.obj, Configuration: RelWithDebInfo x64 ------
2>Generating bint-x64-win.obj
2>Microsoft (R) Macro Assembler (x64) Version 14.29.30038.1
2>Copyright (C) Microsoft Corporation.  All rights reserved.
2>
2> Assembling: D:/.../bls/mcl/src/asm/bint-x64-win.asm
2>Building Custom Rule D:/.../bls/mcl/CMakeLists.txt
3>------ Build started: Project: mcl, Configuration: RelWithDebInfo x64 ------
4>------ Build started: Project: mcl_st, Configuration: RelWithDebInfo x64 ------
3>Building Custom Rule D:/.../bls/mcl/CMakeLists.txt
4>Building Custom Rule D:/.../bls/mcl/CMakeLists.txt
3>fp.cpp
4>fp.cpp
3>C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\include\ostream(375,1): warning C4530: C++ exception handler used, but unwind semantics are not enabled. Specify /EHsc
3>C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\include\ostream(368): message : while compiling class template member function 'std::basic_ostream<char,std::char_traits<char>> &std::basic_ostream<char,std::char_traits<char>>::operator <<(unsigned __int64)'
3>D:\...\mcl\include\cybozu/exception.hpp(176): message : see reference to function template instantiation 'std::basic_ostream<char,std::char_traits<char>> &std::basic_ostream<char,std::char_traits<char>>::operator <<(unsigned __int64)' being compiled
3>C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\include\ostream(641): message : see reference to class template instantiation 'std::basic_ostream<char,std::char_traits<char>>' being compiled
4>C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\include\ostream(375,1): warning C4530: C++ exception handler used, but unwind semantics are not enabled. Specify /EHsc
4>C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\include\ostream(368): message : while compiling class template member function 'std::basic_ostream<char,std::char_traits<char>> &std::basic_ostream<char,std::char_traits<char>>::operator <<(unsigned __int64)'
4>D:\...\mcl\include\cybozu/exception.hpp(176): message : see reference to function template instantiation 'std::basic_ostream<char,std::char_traits<char>> &std::basic_ostream<char,std::char_traits<char>>::operator <<(unsigned __int64)' being compiled
4>C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\include\ostream(641): message : see reference to class template instantiation 'std::basic_ostream<char,std::char_traits<char>>' being compiled
4>mcl_st.vcxproj -> D:\...\build\lib\RelWithDebInfo\mcl.lib
3>mcl.vcxproj -> D:\...\build\bin\RelWithDebInfo\mcl.dll
4>Done building project "mcl_st.vcxproj".
3>Done building project "mcl.vcxproj".
5>------ Build started: Project: mclbn256, Configuration: RelWithDebInfo x64 ------
6>------ Build started: Project: mclbn384, Configuration: RelWithDebInfo x64 ------
7>------ Build started: Project: mclbn384_256, Configuration: RelWithDebInfo x64 ------
5>Building Custom Rule D:/.../bls/mcl/CMakeLists.txt
6>Building Custom Rule D:/.../bls/mcl/CMakeLists.txt
6>bn_c384.cpp
5>bn_c256.cpp
7>Building Custom Rule D:/.../bls/mcl/CMakeLists.txt
7>bn_c384_256.cpp
6>C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\include\ostream(375,1): warning C4530: C++ exception handler used, but unwind semantics are not enabled. Specify /EHsc
6>C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\include\ostream(368): message : while compiling class template member function 'std::basic_ostream<char,std::char_traits<char>> &std::basic_ostream<char,std::char_traits<char>>::operator <<(unsigned __int64)'
6>D:\...\mcl\include\cybozu/exception.hpp(176): message : see reference to function template instantiation 'std::basic_ostream<char,std::char_traits<char>> &std::basic_ostream<char,std::char_traits<char>>::operator <<(unsigned __int64)' being compiled
6>C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\include\ostream(641): message : see reference to class template instantiation 'std::basic_ostream<char,std::char_traits<char>>' being compiled
5>C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\include\ostream(375,1): warning C4530: C++ exception handler used, but unwind semantics are not enabled. Specify /EHsc
5>C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\include\ostream(368): message : while compiling class template member function 'std::basic_ostream<char,std::char_traits<char>> &std::basic_ostream<char,std::char_traits<char>>::operator <<(unsigned __int64)'
5>D:\...\mcl\include\cybozu/exception.hpp(176): message : see reference to function template instantiation 'std::basic_ostream<char,std::char_traits<char>> &std::basic_ostream<char,std::char_traits<char>>::operator <<(unsigned __int64)' being compiled
5>C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\include\ostream(641): message : see reference to class template instantiation 'std::basic_ostream<char,std::char_traits<char>>' being compiled
7>C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\include\ostream(375,1): warning C4530: C++ exception handler used, but unwind semantics are not enabled. Specify /EHsc
7>C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\include\ostream(368): message : while compiling class template member function 'std::basic_ostream<char,std::char_traits<char>> &std::basic_ostream<char,std::char_traits<char>>::operator <<(unsigned __int64)'
7>D:\...\mcl\include\cybozu/exception.hpp(176): message : see reference to function template instantiation 'std::basic_ostream<char,std::char_traits<char>> &std::basic_ostream<char,std::char_traits<char>>::operator <<(unsigned __int64)' being compiled
7>C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\include\ostream(641): message : see reference to class template instantiation 'std::basic_ostream<char,std::char_traits<char>>' being compiled
5>   Creating library D:/.../bls/build/lib/RelWithDebInfo/mclbn256.lib and object D:/.../bls/build/lib/RelWithDebInfo/mclbn256.exp
6>   Creating library D:/.../bls/build/lib/RelWithDebInfo/mclbn384.lib and object D:/.../bls/build/lib/RelWithDebInfo/mclbn384.exp
5>mclbn256.vcxproj -> D:\...\build\bin\RelWithDebInfo\mclbn256.dll
6>mclbn384.vcxproj -> D:\...\build\bin\RelWithDebInfo\mclbn384.dll
5>Done building project "mclbn256.vcxproj".
6>Done building project "mclbn384.vcxproj".
8>------ Build started: Project: bls256, Configuration: RelWithDebInfo x64 ------
9>------ Build started: Project: bls384, Configuration: RelWithDebInfo x64 ------
9>Building Custom Rule D:/.../bls/CMakeLists.txt
8>Building Custom Rule D:/.../bls/CMakeLists.txt
9>bls_c384.cpp
8>bls_c256.cpp
9>D:\...\include\bls/bls.h(9,1): warning C4005: 'MCLBN_NO_AUTOLINK': macro redefinition
9>D:\...\src\bls_c384.cpp : message : see previous definition of 'MCLBN_NO_AUTOLINK'
8>D:\...\include\bls/bls.h(9,1): warning C4005: 'MCLBN_NO_AUTOLINK': macro redefinition
8>D:\...\src\bls_c256.cpp : message : see previous definition of 'MCLBN_NO_AUTOLINK'
7>   Creating library D:/.../bls/build/lib/RelWithDebInfo/mclbn384_256.lib and object D:/.../bls/build/lib/RelWithDebInfo/mclbn384_256.exp
7>mclbn384_256.vcxproj -> D:\...\build\bin\RelWithDebInfo\mclbn384_256.dll
7>Done building project "mclbn384_256.vcxproj".
10>------ Build started: Project: bls384_256, Configuration: RelWithDebInfo x64 ------
9>C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\include\ostream(375,1): warning C4530: C++ exception handler used, but unwind semantics are not enabled. Specify /EHsc
9>C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\include\ostream(368): message : while compiling class template member function 'std::basic_ostream<char,std::char_traits<char>> &std::basic_ostream<char,std::char_traits<char>>::operator <<(unsigned __int64)'
9>D:\...\mcl\include\cybozu/exception.hpp(176): message : see reference to function template instantiation 'std::basic_ostream<char,std::char_traits<char>> &std::basic_ostream<char,std::char_traits<char>>::operator <<(unsigned __int64)' being compiled
9>C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\include\ostream(641): message : see reference to class template instantiation 'std::basic_ostream<char,std::char_traits<char>>' being compiled
10>Building Custom Rule D:/.../bls/CMakeLists.txt
8>C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\include\ostream(375,1): warning C4530: C++ exception handler used, but unwind semantics are not enabled. Specify /EHsc
8>C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\include\ostream(368): message : while compiling class template member function 'std::basic_ostream<char,std::char_traits<char>> &std::basic_ostream<char,std::char_traits<char>>::operator <<(unsigned __int64)'
8>D:\...\mcl\include\cybozu/exception.hpp(176): message : see reference to function template instantiation 'std::basic_ostream<char,std::char_traits<char>> &std::basic_ostream<char,std::char_traits<char>>::operator <<(unsigned __int64)' being compiled
8>C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\include\ostream(641): message : see reference to class template instantiation 'std::basic_ostream<char,std::char_traits<char>>' being compiled
10>bls_c384_256.cpp
10>D:\...\include\bls/bls.h(9,1): warning C4005: 'MCLBN_NO_AUTOLINK': macro redefinition
10>D:\...\src\bls_c384_256.cpp : message : see previous definition of 'MCLBN_NO_AUTOLINK'
9>   Creating library D:/.../bls/build/lib/RelWithDebInfo/bls384.lib and object D:/.../bls/build/lib/RelWithDebInfo/bls384.exp
10>C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\include\ostream(375,1): warning C4530: C++ exception handler used, but unwind semantics are not enabled. Specify /EHsc
10>C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\include\ostream(368): message : while compiling class template member function 'std::basic_ostream<char,std::char_traits<char>> &std::basic_ostream<char,std::char_traits<char>>::operator <<(unsigned __int64)'
10>D:\...\mcl\include\cybozu/exception.hpp(176): message : see reference to function template instantiation 'std::basic_ostream<char,std::char_traits<char>> &std::basic_ostream<char,std::char_traits<char>>::operator <<(unsigned __int64)' being compiled
10>C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\include\ostream(641): message : see reference to class template instantiation 'std::basic_ostream<char,std::char_traits<char>>' being compiled
8>   Creating library D:/.../bls/build/lib/RelWithDebInfo/bls256.lib and object D:/.../bls/build/lib/RelWithDebInfo/bls256.exp
9>bls384.vcxproj -> D:\...\build\bin\RelWithDebInfo\bls384.dll
9>Done building project "bls384.vcxproj".
8>bls256.vcxproj -> D:\...\build\bin\RelWithDebInfo\bls256.dll
8>Done building project "bls256.vcxproj".
10>   Creating library D:/.../bls/build/lib/RelWithDebInfo/bls384_256.lib and object D:/.../bls/build/lib/RelWithDebInfo/bls384_256.exp
10>bls384_256.vcxproj -> D:\...\build\bin\RelWithDebInfo\bls384_256.dll
10>Done building project "bls384_256.vcxproj".
11>------ Build started: Project: ALL_BUILD, Configuration: RelWithDebInfo x64 ------
11>Building Custom Rule D:/.../bls/CMakeLists.txt
========== Build: 11 succeeded, 0 failed, 0 up-to-date, 0 skipped ==========

This is what returns when I run the bls.exe (C# binding) with previously built dll from CMake, and I tried to Init() with BLS12_381 and it also fails.

curveType=0
ERR=System.ArgumentException: blsInit
   Location: mcl.BLS.Init(Int32 curveType) File ...bls.cs:Line 137
   Location: mcl.BLSTest.Main(String[] args) File ...:Line 241

Some digging, I guess it seems to be a MCLBN_COMPILED_TIME_VAR related error. the err returns -246046.

https://github.com/herumi/bls/blob/4a99be9f98669a59ef930765ddc546845c42dae7/ffi/cs/bls.cs#L15

https://github.com/herumi/bls/blob/4a99be9f98669a59ef930765ddc546845c42dae7/include/bls/bls.h#L12-L19

[colibrishin]

After changing COMPILED_TIME_VAR, MemoryAccessViolation is thrown. Because public key and signature array sizes did not swap in bls binding. Had to change the const declaration and after than it worked.

[herumi]

Sorry for the late response. I've added some codes such as bls_eth.cs for the DLL with BLS_ETH. https://github.com/herumi/bls/commit/4feb8e14e577496d59855fc05fbccb3cb1017f0f

[herumi]

I want to use

private fixed ulong v[SECRETKEY_UNIT_SIZE];

, then SECRETKEY_UNIT_SIZE must be constant, so I think it is difficult to change those values in a constructor. Do you have any good idea?

[colibrishin]

Thanks for your response. Sorry if I bothered you by closing the issue by myself.

No, I don't have any better idea. That's why I also go for hard-code. :( I'll give it a try with your dev branch.

And, one more thing. In C# binding, the ethereum's bls12_381 tests were failing. I don't have a clear reason why-, but changing string to byte[] solves the problem. It seems taking a message as a string causes some difference during conversion (UTF8, ASCII or endianess?). https://github.com/colibrishin/bls/commit/da82c1f787d912c40afbb9a0ec0b1de352bc1ea6

[colibrishin]

The dev branch works well on my side 👍 , but Serialize() returns an unnecessarily large array of bytes 288 (e.g.. Signature.Serialize()) by SIGNATURE_SERIALIZE_SIZE = SIGNATURE_UNIT_SIZE * 8.

[herumi]

Thank you for the report. I've fixed it at https://github.com/herumi/bls/commit/6b0f6fe84968936559eee39d5ac2ce77bc9d53a1 .

[herumi]

I don't have a clear reason why-, but changing string to byte[] solves the problem.

Thank you for the advice. It's old apis. I supported it at https://github.com/herumi/bls/commit/3af40c71b42166b42c3e8737e5ce85fdbaf0f26f .

[colibrishin]

Thanks! Cross-validation with https://github.com/ethereum/bls12-381-tests is stuck at AggreateVerify. I'll investigate further for debug.

In Progress...

• FastAggreateVerify is also failing however, If i mimic the behavior in C# (Aggregate Public Keys and call Verify()) works fine.

These are the keys, message and signature used for test.

input:
  pubkeys: ['0xa491d1b0ecd9bb917989f0e74f0dea0422eac4a873e5e2644f368dffb9a6e20fd6e10c1b77654d067c0618f6e5a7f79a',
    '0xb301803f8b5ac4a1133581fc676dfedc60d891dd5fa99028805e5ea5b08d3491af75d0707adab3b70c6a6a580217bf81',
    '0xb53d21a4cfd562c469cc81514d4ce5a6b577d8403d32a394dc265dd190b47fa9f829fdd7963afdf972e5e77854051f6f']
  message: '0xabababababababababababababababababababababababababababababababab'
  signature: '0x9712c3edd73a209c742b8250759db12549b3eaf43b5ca61376d9f30e2747dbcf842d8b2ac0901d2a093713e20284a7670fcf6954e9ab93de991bb9b313e664785a075fc285806fa5224c82bde146561b446ccfc706a64b8579513cfc4ff1d930'
output: true

• AggregateVerify is bit complicated for me to mimic the behavior in C#. I tried to invoking with custom marshaler, and vectorized byte array, still fails.

  
  input:
  pubkeys: ['0xa491d1b0ecd9bb917989f0e74f0dea0422eac4a873e5e2644f368dffb9a6e20fd6e10c1b77654d067c0618f6e5a7f79a',
  '0xb301803f8b5ac4a1133581fc676dfedc60d891dd5fa99028805e5ea5b08d3491af75d0707adab3b70c6a6a580217bf81',
  '0xb53d21a4cfd562c469cc81514d4ce5a6b577d8403d32a394dc265dd190b47fa9f829fdd7963afdf972e5e77854051f6f']
  messages: ['0x0000000000000000000000000000000000000000000000000000000000000000',
  '0x5656565656565656565656565656565656565656565656565656565656565656', '0xabababababababababababababababababababababababababababababababab']
  signature: '0x9104e74b9dfd3ad502f25d6a5ef57db0ed7d9a0e00f3500586d8ce44231212542fcfaf87840539b398bf07626705cf1105d246ca1062c6c2e1a53029a0f790ed5e3cb1f52f8234dc5144c45fc847c0cd37a92d68e7c5ba7c648a8a339f171244'
  output: true

- Both of them use `PublicKey` struct array.

[herumi]

Okay, I'll check it later. Please wait for a while because I have a lot of work to do.

[herumi]

I added FastAggregateVerify and tests at https://github.com/herumi/bls/commit/0eacdc5788c5fd79a19d6247b40c853041221a9d . It seems okay.

[colibrishin]

Correct me if i'm wrong. What I have understood, FastAggregateVerify uses all public keys with aggregation. From the bls_c_impl.hpp, the definition of blsFastAggregateVerify is taking a vector of publickey (or array of publicKey), but on the C# binding, the function defined use only one.

https://github.com/herumi/bls/blob/4a99be9f98669a59ef930765ddc546845c42dae7/src/bls_c_impl.hpp#L428-L436

https://github.com/herumi/bls/blob/467dec02ecc89c05232953b1d0d20c3250fd87e6/ffi/cs/bls.cs#L466-L471

Passing the whole array fails the test on my side. Could you check whether it successes?

[herumi]

The TestFastAggregate runs well on my environments. Does not it run on yours?

but on the C# binding, the function defined use only one.

in pubVec[0] of blsFastAggregateVerify means to pass the head pointer of pubVec and blsFastAggregateVerify checks all of the array. For example, if I comment out one element of pubVec (https://github.com/herumi/bls/blob/dev/ffi/cs/bls_test.cs#L281), then the test fails.

[colibrishin]

Oh, Now I get it. Yes, it works on my machine too. passing the first address of array. I thought it would not be passed as pointer but only an array of index #0. Learned new thing.

Interesting to see that passing just array name with ref keyword is not working. 🤔

I have never thought about commenting out inputs. I'll try on AggregateVerify and see if it's work.

[colibrishin]

I've tried to pass the messages array with msg[0], and it seems like needs to be passed in another way. Do you have any clue about this..? https://github.com/colibrishin/bls/commit/42e6c38207c34e71c36e4967fc06f5e0d5903604

[herumi]

MSG_SIZE is 32 on Ethereum and byte[][] does not mean C array. I've added AggregateVerify to the dev branch. Please see it. https://github.com/herumi/bls/tree/dev

[colibrishin]

Thanks for your informative reply! I might have to hash the message before signing.

• P.S : I have added a semaphore in code due to sharing local/static variables causing data corruption. What is your aspect of doing so (e.g.. Maybe like, static variable g_Q should not be shared between other keys)?

[herumi]

I have added a semaphore in code due to sharing local/static variables causing data corruption. What is your aspect of doing so (e.g.. Maybe like, static variable g_Q should not be shared between other keys)?

All functions of mcl/bls except for init and some global configuration functions are thread-safe. So we can use those functions in multi-thread after calling init(). g_Q is constant after initialization.

[colibrishin]

If there are multiple instance of BLS, that saying multiple initialization is called, data shows different result. I'll prove with code later on. (Or, It might should track if it is already initialized once, not applying a semaphore for methods.)

[herumi]

If there are multiple instance of BLS, that saying multiple initialization is called, data shows different result.

If you do it, then the result will happen because blsInit is not thread-safe. https://github.com/herumi/bls/blob/master/include/bls/bls.h#L84

Why do you make multiple instances of BLS?

[colibrishin]

Sorry if it sounds aggressive but I'm not intended in such. My english skill is not good enough.

If I could, I wouldn't. As far as I know, the shared library created only once and requests are processed by one the loaded. Some classes using BLS (e.g., BLSPrivateKey or BLSPublicKey) will hold BLS library invoker for its own and it might be run in parallel.

For example, verifying block transactions for already in block chain and newly incoming block. Each cases will create two PublicKey and BLS library and they don't know whether library is initialized. (Fortunately, not thread-safeness has caught early in unit test. I wasn't aware of it.)

I think, If one doesn't create a PublicKey instance for required situation, then the BLS library will be passed and placed everywhere. That's why i wanted to do it such way. I would like to hear any suggestion.

(P.S. I added MultiVerify in cs binding https://github.com/colibrishin/bls/commit/4a992583e96f5bbd1446e091efbda998261e9649 and it seems work.) (P.S.2. I might have found the solution. I'll notify after double-check if it is working.)

[herumi]

Okay, I understand what you want. How about https://github.com/herumi/bls/commit/4e7800ce284f4367f98f35f604d92752dfb7e829 ?

[colibrishin]

Thanks! That's what i was going to do and wanted.

[colibrishin]

https://github.com/planetarium/bls-herumi/tree/master/ffi/cs

After some struggle, have done some modifications in C# binding for cross-platform. Would it be okay to publish in a NuGet package? and could you help me with what Id is used for, and how the Share/Recover[Type]() methods work?

[colibrishin]

I'll close this issue and create another due to the blsInit problem has been solved.