morganstanley / binlog

A high performance C++ log library, producing structured binary logs
http://opensource.morganstanley.com/binlog/
Apache License 2.0
608 stars 72 forks source link

not support __int128 output #163

Open stockxiay opened 1 year ago

stockxiay commented 1 year ago

Question1: template <typename T, int32t Digits> class TestFixedPoint { public: TestFixedPoint(T t) : value(t) {}

private: T value_; BINLOG_ADAPT_STRUCT_FRIEND; };

using TestPrice = TestFixedPoint<int64_t, 9>; BINLOG_ADAPTTEMPLATE((typename Interl, int Digits), (TestFixedPoint<Interl, Digits>), value) BINLOG_INFO_W(writer, "my log MyPrice: {}", myPrice); Compile time error: binlog/include/mserialize/make_templateserializable.hpp:75:84: 错误:‘long int TestFixedPoint<long int, 9>::value’ is private within this context Question2: if T == __int128 then throw "This Arithmetic type is not taggable" at tag.hpp line 79

erenon commented 1 year ago

I'm unable to reproduce the first issue:

template <typename T, int32_t Digits>
class TestFixedPoint {
public:
  TestFixedPoint(T t) : value_(t) {}

private:
  T value_;
  BINLOG_ADAPT_STRUCT_FRIEND;
};

using TestPrice = TestFixedPoint<int64_t, 9>;
BINLOG_ADAPT_TEMPLATE((typename Interl, int Digits), (TestFixedPoint<Interl, Digits>), value_)

//...

  TestPrice myPrice(1);
  BINLOG_INFO("my log MyPrice: {}", myPrice);

This works as expected.

Regarding __int128: It is not a standard type, and therefore not supported. As a workaround, you can:

chowtin commented 1 year ago

Hi Erenon, Thanks for your reply.

The origin requirement is that a custom type (similar to java.math.BigDecimal) is introduced represent arbitrary-precision signed decimal numbers, definition is as following, integerT = __int128

template<typename IntegerT, int32_t Digits>
class FixedPoint {
// custom defined operator + - * /

std::toString() const {
 ...
}

private:
IntegerT _value;
};

using ValueT = FixedPoint<__int128, 13>;

ValueT::toString() involves converting __int128 to string, which is time-consuming, binlog is a good candidate to solve this problem.

we can write FixedPoint as 3 elements per your 1st suggestion.

  1. Add two getters, that return the upper and lower halfs, and log them.
  2. Digits
template<typename IntegerT, int32_t Digits>
class FixedPoint {
// custom defined operator + - * /

std::toString() const {
 ...
}

int64_t getUper() const ;
int64_t getLower() const;
int32_t getDigits() const;

private:
IntegerT _value;
};

If we changed the code as above, binlog bread tool would take FixedPoint as normal json instead of the expected output fromFixedPoint::toString

so the question is how to customize the output of a struct? how to change bread tool code.

erenon commented 1 year ago

You can modify PrettyPrinter here: https://github.com/morganstanley/binlog/blob/main/include/binlog/PrettyPrinter.cpp#L128 to make it recognize your custom type (if (sb.name == "FixedPoint")) and format the output accordingly. There are already a few example there, that should help you get started.

chowtin commented 1 year ago

Hi Erenon, Thanks for your advice, we managed to print out FixedPoint<int128_t>. now we 're about to turnover to prod, but found the output message is cluttered with types, things can go worse if FixedPoint<int128_t> is used in a struct.

for example , bread give output _orderQtyEq: engine::datatype::FixedPoint{ getBinlogValue: 0.003 } expected is _orderQtyEq: 0.003

how to remove the type prefix , only keep the value?

erenon commented 1 year ago

did you change PrettyPrinter.cpp?

chowtin commented 1 year ago

Erenon, thanks for your support.

After binlog turnedover to prod, it reduced CPU usage to one-half and reduced the CPU spikes with ~60M memory usage increase.