morganstanley / hobbes

A language and an embedded JIT compiler
http://hobbes.readthedocs.io/
Apache License 2.0
1.16k stars 105 forks source link

Failed to compress slog file with enum type #398

Open dawa79 opened 3 years ago

dawa79 commented 3 years ago

Here is the test case, you can see there is no issue for raw storage, but will generate core dump(not always but very offen) file for compressd file :

`

include

include <hobbes/hobbes.H>

include <hobbes/storage.H>

include <hobbes/db/file.H>

include <hobbes/db/series.H>

include <hobbes/lang/tylift.H>

const int option = 100; struct TestEnum { enum Enum : uint32_t{ Invalid = 0, Option = option }; Enum _value; using is_enum = void; };

namespace hobbes { template struct liftTestEnum { static MonoTypePtr type(typedb&) { Variant::Members vms; auto i = 0; vms.push_back(Variant::Member("Invalid", primty("unit"), 0)); vms.push_back(Variant::Member("Option", primty("unit"), option)); return MonoTypePtr(Variant::make(vms)); } }; template struct lift<T*, false, typename T::is_enum> : public liftTestEnum {}; template struct lift<T, true, typename T::is_enum> : public liftTestEnum {}; }

DEFINE_STRUCT( FixarrayS, (TestEnum, value), (const hobbes::array*, body) );

void run(const std::string& fname, bool compressed) { hobbes::cc c; FixarrayS v1; v1.body = hobbes::makeString("Hello world!"); v1.value._value = TestEnum::Option;

hobbes::writer writer{fname}; hobbes::series ss1(&c, &writer, "udata", 10000, compressed? hobbes::StoredSeries::Compressed : hobbes::StoredSeries::Raw);
ss1(v1);
std::cout << "------------testing -----------------" << std::endl; c.define("f", "inputFile :: (LoadFile \"" + fname + "\" w) => w"); c.compileFn<void()>("print([(x.value, (unsafeCast(x.value)::{t:int}).t) | x<-f.udata])")(); }

int main() { std::cout << "=============Create Raw Stream [OK]=============" << std::endl; run("/var/tmp/test.log", false); std::cout << "=============Create Compressed Stream [Failed]=============" << std::endl; run("/var/tmp/testc.log", true); return 0; } `

dawa79 commented 3 years ago

it seems need change here: https://github.com/morganstanley/hobbes/blob/90c9865cfbfb3c4c08437dcd27bd9a090028ff99/lib/hobbes/db/cbindings.C#L17

right now hobbes only support the enum without customized values

dawa79 commented 3 years ago

https://github.com/morganstanley/hobbes/blob/main/include/hobbes/cfregion.H#L850

smunix commented 3 years ago

Your enum here is an Int in disguise, not a variant. With the changes made here, you can keep your type as enums while keeping the static information as well.

Is this something you can do?

hi : an interactive shell for hobbes
      type ':h' for help on commands

> db=inputFile :: (Argument "file" p, LoadFile p _) => _
> :t db.udata
(cseq { test:(test |invalid:(Test 0L), option:(Test 100L)|), string:[char] } ({ pmodels:[:{ freqs:[:int|256L:], activeFreqs:[:int|256L:], c:int }|4L:] } * ({ pmodels:[:{ freqs:[:int|256L:], activeFreqs:[:int|256L:], c:int }|8L:] } * { pmodels:[:{ freqs:[:int|256L:], activeFreqs:[:int|256L:], c:int }|1L:] })) 10000L)
dawa79 commented 3 years ago

seem use fregion is an options. will have a try