capnp transcoder and support in mcap cat --json output

foxglove / community

This repo hosts our org-wide discussion board

0 stars 0 forks source link

capnp transcoder and support in mcap cat --json output #657

Closed jon-chuang closed 1 year ago

jon-chuang commented 1 year ago

Description We want to be able to decode capnp with dynamic schema in mcap-support e.g. as is done for protobuf: https://github.com/foxglove/studio/blob/3767475a507097b3e4e046fc4d8b3a20135101f8/packages/mcap-support/src/parseChannel.ts#L125-L134

As a POC, we should first implement this in python and go for the CLI. The go API does not support reflection. I am thinking of a C-ABI FFI that we can bind to cgo. WASM compilation for browser can come later...

@jtbandes RE Slack lounge discussion

foxhubber[bot] commented 1 year ago

Linear: FG-2433

jon-chuang commented 1 year ago

Looking at all the possible pathways, while it may be fun to explore WASM compilation (see e.g. https://medium.com/cyberbotics/porting-a-c-rendering-engine-to-webassembly-9c32d76c31f1) or using cgo to bind to the JSON transcoder (https://github.com/capnproto/capnproto/blob/master/c%2B%2B/src/capnp/compat/json.h), given the poor support of Capnproto in various languages, especially for browser, It does seem to make much more sense to encode the original capnp recording into a compatible format like JSON (or CBOR for compactness) at time of storage. Anything that decodes capnproto in a browser environment will not be zero-copy anyway.

defunctzombie commented 1 year ago

@jon-chuang Please me mindful when using the bug label. The bug label is for broken functionality. If you believe this issue is a bug please explain what existing functionality that we've documented is not working, provide steps to reproduce, and describe what you expect to happen.

jtbandes commented 1 year ago

I'm going to convert this to a discussion since we haven't yet settled on the spec updates to "well-known" encodings for capnp usage.