search

deephaven / deephaven-core

Deephaven Community Core
Other
254 stars 80 forks source link

Research using Arrow more fundamentally as in-memory data structure #1764

Open pete-petey opened 2 years ago

pete-petey commented 2 years ago

Research using Arrow value vectors (arrays) as the backing store for chunks and in-memory column sources.

Allows us to exchange data in a flat way between processes without any marshaling (serialization - deserialization).

pete-petey commented 2 years ago

Devin made the comment about using Arrow for memory-maps. The conversation migrated into sidebars about whether this would be relevant for persistence or only for other use cases -- yada yada.