search

Igalia / snabb

Snabb Switch: Fast open source packet processing
Apache License 2.0
47 stars 5 forks source link

[WIP] Use manual managed memory in mem.lua #1146

Closed dpino closed 5 years ago

dpino commented 6 years ago

Attempt to fix #829. The issue states is not possible to retrieve large binding tables using config get, even if using RaptorJIT.

The main issue here is that even though RaptorJIT is aimed for Intel 64-bit architecture, it still have limitations on string size inherited from LuaJIT. It's not possible to manage strings bigger than 2GB.

When pulling a 10M binding-table config using config get the output string is larger than 2GB.

In this PR I modified lib/stream/mem.lua to back up on C's runtime memory management functions, however call_with_output_string converts the buffered string to a Lua string using FFI, so backing up on C's runtime memory functions does not solve the issue.

$ sudo ./snabb lwaftr run --cpu 11 --name lwaftr --conf lwaftr.conf --on-a-stick 82:00.0
string length overflow:         2387430001
$ sudo ./snabb config get lwaftr /

2387430001 bytes is approximately 2.3GB.

dpino commented 6 years ago

call_with_output_string converts the string to a Lua string, because that's what all the data printers expect. I think either RaptorJIT is extendeded so it can manage strings bigger than 2GB or we modify the printers somehow.

wingo commented 5 years ago

It would certainly be possible for raptorjit to support longer strings. But it would be better if we never used "call_with_output_string" anywhere: i.e. never build up a big result string like that. Always stream. That would require changing the snabb-config-leader-v1 schema to represent the config as YANG 1.1 anydata (https://tools.ietf.org/html/rfc7950#section-7.10) instead of a string.

dpino commented 5 years ago

Created new issue https://github.com/Igalia/snabb/issues/1177