Large RAM use for large messages

[pgab]

Our application uses some bigger messages. Those messages can get close to several gigabytes. We observed a fairly high RAM use related to those messages. I've created a test case at https://github.com/pgab/grpc-crash/tree/ram_use.

It clearly demonstrates that the more than double of RAM is being used on both server and client:

/usr/bin/time --verbose ../target/release/server --config config.toml 
^CCommand terminated by signal 2
    Command being timed: "../target/release/server --config config.toml"
    User time (seconds): 8.37
    System time (seconds): 3.55
    Percent of CPU this job got: 65%
    Elapsed (wall clock) time (h:mm:ss or m:ss): 0:18.31
    Average shared text size (kbytes): 0
    Average unshared data size (kbytes): 0
    Average stack size (kbytes): 0
    Average total size (kbytes): 0
    Maximum resident set size (kbytes): 6293996
    Average resident set size (kbytes): 0
    Major (requiring I/O) page faults: 0
    Minor (reclaiming a frame) page faults: 2359595
    Voluntary context switches: 115070
    Involuntary context switches: 978
    Swaps: 0
    File system inputs: 0
    File system outputs: 0
    Socket messages sent: 0
    Socket messages received: 0
    Signals delivered: 0
    Page size (bytes): 4096
    Exit status: 0

and on the client side:

/usr/bin/time --verbose ../target/release/client-test
3072 MiB
    Command being timed: "../target/release/client-test"
    User time (seconds): 2.81
    System time (seconds): 4.02
    Percent of CPU this job got: 43%
    Elapsed (wall clock) time (h:mm:ss or m:ss): 0:15.81
    Average shared text size (kbytes): 0
    Average unshared data size (kbytes): 0
    Average stack size (kbytes): 0
    Average total size (kbytes): 0
    Maximum resident set size (kbytes): 7344300
    Average resident set size (kbytes): 0
    Major (requiring I/O) page faults: 1
    Minor (reclaiming a frame) page faults: 1835317
    Voluntary context switches: 173080
    Involuntary context switches: 259
    Swaps: 0
    File system inputs: 48
    File system outputs: 0
    Socket messages sent: 0
    Socket messages received: 0
    Signals delivered: 0
    Page size (bytes): 4096
    Exit status: 0

[stepancheg]

Thank you for the report, sometimes grpc does unnecessary message copies/allocations. That need to be dealt with.