High memory usage issue while streaming data as per gNMI.proto in ASCII/Proto format

[akhandel-1]

What version of protobuf and what language are you using? Version: 3.12.x (Same was tried on latest versions as well) Language: C++ libprotoc - 3.15.8 gRPC stack - v1.0.0 gNMI version - 070

What operating system (Linux, Windows, ...) and version? Linux

What runtime / compiler are you using (e.g., python version or gcc version) gcc

Problem Statement High memory usage issue while streaming data as per gNMI.proto in ASCII/Proto format

Sample Yang Data model used in tree format The snippet of the data which we want to serialize is

+--rw fps | +--rw fp* [name] | +--rw name string | +--rw description? string | +--rw fd-name? -> /mef-fd:fds/fd/name | +--rw logical-port? mef-logical-port:logical-port-ref | +--rw (type)? | | +--:(q-in-q) | | | +--rw svlan? uint32 | | +--:(mpls-pw) | | | +--rw mpls-pw? empty | | +--:(uni) | | | +--rw uni? empty | | +--:(other) | | +--rw other? empty

APIs used to monitor heap usage

if (getrusage(RUSAGE_SELF, &usage) != 0) {
    perror("getrusage");
    exit(1);
}
data_size = usage.ru_maxrss;
printf("Heap utilization(after): %ld KB\n", data_size);

Standalone Code snippet

inline static void addPrefix2Notif(::gnmi::Notification *notif, ::gnmi::Path& yPrefix)
{ 
   ::gnmi::Path*           prefix  = new ::gnmi::Path(yPrefix);

   assert(prefix != NULL);
   notif->clear_prefix();

   notif->set_allocated_prefix(prefix);

   return;
}

void serializeDataIntoGetResp(::gnmi::GetResponse    *protoResp,
                                                     ::gnmi::Path&  hierarchyBox)
{
    int notif_count = 250000, i;  //Notif count taken to increase the hierarchal data
    int update_count = 4, j;    //update_count taken as 4 aprox
    ::gnmi::Notification *notif = NULL;
    uint64  epochTm = 2221;

    printf("Data to be published is bytes %ld\n", notif_count * 4 + update_count * 4);   //approx., excluding the object creation memory done internally
    hierarchyBox.add_elem()->set_name("ma");            

    for (i = 0; i < notif_count; i++) {     //Loop to create hierarchal Notififcation data(as mention in the summary)
        notif   = protoResp->add_notification();               

        notif->set_timestamp(epochTm);                         

        notif->mutable_prefix()->CopyFrom(hierarchyBox);
//      addPrefix2Notif(notif, hierarchyBox); 

        for (j = 0; j < update_count; j++) {    //Loop to create hierarchal Update data(as mention in the summary)

            ::gnmi::Update* update = notif->add_update();     
            update->mutable_path()->add_elem()->set_name("id"); 

            update->mutable_val()->set_string_val("se");      
        }
    }
    return;

The complete code is added here - https://github.com/akhandel-1/grpc-libproto/

Running valgrind to gather heap profile information

To gather heap profiling information about the program we can run valgrind using valgrind --tool=massif ./proto_heap. Upon completion, no summary statistics are printed to Valgrind's commentary; all of Massif's profiling data is written to a file. By default, this file is called massif.out., where is the process ID.

To see the information gathered by Massif in an easy-to-read form, use ms_print. If the output file's name is massif.out., typems_print massif.out.

To check the complete valgrind output, the details can be found here - https://github.com/akhandel-1/grpc-libproto/blob/main/massif.out.23803

Detailed analysis

After running the above changes, the total memory allocated was captured, details of which are mentioned below -

<style> </style>

 

Data Size to be published | Heap Utilization (Before serialization) | Heap Utilization (After serialization) | Heap Utilization Difference | getResp byte size( After serialization) -- | -- | -- | -- | -- 10 KB | 12.81 MB | 23.28 MB | 10.47 MB | 300KB 1 MB | 12.65 MB | 836.26 MB | 823.61 MB | 30MB 1 KB | 12.62 MB | 15.77 MB | 3.15 MB | 30KB 100 KB | 12.64 MB | 97.16 MB | 84.52 MB | 3MB

What did you expect to see We don't expect this much memory usage of serializing small response.

What did you see instead? a) 1 KB Data: When we pushed 1 KB data, the byte size increased to 30.00 KB. This indicates that the operation caused additional memory consumption of 28.98 KB. b) 10 KB Data: For 10 KB data, the byte size expanded to 300.00 KB, resulting in a memory consumption increase of 290.00 KB. c) 100 KB Data: With 100 KB data, the byte size grew to 3.00 MB, signifying an additional memory consumption of 2.90 MB. d) 1 MB Data: Upon pushing 1 MB data, the byte size escalated to 30.00 MB, representing a memory consumption increase of 29.00 MB.

The data size published to getRresp byte size ratio is 1:30. This will be increased more if we are publishing a bigger notification string or bigger update level data.

Valgrind analysis During the execution of the "add_elem() and add_update()" The add_update() API is used to add a new update message to a notification message, while the add_elem() API appends a new element to a repeated field in a message. The memory consumption graph generated by Valgrind's massif tool provides insights into the memory usage patterns, highlighting the significant impact of this API on memory consumption during the execution of the program. - massif.out.23803

81.42% (653,113,774B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
->19.95% (160,000,128B) 0x411599: CreateMessageInternalgnmi::PathElem (arena.h:501)
| ->19.95% (160,000,128B) 0x411599: gnmi::PathElem* google::protobuf::Arena::CreateMaybeMessagegnmi::PathElem(google::protobuf::Arena*) (gNMI.pb.cc:8684)
| ->15.96% (128,000,000B) 0x404A2A: New (repeated_field.h:802)
| | ->15.96% (128,000,000B) 0x404A2A: NewFromPrototype (repeated_field.h:832)
| | ->15.96% (128,000,000B) 0x404A2A: Add<google::protobuf::RepeatedPtrFieldgnmi::PathElem::TypeHandler> (repeated_field.h:1738)
| | ->15.96% (128,000,000B) 0x404A2A: Add (repeated_field.h:2181)
| | ->15.96% (128,000,000B) 0x404A2A: _internal_add_elem (gNMI.pb.h:7164)
| | ->15.96% (128,000,000B) 0x404A2A: add_elem (gNMI.pb.h:7168)
| | ->15.96% (128,000,000B) 0x404A2A: serializeDataIntoGetResp (proto_gnmi.c:48)
| | ->15.96% (128,000,000B) 0x404556: main (main.c:23)

Make sure you include information that can help us debug (full error message, exception listing, stack trace, logs). The code is added here - https://github.com/akhandel-1/grpc-libproto/

[acozzette]

I suspect this is probably working as intended. Protos can take up much more space in their in-memory format than they do in the wire format, mostly because unused fields still take up space in the memory layout.

If you're using arenas then that could be another reason for the high memory usage. An arena's memory is not freed until it is destroyed, and so for example as you grow the notification repeated field, whenever the array length is doubled the old array will not be freed immediately. You might get better results by reserving the full length of the repeated field in advance with something like protoResp->mutable_notification()->Reserve(notif_count);.

[akhandel-1]

Hi @acozzette,

Can you please share a reference that guides us on how to utilize the proposed "Reserve" keyword in our code and which protobuf version supports it.

[acozzette]

There's a reserve keyword in the .proto file language, but I'm just referring to the Reserve() method on RepeatedField and RepeatedPtrField. It works the same way as std::vector::reserve and lets you pre-allocate the array in advance. I'm not sure when exactly we added it but I think it's been around for a long time.

[akhandel-1]

Hi @acozzette ,

Based on the discussions and code changes made, an attempt was made to reduce heap memory usage by reserving memory for the Update field using "Reserve". However, it seems that the heap memory is still partially increasing, indicating that further optimization is needed to achieve the desired reduction in memory usage.

To provide context, before applying the "reserve" keyword changes, the heap memory usage was measured at 239,795,304 bytes. After implementing the "reserve" keyword changes, the heap memory usage increased slightly to 240,271,656 bytes. This data was taken for total 100000 Updates

The following code changes were made:

::gnmi::Update* update;
::google::protobuf::RepeatedPtrField<::gnmi::Update>* update_field = notif->mutable_update();
update_field->Reserve(MAX_UPDATE_ITERATION);
update = update_field->Add();

In this updated code, memory reservation for the Update field is achieved by using the Reserve() function. The MAX_UPDATE_ITERATION represents the desired number of "Update" iterations for which memory is reserved. Can you please suggest what else we can try? Please let us know if we have missed anything in this code.

[acozzette]

I still think this code is probably working as intended. The thing is that fields in the in-memory representation take up space even if they're unset. Each time you create a new submessage you're allocating space for all fields even if you don't set all of them. You might find it useful to experiment with the SpaceUsedLong() method on messages to get an estimate of how much space is used.

[github-actions[bot]]

We triage inactive PRs and issues in order to make it easier to find active work. If this issue should remain active or becomes active again, please add a comment.

This issue is labeled inactive because the last activity was over 90 days ago.

[github-actions[bot]]

We triage inactive PRs and issues in order to make it easier to find active work. If this issue should remain active or becomes active again, please reopen it.

This issue was closed and archived because there has been no new activity in the 14 days since the inactive label was added.