Benchmark all queue sizes to identify optimal size

peplin commented 11 years ago

I think some of the in-memory queues are grossly oversized right now, adding to the statically allocated memory pressure. As a first pass before changing to dynamic allocation, I want to get an idea of the appropriate sizes. The buffers in question:

CanBus.buffer - controlled by BUS_MEMORY_BUFFER_SIZE and currently set to 2 * 8 * 16 == 256 bytes.
- This is only needed by the PIC32 build. This size allows for 2 channels, 8 messages per channel with 16 bytes per message.
CanBus.sendQueue - currently 64 CAN messages wide (each CanMessage struct is 32 + 32 + 64 == 128 bytes) for a total of 8192 bytes! Not all of this is allocated, though.
CanBus.receiveQueue - also 64 CAN messages, same issue as with sendQueue.
UsbDevice.sendQueue - 1024 bytes
UsbDevice.receiveQueue - 1024 bytes
UsbDevice.sendBuffer - 512 bytes
UsbDevice.receiveBuffer - 64 bytes, the endpoint size
UartDevice.sendQueue - 1024 bytes
UartDevice.receiveQueue - 1024 bytes

peplin commented 11 years ago

I added queue length logging for the CanBus receive and send queues in 8d91ecf. I noticed it's always zero, and with further investigation found that the only time we ever stress the queue is in short bursts. We would need a queue of ~200 messages to handle this example burst. If we allocate that it's around 25KB, which will likely be tough with 32KB of RAM. I'm not sure what do do about this.

peplin commented 11 years ago

I've added queue size logging for the USB, UART and network queues - they are never more than 3% used on average, so we can save some space there.

tgiuli commented 11 years ago

Wow, a 200 message burst -- not to be obsessive, but could you plot the queue size vs time? Do you have a sense of whether the burst occurs on the CAN bus, or if in this case the MCU is busy parsing other messages and so incoming CAN messages accumulate over time in the buffer without being serviced?

peplin commented 11 years ago

I'll obviously want to spend a little more time on it to make sure but what I saw today could be graphed like this:

                  |           FULL FULL FULL            FULL FULL FULL
Queue Size        |
                  |
                  |   0 0 0 0                0 0 0 0 0 
                  |____________________________________________

                                              Time

We're spinning through the main loop no sweat, usually no more than 1 CAN message is queued up to be processed so I can hardly even register anything being in the queue unless I record in exactly the right spot until BAM, the CAN interrupt handler just takes over with 150+ messages in a row.

peplin commented 11 years ago

I've decreased the size of the send/receive queues for the 3 output interfaces and added an alternative process mechanism to make sure we don't overflow the buffer (in a8843d0). The code size went from:

> arm-none-eabi-size build/FORDBOARD/vi-firmware-FORDBOARD.elf 
   text    data     bss     dec     hex filename
 132120   19112    4828  156060   2619c build/FORDBOARD/vi-firmware-FORDBOARD.elf

to

> arm-none-eabi-size build/FORDBOARD/vi-firmware-FORDBOARD.elf
   text    data     bss     dec     hex filename
 132216   15528    4828  152572   253fc build/FORDBOARD/vi-firmware-FORDBOARD.elf

a savings of 3.5KB of RAM.

peplin commented 11 years ago

Moving a PIC32-only buffer to the PIC32 build only dropped from:

   text    data     bss     dec     hex filename
 130752   12312    4724  147788   2414c build/FORDBOARD/vi-firmware-FORDBOARD.elf

to

   text    data     bss     dec     hex filename
 130752   11800    4724  147276   23f4c build/FORDBOARD/vi-firmware-FORDBOARD.elf

another 512 bytes of RAM. The PIC32 has 128KB of RAM so I don't think this will make the builds lopsided.

peplin commented 11 years ago

Make the CanSignalState array a const array dropped it again (in 60ff054):

   text    data     bss     dec     hex filename
 132528   15240    4868  152636   2543c build/FORDBOARD/vi-firmware-FORDBOARD.elf

to

   text    data     bss     dec     hex filename
 135120   12648    4868  152636   2543c build/FORDBOARD/vi-firmware-FORDBOARD.elf

saving 2.5KB (this is now the C-Max build, that's why the overall size is a bit larger than the previous comments which were with the Flex 2011).

peplin commented 11 years ago

We're down to these top RAM hogs:

> arm-none-eabi-nm build/FORDBOARD/vi-firmware-FORDBOARD.elf --size-sort | grep " D "
<truncated...>
00000200 D iso_buffer
00000200 D usb_data_buffer
00000200 D usb_data_buffer_IN
00000200 D usb_data_buffer_OUT
0000032c D USB_DEVICE
00000408 D __malloc_av_
00000804 D SIGNALS
000010a0 D CAN_BUSES

The iso_buffer and usbdata* are all from the nxpUSBlib. It's possible a newer version of that library is more efficient (they made a radical change to how they distribute the library so I've been avoiding upgrading, but this might be a good enough reason)
The biggest part of the USB_DEVICE is the 512 byte packet buffer - nothing we can change there, we have to have it to send a full packet.
malloc - huh, I guess one of the libraries we use must pull in malloc - I've been avoiding using it because of this big memory hit but since we're already paying the price...
SIGNALS - each CanSignal initialized is 76 bytes. This particular build has 27 signals. Likely some minor savings here which will add up when you start adding more signals.
CAN_BUSES - the send/receive queues for CanMessage objects are the killer here. If we're going to be screwed anyway with rapid fire bursts (when we don't have any CAN filters enabled) we may want to just bail and not devote so much RAM to it. I'll run some experiments to fully understand the burstiness and find an optimal setting.

peplin commented 11 years ago

After optimizing some types, the new totals:

> arm-none-eabi-nm build/FORDBOARD/vi-firmware-FORDBOARD.elf --size-sort | grep " D "
<truncated...>
00000200 D iso_buffer
00000200 D usb_data_buffer
00000200 D usb_data_buffer_IN
00000200 D usb_data_buffer_OUT
00000320 D USB_DEVICE
00000408 D __malloc_av_
0000072c D SIGNALS
000010a0 D CAN_BUSES

taking it from:

> arm-none-eabi-size build/FORDBOARD/vi-firmware-FORDBOARD.elf
   text    data     bss     dec     hex filename
 135120   12648    4868  152636   2543c build/FORDBOARD/vi-firmware-FORDBOARD.elf

to

   text    data     bss     dec     hex filename
 135128   12408    4868  152404   25354 build/FORDBOARD/vi-firmware-FORDBOARD.elf

for a savings of 240 bytes (CanSignal structs are down to 68 bytes each from 76).

peplin commented 11 years ago

I'll split off a new issue re: optimizing the CAN message queues and close this one as I'm happy with the space savings. For the C-Max 2013 build, at the start we had:

> arm-none-eabi-size build/FORDBOARD/vi-firmware-FORDBOARD.elf
   text    data     bss     dec     hex filename
 132272   19472    4844  156588   263ac build/FORDBOARD/vi-firmware-FORDBOARD.elf

and now in the next branch we have:

> arm-none-eabi-size build/FORDBOARD/vi-firmware-FORDBOARD.elf
   text    data     bss     dec     hex filename
 135128   12408    4868  152404   25354 build/FORDBOARD/vi-firmware-FORDBOARD.elf

7064 bytes of RAM use was alleviated
- ~2856 of those bytes were moved to the .text section (flash) (made things const if possible)
- Some amount was made more efficient by initializing at runtime (and thus compressing into .bss, in RAM)
Total RAM usage as startup is ~29% less (24316 bytes to 17276) and nothing was moved to dynamic allocation, so we can consider that just more headroom for additional CAN messages and signals

openxc / vi-firmware

Benchmark all queue sizes to identify optimal size #162