search

Juliasunn / video-web

Website for publishing and watching videos, etc...
0 stars 0 forks source link

Implent server memory usage analyze with valgrind #19

Open Juliasunn opened 4 months ago

Juliasunn commented 4 months ago

valgrind --leak-check=full ./HttpLiveStreaming

/main, /channel?u=705e39be-cd3c-4f31-b597-4134941e425c

==10600== LEAK SUMMARY:
==10600==    definitely lost: 96 bytes in 1 blocks
==10600==    indirectly lost: 0 bytes in 0 blocks
==10600==      possibly lost: 2,408 bytes in 20 blocks
==10600==    still reachable: 6,626,344 bytes in 5,018 blocks
==10600==                       of which reachable via heuristic:
==10600==                         newarray           : 1,536 bytes in 16 blocks
==10600==         suppressed: 0 bytes in 0 blocks

possibly lost in /usr/lib/x86_64-linux-gnu/ld-2.31.so mongocxx::v_noabi::pool::acquire()

Juliasunn commented 4 months ago

The "still reachable" category: "Memory was allocated and was not subsequently freed before the program terminated."

Other categories: "Memory was allocated and cannot be subsequently freed because the program no longer has any pointers to the allocated memory block."

Juliasunn commented 4 months ago

Просмотр видео:

==11483== LEAK SUMMARY:
==11483==    definitely lost: 11,700,096 bytes in 118 blocks
==11483==    indirectly lost: 0 bytes in 0 blocks
==11483==      possibly lost: 1,002,408 bytes in 30 blocks
==11483==    still reachable: 6,636,447 bytes in 5,113 blocks
==11483==                       of which reachable via heuristic:
==11483==                         newarray           : 1,536 bytes in 16 blocks
==11483==         suppressed: 0 bytes in 0 blocks
==11483== Reachable blocks (those to which a pointer was found) are not shown.
==11483== To see them, rerun with: --leak-check=full --show-leak-kinds=all

ns_server::MediaRequestHandler::setPayload operator new[](unsigned long)

Причина: std::string::assign(char *, size_t) не передает во владение выделенный указатель, а копирует данные. Динамически выделенный массив не очищается.

После устранения утечки:

==13732== LEAK SUMMARY:
==13732==    definitely lost: 96 bytes in 1 blocks
==13732==    indirectly lost: 0 bytes in 0 blocks
==13732==      possibly lost: 2,408 bytes in 20 blocks
==13732==    still reachable: 6,638,023 bytes in 5,128 blocks
==13732==                       of which reachable via heuristic:
==13732==                         newarray           : 1,536 bytes in 16 blocks
==13732==         suppressed: 0 bytes in 0 blocks

TODO: гарантировать что при копировании MediaRequestHandler указатель на буфер у каждого будет свой (глубокое копирование)

==25613== HEAP SUMMARY:
==25613==     in use at exit: 6,638,113 bytes in 5,138 blocks
==25613==   total heap usage: 81,745 allocs, 76,607 frees, 268,028,594 bytes allocated

Почему столько (81,745) алокаций?

Преположение:

   // res is response object that is allocated on each server's http response
    res.body().assign(buf, chunkSize);

При копировании из буфера каждый раз динамически выделяем память под тело http-ответа. Варианты решения:

Результат:

==6615== HEAP SUMMARY:
==6615==     in use at exit: 6,636,809 bytes in 5,123 blocks
==6615==   total heap usage: 67,807 allocs, 62,684 frees, 208,707,421 bytes allocated

Вывод: количество алокаций не сократилось.

Причины: на запрос каждого фрагмента видео создается своя сессия и по новой выделяется память на запрос, так как запросы содержат заголовок Connection: close хотя в браузере отображается Connection: keep-alive

Решение: A) Понять почему запросы приходят с другим заголовком Connection (возможно причина в том как их проксирует nginx HTTP/1.0 по умолчанию имеет поле close).

Б) Создать пул объектов response на каждый поток (чтобы избежать блокировок), сначала пробовать получить свободный объект, если нет свободного - создавать, когда объект не нужен - возвращать в пул. За удаление объектоа ответственен пул.

Juliasunn commented 4 months ago

/private/upload

==24818== LEAK SUMMARY:
==24818==    definitely lost: 96 bytes in 1 blocks
==24818==    indirectly lost: 0 bytes in 0 blocks
==24818==      possibly lost: 2,408 bytes in 20 blocks
==24818==    still reachable: 6,639,870 bytes in 5,080 blocks
==24818==                       of which reachable via heuristic:
==24818==                         newarray           : 1,536 bytes in 16 blocks
==24818==         suppressed: 0 bytes in 0 blocks
==24818== Reachable blocks (those to which a pointer was found) are not shown.
==24818== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==24818== HEAP SUMMARY:
==24818==     in use at exit: 6,642,374 bytes in 5,101 blocks
==24818==   total heap usage: 30,148 allocs, 25,047 frees, 67,726,049 bytes allocated
Juliasunn commented 4 months ago

Запуск без обработки запросов:

==25453== HEAP SUMMARY:
==25453==     in use at exit: 6,626,962 bytes in 5,045 blocks
==25453==   total heap usage: 7,462 allocs, 2,417 frees, 8,096,226 bytes allocated
==25453== LEAK SUMMARY:
==25453==    definitely lost: 96 bytes in 1 blocks
==25453==    indirectly lost: 0 bytes in 0 blocks
==25453==      possibly lost: 2,408 bytes in 20 blocks
==25453==    still reachable: 6,624,458 bytes in 5,024 blocks
==25453==                       of which reachable via heuristic:
==25453==                         newarray           : 1,536 bytes in 16 blocks
Juliasunn commented 4 months ago

/main (отправляются json-ы видео)

==25559== HEAP SUMMARY:
==25559==     in use at exit: 6,629,154 bytes in 5,054 blocks
==25559==   total heap usage: 9,448 allocs, 4,394 frees, 19,011,171 bytes allocated

==25559== LEAK SUMMARY:
==25559==    definitely lost: 96 bytes in 1 blocks
==25559==    indirectly lost: 0 bytes in 0 blocks
==25559==      possibly lost: 2,408 bytes in 20 blocks
==25559==    still reachable: 6,626,650 bytes in 5,033 blocks
==25559==                       of which reachable via heuristic:
==25559==                         newarray           : 1,536 bytes in 16 blocks
Juliasunn commented 4 months ago

Анализ выделенной памяти в программе (в каких функциях выделялась и сколько %)

$valgrind --tool=massif ./HttpLiveStreaming
$ ls
massif.out.7079
$ms_print massif.out.7079
94.86% (1,476,581B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
->67.37% (1,048,576B) 0xC8D977: static_buffer::static_buffer(unsigned long) (buffers.h:59)
....