ManimCommunity / manim

A community-maintained Python framework for creating mathematical animations.
https://www.manim.community
MIT License
20.85k stars 1.52k forks source link

Manim stuck in python 3.12.3 on arch linux, and keeps allocating memory until no memory available. #3743

Closed astro-angelfish closed 4 months ago

astro-angelfish commented 4 months ago

Description of bug / unexpected behavior

I tried to generate a video after a full system upgrade, but manim stopped responding in self.play of manim.Scene, and ate all of memory of my laptop.

Expected behavior

It should not use so much memory and should give me a video.

How to reproduce the issue

Compile an binary and name that variable_distribution. The command I used to compile was gcc variable_distribution.c -g -no-pie -O0 -o variable_distribution

Code for reproducing the problem ```py import manim import pwn import numpy class VariableDistribution(manim.Scene): def construct(self): example_program = pwn.gdb.debug(["./variable_distribution"], api=True) example_program.gdb.Breakpoint('main', temporary=True) example_program.gdb.continue_and_wait() stack_view = manim.Rectangle(color=manim.WHITE, width=3, height=5, fill_color=manim.BLUE, fill_opacity=0.75) stack_view.move_to(numpy.array([-4.5, 0, 0])) heap_view = manim.Rectangle(color=manim.WHITE, width=3, height=5, fill_color=manim.BLUE, fill_opacity=0.75) heap_view.move_to(numpy.array([-2.5, 0, 0])) code_view = manim.Code("variable_distribution.c", language="C") code_view.scale(0.35) code_view.move_to(numpy.array([2.5, 0, 0])) example_program.gdb.quit() example_program.close() self.play(manim.DrawBorderThenFill(stack_view), manim.DrawBorderThenFill(heap_view), manim.Write(code_view)) self.wait() ``` `variable_distribution.c`: ```c #include #include struct Baka { int cc; long dd; }; union Sss { struct { int aa; int bb; } qq; long cc; }; void func() { int a; scanf("%d", &a); printf("%d\n", a); } int main() { int a; char b[0x10]; long c; int* d; struct Baka e; struct Baka* f; union Sss g; scanf("%d", &a); fgets(b, 0x10, stdin); scanf("%ld", &c); d = malloc(sizeof(int)); scanf("%d", d); scanf("%d %ld", &e.cc, &e.dd); f = malloc(sizeof(struct Baka)); scanf("%d %ld", &f->cc, &f->dd); scanf("%ld", &g.cc); printf("%d %s %ld %d %d %ld %d %ld %d %d\n", a, b, c, *d, e.cc, e.dd, f->cc, f->dd, g.qq.aa, g.qq.bb); func(); return 0; } ```

Additional media files

Images/GIFs No media available. My laptop fails to help manim to generate.

Logs

Terminal output ``` +-$ manim -pql scene.py VariableDistribution -v DEBUG Manim Community v0.18.1 [+] Starting local process '/usr/bin/gdbserver' argv=[b'/usr/bin/gdbserver', b'--multi', b'--no-disable-randomization', b'localhost:0', b'./variable_distribution'] : pid 5023 [DEBUG] Received 0x4c bytes: b'Process ./variable_distribution created; pid = 5026\n' b'Listening on port 45005\n' [DEBUG] Wrote gdb script to '/tmp/pwnrsxl9zmm.gdb' python socket_path = '/tmp/tmpbvjzml5b/socket' source /home/astro-angelfish/program-dev/tutorials/ep1-information-gathering-memory-layout-cpu-structure/.venv/lib/python3.12/site-packages/pwnlib/gdb_api_bridge.py target remote 127.0.0.1:45005 [*] running in new terminal: ['/usr/bin/gdb', '-q', './variable_distribution', '-x', '/tmp/pwnrsxl9zmm.gdb'] [DEBUG] Created script for new terminal: #!/home/astro-angelfish/program-dev/tutorials/ep1-information-gathering-memory-layout-cpu-structure/.venv/bin/python import os os.execve('/usr/bin/gdb', ['/usr/bin/gdb', '-q', './variable_distribution', '-x', '/tmp/pwnrsxl9zmm.gdb'], os.environ) [DEBUG] Launching a new terminal: ['/usr/bin/konsole', '-e', '/tmp/tmpte67demj'] [DEBUG] Received 0x38 bytes: b'Remote debugging from host ::ffff:127.0.0.1, port 44924\n' [*] Stopped process './variable_distribution' (pid 5026) Aborted! Exception ignored in: Traceback (most recent call last): File "/usr/lib/python3.12/threading.py", line 1573, in _shutdown def _shutdown(): KeyboardInterrupt: ```

System specifications

System Details - OS (with version, e.g., Windows 10 v2004 or macOS 10.15 (Catalina)):`Linux AL-1S 6.8.8-arch1-1 #1 SMP PREEMPT_DYNAMIC Sun, 28 Apr 2024 15:59:47 +0000 x86_64 GNU/Linux` RAM: 16G Python version (python/py/python3 --version): Python 3.12.3 - Installed modules (provide output from `pip list`): ``` Package Version ------------------ ----------- bcrypt 4.1.2 cairocffi 1.7.0 capstone 5.0.1 certifi 2024.2.2 cffi 1.16.0 charset-normalizer 3.3.2 click 8.1.7 cloup 3.0.5 colorama 0.4.6 colored-traceback 0.3.0 colour 0.1.5 commonmark 0.9.1 cryptography 42.0.5 decorator 5.1.1 glcontext 2.5.0 idna 3.7 intervaltree 3.1.0 isosurfaces 0.1.2 Mako 1.3.3 manim 0.18.1 ManimPango 0.5.0 mapbox-earcut 1.0.1 markdown-it-py 3.0.0 MarkupSafe 2.1.5 mdurl 0.1.2 moderngl 5.10.0 moderngl-window 2.4.6 multipledispatch 1.0.0 networkx 3.3 numpy 1.26.4 packaging 24.0 pangocairocffi 0.4.0 pangocffi 0.8.0 paramiko 3.4.0 pillow 10.3.0 pip 23.2.1 plumbum 1.8.3 progressbar 2.5 psutil 5.9.8 pwndbg 0.1.1 pwntools 4.12.0 pycairo 1.26.0 pycparser 2.22 pydub 0.25.1 pyelftools 0.31 pyglet 2.0.15 Pygments 2.17.2 PyNaCl 1.5.0 pyrr 0.10.3 pyserial 3.5 PySocks 1.7.1 python-dateutil 2.9.0.post0 requests 2.31.0 rich 13.7.1 ROPGadget 7.4 rpyc 6.0.0 scipy 1.13.0 screeninfo 0.8.1 six 1.16.0 skia-pathops 0.8.0.post1 sortedcontainers 2.4.0 srt 3.5.3 svgelements 1.9.6 tqdm 4.66.2 typing_extensions 4.11.0 unicorn 2.0.1.post1 unix-ar 0.2.1 urllib3 2.2.1 watchdog 4.0.0 zstandard 0.22.0 ```
LaTeX details + LaTeX distribution (e.g. TeX Live 2020): texlive-basic 2024.2-1 + Installed LaTeX packages: https://github.com/ManimCommunity/manim/files/15051473/tlmgrlist.txt I dont think my tlmgr list changed.
FFMPEG Output of `ffmpeg -version`: ``` ffmpeg version n6.1.1 Copyright (c) 2000-2023 the FFmpeg developers built with gcc 13.2.1 (GCC) 20230801 configuration: --prefix=/usr --disable-debug --disable-static --disable-stripping --enable-amf --enable-avisynth --enable-cuda-llvm --enable-lto --enable-fontconfig --enable-frei0r --enable-gmp --enable-gnutls --enable-gpl --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libdav1d --enable-libdrm --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libharfbuzz --enable-libiec61883 --enable-libjack --enable-libjxl --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libplacebo --enable-libpulse --enable-librav1e --enable-librsvg --enable-librubberband --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpl --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxml2 --enable-libxvid --enable-libzimg --enable-nvdec --enable-nvenc --enable-opencl --enable-opengl --enable-shared --enable-vapoursynth --enable-version3 --enable-vulkan libavutil 58. 29.100 / 58. 29.100 libavcodec 60. 31.102 / 60. 31.102 libavformat 60. 16.100 / 60. 16.100 libavdevice 60. 3.100 / 60. 3.100 libavfilter 9. 12.100 / 9. 12.100 libswscale 7. 5.100 / 7. 5.100 libswresample 4. 12.100 / 4. 12.100 libpostproc 57. 3.100 / 57. 3.100 ```

Additional comments

Evidence from top ``` top - 18:03:34 up 15 min, 1 user, load average: 2.95, 2.73, 2.00 Tasks: 378 total, 2 running, 375 sleeping, 0 stopped, 1 zombie %Cpu(s): 9.1 us, 0.7 sy, 0.0 ni, 89.8 id, 0.2 wa, 0.1 hi, 0.0 si, 0.0 st MiB Mem : 15815.0 total, 577.2 free, 12507.4 used, 3699.0 buff/cache MiB Swap: 0.0 total, 0.0 free, 0.0 used. 3307.6 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 5334 astro-a+ 20 0 9297572 7.9g 44492 R 99.7 51.2 2:13.41 manim 4218 astro-a+ 20 0 11.3g 423372 207200 S 13.6 2.6 1:09.67 firefox 2886 astro-a+ 20 0 678192 13268 8672 S 1.3 0.1 0:16.54 conky 2535 astro-a+ 20 0 644328 61108 37412 S 1.0 0.4 0:13.43 Xwayland 1618 astro-a+ -2 0 3360176 249324 143032 S 0.7 1.5 0:47.13 kwin_wayland 1799 root 20 0 7144232 149632 23576 S 0.7 0.9 0:03.79 v2ray ```
behackl commented 4 months ago

I tried to reproduce this, but failed due to all sorts of things not being installed and setup correctly on my machine. I needed to import pwnlib as pwn instead of import pwn to get anywhere though, are you sure the snippet is correct?

In any case, I don't quite see where we even have the potential for a memory leak here. Is it vital that your example_program debugging happens within construct? I guess you will want to interact with it somehow in your actual example. Can you verify that the parts of your code where you interact with pwn run independently in the same Python environment you use to render your Scene? I.e., does running

import manim
import pwn
import numpy

example_program = pwn.gdb.debug(["./variable_distribution"], api=True)
example_program.gdb.Breakpoint('main', temporary=True)
example_program.gdb.continue_and_wait()

example_program.gdb.quit()
example_program.close()

work for you as intended?

astro-angelfish commented 4 months ago

I tried to reproduce this, but failed due to all sorts of things not being installed and setup correctly on my machine. I needed to import pwnlib as pwn instead of import pwn to get anywhere though, are you sure the snippet is correct?

In any case, I don't quite see where we even have the potential for a memory leak here. Is it vital that your example_program debugging happens within construct? I guess you will want to interact with it somehow in your actual example. Can you verify that the parts of your code where you interact with pwn run independently in the same Python environment you use to render your Scene? I.e., does running

import manim
import pwn
import numpy

example_program = pwn.gdb.debug(["./variable_distribution"], api=True)
example_program.gdb.Breakpoint('main', temporary=True)
example_program.gdb.continue_and_wait()

example_program.gdb.quit()
example_program.close()

work for you as intended?

Yes these code works for me. I got pwntools by pip install pwntools directly and gave me a pwn package.

Everything works fine until the code reaches self.play(...).

Maybe I should provide my source of variable_distribution.c. I'll add that by editing.

astro-angelfish commented 4 months ago

I did a little research and I found when manim trying to use a logger, it traps into pwntools and there seems to be a dead loop there

Full call stack, if needed ``` parse, .venv/lib/python3.12/site-packages/pwnlib/term/term.py:298 output, .venv/lib/python3.12/site-packages/pwnlib/term/term.py:508 write, .venv/lib/python3.12/site-packages/pwnlib/term/term.py:144 _check_buffer, .venv/lib/python3.12/site-packages/rich/console.py:2060 _exit_buffer, .venv/lib/python3.12/site-packages/rich/console.py:823 __exit__, .venv/lib/python3.12/site-packages/rich/console.py:865 print, .venv/lib/python3.12/site-packages/rich/console.py:1673 emit, .venv/lib/python3.12/site-packages/rich/logging.py:170 handle, /usr/lib/python3.12/logging/__init__.py:1028 callHandlers, /usr/lib/python3.12/logging/__init__.py:1762 handle, /usr/lib/python3.12/logging/__init__.py:1700 _log, /usr/lib/python3.12/logging/__init__.py:1684 debug, /usr/lib/python3.12/logging/__init__.py:1527 get_hash_from_play_call, .venv/lib/python3.12/site-packages/manim/utils/hashing.py:352 play, cairo_renderer.py:81 play, .venv/lib/python3.12/site-packages/manim/renderer/cairo_renderer.py:1092 construct, scene.py:722 render, .venv/lib/python3.12/site-packages/manim/scene/scene.py:229 render, .venv/lib/python3.12/site-packages/manim/cli/render/commands.py:120 invoke, .venv/lib/python3.12/site-packages/click/core.py:783 invoke, .venv/lib/python3.12/site-packages/click/core.py:1434 invoke, .venv/lib/python3.12/site-packages/click/core.py:1688 main, .venv/lib/python3.12/site-packages/click/core.py:1078 __call__, .venv/lib/python3.12/site-packages/click/core.py:1157 , .venv/bin/manim:8 ```

j = s.index('\x1b\\', i) where s is a byte array and i is an integer, just raised an TypeError and causing the dead-loop. It seems to be fixed in https://github.com/Gallopsled/pwntools/commit/43071cd6e52e7518baa4e7f9d7ef57dd9e691c4c#diff-d6c967c202f605366647f784a21a9745a41941dc319bdd2493429df57680a854. I guess it is just a incompatibility of old pwntools and python 3.12. because everything before that was fine.

The workaround seems to be use dev branch of pwntools or downgrade python back to 3.11. The fastest way is to add PWNLIB_NOTERM=1 before the manim command.

Closing this issue because it seems nothing to do with manim.