kedro-org / kedro-viz

Visualise your Kedro data and machine-learning pipelines and track your experiments.
https://demo.kedro.org
Apache License 2.0
674 stars 111 forks source link

CPU pinned at 100% #2135

Open pascalwhoop opened 2 hours ago

pascalwhoop commented 2 hours ago

Description

Regularly observing 1 python process being pinned at 100% CPU utilization. Turns out it's kedro viz

$ ps ax | grep 71551                                                                                                                                                                                          4994ms
71551 s023  R+   319:19.17 /opt/homebrew/Cellar/python@3.11/3.11.9_1/Frameworks/Python.framework/Versions/3.11/Resources/Python.app/Contents/MacOS/Python -c from multiprocessing.spawn import spawn_main; spawn_main(tracker_fd=7, pipe_handle=9) --multiprocessing-fork
85432 s028  S+     0:00.00 grep --color=auto 71551
$ pstree -p 71551                          394ms
-+= 00001 root /sbin/launchd
 \-+= 00836 pascalwhoop /Applications/iTerm.app/Contents/MacOS/iTerm2
   \-+- 00901 pascalwhoop /Users/pascalwhoop/Library/Application Support/iTerm2/iTermServer-3.5.5 /Users/pascalwhoop/Library/Application Support/iTerm2/iterm2-daemon-1.socket
     \-+= 70530 root /usr/bin/login -fpl pascalwhoop /Applications/iTerm.app/Contents/MacOS/ShellLauncher --launch_shell SHELL=/opt/homebrew/bin/fish
       \-+= 70533 pascalwhoop -fish
         \-+= 71535 pascalwhoop /opt/homebrew/Cellar/python@3.11/3.11.9_1/Frameworks/Python.framework/Versions/3.11/Resources/Python.app/Contents/MacOS/Python /Users/pascalwhoop/Code/everycure/matrix/pipelines/matrix/.venv/bin/kedro viz --aut
           \-+- 71551 pascalwhoop /opt/homebrew/Cellar/python@3.11/3.11.9_1/Frameworks/Python.framework/Versions/3.11/Resources/Python.app/Contents/MacOS/Python -c from multiprocessing.spawn import spawn_main; spawn_main(tracker_fd=7, pipe_ha
             \--- 71563 pascalwhoop /opt/homebrew/Cellar/python@3.11/3.11.9_1/Frameworks/Python.framework/Versions/3.11/Resources/Python.app/Contents/MacOS/Python -c from multiprocessing.spawn import spawn_main; spawn_main(tracker_fd=7, pipe_

Steps to Reproduce

it may be related to our kedro pipeline, it's dynamic and has 180+ nodes. Happy to share some telemetry if you guide me how. I can't share the code (yet) unfortunately.

Your Environment

Include as many relevant details as possible about the environment you experienced the bug in:

Checklist

rashidakanchwala commented 2 hours ago

Thanks, please do share telemetry for it.

rashidakanchwala commented 2 hours ago

@ravi-kumar-pilla