Closed joeyy-watts closed 8 months ago
joeyy-watts
commented
10 months ago probably not network; ping times are pretty much identical between RPi and laptop, even from Docker container
might be a big change to optimize, some ideas:
joeyy-watts
commented
10 months ago Top cProfile (from laptop run), for reference:
Ordered by: call count
ncalls tottime percall cumtime percall filename:lineno(function)
2147328 0.348 0.000 0.348 0.000 {built-in method builtins.round}
1466830 0.210 0.000 0.210 0.000 {method 'to_bytes' of 'int' objects}
1433964 0.152 0.000 0.152 0.000 {built-in method builtins.hash}
1433961 0.291 0.000 0.443 0.000 enum.py:774(__hash__)
746538/746471 0.069 0.000 0.069 0.000 {built-in method builtins.len}
734661/734649 0.071 0.000 0.072 0.000 {built-in method builtins.isinstance}
715776 1.371 0.000 3.124 0.000 channel.py:100(set_values)
715776 0.243 0.000 0.311 0.000 universe.py:52(get_channel)
715776 0.203 0.000 0.514 0.000 universe.py:132(__getitem__)
486604 0.068 0.000 0.068 0.000 background_task.py:30(start)
486603 0.733 0.000 0.941 0.000 channel.py:129(to_buffer)
486603 0.332 0.000 1.341 0.000 universe.py:36(channel_changed)
102509 0.014 0.000 0.014 0.000 {built-in method builtins.max}
56323 0.015 0.000 0.015 0.000 {built-in method builtins.all}
56320 0.014 0.000 0.029 0.000 effects_utils.py:7(is_black)
42174 0.008 0.000 0.008 0.000 {built-in method builtins.min}
30282 0.005 0.000 0.005 0.000 {built-in method time.monotonic}
29289 0.005 0.000 0.005 0.000 base_events.py:1923(get_debug)
24720 0.005 0.000 0.005 0.000 base_events.py:513(_check_closed)
19877 0.003 0.000 0.003 0.000 {method 'popleft' of 'collections.deque' objects}
19834 0.004 0.000 0.004 0.000 {method 'append' of 'collections.deque' objects}
19794 0.021 0.000 0.025 0.000 events.py:31(__init__)
19785 0.022 0.000 6.372 0.000 events.py:78(_run)
19785 0.032 0.000 6.350 0.000 {method 'run' of 'Context' objects}
17887 0.017 0.000 0.067 0.000 base_events.py:741(call_soon)
17887 0.023 0.000 0.047 0.000 base_events.py:770(_call_soon)
14570 0.015 0.000 0.058 0.000 tasks.py:633(sleep)
11620 0.002 0.000 0.002 0.000 {method 'append' of 'list' objects}
11506 0.007 0.000 0.009 0.000 base_events.py:694(time)
10760 0.001 0.000 0.001 0.000 tasks.py:621(__sleep0)
9805 0.001 0.000 0.001 0.000 {method 'cancelled' of '_asyncio.Task' objects}
9791 0.001 0.000 0.001 0.000 {method 'exception' of '_asyncio.Task' objects}
8916 19.388 0.002 19.388 0.002 {built-in method _overlapped.GetQueuedCompletionStatus}
8881 0.003 0.000 0.003 0.000 {built-in method math.ceil}
8878 0.027 0.000 19.421 0.002 windows_events.py:775(_poll)
8878 0.002 0.000 0.002 0.000 {method 'clear' of 'list' objects}
8877 0.079 0.000 25.919 0.003 base_events.py:1830(_run_once)
8877 0.011 0.000 19.432 0.002 windows_events.py:437(select)
8876 0.002 0.000 0.002 0.000 proactor_events.py:855(_process_events)
8770 0.008 0.000 0.009 0.000 events.py:120(__lt__)
5876 0.002 0.000 0.045 0.000 background_task.py:61(coro_wrap)
5875 0.016 0.000 0.043 0.000 base_node.py:116(_periodic_refresh_worker)
4983 0.001 0.000 0.001 0.000 {method 'done' of '_asyncio.Future' objects}
4967 0.002 0.000 0.002 0.000 {method 'add' of 'set' objects}
4934 0.006 0.000 0.008 0.000 _weakrefset.py:86(add)
4919 0.003 0.000 0.004 0.000 _weakrefset.py:39(_remove)
4919 0.001 0.000 0.001 0.000 {method 'discard' of 'set' objects}
4918 0.022 0.000 0.049 0.000 base_events.py:431(create_task)
4910 0.002 0.000 0.002 0.000 coroutines.py:177(iscoroutine)
4909 0.008 0.000 0.060 0.000 tasks.py:657(ensure_future)
4906 0.001 0.000 0.001 0.000 {method 'add_done_callback' of '_asyncio.Task' objects}
4902 0.009 0.000 0.015 0.000 tasks.py:767(_done_callback)
4896 0.001 0.000 0.001 0.000 {method 'result' of '_asyncio.Task' objects}
4893 0.836 0.000 4.916 0.001 artnet_handler.py:130(__async_set_pixels)
4725 0.002 0.000 0.002 0.000 __init__.py:1689(isEnabledFor)
4223 0.001 0.000 0.001 0.000 {method 'replace' of 'str' objects}
4072 0.000 0.000 0.000 0.000 {built-in method nt.fspath}
3910 0.001 0.000 0.001 0.000 {method 'startswith' of 'str' objects}
3573 0.000 0.000 0.000 0.000 {method 'append' of 'bytearray' objects}
3510 0.009 0.000 0.122 0.000 base_node.py:70(_send_data)
3510 0.007 0.000 0.159 0.000 universe.py:47(send_data)
3510 0.003 0.000 0.003 0.000 seq_counter.py:13(value)
3510 0.023 0.000 0.152 0.000 node.py:42(_send_universe)
3510 0.112 0.000 0.112 0.000 {method 'sendto' of '_socket.socket' objects}
2644 0.008 0.000 0.027 0.000 {method 'set_result' of '_asyncio.Future' objects}
2641 0.000 0.000 0.000 0.000 {method 'lower' of 'str' objects}
2209 0.000 0.000 0.000 0.000 {method 'endswith' of 'str' objects}
1918 0.001 0.000 0.001 0.000 {built-in method _contextvars.copy_context}
1916 0.004 0.000 0.005 0.000 base_events.py:427(create_future)
1908 0.003 0.000 0.003 0.000 events.py:64(cancel)
1908 0.007 0.000 0.011 0.000 events.py:148(cancel)
1908 0.001 0.000 0.001 0.000 {built-in method _asyncio.get_running_loop}
1907 0.005 0.000 0.020 0.000 base_events.py:725(call_at)
1907 0.001 0.000 0.001 0.000 base_events.py:1825(_timer_handle_cancelled)
1907 0.004 0.000 0.009 0.000 events.py:103(__init__)
1907 0.003 0.000 0.005 0.000 {built-in method _heapq.heappush}
1907 0.008 0.000 0.015 0.000 {built-in method _heapq.heappop}
1905 0.005 0.000 0.026 0.000 base_events.py:703(call_later)
1902 0.001 0.000 0.001 0.000 {method 'cancelled' of '_asyncio.Future' objects}
1899 0.004 0.000 0.029 0.000 futures.py:308(_set_result_unless_cancelled)
1434 0.000 0.000 0.001 0.000 inspect.py:700(<genexpr>)
1424 0.000 0.000 0.000 0.000 inspect.py:703(<genexpr>)
1411 0.007 0.000 0.146 0.000 animations.py:51(_main_function)
1401 0.001 0.000 0.699 0.000 async_utils.py:67(__main_loop)
1398 0.033 0.000 0.119 0.000 artnet_handler.py:96(set_pixels)
1341 0.001 0.000 0.002 0.000 ntpath.py:44(normcase)
1340 0.002 0.000 0.003 0.000 ntpath.py:124(splitdrive)
1264 0.000 0.000 0.000 0.000 {method 'items' of 'dict' objects}
1206 0.000 0.000 0.000 0.000 {method 'values' of 'dict' objects}
1204 0.000 0.000 0.001 0.000 __init__.py:1424(debug)
1198 0.001 0.000 0.001 0.000 output_correction.py:5(__init__)
1190 0.000 0.000 0.000 0.000 channel.py:64(<listcomp>)
1190 0.004 0.000 0.005 0.000 channel.py:29(__init__)
1190 0.000 0.000 0.000 0.000 channel.py:83(_apply_output_correction)
1190 0.014 0.000 0.049 0.000 universe.py:65(add_channel)
1190 0.014 0.000 0.028 0.000 universe.py:106(_resize_universe)
1142 0.000 0.000 0.000 0.000 {method 'join' of 'str' objects}
1035 0.000 0.000 0.001 0.000 inspect.py:64(ismodule)
1001 0.001 0.000 0.001 0.000 {method 'split' of 'str' objects}
981 0.001 0.000 0.002 0.000 {built-in method builtins.any}
950 0.000 0.000 0.000 0.000 {method 'lstrip' of 'str' objects}
895 0.005 0.000 0.009 0.000 ntpath.py:450(normpath)
888 0.061 0.000 0.061 0.000 {built-in method nt._getfinalpathname}
765 0.000 0.000 0.000 0.000 {method 'upper' of 'str' objects}
752 0.000 0.000 0.000 0.000 {built-in method builtins.hasattr}
744 0.000 0.000 0.001 0.000 _collections_abc.py:849(__iter__)
720 0.001 0.000 0.001 0.000 futures.py:296(_get_loop)
720 0.000 0.000 0.000 0.000 {method 'get_loop' of '_asyncio.Future' objects}
719 0.001 0.000 0.001 0.000 {built-in method _asyncio.get_event_loop}
703 0.003 0.000 0.003 0.000 tasks.py:701(__init__)
703 0.014 0.000 0.080 0.000 tasks.py:759(_gather)
702 0.003 0.000 0.083 0.000 tasks.py:721(gather)
700 0.000 0.000 0.001 0.000 os.py:674(__getitem__)
700 0.000 0.000 0.000 0.000 os.py:740(check_str)
700 0.000 0.000 0.001 0.000 os.py:746(encodekey)
699 0.003 0.000 0.003 0.000 artnet_handler.py:99(<listcomp>)
697 0.000 0.000 0.000 0.000 {built-in method builtins.getattr}
686 0.006 0.000 0.185 0.000 base_node.py:77(_process_values_task)
686 0.001 0.000 0.185 0.000 background_task.py:45(coro_wrap)
680 0.000 0.000 0.000 0.000 {method 'get' of 'dict' objects}
672 0.000 0.000 0.000 0.000 os.py:697(__iter__)
667 0.000 0.000 0.000 0.000 {built-in method builtins.ord}
534 0.020 0.000 0.020 0.000 {built-in method nt.stat}
523 0.000 0.000 0.000 0.000 {method 'decode' of 'bytes' objects}
499 0.000 0.000 0.001 0.000 inspect.py:655(getfile)
498 0.000 0.000 0.000 0.000 sre_parse.py:165(__getitem__)
478 0.002 0.000 0.022 0.000 inspect.py:693(getsourcefile)
469 0.000 0.000 0.018 0.000 genericpath.py:16(exists)
451 0.000 0.000 0.000 0.000 ntpath.py:34(_get_bothseps)
451 0.000 0.000 0.006 0.000 ntpath.py:524(abspath)
451 0.001 0.000 0.029 0.000 inspect.py:715(getabsfile)
451 0.001 0.000 0.001 0.000 {built-in method nt._getfullpathname}
444 0.001 0.000 0.002 0.000 ntpath.py:61(isabs)
444 0.002 0.000 0.071 0.000 ntpath.py:625(realpath)
444 0.000 0.000 0.000 0.000 {built-in method nt.getcwd}
359 0.000 0.000 0.000 0.000 sre_parse.py:234(__next)
298 0.000 0.000 0.000 0.000 {method 'rstrip' of 'str' objects}
256 0.000 0.000 0.000 0.000 sre_parse.py:255(get)
204/19 0.000 0.000 0.000 0.000 abc.py:121(__subclasscheck__)
204/19 0.000 0.000 0.000 0.000 {built-in method _abc._abc_subclasscheck}
196 0.000 0.000 0.000 0.000 sre_parse.py:173(append)
192/188 0.000 0.000 0.000 0.000 {method 'encode' of 'str' objects}
191 0.000 0.000 0.000 0.000 {method 'match' of 're.Pattern' objects}
180 0.000 0.000 0.000 0.000 _collections_abc.py:409(__subclasshook__)
171 0.000 0.000 0.000 0.000 {method 'rpartition' of 'str' objects}
146 0.002 0.000 0.002 0.000 {built-in method __new__ of type object at 0x00007FFD47A54C60}
112 0.000 0.000 0.000 0.000 {method 'partition' of 'str' objects}
111 0.000 0.000 0.000 0.000 sre_parse.py:250(match)
110 0.000 0.000 0.000 0.000 structures.py:46(__setitem__)
103 0.000 0.000 0.000 0.000 sre_parse.py:161(__len__)
98 0.000 0.000 0.351 0.004 {method 'readline' of '_io.BufferedReader' objects}
98 0.000 0.000 0.000 0.000 {built-in method _imp.acquire_lock}
98 0.000 0.000 0.000 0.000 {built-in method _imp.release_lock}
92 0.000 0.000 0.001 0.000 abc.py:117(__instancecheck__)
92 0.000 0.000 0.001 0.000 {built-in method _abc._abc_instancecheck}
91 0.000 0.000 0.000 0.000 structures.py:58(<genexpr>)
88/38 0.000 0.000 0.076 0.002 {method 'join' of 'bytes' objects}
88 0.000 0.000 0.000 0.000 {method 'pop' of 'dict' objects}
87 0.000 0.000 0.000 0.000 _collections.py:257(__getitem__)
86 0.000 0.000 0.000 0.000 parse.py:103(_noop)
86 0.000 0.000 0.000 0.000 parse.py:114(_coerce_args)
84 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:231(_verbose_message)
84 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:129(<genexpr>)
82 0.000 0.000 0.000 0.000 ssl.py:1079(_checkClosed)
80 0.004 0.000 0.004 0.000 {method 'acquire' of '_thread.lock' objects}
77 0.000 0.000 0.000 0.000 _weakrefset.py:75(__contains__)
77 0.000 0.000 0.000 0.000 {method '__exit__' of '_thread.lock' objects}
75 0.000 0.000 0.000 0.000 inspect.py:81(ismethod)
75 0.000 0.000 0.000 0.000 utils.py:51(_has_surrogates)
73 0.000 0.000 0.000 0.000 {method 'find' of 'str' objects}
72 0.000 0.000 0.000 0.000 _policybase.py:281(_sanitize_header)
72 0.000 0.000 0.000 0.000 _policybase.py:311(header_fetch_parse)
70 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:874(__enter__)
70 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:878(__exit__)
70 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:114(<listcomp>)
70 0.000 0.000 0.001 0.000 <frozen importlib._bootstrap_external>:91(_path_join)
68/20 0.000 0.000 0.000 0.000 sre_parse.py:175(getwidth)
68 0.000 0.000 0.000 0.000 inspect.py:237(istraceback)
68 0.000 0.000 0.000 0.000 inspect.py:247(isframe)
67 0.000 0.000 0.000 0.000 inspect.py:159(isfunction)
67 0.000 0.000 0.000 0.000 feedparser.py:78(readline)
66 0.000 0.000 0.000 0.000 structures.py:51(__getitem__)
66 0.000 0.000 0.000 0.000 {method 'setdefault' of 'dict' objects}
66 0.000 0.000 0.000 0.000 {built-in method builtins.next}
65 0.000 0.000 0.000 0.000 _collections_abc.py:760(get)
65 0.000 0.000 0.000 0.000 feedparser.py:128(__next__)
64 0.000 0.000 0.000 0.000 {method 'read' of '_io.BufferedReader' objects}
61 0.000 0.000 0.000 0.000 {method 'rfind' of 'str' objects}
60 0.000 0.000 0.000 0.000 client.py:442(isclosed)
60 0.000 0.000 0.000 0.000 {method 'strip' of 'str' objects}
58 0.000 0.000 0.000 0.000 sre_parse.py:112(__init__)
58 0.000 0.000 0.000 0.000 sre_parse.py:287(tell)
58 0.000 0.000 0.000 0.000 response.py:697(_error_catcher)
57 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:398(parent)
57 0.000 0.000 0.000 0.000 enum.py:358(__call__)
57 0.000 0.000 0.000 0.000 enum.py:670(__new__)
57 0.000 0.000 0.000 0.000 {built-in method builtins.abs}
57 0.000 0.000 0.000 0.000 {method 'fileno' of '_socket.socket' objects}
55/6 0.000 0.000 0.001 0.000 sre_compile.py:87(_compile)
55 0.029 0.001 0.058 0.001 base_effects.py:34(<listcomp>)
55 0.000 0.000 0.000 0.000 base_effects.py:150(func)
55 0.000 0.000 0.000 0.000 {built-in method math.sin}
54 0.000 0.000 0.000 0.000 response.py:244(__len__)
53 0.000 0.000 0.000 0.000 enum.py:792(value)
53 0.000 0.000 0.000 0.000 types.py:171(__get__)
53 0.000 0.000 0.000 0.000 _collections.py:289(__iter__)
52 0.000 0.000 0.000 0.000 enum.py:438(<genexpr>)
52 0.000 0.000 0.000 0.000 _policybase.py:293(header_source_parse)
52 0.000 0.000 0.000 0.000 message.py:479(set_raw)
52 0.000 0.000 0.000 0.000 response.py:429(_decode)
52 0.000 0.000 0.000 0.000 {built-in method _thread.allocate_lock}
51 0.000 0.000 0.000 0.000 inspect.py:73(isclass)
51 0.000 0.000 0.000 0.000 inspect.py:261(iscode)
50 0.000 0.000 0.000 0.000 client.py:610(_safe_read)
50 0.000 0.000 0.000 0.000 _collections.py:300(add)
49 0.000 0.000 0.000 0.000 codecs.py:319(decode)
49 0.000 0.000 0.000 0.000 {built-in method _codecs.utf_8_decode}
48 0.000 0.000 0.000 0.000 threading.py:256(__enter__)
48 0.000 0.000 0.000 0.000 threading.py:259(__exit__)
48 0.000 0.000 0.000 0.000 {method '__enter__' of '_thread.lock' objects}
47 0.000 0.000 0.000 0.000 threading.py:271(_is_owned)
47 0.000 0.000 0.000 0.000 _url.py:534(raw_path)
47 0.000 0.000 0.000 0.000 {method 'search' of 're.Pattern' objects}
47 0.000 0.000 0.000 0.000 {built-in method _thread.get_ident}
45 0.000 0.000 0.000 0.000 util.py:19(to_str)
43 0.000 0.000 0.000 0.000 windows_events.py:423(_check_closed)
43 0.000 0.000 0.000 0.000 {built-in method from_bytes}
43 0.000 0.000 0.000 0.000 {method 'format' of 'str' objects}
42 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:79(_unpack_uint32)
42 0.000 0.000 0.002 0.000 <frozen importlib._bootstrap_external>:135(_path_stat)
42 0.000 0.000 0.000 0.000 web_urldispatcher.py:364(resolve)
42 0.000 0.000 0.000 0.000 web_urldispatcher.py:413(_match)
42 0.000 0.000 0.000 0.000 {method 'read' of '_io.BytesIO' objects}
41 0.000 0.000 0.000 0.000 socket.py:729(readable)
41 0.000 0.000 0.000 0.000 {method 'add_done_callback' of '_asyncio.Future' objects}
40 0.000 0.000 0.000 0.000 {method 'find' of 'bytearray' objects}
40 0.000 0.000 0.000 0.000 {method 'result' of '_asyncio.Future' objects}
39 0.000 0.000 0.000 0.000 sre_parse.py:82(groups)
39 0.000 0.000 0.000 0.000 windows_events.py:54(__init__)
39 0.000 0.000 0.000 0.000 windows_events.py:718(_register_with_iocp)
39 0.000 0.000 0.000 0.000 windows_events.py:728(_register)
38 0.000 0.000 0.423 0.011 socket.py:690(readinto)
38 0.000 0.000 0.423 0.011 ssl.py:1091(read)
38 0.000 0.000 0.423 0.011 ssl.py:1231(recv_into)App uses >70% of CPU on RPi 3B+
joeyy-watts
commented
10 months ago If optimizing on Python doesn't pan out, this might be another project completely to re-make this in C or something..
Will pinpoint the root cause first, then we'll see..
joeyy-watts
commented
10 months ago Might use set_fade instead of set_values https://pyartnet.readthedocs.io/en/latest/pyartnet.html, to reduce number of ArtNet packets the need to be sent
joeyy-watts
commented
10 months ago Culprit is probably set_values. Below is cProfile from container in RPi (from like 30-40 seconds of app runtime)
joeyy-watts
commented
9 months ago After running an adhoc testing script on the Pi, outside of container, it's most likely due to Python performance on the Pi.
Will try to do some research and confirm this.
Most likely will have to port this over to a more performant language. A good opportunity for me to learn something new I guess..
joeyy-watts
commented
8 months ago Decided to develop in Rust: https://github.com/joeyy-watts/rustify-wled
In 20231224, the app is now dockerized.
Runs fine on laptop. Running on RPi, the performance is significantly degraded.
Have to pinpoint the cause: