joeyy-watts / SpotifyWLED

A script to listen to the currently playing track on your Spotify account and forward it to a WLED (https://github.com/Aircoookie/WLED) LED matrix with support for animations.
2 stars 0 forks source link

Optimize against Raspberry Pi #27

Closed joeyy-watts closed 8 months ago

joeyy-watts commented 10 months ago

In 20231224, the app is now dockerized.

Runs fine on laptop. Running on RPi, the performance is significantly degraded.

Have to pinpoint the cause:

joeyy-watts commented 10 months ago

probably not network; ping times are pretty much identical between RPi and laptop, even from Docker container

might be a big change to optimize, some ideas:

joeyy-watts commented 10 months ago

Top cProfile (from laptop run), for reference:

Ordered by: call count

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
  2147328    0.348    0.000    0.348    0.000 {built-in method builtins.round}
  1466830    0.210    0.000    0.210    0.000 {method 'to_bytes' of 'int' objects}
  1433964    0.152    0.000    0.152    0.000 {built-in method builtins.hash}
  1433961    0.291    0.000    0.443    0.000 enum.py:774(__hash__)
746538/746471    0.069    0.000    0.069    0.000 {built-in method builtins.len}
734661/734649    0.071    0.000    0.072    0.000 {built-in method builtins.isinstance}
   715776    1.371    0.000    3.124    0.000 channel.py:100(set_values)
   715776    0.243    0.000    0.311    0.000 universe.py:52(get_channel)
   715776    0.203    0.000    0.514    0.000 universe.py:132(__getitem__)
   486604    0.068    0.000    0.068    0.000 background_task.py:30(start)
   486603    0.733    0.000    0.941    0.000 channel.py:129(to_buffer)
   486603    0.332    0.000    1.341    0.000 universe.py:36(channel_changed)
   102509    0.014    0.000    0.014    0.000 {built-in method builtins.max}
    56323    0.015    0.000    0.015    0.000 {built-in method builtins.all}
    56320    0.014    0.000    0.029    0.000 effects_utils.py:7(is_black)
    42174    0.008    0.000    0.008    0.000 {built-in method builtins.min}
    30282    0.005    0.000    0.005    0.000 {built-in method time.monotonic}
    29289    0.005    0.000    0.005    0.000 base_events.py:1923(get_debug)
    24720    0.005    0.000    0.005    0.000 base_events.py:513(_check_closed)
    19877    0.003    0.000    0.003    0.000 {method 'popleft' of 'collections.deque' objects}
    19834    0.004    0.000    0.004    0.000 {method 'append' of 'collections.deque' objects}
    19794    0.021    0.000    0.025    0.000 events.py:31(__init__)
    19785    0.022    0.000    6.372    0.000 events.py:78(_run)
    19785    0.032    0.000    6.350    0.000 {method 'run' of 'Context' objects}
    17887    0.017    0.000    0.067    0.000 base_events.py:741(call_soon)
    17887    0.023    0.000    0.047    0.000 base_events.py:770(_call_soon)
    14570    0.015    0.000    0.058    0.000 tasks.py:633(sleep)
    11620    0.002    0.000    0.002    0.000 {method 'append' of 'list' objects}
    11506    0.007    0.000    0.009    0.000 base_events.py:694(time)
    10760    0.001    0.000    0.001    0.000 tasks.py:621(__sleep0)
     9805    0.001    0.000    0.001    0.000 {method 'cancelled' of '_asyncio.Task' objects}
     9791    0.001    0.000    0.001    0.000 {method 'exception' of '_asyncio.Task' objects}
     8916   19.388    0.002   19.388    0.002 {built-in method _overlapped.GetQueuedCompletionStatus}
     8881    0.003    0.000    0.003    0.000 {built-in method math.ceil}
     8878    0.027    0.000   19.421    0.002 windows_events.py:775(_poll)
     8878    0.002    0.000    0.002    0.000 {method 'clear' of 'list' objects}
     8877    0.079    0.000   25.919    0.003 base_events.py:1830(_run_once)
     8877    0.011    0.000   19.432    0.002 windows_events.py:437(select)
     8876    0.002    0.000    0.002    0.000 proactor_events.py:855(_process_events)
     8770    0.008    0.000    0.009    0.000 events.py:120(__lt__)
     5876    0.002    0.000    0.045    0.000 background_task.py:61(coro_wrap)
     5875    0.016    0.000    0.043    0.000 base_node.py:116(_periodic_refresh_worker)
     4983    0.001    0.000    0.001    0.000 {method 'done' of '_asyncio.Future' objects}
     4967    0.002    0.000    0.002    0.000 {method 'add' of 'set' objects}
     4934    0.006    0.000    0.008    0.000 _weakrefset.py:86(add)
     4919    0.003    0.000    0.004    0.000 _weakrefset.py:39(_remove)
     4919    0.001    0.000    0.001    0.000 {method 'discard' of 'set' objects}
     4918    0.022    0.000    0.049    0.000 base_events.py:431(create_task)
     4910    0.002    0.000    0.002    0.000 coroutines.py:177(iscoroutine)
     4909    0.008    0.000    0.060    0.000 tasks.py:657(ensure_future)
     4906    0.001    0.000    0.001    0.000 {method 'add_done_callback' of '_asyncio.Task' objects}
     4902    0.009    0.000    0.015    0.000 tasks.py:767(_done_callback)
     4896    0.001    0.000    0.001    0.000 {method 'result' of '_asyncio.Task' objects}
     4893    0.836    0.000    4.916    0.001 artnet_handler.py:130(__async_set_pixels)
     4725    0.002    0.000    0.002    0.000 __init__.py:1689(isEnabledFor)
     4223    0.001    0.000    0.001    0.000 {method 'replace' of 'str' objects}
     4072    0.000    0.000    0.000    0.000 {built-in method nt.fspath}
     3910    0.001    0.000    0.001    0.000 {method 'startswith' of 'str' objects}
     3573    0.000    0.000    0.000    0.000 {method 'append' of 'bytearray' objects}
     3510    0.009    0.000    0.122    0.000 base_node.py:70(_send_data)
     3510    0.007    0.000    0.159    0.000 universe.py:47(send_data)
     3510    0.003    0.000    0.003    0.000 seq_counter.py:13(value)
     3510    0.023    0.000    0.152    0.000 node.py:42(_send_universe)
     3510    0.112    0.000    0.112    0.000 {method 'sendto' of '_socket.socket' objects}
     2644    0.008    0.000    0.027    0.000 {method 'set_result' of '_asyncio.Future' objects}
     2641    0.000    0.000    0.000    0.000 {method 'lower' of 'str' objects}
     2209    0.000    0.000    0.000    0.000 {method 'endswith' of 'str' objects}
     1918    0.001    0.000    0.001    0.000 {built-in method _contextvars.copy_context}
     1916    0.004    0.000    0.005    0.000 base_events.py:427(create_future)
     1908    0.003    0.000    0.003    0.000 events.py:64(cancel)
     1908    0.007    0.000    0.011    0.000 events.py:148(cancel)
     1908    0.001    0.000    0.001    0.000 {built-in method _asyncio.get_running_loop}
     1907    0.005    0.000    0.020    0.000 base_events.py:725(call_at)
     1907    0.001    0.000    0.001    0.000 base_events.py:1825(_timer_handle_cancelled)
     1907    0.004    0.000    0.009    0.000 events.py:103(__init__)
     1907    0.003    0.000    0.005    0.000 {built-in method _heapq.heappush}
     1907    0.008    0.000    0.015    0.000 {built-in method _heapq.heappop}
     1905    0.005    0.000    0.026    0.000 base_events.py:703(call_later)
     1902    0.001    0.000    0.001    0.000 {method 'cancelled' of '_asyncio.Future' objects}
     1899    0.004    0.000    0.029    0.000 futures.py:308(_set_result_unless_cancelled)
     1434    0.000    0.000    0.001    0.000 inspect.py:700(<genexpr>)
     1424    0.000    0.000    0.000    0.000 inspect.py:703(<genexpr>)
     1411    0.007    0.000    0.146    0.000 animations.py:51(_main_function)
     1401    0.001    0.000    0.699    0.000 async_utils.py:67(__main_loop)
     1398    0.033    0.000    0.119    0.000 artnet_handler.py:96(set_pixels)
     1341    0.001    0.000    0.002    0.000 ntpath.py:44(normcase)
     1340    0.002    0.000    0.003    0.000 ntpath.py:124(splitdrive)
     1264    0.000    0.000    0.000    0.000 {method 'items' of 'dict' objects}
     1206    0.000    0.000    0.000    0.000 {method 'values' of 'dict' objects}
     1204    0.000    0.000    0.001    0.000 __init__.py:1424(debug)
     1198    0.001    0.000    0.001    0.000 output_correction.py:5(__init__)
     1190    0.000    0.000    0.000    0.000 channel.py:64(<listcomp>)
     1190    0.004    0.000    0.005    0.000 channel.py:29(__init__)
     1190    0.000    0.000    0.000    0.000 channel.py:83(_apply_output_correction)
     1190    0.014    0.000    0.049    0.000 universe.py:65(add_channel)
     1190    0.014    0.000    0.028    0.000 universe.py:106(_resize_universe)
     1142    0.000    0.000    0.000    0.000 {method 'join' of 'str' objects}
     1035    0.000    0.000    0.001    0.000 inspect.py:64(ismodule)
     1001    0.001    0.000    0.001    0.000 {method 'split' of 'str' objects}
      981    0.001    0.000    0.002    0.000 {built-in method builtins.any}
      950    0.000    0.000    0.000    0.000 {method 'lstrip' of 'str' objects}
      895    0.005    0.000    0.009    0.000 ntpath.py:450(normpath)
      888    0.061    0.000    0.061    0.000 {built-in method nt._getfinalpathname}
      765    0.000    0.000    0.000    0.000 {method 'upper' of 'str' objects}
      752    0.000    0.000    0.000    0.000 {built-in method builtins.hasattr}
      744    0.000    0.000    0.001    0.000 _collections_abc.py:849(__iter__)
      720    0.001    0.000    0.001    0.000 futures.py:296(_get_loop)
      720    0.000    0.000    0.000    0.000 {method 'get_loop' of '_asyncio.Future' objects}
      719    0.001    0.000    0.001    0.000 {built-in method _asyncio.get_event_loop}
      703    0.003    0.000    0.003    0.000 tasks.py:701(__init__)
      703    0.014    0.000    0.080    0.000 tasks.py:759(_gather)
      702    0.003    0.000    0.083    0.000 tasks.py:721(gather)
      700    0.000    0.000    0.001    0.000 os.py:674(__getitem__)
      700    0.000    0.000    0.000    0.000 os.py:740(check_str)
      700    0.000    0.000    0.001    0.000 os.py:746(encodekey)
      699    0.003    0.000    0.003    0.000 artnet_handler.py:99(<listcomp>)
      697    0.000    0.000    0.000    0.000 {built-in method builtins.getattr}
      686    0.006    0.000    0.185    0.000 base_node.py:77(_process_values_task)
      686    0.001    0.000    0.185    0.000 background_task.py:45(coro_wrap)
      680    0.000    0.000    0.000    0.000 {method 'get' of 'dict' objects}
      672    0.000    0.000    0.000    0.000 os.py:697(__iter__)
      667    0.000    0.000    0.000    0.000 {built-in method builtins.ord}
      534    0.020    0.000    0.020    0.000 {built-in method nt.stat}
      523    0.000    0.000    0.000    0.000 {method 'decode' of 'bytes' objects}
      499    0.000    0.000    0.001    0.000 inspect.py:655(getfile)
      498    0.000    0.000    0.000    0.000 sre_parse.py:165(__getitem__)
      478    0.002    0.000    0.022    0.000 inspect.py:693(getsourcefile)
      469    0.000    0.000    0.018    0.000 genericpath.py:16(exists)
      451    0.000    0.000    0.000    0.000 ntpath.py:34(_get_bothseps)
      451    0.000    0.000    0.006    0.000 ntpath.py:524(abspath)
      451    0.001    0.000    0.029    0.000 inspect.py:715(getabsfile)
      451    0.001    0.000    0.001    0.000 {built-in method nt._getfullpathname}
      444    0.001    0.000    0.002    0.000 ntpath.py:61(isabs)
      444    0.002    0.000    0.071    0.000 ntpath.py:625(realpath)
      444    0.000    0.000    0.000    0.000 {built-in method nt.getcwd}
      359    0.000    0.000    0.000    0.000 sre_parse.py:234(__next)
      298    0.000    0.000    0.000    0.000 {method 'rstrip' of 'str' objects}
      256    0.000    0.000    0.000    0.000 sre_parse.py:255(get)
   204/19    0.000    0.000    0.000    0.000 abc.py:121(__subclasscheck__)
   204/19    0.000    0.000    0.000    0.000 {built-in method _abc._abc_subclasscheck}
      196    0.000    0.000    0.000    0.000 sre_parse.py:173(append)
  192/188    0.000    0.000    0.000    0.000 {method 'encode' of 'str' objects}
      191    0.000    0.000    0.000    0.000 {method 'match' of 're.Pattern' objects}
      180    0.000    0.000    0.000    0.000 _collections_abc.py:409(__subclasshook__)
      171    0.000    0.000    0.000    0.000 {method 'rpartition' of 'str' objects}
      146    0.002    0.000    0.002    0.000 {built-in method __new__ of type object at 0x00007FFD47A54C60}
      112    0.000    0.000    0.000    0.000 {method 'partition' of 'str' objects}
      111    0.000    0.000    0.000    0.000 sre_parse.py:250(match)
      110    0.000    0.000    0.000    0.000 structures.py:46(__setitem__)
      103    0.000    0.000    0.000    0.000 sre_parse.py:161(__len__)
       98    0.000    0.000    0.351    0.004 {method 'readline' of '_io.BufferedReader' objects}
       98    0.000    0.000    0.000    0.000 {built-in method _imp.acquire_lock}
       98    0.000    0.000    0.000    0.000 {built-in method _imp.release_lock}
       92    0.000    0.000    0.001    0.000 abc.py:117(__instancecheck__)
       92    0.000    0.000    0.001    0.000 {built-in method _abc._abc_instancecheck}
       91    0.000    0.000    0.000    0.000 structures.py:58(<genexpr>)
    88/38    0.000    0.000    0.076    0.002 {method 'join' of 'bytes' objects}
       88    0.000    0.000    0.000    0.000 {method 'pop' of 'dict' objects}
       87    0.000    0.000    0.000    0.000 _collections.py:257(__getitem__)
       86    0.000    0.000    0.000    0.000 parse.py:103(_noop)
       86    0.000    0.000    0.000    0.000 parse.py:114(_coerce_args)
       84    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:231(_verbose_message)
       84    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap_external>:129(<genexpr>)
       82    0.000    0.000    0.000    0.000 ssl.py:1079(_checkClosed)
       80    0.004    0.000    0.004    0.000 {method 'acquire' of '_thread.lock' objects}
       77    0.000    0.000    0.000    0.000 _weakrefset.py:75(__contains__)
       77    0.000    0.000    0.000    0.000 {method '__exit__' of '_thread.lock' objects}
       75    0.000    0.000    0.000    0.000 inspect.py:81(ismethod)
       75    0.000    0.000    0.000    0.000 utils.py:51(_has_surrogates)
       73    0.000    0.000    0.000    0.000 {method 'find' of 'str' objects}
       72    0.000    0.000    0.000    0.000 _policybase.py:281(_sanitize_header)
       72    0.000    0.000    0.000    0.000 _policybase.py:311(header_fetch_parse)
       70    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:874(__enter__)
       70    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:878(__exit__)
       70    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap_external>:114(<listcomp>)
       70    0.000    0.000    0.001    0.000 <frozen importlib._bootstrap_external>:91(_path_join)
    68/20    0.000    0.000    0.000    0.000 sre_parse.py:175(getwidth)
       68    0.000    0.000    0.000    0.000 inspect.py:237(istraceback)
       68    0.000    0.000    0.000    0.000 inspect.py:247(isframe)
       67    0.000    0.000    0.000    0.000 inspect.py:159(isfunction)
       67    0.000    0.000    0.000    0.000 feedparser.py:78(readline)
       66    0.000    0.000    0.000    0.000 structures.py:51(__getitem__)
       66    0.000    0.000    0.000    0.000 {method 'setdefault' of 'dict' objects}
       66    0.000    0.000    0.000    0.000 {built-in method builtins.next}
       65    0.000    0.000    0.000    0.000 _collections_abc.py:760(get)
       65    0.000    0.000    0.000    0.000 feedparser.py:128(__next__)
       64    0.000    0.000    0.000    0.000 {method 'read' of '_io.BufferedReader' objects}
       61    0.000    0.000    0.000    0.000 {method 'rfind' of 'str' objects}
       60    0.000    0.000    0.000    0.000 client.py:442(isclosed)
       60    0.000    0.000    0.000    0.000 {method 'strip' of 'str' objects}
       58    0.000    0.000    0.000    0.000 sre_parse.py:112(__init__)
       58    0.000    0.000    0.000    0.000 sre_parse.py:287(tell)
       58    0.000    0.000    0.000    0.000 response.py:697(_error_catcher)
       57    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:398(parent)
       57    0.000    0.000    0.000    0.000 enum.py:358(__call__)
       57    0.000    0.000    0.000    0.000 enum.py:670(__new__)
       57    0.000    0.000    0.000    0.000 {built-in method builtins.abs}
       57    0.000    0.000    0.000    0.000 {method 'fileno' of '_socket.socket' objects}
     55/6    0.000    0.000    0.001    0.000 sre_compile.py:87(_compile)
       55    0.029    0.001    0.058    0.001 base_effects.py:34(<listcomp>)
       55    0.000    0.000    0.000    0.000 base_effects.py:150(func)
       55    0.000    0.000    0.000    0.000 {built-in method math.sin}
       54    0.000    0.000    0.000    0.000 response.py:244(__len__)
       53    0.000    0.000    0.000    0.000 enum.py:792(value)
       53    0.000    0.000    0.000    0.000 types.py:171(__get__)
       53    0.000    0.000    0.000    0.000 _collections.py:289(__iter__)
       52    0.000    0.000    0.000    0.000 enum.py:438(<genexpr>)
       52    0.000    0.000    0.000    0.000 _policybase.py:293(header_source_parse)
       52    0.000    0.000    0.000    0.000 message.py:479(set_raw)
       52    0.000    0.000    0.000    0.000 response.py:429(_decode)
       52    0.000    0.000    0.000    0.000 {built-in method _thread.allocate_lock}
       51    0.000    0.000    0.000    0.000 inspect.py:73(isclass)
       51    0.000    0.000    0.000    0.000 inspect.py:261(iscode)
       50    0.000    0.000    0.000    0.000 client.py:610(_safe_read)
       50    0.000    0.000    0.000    0.000 _collections.py:300(add)
       49    0.000    0.000    0.000    0.000 codecs.py:319(decode)
       49    0.000    0.000    0.000    0.000 {built-in method _codecs.utf_8_decode}
       48    0.000    0.000    0.000    0.000 threading.py:256(__enter__)
       48    0.000    0.000    0.000    0.000 threading.py:259(__exit__)
       48    0.000    0.000    0.000    0.000 {method '__enter__' of '_thread.lock' objects}
       47    0.000    0.000    0.000    0.000 threading.py:271(_is_owned)
       47    0.000    0.000    0.000    0.000 _url.py:534(raw_path)
       47    0.000    0.000    0.000    0.000 {method 'search' of 're.Pattern' objects}
       47    0.000    0.000    0.000    0.000 {built-in method _thread.get_ident}
       45    0.000    0.000    0.000    0.000 util.py:19(to_str)
       43    0.000    0.000    0.000    0.000 windows_events.py:423(_check_closed)
       43    0.000    0.000    0.000    0.000 {built-in method from_bytes}
       43    0.000    0.000    0.000    0.000 {method 'format' of 'str' objects}
       42    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap_external>:79(_unpack_uint32)
       42    0.000    0.000    0.002    0.000 <frozen importlib._bootstrap_external>:135(_path_stat)
       42    0.000    0.000    0.000    0.000 web_urldispatcher.py:364(resolve)
       42    0.000    0.000    0.000    0.000 web_urldispatcher.py:413(_match)
       42    0.000    0.000    0.000    0.000 {method 'read' of '_io.BytesIO' objects}
       41    0.000    0.000    0.000    0.000 socket.py:729(readable)
       41    0.000    0.000    0.000    0.000 {method 'add_done_callback' of '_asyncio.Future' objects}
       40    0.000    0.000    0.000    0.000 {method 'find' of 'bytearray' objects}
       40    0.000    0.000    0.000    0.000 {method 'result' of '_asyncio.Future' objects}
       39    0.000    0.000    0.000    0.000 sre_parse.py:82(groups)
       39    0.000    0.000    0.000    0.000 windows_events.py:54(__init__)
       39    0.000    0.000    0.000    0.000 windows_events.py:718(_register_with_iocp)
       39    0.000    0.000    0.000    0.000 windows_events.py:728(_register)
       38    0.000    0.000    0.423    0.011 socket.py:690(readinto)
       38    0.000    0.000    0.423    0.011 ssl.py:1091(read)
       38    0.000    0.000    0.423    0.011 ssl.py:1231(recv_into)
joeyy-watts commented 10 months ago

App uses >70% of CPU on RPi 3B+ image

joeyy-watts commented 10 months ago

If optimizing on Python doesn't pan out, this might be another project completely to re-make this in C or something..

Will pinpoint the root cause first, then we'll see..

joeyy-watts commented 10 months ago

Might use set_fade instead of set_values https://pyartnet.readthedocs.io/en/latest/pyartnet.html, to reduce number of ArtNet packets the need to be sent

joeyy-watts commented 10 months ago

Culprit is probably set_values. Below is cProfile from container in RPi (from like 30-40 seconds of app runtime) image

joeyy-watts commented 9 months ago

After running an adhoc testing script on the Pi, outside of container, it's most likely due to Python performance on the Pi.

Will try to do some research and confirm this.

Most likely will have to port this over to a more performant language. A good opportunity for me to learn something new I guess..

joeyy-watts commented 8 months ago

Decided to develop in Rust: https://github.com/joeyy-watts/rustify-wled