search

bigretromike / nakamori

Nakamori is Kodi addon that use Shoko (known as Japanese Media Manager (JMM)) Server as back-end for metadata information.
https://shokunin.monogatari.pl/nakamori/
49 stars 5 forks source link

Profiling Results #217

Closed da3dsoul closed 7 years ago

da3dsoul commented 7 years ago

I ran a profiler, and these are the results. It looks like tag parsing takes the most time. It might be better to move it to server.

ncalls tottime percall cumtime percall filename:lineno(function)
12951 1.737 0 3.229 0 TagBlacklist.py:132(processTags)
729 0.908 0.001 0.908 0.001 {method 'recv' of '_socket.socket' objects}
112 0.799 0.007 0.799 0.007 decoder.py:370(raw_decode)
1355 0.654 0 0.654 0 {method 'setArt' of 'xbmcgui.ListItem' objects}
1532113 0.606 0 0.606 0 {method 'startswith' of 'str' objects}
1153515 0.41 0 0.41 0 {method 'endswith' of 'str' objects}
399241 0.354 0 0.354 0 {method 'lower' of 'str' objects}
2 0.142 0.071 0.142 0.071 {method 'close' of 'xbmcgui.DialogProgress' objects}
411970 0.11 0 0.11 0 {method 'strip' of 'str' objects}
198 0.055 0 3.356 0.017 nakamori.py:528(get_tags)
198 0.047 0 0.047 0 {method 'setInfo' of 'xbmcgui.ListItem' objects}
2 0.041 0.021 0.041 0.021 {method 'create' of 'xbmcgui.DialogProgress' objects}
108 0.029 0 0.029 0 {xbmc.executeJSONRPC}
2 0.023 0.012 1.357 0.679 util.py:182(get_data)
13547 0.023 0 0.023 0 {method 'encode' of 'unicode' objects}
13547 0.02 0 0.053 0 util.py:317(encode)
13167 0.016 0 0.016 0 {method 'join' of 'str' objects}
39230 0.014 0 0.014 0 {isinstance}
753 0.012 0 0.012 0 {method 'write' of 'cStringIO.StringO' objects}
152 0.012 0 2.488 0.016 nakamori.py:700(add_serie_item)
9867 0.011 0 0.011 0 {method 'remove' of 'list' objects}
198 0.008 0 0.724 0.004 nakamori.py:248(add_gui_item)
398 0.007 0 0.007 0 {method 'sub' of '_sre.SRE_Pattern' objects}
198 0.006 0 0.006 0 {method 'addContextMenuItems' of 'xbmcgui.ListItem' objects}
4 0.004 0.001 0.004 0.001 {method 'getvalue' of 'cStringIO.StringO' objects}
1238 0.004 0 0.011 0 util.py:497(set_parameter)
729/2 0.004 0 0.044 0.022 socket.py:340(read)
198 0.004 0 0.004 0 {xbmcplugin.addDirectoryItem}
1 0.003 0.003 6.104 6.104 nakamori.py:1133(build_groups_menu)
642 0.003 0 0.003 0 {method 'setProperty' of 'xbmcgui.ListItem' objects}
7537 0.003 0 0.003 0 {method 'get' of 'dict' objects}
19630 0.003 0 0.003 0 {method 'append' of 'list' objects}
46 0.003 0 1.673 0.036 nakamori.py:869(add_group_item)
1192 0.002 0 0.005 0 urllib.py:1262(quote)
729 0.002 0 0.026 0 httplib.py:564(read)
198 0.002 0 0.002 0 {map}
1150 0.002 0 0.002 0 {method 'getSetting' of 'xbmcaddon.Addon' objects}
900 0.002 0 0.002 0 util.py:88(safeInt)
1044 0.001 0 0.001 0 {method 'split' of 'str' objects}
242 0.001 0 0.001 0 {method 'match' of '_sre.SRE_Pattern' objects}
777 0.001 0 0.001 0 {method 'seek' of 'cStringIO.StringO' objects}
396 0.001 0 0.002 0 re.py:230(_compile)
1192 0.001 0 0.007 0 urllib.py:1301(quote_plus)
108 0.001 0 0.034 0 util.py:436(get_kodi_setting_bool)
1 0.001 0.001 0.001 0.001 {xbmcgui.getCurrentWindowId}
198 0.001 0 0.01 0 util.py:60(remove_anidb_links)
203 0.001 0 0.001 0 {method 'update' of 'xbmcgui.DialogProgress' objects}
112 0.001 0 0.801 0.007 decoder.py:359(decode)
198 0.001 0 0.002 0 nakamori.py:472(get_title)
1192 0.001 0 0.001 0 {method 'rstrip' of 'str' objects}
4842/4837 0.001 0 0.001 0 {len}
9866 0.001 0 0.001 0 {method 'add' of 'set' objects}
2 0 0 0.89 0.445 urllib2.py:1152(do_open)
2 0 0 0 0 httplib.py:785(set_debuglevel)
2 0 0 0 0 urllib.py:1054(_is_unicode)
4 0 0 0 0 urllib2.py:314(add_header)
2 0 0 0 0 httplib.py:1044(_set_content_length)
152 0 0 0 0 nakamori.py:559(get_cast_and_role)
4 0 0 0.89 0.223 urllib2.py:399(_call_chain)
2 0 0 0.001 0 httplib.py:818(connect)
2 0 0 0.001 0 rfc822.py:88(init)
2 0 0 0 0 urllib2.py:293(get_selector)
20 0 0 0 0 httplib.py:257(addheader)
4 0 0 0 0 urllib2.py:322(has_header)
1 0 0 0 0 {xbmcplugin.endOfDirectory}
2 0 0 0 0 urllib.py:958(init)
2 0 0 0 0 httplib.py:364(init)
2 0 0 0 0 {method 'sendall' of '_socket.socket' objects}
24 0 0 0 0 sre_parse.py:149(append)
14 0 0 0 0 sre_parse.py:137(len)
22 0 0 0 0 {method 'readline' of 'cStringIO.StringO' objects}
2 0 0 0 0 urllib2.py:305(has_proxy)
4 0 0 0 0 socket.py:289(del)
2 0 0 0.891 0.445 urllib2.py:131(urlopen)
2 0 0 0 0 sre_parse.py:706(parse)
42920 0 0 0 0 sre_parse.py:151(getwidth)
4 0 0 0 0 {method 'extend' of 'list' objects}
41 0 0 0 0 sre_parse.py:212(get)
4 0 0 0 0 urllib2.py:267(has_data)
2 0 0 0.891 0.445 urllib2.py:411(open)
2 0 0 0.888 0.444 httplib.py:392(_read_status)
2 0 0 0 0 {method 'settimeout' of '_socket.socket' objects}
14 0 0 0 0 rfc822.py:290(getheader)
753 0 0 0 0 {method 'tell' of 'cStringIO.StringO' objects}
24 0 0 0 0 {setattr}
4 0 0 0 0 urllib2.py:279(get_type)
17 0 0 0 0 {method 'split' of 'unicode' objects}
20 0 0 0 0 rfc822.py:202(isheader)
1 0 0 0 0 sre_parse.py:74(opengroup)
24 0 0 0.889 0.037 socket.py:410(readline)
2 0 0 0.001 0 sre_compile.py:567(compile)
1 0 0 0 0 sre_parse.py:85(closegroup)
14 0 0 0 0 httplib.py:860(_output)
2 0 0 0 0 sre_compile.py:552(_code)
2 0 0 0 0 urllib2.py:208(request_host)
4 0 0 0 0 urllib2.py:318(add_unredirected_header)
12 0 0 0 0 {method 'search' of '_sre.SRE_Pattern' objects}
2 0 0 0.001 0 mimetools.py:24(init)
2 0 0 0 0 {method 'connect' of '_socket.socket' objects}
2 0 0 0.889 0.445 httplib.py:431(begin)
327 0 0 0 0 {method 'replace' of 'str' objects}
23 0 0 0 0 {ord}
6 0 0 0 0 {method 'group' of '_sre.SRE_Match' objects}
2 0 0 0 0 sre_parse.py:216(tell)
2 0 0 0 0 {method 'upper' of 'str' objects}
3 0 0 0 0 sre_compile.py:256(_optimize_charset)
11 0 0 0 0 {method 'find' of 'bytearray' objects}
4 0 0 0 0 socket.py:296(flush)
7 0 0 0 0 sre_parse.py:92(init)
51 0 0 0 0 sre_parse.py:193(__next)
3 0 0 0 0 sre_compile.py:228(_compile_charset)
1 0 0 0 0 {xbmcplugin.setContent}
198 0 0 0 0 nakamori.py:456(set_watch_flag)
2 0 0 0 0 httplib.py:517(_check_close)
2 0 0 0 0 socket.py:196(close)
2 0 0 0 0 urllib2.py:226(init)
22 0 0 0 0 {method 'read' of 'cStringIO.StringO' objects}
729 0 0 0 0 {max}
48 0 0 0 0 {method 'isnumeric' of 'unicode' objects}
757 0 0 0 0 {cStringIO.StringIO}
4 0 0 0 0 socket.py:250(init)
2 0 0 0 0 sre_compile.py:428(_simple)
8 0 0 0 0 {method 'items' of 'dict' objects}
2 0 0 0.001 0 socket.py:541(create_connection)
12 0 0 0 0 httplib.py:1005(putheader)
42857 0 0 0 0 sre_compile.py:64(_compile)
22 0 0 0 0 sre_parse.py:141(getitem)
2 0 0 0 0 urllib.py:1069(unwrap)
2 0 0 0.891 0.445 urllib2.py:439(_open)
2 0 0 0 0 httplib.py:721(init)
4 0 0 0 0 {method 'index' of 'str' objects}
2 0 0 0 0 httplib.py:826(close)
2 0 0 0 0 {_sre.compile}
2 0 0 0.399 0.2 util.py:132(parse_possible_error)
2 0 0 0 0 :8(new)
42827 0 0 0 0 sre_parse.py:395(_parse)
2 0 0 0 0 {built-in method fromkeys}
2 0 0 0 0 httplib.py:766(_get_hostport)
2 0 0 0.889 0.445 httplib.py:1084(getresponse)
396 0 0 0.002 0 re.py:192(compile)
212 0 0 0 0 {min}
2 0 0 0 0 urlparse.py:168(urlsplit)
6 0 0 0.001 0 socket.py:227(meth)
4 0 0 0 0 urllib2.py:286(get_host)
112 0 0 0.801 0.007 init.py:294(loads)
2 0 0 0 0 mimetools.py:33(parsetype)
2 0 0 0 0 urllib.py:1183(splittag)
2 0 0 0 0 urllib2.py:256(get_method)
2 0 0 0 0 urllib2.py:541(http_response)
2 0 0 0 0 httplib.py:888(putrequest)
3 0 0 0 0 {method 'update' of 'dict' objects}
2 0 0 0.001 0 httplib.py:1040(request)
2 0 0 1.357 0.679 util.py:171(get_json)
6 0 0 0 0 urllib2.py:1171()
30 0 0 0 0 {method 'find' of 'str' objects}
2 0 0 0 0 urllib.py:1017(init)
12 0 0 0 0 {hasattr}
2 0 0 0.001 0 httplib.py:1067(_send_request)
4 0 0 0 0 {method 'rfind' of 'str' objects}
2 0 0 0 0 urllib2.py:1123(dorequest)
4 0 0 0 0 sre_compile.py:546(isstring)
19 0 0 0 0 sre_parse.py:206(match)
4 0 0 0 0 {built-in method new of type object at 0x0FDF3318}
2 0 0 0 0 socket.py:216(makefile)
4 0 0 0 0 urllib2.py:273(get_full_url)
2 0 0 0 0 socket.py:189(init)
2 0 0 0 0 urllib.py:976(close)
4 0 0 0 0 httplib.py:547(close)
10 0 0 0 0 {method 'title' of 'str' objects}
12 0 0 0 0 urllib2.py:1182()
2 0 0 0 0 urlparse.py:137(urlparse)
2 0 0 0.001 0 httplib.py:840(send)
1 0 0 0.001 0.001 nakamori.py:145(set_window_heading)
2 0 0 0.001 0 httplib.py:1025(endheaders)
2 0 0 0 0 sre_parse.py:189(init)
22 0 0 0 0 rfc822.py:224(iscomment)
2 0 0 0 0 urlparse.py:160(_splitnetloc)
2 0 0 0 0 {_socket.getaddrinfo}
198 0 0 0 0 {xbmcplugin.setPluginCategory}
22 0 0 0 0 rfc822.py:214(islast)
24 0 0 0 0 {getattr}
4 0 0 0 0 {range}
2 0 0 0 0 sre_parse.py:145(setitem)
2 0 0 0 0 sre_parse.py:268(_escape)
2 0 0 0 0 {method 'setProperty' of 'xbmcgui.Window' objects}
2 0 0 0 0 sre_compile.py:433(_compile_info)
224 0 0 0 0 {method 'end' of '_sre.SRE_Match' objects}
8 0 0 0 0 socket.py:280(close)
10 0 0 0 0 {method 'capitalize' of 'str' objects}
2 0 0 0.001 0 httplib.py:271(readheaders)
2 0 0 0 0 mimetools.py:50(parseplist)
1 0 0 0 0 {method 'copy' of 'dict' objects}
42796 0 0 0 0 sre_parse.py:317(_parse_sub)
4 0 0 0 0 urllib.py:1023(info)
2 0 0 0 0 urllib.py:1078(splittype)
2 0 0 0 0 urllib.py:1092(splithost)
2 0 0 0.89 0.445 urllib2.py:1227(http_open)
2 0 0 0 0 sre_parse.py:67(init)
2 0 0 0 0 urllib.py:1221(unquote)
2 0 0 0 0 {method 'iteritems' of 'dict' objects}
1 0 0 0 0 {method 'disable' of '_lsprof.Profiler' objects}
2 0 0 0.001 0 httplib.py:867(_send_output)
bigretromike commented 7 years ago

or make it quicker :-) also server wont benefit from it, only 3rd party that use this (nakamori, and that plex plugin - if he still use this ) Maybe by default we turn this off if it really THAT slow.

There are many slow function.. also I see the glory of your profiler here ;p not sure about all colums but no-time-for-that-right-now 👍

bigretromike commented 7 years ago

Also we could cut our most lower or replace startswith with string[tag.lenght:] ==tag or something like that. Also there is few parts that I would like to rewrite because I cannot look at that mess in code :-) and maybe then there will be place to use some python-speed-hacks

da3dsoul commented 7 years ago

I mean that c# compiles with faster string operations than Python

bigretromike commented 7 years ago

yeah, Python ain't the fastest but it not Java also ;-)

da3dsoul commented 7 years ago

You'd be surprised, Java string operations and enumerables are actually pretty fast

da3dsoul commented 7 years ago

It's cleaning up after that it'll have issues with

bigretromike commented 7 years ago

I'm more than sure that there are some code hacks we could use to make it quicker. But if you think that Shoko would benefit from this then maybe it should be on server end... cant tell. im not sure, also it would take ages until we would have it there in release

While Im busy at work, I will try to take a look at that profiler you added and start cleaning up and refactoring code

da3dsoul commented 7 years ago

It would take like a day to get released. It's almost done and we're planning a .1

bigretromike commented 7 years ago

If it's so fast then we can move this part out cazzar would also enjoy it ;-)

On September 6, 2017 3:37:29 PM GMT+02:00, da3dsoul notifications@github.com wrote:

It would take like a day to get released. It's almost done and we're planning a .1

-- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/bigretromike/nakamori/issues/217#issuecomment-327485474

-- Sent from my Android device with K-9 Mail. Please excuse my brevity.

da3dsoul commented 7 years ago

TBH, another major reason is that I am familiar with C# multiprocessing, but not Python....

da3dsoul commented 7 years ago

1.5 million operations should be split up

bigretromike commented 7 years ago

No problem, it was here because you made it here and it was great why not intercorporate it into bigger app so not only us will benefit it :-) of course only if it won't add bigger delay in the end or if you adding it as a tag cleaning took the there won't be no overhead as now as we lost and clean all records.

On September 6, 2017 3:54:28 PM GMT+02:00, da3dsoul notifications@github.com wrote:

TBH, another major reason is that I am familiar with C# multiprocessing, but not Python....

-- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/bigretromike/nakamori/issues/217#issuecomment-327490468

-- Sent from my Android device with K-9 Mail. Please excuse my brevity.