search

andgineer / lexiflux

AI-powered foreign text reader for language learners (Django)
1 stars 0 forks source link

Optimize EPUB import #71

Closed andgineer closed 6 days ago

andgineer commented 2 months ago

War and peace imported more than minute Interesting that from django command that takes only 5 seconds

andgineer commented 1 week ago

is is slow only in SSL mode with RunServerPlus from django_extensions and all the time is spent in BookLoaderEpub.create

andgineer commented 1 week ago

running with SSL

     40803912 function calls (40767430 primitive calls) in 9.355 seconds

Ordered by: cumulative time ncalls tottime percall cumtime percall filename:lineno(function) 136/133 0.001 0.000 20.268 0.152 /Users/andrei_sorokin2/projects/lexiflux/.venv/lib/python3.12/site-packages/watchdog/observers/api.py:378(dispatch_events) 796/202 0.004 0.000 16.845 0.083 /Users/andrei_sorokin2/projects/lexiflux/.venv/lib/python3.12/site-packages/django/db/models/manager.py:85(manager_method) 136/134 0.000 0.000 15.278 0.114 /Users/andrei_sorokin2/projects/lexiflux/.venv/lib/python3.12/site-packages/watchdog/events.py:342(dispatch) 134 0.001 0.000 12.363 0.092 /Users/andrei_sorokin2/projects/lexiflux/.venv/lib/python3.12/site-packages/watchdog/utils/patterns.py:81(match_any_paths) 48953 0.023 0.000 11.353 0.000 /Users/andrei_sorokin2/projects/lexiflux/.venv/lib/python3.12/site-packages/watchdog/utils/patterns.py:45() 48686 0.096 0.000 9.370 0.000 /Users/andrei_sorokin2/.pyenv/versions/3.12.4/lib/python3.12/pathlib.py:779(match) 199 0.001 0.000 8.677 0.044 /Users/andrei_sorokin2/projects/lexiflux/.venv/lib/python3.12/site-packages/django/db/models/query.py:663(create) 196 0.001 0.000 8.258 0.042 /Users/andrei_sorokin2/projects/lexiflux/lexiflux/ebook/book_loader_epub.py:92(pages) 267 0.002 0.000 7.771 0.029 /Users/andrei_sorokin2/projects/lexiflux/.venv/lib/python3.12/site-packages/watchdog/utils/patterns.py:24(_match_path) 48686 0.316 0.000 7.195 0.000 /Users/andrei_sorokin2/.pyenv/versions/3.12.4/lib/python3.12/pathlib.py:112(_compile_pattern_lines) 195 0.001 0.000 6.738 0.035 /Users/andrei_sorokin2/projects/lexiflux/lexiflux/models.py:299(save) 630002 2.430 0.000 6.277 0.000 /Users/andrei_sorokin2/.pyenv/versions/3.12.4/lib/python3.12/fnmatch.py:74(translate) 199 0.001 0.000 5.826 0.029 /Users/andrei_sorokin2/projects/lexiflux/.venv/lib/python3.12/site-packages/django/db/models/base.py:820(save) 199 0.010 0.000 5.239 0.026 /Users/andrei_sorokin2/projects/lexiflux/.venv/lib/python3.12/site-packages/django/db/models/base.py:951(save_base) 2586/153 0.004 0.000 5.133 0.034 {built-in method builtins.any} 133 0.001 0.000 4.931 0.037 /Users/andrei_sorokin2/projects/lexiflux/.venv/lib/python3.12/site-packages/watchdog/utils/patterns.py:48(filter_paths) 195 0.003 0.000 4.111 0.021 /Users/andrei_sorokin2/projects/lexiflux/lexiflux/ebook/book_loader_epub.py:334(clear_html) 199 0.001 0.000 3.667 0.018 /Users/andrei_sorokin2/projects/lexiflux/.venv/lib/python3.12/site-packages/django/db/models/base.py:1070(_save_table) 199 0.002 0.000 2.786 0.014 /Users/andrei_sorokin2/projects/lexiflux/.venv/lib/python3.12/site-packages/django/db/models/base.py:1196(_do_insert) 5892286 1.215 0.000 2.755 0.000 /Users/andrei_sorokin2/.pyenv/versions/3.12.4/lib/python3.12/re/init.py:255(escape) 596 0.003 0.000 2.661 0.004 /Users/andrei_sorokin2/projects/lexiflux/.venv/lib/python3.12/site-packages/django/db/backends/utils.py:120(execute)

without ssl 4827480 function calls (4786892 primitive calls) in 1.618 seconds Ordered by: cumulative time ncalls tottime percall cumtime percall filename:lineno(function) 197/196 0.002 0.000 1.297 0.007 /Users/andrei_sorokin2/projects/lexiflux/lexiflux/ebook/book_loader_epub.py:92(pages) 797/202 0.002 0.000 1.031 0.005 /Users/andrei_sorokin2/projects/lexiflux/.venv/lib/python3.12/site-packages/django/db/models/manager.py:85(manager_method) 589 0.003 0.000 0.650 0.001 /Users/andrei_sorokin2/projects/lexiflux/.venv/lib/python3.12/site-packages/bs4/init.py:122(init) 589 0.001 0.000 0.625 0.001 /Users/andrei_sorokin2/projects/lexiflux/.venv/lib/python3.12/site-packages/bs4/init.py:471(_feed) 589 0.001 0.000 0.596 0.001 /Users/andrei_sorokin2/projects/lexiflux/.venv/lib/python3.12/site-packages/bs4/builder/_htmlparser.py:372(feed) 589 0.000 0.000 0.590 0.001 /Users/andrei_sorokin2/.pyenv/versions/3.12.4/lib/python3.12/html/parser.py:104(feed) 1178 0.053 0.000 0.588 0.000 /Users/andrei_sorokin2/.pyenv/versions/3.12.4/lib/python3.12/html/parser.py:134(goahead) 702 0.001 0.000 0.480 0.001 /Users/andrei_sorokin2/.pyenv/versions/3.12.4/lib/python3.12/pathlib.py:1228(resolve) 7878 0.003 0.000 0.445 0.000 /Users/andrei_sorokin2/projects/lexiflux/.venv/lib/python3.12/site-packages/bs4/element.py:1657(unicode) 195 0.003 0.000 0.418 0.002 /Users/andrei_sorokin2/projects/lexiflux/lexiflux/ebook/book_loader_epub.py:334(clear_html) 7878 0.051 0.000 0.373 0.000 /Users/andrei_sorokin2/projects/lexiflux/.venv/lib/python3.12/site-packages/bs4/element.py:1689(decode) 17716 0.053 0.000 0.336 0.000 /Users/andrei_sorokin2/.pyenv/versions/3.12.4/lib/python3.12/html/parser.py:301(parse_starttag) 200/199 0.001 0.000 0.321 0.002 /Users/andrei_sorokin2/projects/lexiflux/.venv/lib/python3.12/site-packages/django/db/models/query.py:663(create) 196/195 0.000 0.000 0.273 0.001 /Users/andrei_sorokin2/projects/lexiflux/lexiflux/models.py:299(save) 394 0.001 0.000 0.218 0.001 /Users/andrei_sorokin2/projects/lexiflux/.venv/lib/python3.12/site-packages/bs4/init.py:777(decode) 4298 0.004 0.000 0.215 0.000 /Users/andrei_sorokin2/projects/lexiflux/.venv/lib/python3.12/site-packages/bs4/element.py:2013(find_all) 17716 0.018 0.000 0.213 0.000 /Users/andrei_sorokin2/projects/lexiflux/.venv/lib/python3.12/site-packages/bs4/builder/_htmlparser.py:105(handle_starttag) 195 0.001 0.000 0.209 0.001 /Users/andrei_sorokin2/projects/lexiflux/lexiflux/ebook/book_loader_epub.py:221(_process_anchors) 4298 0.016 0.000 0.209 0.000 /Users/andrei_sorokin2/projects/lexiflux/.venv/lib/python3.12/site-packages/bs4/element.py:792(_find_all) 17716 0.024 0.000 0.186 0.000 /Users/andrei_sorokin2/projects/lexiflux/.venv/lib/python3.12/site-packages/bs4/init.py:722(handle_starttag) 199 0.001 0.000 0.167 0.001 /Users/andrei_sorokin2/projects/lexiflux/.venv/lib/python3.12/site-packages/django/db/models/base.py:820(save)