@ultrabug
I noticed another challenge when creating a PR a few days ago, so I am issuing this PR. Please review it.
description
I improved performance of reconfigure_search_duplicates and fix (maybe) a bug.
effects of Improvements
Bulid took 9m47.324s so far, but takes 0m12.866s after this improment.
# generate many files
$ ./copy.sh
$ time mkdocs build
INFO - mkdocs_static_i18n: Adding 'ja' to the 'search' plugin 'lang' option
INFO - Cleaning site directory
INFO - Building documentation to directory: /home/yosuke/work/git/sample-mkdocs-prj/site
INFO - mkdocs_static_i18n: Building 'ja' documentation to directory: /home/yosuke/work/git/sample-mkdocs-prj/site
INFO - Documentation built in 6.30 seconds
INFO - Documentation built in 587.09 seconds
real 9m47.324s
user 9m47.103s
sys 0m0.220s
$ time mkdocs build
INFO - Cleaning site directory
INFO - Building documentation to directory: /home/yosuke/work/git/sample-mkdocs-prj/site
INFO - mkdocs_static_i18n: Building 'ja' documentation to directory: /home/yosuke/work/git/sample-mkdocs-prj/site
INFO - Documentation built in 6.31 seconds
INFO - Documentation built in 12.64 seconds
real 0m12.866s
user 0m12.641s
sys 0m0.220s
The first number in the line is the percentage of the total execution time of the function. As you can see, reconfigure_search_duplicates function takes too much time.
Then, I logged with line_profiler. See following result.
$ python3 -m line_profiler -rmt "__main__.py.lprof"
Timer unit: 1e-06 s
Total time: 638.575 s
File: /home/yosuke/work/git/mkdocs-static-i18n/mkdocs_static_i18n/reconfigure.py
Function: reconfigure_search_duplicates at line 449
Line # Hits Time Per Hit % Time Line Contents
==============================================================
449 @profile
450 def reconfigure_search_duplicates(self, search_index_entries):
451 """
452 We want to avoid indexing the same pages twice if the default language
453 has its own version built as well as the /language version too as this
454 would pollute the search results.
455 When this happens, we favor the default language location if its
456 content is the same as its /language counterpart.
457 """
458 2 2.1 1.1 0.0 default_lang_entries = filter(
459 1 1.0 1.0 0.0 lambda x: not x["location"].startswith(
460 tuple([f"{lang}/" for lang in self.build_languages])
461 ),
462 1 0.1 0.1 0.0 search_index_entries,
463 )
464 2 110505.5 55252.8 0.0 target_lang_entries = list(
465 2 0.8 0.4 0.0 filter(
466 1 0.3 0.3 0.0 lambda x: x["location"].startswith(
467 tuple([f"{lang}/" for lang in self.build_languages])
468 ),
469 1 0.1 0.1 0.0 search_index_entries,
470 )
471 )
472 7526 292071.2 38.8 0.0 for default_lang_entry in default_lang_entries:
473 15050 8913.7 0.6 0.0 duplicated_entries = filter(
474 7525 3248.7 0.4 0.0 lambda x: x["title"] == default_lang_entry["title"]
475 and x["location"].endswith(x["location"])
476 and x["text"] == default_lang_entry["text"],
477 7525 1033.0 0.1 0.0 target_lang_entries,
478 )
479 2182249 29943908.1 13.7 4.7 for duplicated_entry in duplicated_entries:
480 2174724 605155199.4 278.3 94.8 if duplicated_entry in search_index_entries:
481 15048 245214.8 16.3 0.0 log.debug(
482 15048 5767.5 0.4 0.0 f"removed duplicated search entry: {duplicated_entry['title']} "
483 7524 1247.3 0.2 0.0 f"{duplicated_entry['location']}"
484 )
485 7524 2807696.3 373.2 0.4 search_index_entries.remove(duplicated_entry)
638.57 seconds - /home/yosuke/work/git/mkdocs-static-i18n/mkdocs_static_i18n/reconfigure.py:449 - reconfigure_search_duplicates
The following two points are important.
x["location"].endswith(x["location"]) is always True and seems a mistake. Maybe this indends x["location"].endswith(default_lang_entry["location"]).
if duplicated_entry in search_index_entries: took much time.
How to improve
x["location"].endswith(x["location"]) maybe intend x["location"].endswith(default_lang_entry["location"]). So I fix as intended.
duplicated_entry is removed from target_lang_entries. if duplicated_entry in search_index_entries: is unnecessary now.
@ultrabug I noticed another challenge when creating a PR a few days ago, so I am issuing this PR. Please review it.
description
I improved performance of
reconfigure_search_duplicates
and fix (maybe) a bug.effects of Improvements
Bulid took 9m47.324s so far, but takes 0m12.866s after this improment.
This is logged with following env.
root cause
reconfigure_search_duplicates
takes too much time. See following result.The first number in the line is the percentage of the total execution time of the function. As you can see,
reconfigure_search_duplicates
function takes too much time.Then, I logged with line_profiler. See following result.
The following two points are important.
x["location"].endswith(x["location"])
is always True and seems a mistake. Maybe this indendsx["location"].endswith(default_lang_entry["location"])
.if duplicated_entry in search_index_entries:
took much time.How to improve
x["location"].endswith(x["location"])
maybe intendx["location"].endswith(default_lang_entry["location"])
. So I fix as intended.if duplicated_entry in search_index_entries:
is unnecessary now.