search

jwons / MultilingualProvenanceDebugger

GNU General Public License v3.0
2 stars 1 forks source link

Possible Performance Degradation when ~/.history file is large #3

Open jyoo980 opened 3 years ago

jyoo980 commented 3 years ago

cc: @Alison-Li

This might be a known issue or something unique to me, but I thought I'd document it anyway. After using provdebug a number of times, I noticed a considerable slowdown on the CLI. I had to wait > 10 sec for steps to be processed.

Some rudimentary profiling with the Activity Monitor on macOS showed me that provdb was consuming at least 99% of the CPU during these slow runtimes.

After investigating, I found the bottleneck to be the following lines of code in pvdebug.py

    while(True):
        try: readline.read_history_file()
        except: pass
        userInput = input(">")
        try: readline.write_history_file()

When the ~/.history file gets too large, there appears to be a considerable slowdown, since the Python process is effectively doing a very expensive disk IO operation on each loop iteration.

I "solved" this (rather a workaround) by deleting the ~/.history file, since I don't need it right now.

Specifications

alison-li commented 3 years ago

Just keeping this documented here as well. The sequence of traversal commands I've executed for an example PROV-JSON has just been n, s, n, n, b but the ~/.history file contains the following:

n
n
s
n
n
s
n
n
n
s
n
n
s
n
n
n
s
n
n
s
n
n
n
s
n
n
s
n
b
alison-li commented 3 years ago

n, s, help, l rsf, q

n
n
s
n
n
s
help
n
n
s
n
n
s
help
l rsf
n
n
s
n
n
s
help
n
n
s
n
n
s
help
l rsf
q
alison-li commented 3 years ago

So I think I might have fixed it? Just by moving the operation for reading the history file outside of the while loop, so now it looks like this:

try: readline.read_history_file()
except: pass

while(True):
        userInput = input(">")
        try: readline.write_history_file()
        except: pass

And now the following traversal commands are reflected exactly in the ~/.history file if that was meant to be the original intention? n, s, help, q

n
s
help
q