Post Results times out

[ercjns]

At WIOL champs something caused the system to stop updating around 12:20. Pages could still be loaded, but new results were not updated.

On site investigation determined that the cause was the post results call timing out, but more investigation is needed.

[ercjns]

Adding timings revealed a bunch of database writes that could be batched, which sped things up significantly as done in 3567b81.

On my desktop, the time to complete the request with a full results file dropped from ~10s to ~1s. At this point about half of the time (0.5s) is spent on getRunners reading/parsing the file into python, with the remainder (0.5s) to load the database and perform placement and score calculations.

On the pi, things are a bit different. The before timing was much longer, estimated at about 25 seconds (not profiled). The current timing is about 10 seconds, with about 7 to 8 seconds spent in the getRunners step.

Need to investigate possible alternatives for BeautifulSoup and lxml to see if this step can be improved.

[ercjns]

defusedXML using the builtin eTree parser rather than beautifulsoup4 shows nearly 10x improvement in parsing. Below is output from testing 76dd0c2 directly on the pi by sending an example XML with 313 results, typical for the middle to end of a large COC event.

pi@raspberrypi:~/code/cocwifi $ git checkout master
Switched to branch 'master'
Your branch is up-to-date with 'origin/master'.
pi@raspberrypi:~/code/cocwifi $ python rundevserver.py
WebSocket transport not available. Install eventlet or gevent and gevent-websocket for improved performance.
 * Running on http://0.0.0.0:8080/ (Press CTRL+C to quit)
7.0519s for getRunners
0.4534s for buildDB
0.4289s for assignPos
0.5587s for assignScore
0.3458s for teamScore
0.2693s for teamPos
9.5658s to complete postResults
192.168.1.13 - - [28/Feb/2018 08:32:42] "POST /api/event/2017-01-07-1/results HTTP/1.1" 200 -

pi@raspberrypi:~/code/cocwifi $ git checkout parsefaster
Switched to branch 'parsefaster'
Your branch is up-to-date with 'origin/parsefaster'.
pi@raspberrypi:~/code/cocwifi $ python rundevserver.py
WebSocket transport not available. Install eventlet or gevent and gevent-websocket for improved performance.
 * Running on http://0.0.0.0:8080/ (Press CTRL+C to quit)
0.8359s for getRunners
0.4331s for buildDB
0.5382s for assignPos
0.5555s for assignScore
0.3341s for teamScore
0.2721s for teamPos
3.3956s to complete postResults
192.168.1.13 - - [28/Feb/2018 08:33:29] "POST /api/event/2017-01-07-1/results HTTP/1.1" 200 -

On desktop, getRunners is now consistently quicker than some of the scoring functions and full execution of the POST request is around 0.6 sec, with no timing outliers.