ELITR / online-text-flow

Online event streaming to improve data and text flows
MIT License
1 stars 0 forks source link

import quart in server makes events slow #14

Closed Gldkslfmsd closed 3 years ago

Gldkslfmsd commented 3 years ago

Hi,

onlinetextflow is starting very slow. I expect this could take around 200 ms, not 1700:

$ time echo 100 101 . | online-text-flow events de
100 101 .

real    0m1.722s
user    0m0.506s
sys 0m0.185s

I found it's because import elitr.onlinetextflow by defaults runs import server, which runs this:

from quart import Quart, Blueprint, json, request, session, websocket
import quart

These import takes around 1.1 second, and it is not used by events. How can we reduce it, so it doesn't break server, and does not need to be loaded by events and others? @srdecny ?

srdecny commented 3 years ago

Hi, I'm afraid I won't be of much help -- I do not understand Python's import idiosyncracies, despite trying many times. I suspect that using the dot notation during importing invokes the magical __init__.py file where the server gets imported. I've tried to change this line to import textflow_protocol, and now it works fast when invoked directly:

root@b5764709240e:/cruise-control/online-text-flow/elitr/onlinetextflow# time echo 100 200 . | python3 ./events.py de
100 101 .

real    0m0.095s
user    0m0.082s
sys 0m0.013s

However, it's still slow when ran on the command line as online-text-flow. I suspect this is because the aforementioned __init__.py file has the add_command() methods and is thus invoked. The solution could be to not have this code in the file and have it somewhere else, so it can import only the main of the module that was required (right now it imports all mains). But again, I do not understand the "Pythonic" way of doing imports.

Gldkslfmsd commented 3 years ago

Your snippet showed me another issue:

https://docs.python.org/3/library/time.html#time.perf_counter_ns

time.perf_counter_ns() → int

    Similar to perf_counter(), but return time as nanoseconds.

    New in version 3.7.

I have only 3.6.9. This function makes my python3 ./events.py de crash. I'm OK if server relies on 3.7, but events shouldn't.

Gldkslfmsd commented 3 years ago

But I hope that Ota's commit, the one mentioned in mail, fixes it.

Gldkslfmsd commented 3 years ago

So, I'm suggesting to split server and the rest into two packages. See my new branch soon.

Gldkslfmsd commented 3 years ago

let's continue in #15 -- it's fixed there