eel.init takes 20seconds

[shrsulav]

Describe the problem In Raspbian Stretch, Python 3.5 is the highest supported version. But for eel Python 3.6 is required. So, I looking at tutorials I compiled Python 3.6 for Raspbian Stretch and got eel working but the line "eel.init" is taking 20 seconds.

How can I debug this issue?

Desktop (please complete the following information):

• OS: Raspbian Stretch
• Browser Chromium

[samuelhwilliams]

eel.init runs through the provided directory and parses files to pull out Javascript function names. If you're sure that is taking 20 seconds (I'm surprised), my first guess would be that you're pointing it at a massive directory and so the parsing is taking ages. Is that possible?

Otherwise you can debug it with the normal tools, eg pdb/ipdb, by setting a trace in the Eel library code for the init function to see exactly what's going on.

[shrsulav]

@samuelhwilliams Hello Samuel, I also had the same guess and I did find some files which were in the range of 75MB. So, I removed them and tried, but it did not help.

The python I am using is not available for Raspbian Stretch and I had to compile it from the source. I think this could have played a major part in it. I will try multiple versions of Python3 and see if there's any improvement, before I try the pdb/ipdb.

[samuelhwilliams]

A very quick (and dirty) check might be to insert a print(name) statement before or just after https://github.com/samuelhwilliams/Eel/blob/master/eel/__init__.py#L111 so that you can quickly see what files are getting checked/processed. Good to know that you've eliminated some of the large files already, but I wonder if there aren't still more. Have you tried pointing it at a separate directory that only contains the raw HTML/JS/CSS files you need and nothing else?

P.S. Eel is only officially supported on 3.6, but depending on your use-case and how much maintenance it needs, you might be able to get away with using it on 3.5 - so possibly worth giving that a shot as well? Obviously not if that's a production use-case though.

[ChrisKnott]

The reason for the whitelist of file extensions was to speed up this bit. Previously I was making a map app which had a folder with hundreds of OpenStreetMap tile images, and it was taking a long time to start up.

I also suggest Sam's debugging idea; you can also print time.perf_counter() if you don't want to watch it

[shrsulav]

@samuelhwilliams I tried to find out which file(s) were causing the initialization to be slow. It turns out that there was one javascript file which was taking 12s and another javascript file which was taking 6s.

I decided not to use the 6s javascript file, and for the 12s javascript file I used the minified version. After doing this, the initialization time has come down to 6s.

Now, that I know the minified versions of javascript file have great impact on the initialization, I will try to minify every javascript file.

[ChrisKnott]

Still too long IMO. Maybe we should be hashing the file, and caching parse results, so that it's faster if the file hasn't changed.

[samuelhwilliams]

Out of curiosity @shrsulav, how big are your JS files?

[shrsulav]

The size itself does not seem big. The unminified version is around 600kB and minified version is 220kB. The number of lines of code is high though. Around 20k lines of code.

[shrsulav]

@samuelhwilliams Earlier you had mentioned, there's a way to use eel with python 3.5. Can you provide the details on how to do that?

[t-walker-wei]

I think should add a allow prefix argument in eel.init() if need.

[Malachov]

Hello,

It's definitely an issue (solution below, so chill down....). I don't know the reason, but suddenly init took ages (order of minutes). I build static assets in Vue (works like a charm in debugging on server (hope i add examples one day)). Build assets are very big - around 7 Mb (plotly library...)) - I don't mind big files (no networking - desktop app) .

I changed webpack optimize options and separated it into small items, then run debug and analyzed where is the problem.

It's not only about big files - if js file has around 100kb, it took about 3 seconds (don't know why...) and it seem that time is not linear with size

Problem is definitely not in parsing file structure in init folder - that's miliseconds. This line is problematic...

matches = EXPOSED_JS_FUNCTIONS.parseString(contents).asList()

The parseString from pyparsing - analysis of js code (probably finding exposed functions) (but i doubt that it's caused by version change).

Maybe it's not an issue, it just takes a time (it's not easy to regex find big files) - but still there have to be a workaround.

Use case

• i want fast eel.init but i have many not exposing js functions

Workaround

I build assets in vue

• it use webpack !!

Webpack separate my functions and 3rd party libraries (then webpack add chunk-vendors to the file name)

I suppose there is not exposed function in chunk-vendors-------.js files.

To __init__.py to def init():..... after this loop

    for root, _, files in os.walk(root_path):
        for name in files:

around line 113 Add this lines

# From init func param or module variable
exclude_chuck_vendors = True

if exclude_chuck_vendors and name.startswith('chunk-vendors'):
    continue

This will skip js parsing for this file...

There is already similar name check, so seems ok...

Now only user code is parsed - and it's fast again... From minutes to much less than second...

I would do merge request, but i don't know if maybe some more general solution will raise...

[t-walker-wei]

Hello,

It's definitely an issue (solution below, so chill down....). I don't know the reason, but suddenly init took ages (order of minutes). I build static assets in Vue (works like a charm in debugging on server (hope i add examples one day)). Build assets are very big - around 7 Mb (plotly library...)) - I don't mind big files (no networking - desktop app) .

I changed webpack optimize options and separated it into small items, then run debug and analyzed where is the problem.

It's not only about big files - if js file has around 100kb, it took about 3 seconds (don't know why...) and it seem that time is not linear with size

Problem is definitely not in parsing file structure in init folder - that's miliseconds. This line is problematic...

matches = EXPOSED_JS_FUNCTIONS.parseString(contents).asList()

The parseString from pyparsing - analysis of js code (probably finding exposed functions) (but i doubt that it's caused by version change).

Maybe it's not an issue, it just takes a time (it's not easy to regex find big files) - but still there have to be a workaround.

Use case

• i want fast eel.init but i have many not exposing js functions

Workaround

I build assets in vue

• it use webpack !!

Webpack separate my functions and 3rd party libraries (then webpack add chunk-vendors to the file name)

I suppose there is not exposed function in chunk-vendors-------.js files.

To __init__.py to def init():..... after this loop

    for root, _, files in os.walk(root_path):
        for name in files:

around line 113 Add this lines

# From init func param or module variable
exclude_chuck_vendors = True

if exclude_chuck_vendors and name.startswith('chunk-vendors'):
    continue

This will skip js parsing for this file...

There is already similar name check, so seems ok...

Now only user code is parsed - and it's fast again... From minutes to much less than second...

I would do merge request, but i don't know if maybe some more general solution will raise...

Hi, I suggest better pass the allow prefixs. This is what I did in init():

def init( path, allowed_extensions=['.js', '.html', '.txt', '.htm','.xhtml', '.vue'], allowed_prefixs=[''], js_result_timeout=10000): global root_path, _js_functions, _js_result_timeout root_path = _get_real_path(path) js_functions = set() for root, _, files in os.walk(root_path): for name in files: if not (any(name.endswith(ext) for ext in allowed_extensions) and any(name.startswith(prefix) for prefix in allowed_prefixs)): continue

Pass the prefixs filter when init eel

eel.init('pages-folder', allowed_prefixs=['start-with-me'])

[Malachov]

Hi... yes, that's the reason why didn't merge request i just made workaround. It's question what's good general solution and it depends on use case. In one project i have most of own js and i prefer exclude logic, in other project i use most of generated 3rd party libraries and i need only few file, i would appreciate to define list of parsed files there. It's an open question how to solve it.

From my point of view add exclude prefix pattern is most simple to implement and not affection other users.

eel.init('pages-folder', excluded_prefixes=['start-with-me'])

When i have time, i'll do merge request...

[Malachov]

I'm not able to create new branch to create pull request. Do you have permissions @t-walker-wei ?

And what about @samuelhwilliams or @ChrisKnott opinion on this.

I think exclude logic is much simplier and has no impact on users that don't want to use it...

Adding excluded_prefixes=[] param into init function and this check

if excluded_prefixes and any(name.startswith(excl) for excl in excluded_prefixes):
    continue

Makes init from one minute to one second in my case - of course this is only issue with framworks like vue, but definitely worth for many users...

[Malachov]

Until it's solved - i created a fork and pushed to Pypi as EelForkExcludeFiles. When it's solved, i'll remove the fork...

[zzxjl1]

How about letting us choose which files are to be scanned? eg: files_to_scan=["path1","path2"]

[Malachov]

Both ways seems legit. Define what files should be used or define what files should be ignored...

It's really minor change of code and it has huge impact on startup time if using framework like Vue or react and some bigger npm libraries. For me it was from about 10 seconds to 1 second...

Personally i prefer regex ignore pattern as for me all npm libraries has always the same prefix, so for all my projects init syntax is the same...

Imho this should be reopen...

[georgepstaylor]

I've just come across this issue - in my web folder I have the font-awesome assets. Took me a while to figure out that was causing the long start up times.

Will try your fork @Malachov

[georgepstaylor]

@Malachov works perfectly - thanks!

This should definitely be re-opened and accepted

[Malachov]

@Malachov works perfectly - thanks!

This should definitely be re-opened and accepted

That would be sweet... I really like this library

Once I have time, I'll create pull request with simple workaround, which may not be generic, but on the other hand it's fully backward compatible

I saw, that some pull requests are accepted, so repo is still alive...

[dstricks]

I saw, that some pull requests are accepted, so repo is still alive...

A bit slow... but still alive. I'll take a look if you submit an updated PR (your previous one is out of date).

[Malachov]

Merge is there. It's backward compatible. Tests added.

I don't know whether should I add Type hints. I saw a merge with pyi files, but not merged, or should I add Type hints directly to the code?