search

YingfanWang / PaCMAP

PaCMAP: Large-scale Dimension Reduction Technique Preserving Both Global and Local Structure
Apache License 2.0
512 stars 52 forks source link

Import performance #40

Closed AndreasGerken closed 2 years ago

AndreasGerken commented 2 years ago

Importing pacmap takes ~4.3 seconds on my machine (core I7/ 8th Generation, 40GB RAM, python 3.8.10). Since i want to include it in my package i overtake these 4.3 seconds into my package and delays everything by a lot.

Could you increase the import performance?

Timed with multiple runs of python3 -m timeit -r 1 "import pacmap"

hyhuang00 commented 2 years ago

I made an investigation into the import process and on my machine it takes even longer to import (~10s). The tuna log doesn't say much about the root cause of the slow import, but I suppose the just in time compiler we used for the functions is the culprit. When you import pacmap, the functions will be compiled. I will try to fix this issue by caching some of the compiled function, which hopefully can reduce import latency.

Screen Shot 2022-07-02 at 3 33 29 PM
hyhuang00 commented 2 years ago

The latest version on GitHub should be able to resolve this issue by reducing the import time to around 30%, and a release on PyPI will follow shortly. Closing this issue.