search

explosion / cython-blis

💥 Fast matrix-multiplication as a self-contained Python library – no system dependencies!
Other
219 stars 37 forks source link

support parallel builds with `setup.py build -j10` #28

Open pabs3 opened 4 years ago

pabs3 commented 4 years ago

When I try to do a parallel build with setup.py build -j10 I still only get one gcc process being run at a time, which means that the build takes a while longer than I would like. It would be great to have support for parallel builds so they take less time.

honnibal commented 4 years ago

I would really appreciate a PR to improve this, sure.

One thing that's really awkward at the moment is that we invoke the compiler separately in a bunch of subprocesses. This is super slow. I don't know how to avoid it though, given the way the jsonl build log works. Open to suggestions.

dimaqq commented 3 years ago

Better yet, allow passing this option via environment variable, so that I could do:

env MAKELAGS="-j" poetry install that installs my project, which installs scapy, which installs blis which then uses as many cores as are available, like make does.

dimaqq commented 3 years ago

Context: this is a notable issue for any musl-based distro, like alpine, where today there can be no wheels on pypi.org