search

libdynd / dynd-python

Python exposure of dynd
http://libdynd.org
Other
119 stars 23 forks source link

Add parallel build option to setup.py. #697

Closed skrah closed 8 years ago

skrah commented 8 years ago

This enables a parallel build by default in setup.py.

skrah commented 8 years ago

Funny: g++-4.9: internal compiler error: Killed (program cc1plus)

Too bad, it works on my machine.

mwiebe commented 8 years ago

That's bizarre, it should work equivalently unless part of the dependencies aren't set up right in the cmakelists

insertinterestingnamehere commented 8 years ago

We've had grief with this sort of thing in the CI systems before. The internal compiler errors come because the CI system runs out of memory. Particularly on Travis CI, the reported number of cores is usually much higher than the number of cores actually available, so too many threads get launched, causing the crash.

mwiebe commented 8 years ago

So there should probably be a num_cpus parameter to setup.py or something to control that on the CI builds.

insertinterestingnamehere commented 8 years ago

Yep, an extra argument to setup.py would work. Last time I tried adjusting these parameters it was in the conda recipe for libdynd. Extra parameters aren't available there.

izaid commented 8 years ago

@skrah This is totally a CI thing with the machines running out of memory. We've seen it many times before, as Ian said.

skrah commented 8 years ago

The latest patch uses -jN as the setup.py argument. This is already present in Python 3.5, where it has a slightly different meaning (parallel build of multiple extensions), but that is harmless here.

So in Python < 3.5: python2 setup.py build_ext -j2

In Python >= 3.5 both happen to work: python3 setup.py build_ext -j2 python3 setup.py build -j2

skrah commented 8 years ago

Closing temporarily -- I need a fresh fork and GitHub apparently won't allow this with open PRs.

mwiebe commented 8 years ago

@skrah can you explain what you mean by a fresh fork? I've generally found I can get pull requests into a desired state by working locally, then force-pushing to the branch the PR is for.

skrah commented 8 years ago

@mwiebe It's hard to say what went wrong since GitHub itself had severe issues today (totally unreachable for 5 min, then not reacting to any commands for at least two hours: https://status.github.com/messages).

Before I read the above status message, I was unable to update my fork: GitHub claimed that the fork was 8 commits behind on the master branch, but my local clone had all the latest commits. Pushing the master branch appeared to work normally but did not have any effect on the web page.

So I tried to get a new fork, but that was impossible as well for about two hours.

I still don't know if it was just GitHub not updating the databases or me messing up a force-push on Sunday.

mwiebe commented 8 years ago

Ugh, that's no fun! Hopefully it was all the github problems, and doesn't come back...