search

Vlad-Shcherbina / icfpc2018-tbd

1 stars 0 forks source link

C++/Python #22

Open Vlad-Shcherbina opened 6 years ago

Vlad-Shcherbina commented 6 years ago

One problem was that it compiled awfully slowly. Apparently pybind11 template magic is too heavy when there are many bindings (and we had quite a few). And even though there were multiple compilation units, distutils recompiled everything on any change. We need to do something about it.

@caudate-julie , @fj128 , were there any other problems?

lukateras commented 6 years ago

CMake would solve that as well.

earthdok commented 6 years ago

I definitely saw multiple recompilations with no changes to the C++ code whatsoever. I think it recompiled every time solver_worker was run?

Vlad-Shcherbina commented 6 years ago

There was a problem with recompiling always, but it was fixed midway through the contest: https://github.com/Vlad-Shcherbina/icfpc2018-tbd/commit/12b082ee7aa63ad1303ccae193f5adb040903397

Are you sure you did not do git pull between runs? Can you observe this behavior now?

Vlad-Shcherbina commented 6 years ago

Assuming separate compilation is made to work with CMake or something, bindings can be partitioned into compilation units: https://pybind11.readthedocs.io/en/stable/faq.html#how-can-i-reduce-the-build-time

earthdok commented 6 years ago

Can you observe this behavior now?

It appears that I was alternating between running $ TBD_RELEASE=1 python solver_worker.py ... and $ python .py ...

I.e. the changing TBD_RELEASE value was triggering a rebuild.

fj128 commented 6 years ago

I wouldn't say it was awfully slow, like 10 seconds I guess. Since it's a one time cost when running stuff repeatedly without git pull I'm OK with that. Though it would be nice if object file compilation was run in parallel of course.

My main problem with Vlad's magic was actually during the previous contest, when it turned out that import distutils on an Anaconda distribution took like 3 seconds for some reason, which meant 3 seconds every single run. That was really fucking annoying. This year, idk if it was because I was using plain Python + pip, or because I had a very nice NVMe SSD (btw I had an SSD in 2017 as well, I shudder to imagine how it was for the HDD people), this pain point went away.

However for those of us who are less fortunate it would be nice to

  1. have the cpp_utils.py do a preliminary dependency analysis itself and don't even import distutils if the target .pyd exists and is newer than all the dependencies (that it knows anyways)

  2. see if distutils can be told to run compilers in parallel.

  3. make TBD_RELEASE somehow more persistent, because of effing course @earthdok it recompiles.

I'm partial to CMake but unless someone shows how it solves the problems above without causing more problems I'm for the devil we know.

earthdok commented 6 years ago

make TBD_RELEASE somehow more persistent, because of effing course @earthdok it recompiles.

You can always export it or have it hardcoded in the makefile.