How to make `pip install klayout` in macOs faster?

joamatab commented 2 years ago

some macOs gdsfactory users have reported that pip install klayout takes 30min for mac

How could we make it faster?

See the issue here https://github.com/gdsfactory/gdsfactory/runs/6214659865?check_suite_focus=true

is there any version for klayout that could make it faster?

@klayoutmatthias @thomaslima @flaporte @tvt173 @lukasc-ubc

thomaslima commented 2 years ago

Yes. I have this in my backlog. We need to create a wheel for the newer macOS versions and upload to pypi. Right now it is compiling from source. Travis stopped their easy free open source service; last I checked they wanted a credit card to offer free builds. I will check GitHub actions (didn't know they offered macOS builds).

joamatab commented 2 years ago

Hi Thomas,

yes, it takes 30min to compile from source :)

we can set up github actions for this https://github.com/KLayout/klayout/pull/1070

let me know what you think Joaquin

klayoutmatthias commented 2 years ago

@joamatab: @thomaslima is correct. Travis has resource restrictions now that basically make it impossible to deploy KLayout packages.

But I see the GitHub actions on the original fork failing and they are only executed on Linux. So is that functional? Is MacOS available with a free plan?

Plus is there going to be active maintenance? I personally do not have the resources for continuous support of another CI queue.

Matthias

joamatab commented 2 years ago

i think is not working yet https://github.com/joamatab/klayout/runs/6220334022?check_suite_focus=true

   g++: error: build/temp.linux-x86_64-3.8/src/tl/tl/tlHttpStreamCurl.o: No such file or directory
    error: command 'g++' failed with exit status 1

once is working, we can deploy wheels that work for all os (linux, macos, windows) as they are available for free in github actions

another great thing would be to have a conda package for klayout

@proppy

klayoutmatthias commented 2 years ago

From what I see the job you're proposing is only pulling the sources from PyPI and generating wheels for particular platforms on a regular basis. So it's not really CI/CD.

So if it's not connected to KLayout's sources here and as this job is not tied to branches or pull requests here I'd prefer to have it separately. I propose to set up a new repository - i.e. klayout/pypi-buildbot - and put the GitHub actions there.

joamatab commented 2 years ago

Hi Matthias,

the CICD job from the PR builds the sources from the klayout repo and uploads them to pypi, so in my opinion would be nice to have as part of the klayout, so they get updated on each new release or PR

let me know what you think Joaquin

klayoutmatthias commented 2 years ago

Ah, you're right. I stand corrected.

I have a few remarks though:

I don't see a dependency of the twine upload on the "v*" tag. You must not upload other builds as you cannot delete files on PyPI, specifically not "master". Ideally builds are done on master and PRs but twine deployment only happens in "v" tags or "on release".
I like the manylinux scheme. Any chance to port it? I'm afraid just offering Ubuntu will not be enough.
Do GitHub actions cache the builds or will every master commit trigger a full rebuild? That will be costly in terms of resources.

I need to say, that I'm actually quite happy to have reduced my dependency on "free" services. These in fact generate some considerable workload by constantly upgrading their environments, certificates, authentication schemes etc. For example I moved away from Azure with Python deployment after they had removed all their MSVC2017 support (MS C++ compilers are magic, so MSVC2019 did not work out of the box ...). To my taste these services do not offer enough configuration options and stability for C++ deployment. I do not expect much more from GitHub actions.

joamatab commented 2 years ago

Hi Matthias,

only tags that start with v will trigger the upload to pypi, I use the python package bumpversion package for this
sure, we can take a look if manylinux works with github actions
only new tagged releases will trigger a rebuild

what is the recommended way to build the python wheels?

Let me know if you can figure out what's the issue of this error https://github.com/joamatab/klayout/runs/6220334022?check_suite_focus=true

klayoutmatthias commented 2 years ago

The error happens because of this:

 src/tl/tl/tlHttpStreamCurl.cc:44:10: fatal error: curl/curl.h: No such file or directory

so the curl package is missing.

klayoutmatthias commented 2 years ago

Apparently adding curl did not help.

This is the package Ubuntu needs: libcurl4-openssl-dev.

But there are some more: for manylinux/CentOS this is the package list I have added: zlib-devel curl-devel expat-devel.

I assume (not tested) that for Ubuntu the package list is: zlib1g-dev libcurl4-openssl-dev libexpat1-dev.

joamatab commented 2 years ago

perfect, thank you Matthias,

I think it is working now :)

thomaslima commented 2 years ago

Sorry I've been awol. =(

Hey @joamatab just curious to check what's the status with the Mac build. I am trying your branch now but I noticed it failed in your GitHub fork.

Caching used to work by enabling ccache and persisting the ccache folder between runs. .travis.yml contains the details.

Manylinux used to work by running python builds inside a docker image they provided. Is that still the case? Only wheels built within that docker earn the manylinux designation and can be uploaded to pypi.

joamatab commented 2 years ago

Hi Thomas,

it's passing now

https://github.com/joamatab/klayout/actions

Matthias

let me know if it's okay on your end

Joaquin

thomaslima commented 2 years ago

@klayoutmatthias @joamatab While I was working on the cache, I discovered an awesome solution called cibuildwheel from pypa. It is a maintained CI tool that builds python projects in all platforms. Check this out: https://cibuildwheel.readthedocs.io/en/stable/

It is meant for various CI workflows but I think what's nice is that we can run locally on a specific hardware, like my M1 Mac, and it will build all py3.x wheels automatically.

I'm testing it here: https://github.com/thomaslima/klayout/actions/workflows/build.yml

joamatab commented 2 years ago

perfect, thank you Thomas, i think you just need to have your PYPI token as a secret

https://github.com/gdsfactory/gdsfactory/blob/master/.github/workflows/pythonpublish.yml

thomaslima commented 2 years ago

@klayoutmatthias @joamatab I would like to report that I am done testing. Thanks to the official cibuildwheel project, we now have a pypi maintained build strategy for macOS and manylinux/musllinux (they also offer windows but I noticed that Matthias has an Azure pipeline for that). It is very straightforward to use and update. The only hack I had to do was related to ccache, but it's worth it, build time is 30 seconds per wheel now. I also constrained to 64-bit systems (couldn't figure out how to handle 32-bit docker environments with ccache so I gave up).

This is how it works:

There is a pypa/cibuildwheel@v2.5.0 GitHub action that I am using directly, which reads configuration from environment variables and/or a pyproject.toml file. I opted to use mostly the pyproject.toml file, to make the Github Actions file as clean as possible in case we need to transition to another CI in the future. By default, this action will automatically download/install all supported python3 versions and trigger a wheel build. After each build, it installs the package from the wheel and runs specified tests (I used the same ones from before). So by running this action on a macOS machine, it will create wheels for all python versions supported by that platform. If ubuntu, it will first prepare a Docker container for manylinux and musllinux and run the builds there.

Because of the ccache strategy I used, we need to create a build job per docker file. This is achieved by a matrix in the build.yml file:

          - os: "macos-latest"
            cibuild: "*macosx*"
          - os: "ubuntu-latest"
            cibuild: "*manylinux*"
          - os: "ubuntu-latest"
            cibuild: "*musllinux*"

After each job is finished, the wheels are uploaded as an artifact, and captured by another job whose goal is to combine all wheels and upload to pypi:

Note: macos-latest at the moment uses an Intel host with 10.11 (Big Sur) OS, but it doesn't matter, as it looks like cibuild builds wheels compatible with 10.9+. Right now, Apple Silicon (arm64) is not yet supported in Github Actions (https://github.com/actions/virtual-environments/issues/2187#issuecomment-737339211). I recommend we wait until it is and add it to the configuration above.

Meanwhile, I think it's possible to run cibuildwheel locally. It's as simple as running pip install cibuildwheel; cibuildwheel --platform macos but it requires platform python to be installed. We need a volunteer to run these in an M1 Mac and provide wheels to us. Since klayout is pretty stable, I think this is a twice a year task until GitHub actions releases M1.

Let me know if you have questions. I think it might be advisable to write this down in a readme file once we merge.

joamatab commented 2 years ago

Awesome, thank you Thomas!

once we merge PR #1070 we will be able to pip install klayout in MacOs

klayoutmatthias commented 2 years ago

Dear @thomaslima and @joamatab,

thanks for the time you spent on this!

I'll merge the changes to master and will try to deploy the solution on 0.27.9 for macos.

Best regards,

Matthias

klayoutmatthias commented 2 years ago

My first run on master wasn't quite successful:

Is that a resource limit I am hitting?

Matthias

thomaslima commented 2 years ago

@klayoutmatthias Have you pushed again? @joamatab included a step in the workflow called "Cancel Workflow Action" that cancels a previously running build if you trigger a new one. I'm not sure what the reasoning was behind that. But that might have been the cause for this getting cancelled. The latest run based on master seems to be working: https://github.com/KLayout/klayout/actions/runs/2277996327

It looks like the push to branch 0.27 failed at the test stage because klayout/testdata/pymod/import_lib.py was missing. Should the default branch be 0.27? Why do we need two parallel branches?

thomaslima commented 2 years ago

By the way, the upload step to pypi also worked: https://test.pypi.org/project/klayout/0.28/#files I realized that the master branch has a version number 0.28, which I imagine is a future version.

joamatab commented 2 years ago

Hi Thomas and Matthias,

from the CICD point of view it seems it worked uploading to testpypi https://github.com/KLayout/klayout/actions/runs/2277996327

How about starting to upload the MacOs wheels and the newer linux versions to pypi?

https://github.com/KLayout/klayout/pull/1074

klayoutmatthias commented 2 years ago

@klayoutmatthias Have you pushed again? @joamatab included a step in the workflow called "Cancel Workflow Action" that cancels a previously running build if you trigger a new one. I'm not sure what the reasoning was behind that. But that might have been the cause for this getting cancelled. The latest run based on master seems to be working: https://github.com/KLayout/klayout/actions/runs/2277996327

Yes, that's the reason. Thanks for clarifying that.

It looks like the push to branch 0.27 failed at the test stage because klayout/testdata/pymod/import_lib.py was missing. Should the default branch be 0.27? Why do we need two parallel branches?

0.27 is my stable branch. "master" will be the future 0.28 and it received some incompatible changes and isn't stable, so it's not ready for release yet.

Best regards,

Matthias

joamatab commented 2 years ago

Awesome,

See PR for 0.27 branch

https://github.com/KLayout/klayout/pull/1077

klayoutmatthias commented 2 years ago

Dear @joamatab and @thomaslima, thanks for the efforts you spent in this!

I have merged all PRs and after adding import_lib.py the actions pass for me. I'll see for the next minor release how things work.

BTW @joamatab: I have created a PR for a major change which targets towards including a Qt-less LayoutView to be included into the Python module. I think you had asked for that some time ago. It needs libpng for image reading and writing, but has a fairly complete API including mouse emulation. I think that could be the beginning of some behind-the-scene deployment of KLayout. The modification is too big to go into 0.27.x, but 0.28 is my target.

joamatab commented 2 years ago

Awesome, thank you Matthias! @tvt173 @flaport and @proppy would also love to see the Qt-less klayout so we can render GDS files from the browser environment, such as jupyter

the pip install klayout still takes 30min for macos, how can we make a new release for it? so we trigger the uploading of the wheels

proppy commented 2 years ago

BTW @joamatab: I have created a PR for a major change which targets towards including a Qt-less LayoutView to be included into the Python module. I think you had asked for that some time ago. It needs libpng for image reading and writing, but has a fairly complete API including mouse emulation. I think that could be the beginning of some behind-the-scene deployment of KLayout. The modification is too big to go into 0.27.x, but 0.28 is my target.

@klayoutmatthias that's awesome to hear! I can't wait to add it to https://colab.research.google.com/gist/proppy/26525ff7d9810247759a8d3ff3c41dcb/openlane-with-conda-eda.ipynb, we're currently relying on mixture of gdstk and cairosvg to do gds -> svg -> png which can turn out to be very slow for bigger design. And having access to the full power of klayout to only render the design we care about could be a game changer. If the API is granular enough we may be even able to integrate it with IPython/matplotlib/bokeh and avoid the filesystem altogether!

klayoutmatthias commented 2 years ago

I tried to create a v0.27.9-1 post version to trigger a MacOS build, but I had to patch the build action because a release was not included in the main selector (https://github.com/KLayout/klayout/commit/0e32c3bd68a658b3226cfcfecfe83dd1b68eaef2). I hope it will now run and deliver the MacOS packages.

Matthias

klayoutmatthias commented 2 years ago

It's done ... MacOS packages for 0.27.9-1 are on PyPI :)

klayoutmatthias commented 2 years ago

@proppy @joamatab I have created another enhancement ticket to discuss further matters of the Qt-less canvas development here: https://github.com/KLayout/klayout/issues/1082

Matthias

joamatab commented 2 years ago

awesome! thank you Matthias!

yes, now pip install klayout is as fast on mac os as in other OS

https://github.com/gdsfactory/gdsfactory/actions/runs/2310609416

thank you

KLayout / klayout

How to make `pip install klayout` in macOs faster? #1069