Closed joamatab closed 2 years ago
Yes. I have this in my backlog. We need to create a wheel for the newer macOS versions and upload to pypi. Right now it is compiling from source. Travis stopped their easy free open source service; last I checked they wanted a credit card to offer free builds. I will check GitHub actions (didn't know they offered macOS builds).
Hi Thomas,
yes, it takes 30min to compile from source :)
we can set up github actions for this https://github.com/KLayout/klayout/pull/1070
let me know what you think Joaquin
@joamatab: @thomaslima is correct. Travis has resource restrictions now that basically make it impossible to deploy KLayout packages.
But I see the GitHub actions on the original fork failing and they are only executed on Linux. So is that functional? Is MacOS available with a free plan?
Plus is there going to be active maintenance? I personally do not have the resources for continuous support of another CI queue.
Matthias
i think is not working yet https://github.com/joamatab/klayout/runs/6220334022?check_suite_focus=true
g++: error: build/temp.linux-x86_64-3.8/src/tl/tl/tlHttpStreamCurl.o: No such file or directory
error: command 'g++' failed with exit status 1
once is working, we can deploy wheels that work for all os (linux, macos, windows) as they are available for free in github actions
another great thing would be to have a conda package for klayout
@proppy
From what I see the job you're proposing is only pulling the sources from PyPI and generating wheels for particular platforms on a regular basis. So it's not really CI/CD.
So if it's not connected to KLayout's sources here and as this job is not tied to branches or pull requests here I'd prefer to have it separately. I propose to set up a new repository - i.e. klayout/pypi-buildbot - and put the GitHub actions there.
Hi Matthias,
the CICD job from the PR builds the sources from the klayout repo and uploads them to pypi, so in my opinion would be nice to have as part of the klayout, so they get updated on each new release or PR
let me know what you think Joaquin
Ah, you're right. I stand corrected.
I have a few remarks though:
I need to say, that I'm actually quite happy to have reduced my dependency on "free" services. These in fact generate some considerable workload by constantly upgrading their environments, certificates, authentication schemes etc. For example I moved away from Azure with Python deployment after they had removed all their MSVC2017 support (MS C++ compilers are magic, so MSVC2019 did not work out of the box ...). To my taste these services do not offer enough configuration options and stability for C++ deployment. I do not expect much more from GitHub actions.
Hi Matthias,
v
will trigger the upload to pypi, I use the python package bumpversion
package for thiswhat is the recommended way to build the python wheels?
Let me know if you can figure out what's the issue of this error https://github.com/joamatab/klayout/runs/6220334022?check_suite_focus=true
The error happens because of this:
src/tl/tl/tlHttpStreamCurl.cc:44:10: fatal error: curl/curl.h: No such file or directory
so the curl package is missing.
Apparently adding curl did not help.
This is the package Ubuntu needs: libcurl4-openssl-dev
.
But there are some more: for manylinux/CentOS this is the package list I have added: zlib-devel curl-devel expat-devel
.
I assume (not tested) that for Ubuntu the package list is: zlib1g-dev libcurl4-openssl-dev libexpat1-dev
.
perfect, thank you Matthias,
I think it is working now :)
Sorry I've been awol. =(
Hey @joamatab just curious to check what's the status with the Mac build. I am trying your branch now but I noticed it failed in your GitHub fork.
Caching used to work by enabling ccache and persisting the ccache folder between runs. .travis.yml
contains the details.
Manylinux used to work by running python builds inside a docker image they provided. Is that still the case? Only wheels built within that docker earn the manylinux designation and can be uploaded to pypi.
Hi Thomas,
it's passing now
https://github.com/joamatab/klayout/actions
Matthias
let me know if it's okay on your end
Joaquin
@klayoutmatthias @joamatab While I was working on the cache, I discovered an awesome solution called cibuildwheel from pypa. It is a maintained CI tool that builds python projects in all platforms. Check this out: https://cibuildwheel.readthedocs.io/en/stable/
It is meant for various CI workflows but I think what's nice is that we can run locally on a specific hardware, like my M1 Mac, and it will build all py3.x wheels automatically.
I'm testing it here: https://github.com/thomaslima/klayout/actions/workflows/build.yml
perfect, thank you Thomas, i think you just need to have your PYPI token as a secret
https://github.com/gdsfactory/gdsfactory/blob/master/.github/workflows/pythonpublish.yml
@klayoutmatthias @joamatab I would like to report that I am done testing. Thanks to the official cibuildwheel project, we now have a pypi maintained build strategy for macOS and manylinux/musllinux (they also offer windows but I noticed that Matthias has an Azure pipeline for that). It is very straightforward to use and update. The only hack I had to do was related to ccache, but it's worth it, build time is 30 seconds per wheel now. I also constrained to 64-bit systems (couldn't figure out how to handle 32-bit docker environments with ccache so I gave up).
This is how it works:
There is a pypa/cibuildwheel@v2.5.0
GitHub action that I am using directly, which reads configuration from environment variables and/or a pyproject.toml
file. I opted to use mostly the pyproject.toml file, to make the Github Actions file as clean as possible in case we need to transition to another CI in the future. By default, this action will automatically download/install all supported python3 versions and trigger a wheel build. After each build, it installs the package from the wheel and runs specified tests (I used the same ones from before). So by running this action on a macOS machine, it will create wheels for all python versions supported by that platform. If ubuntu, it will first prepare a Docker container for manylinux and musllinux and run the builds there.
Because of the ccache strategy I used, we need to create a build job per docker file. This is achieved by a matrix in the build.yml
file:
- os: "macos-latest"
cibuild: "*macosx*"
- os: "ubuntu-latest"
cibuild: "*manylinux*"
- os: "ubuntu-latest"
cibuild: "*musllinux*"
After each job is finished, the wheels are uploaded as an artifact, and captured by another job whose goal is to combine all wheels and upload to pypi:
Note: macos-latest at the moment uses an Intel host with 10.11 (Big Sur) OS, but it doesn't matter, as it looks like cibuild builds wheels compatible with 10.9+. Right now, Apple Silicon (arm64) is not yet supported in Github Actions (https://github.com/actions/virtual-environments/issues/2187#issuecomment-737339211). I recommend we wait until it is and add it to the configuration above.
Meanwhile, I think it's possible to run cibuildwheel locally. It's as simple as running pip install cibuildwheel; cibuildwheel --platform macos
but it requires platform python to be installed. We need a volunteer to run these in an M1 Mac and provide wheels to us. Since klayout is pretty stable, I think this is a twice a year task until GitHub actions releases M1.
Let me know if you have questions. I think it might be advisable to write this down in a readme file once we merge.
Awesome, thank you Thomas!
once we merge PR #1070 we will be able to pip install klayout
in MacOs
Dear @thomaslima and @joamatab,
thanks for the time you spent on this!
I'll merge the changes to master and will try to deploy the solution on 0.27.9 for macos.
Best regards,
Matthias
My first run on master wasn't quite successful:
Is that a resource limit I am hitting?
Matthias
@klayoutmatthias Have you pushed again? @joamatab included a step in the workflow called "Cancel Workflow Action" that cancels a previously running build if you trigger a new one. I'm not sure what the reasoning was behind that. But that might have been the cause for this getting cancelled. The latest run based on master seems to be working: https://github.com/KLayout/klayout/actions/runs/2277996327
It looks like the push to branch 0.27 failed at the test stage because klayout/testdata/pymod/import_lib.py
was missing. Should the default branch be 0.27? Why do we need two parallel branches?
By the way, the upload step to pypi also worked: https://test.pypi.org/project/klayout/0.28/#files I realized that the master branch has a version number 0.28, which I imagine is a future version.
Hi Thomas and Matthias,
from the CICD point of view it seems it worked uploading to testpypi https://github.com/KLayout/klayout/actions/runs/2277996327
How about starting to upload the MacOs wheels and the newer linux versions to pypi?
@klayoutmatthias Have you pushed again? @joamatab included a step in the workflow called "Cancel Workflow Action" that cancels a previously running build if you trigger a new one. I'm not sure what the reasoning was behind that. But that might have been the cause for this getting cancelled. The latest run based on master seems to be working: https://github.com/KLayout/klayout/actions/runs/2277996327
Yes, that's the reason. Thanks for clarifying that.
It looks like the push to branch 0.27 failed at the test stage because
klayout/testdata/pymod/import_lib.py
was missing. Should the default branch be 0.27? Why do we need two parallel branches?
0.27 is my stable branch. "master" will be the future 0.28 and it received some incompatible changes and isn't stable, so it's not ready for release yet.
Best regards,
Matthias
Dear @joamatab and @thomaslima, thanks for the efforts you spent in this!
I have merged all PRs and after adding import_lib.py the actions pass for me. I'll see for the next minor release how things work.
BTW @joamatab: I have created a PR for a major change which targets towards including a Qt-less LayoutView to be included into the Python module. I think you had asked for that some time ago. It needs libpng for image reading and writing, but has a fairly complete API including mouse emulation. I think that could be the beginning of some behind-the-scene deployment of KLayout. The modification is too big to go into 0.27.x, but 0.28 is my target.
Awesome, thank you Matthias! @tvt173 @flaport and @proppy would also love to see the Qt-less klayout so we can render GDS files from the browser environment, such as jupyter
the pip install klayout
still takes 30min for macos, how can we make a new release for it? so we trigger the uploading of the wheels
BTW @joamatab: I have created a PR for a major change which targets towards including a Qt-less LayoutView to be included into the Python module. I think you had asked for that some time ago. It needs libpng for image reading and writing, but has a fairly complete API including mouse emulation. I think that could be the beginning of some behind-the-scene deployment of KLayout. The modification is too big to go into 0.27.x, but 0.28 is my target.
@klayoutmatthias that's awesome to hear! I can't wait to add it to https://colab.research.google.com/gist/proppy/26525ff7d9810247759a8d3ff3c41dcb/openlane-with-conda-eda.ipynb, we're currently relying on mixture of gdstk
and cairosvg
to do gds -> svg -> png which can turn out to be very slow for bigger design. And having access to the full power of klayout to only render the design we care about could be a game changer. If the API is granular enough we may be even able to integrate it with IPython/matplotlib/bokeh and avoid the filesystem altogether!
I tried to create a v0.27.9-1 post version to trigger a MacOS build, but I had to patch the build action because a release was not included in the main selector (https://github.com/KLayout/klayout/commit/0e32c3bd68a658b3226cfcfecfe83dd1b68eaef2). I hope it will now run and deliver the MacOS packages.
Matthias
It's done ... MacOS packages for 0.27.9-1 are on PyPI :)
@proppy @joamatab I have created another enhancement ticket to discuss further matters of the Qt-less canvas development here: https://github.com/KLayout/klayout/issues/1082
Matthias
awesome! thank you Matthias!
yes, now pip install klayout
is as fast on mac os as in other OS
https://github.com/gdsfactory/gdsfactory/actions/runs/2310609416
thank you
some macOs gdsfactory users have reported that
pip install klayout
takes 30min for macHow could we make it faster?
See the issue here https://github.com/gdsfactory/gdsfactory/runs/6214659865?check_suite_focus=true
is there any version for klayout that could make it faster?
@klayoutmatthias @thomaslima @flaporte @tvt173 @lukasc-ubc