klayoutmatthias commented 6 years ago

I have started development on this topic and even if it's in an early stage, I'd like to get some feedback on this.

Details are here: https://github.com/klayoutmatthias/klayout/wiki/pykl---Standalone-KLayout-Python-Module

Thanks,

Matthias

amccaugh commented 6 years ago

I would find this very, very, useful. I like the idea of KLayout interfacing more naturally with external scripting languages.

thomaslima commented 6 years ago

I am still reading the technical documents and testing your branch. It is far more complicated than I had imagined. Thank you for starting this work!

ejprinz commented 6 years ago

I cloned the repo from github (pymod branch) and tried to build. The build completes (on Ubuntu) but the install fails at: /usr/bin/ld: cannot find -lklayout_ext So I can't start the module.

ejprinz commented 6 years ago

@thomaslima, does your build complete and if so, what are your build.sh options? Thx, Erwin

thomaslima commented 6 years ago

Dear Erwin. My build doesn’t complete. It verbosely says it completes but there is an error generated. I created an issue with my error. Matthias is working on a few fixes. We should try again as soon as he’s done.

On 17 Jun 2018, at 22:54, Erwin J Prinz notifications@github.com<mailto:notifications@github.com> wrote:

@thomaslimahttps://github.com/thomaslima, does your build complete and if so, what are your build.sh options? Thx, Erwin

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/klayoutmatthias/klayout/issues/130#issuecomment-397929832, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ABCck5GsV06C6kW1XBusdttXLCjCDImSks5t9xZcgaJpZM4Uh2kH.

thomaslima commented 6 years ago

More two cents:

I think the module in python could be simply called klayout. It is still free so far: https://pypi.org/search/?q=klayout
Since the C++-code is very type oriented, maybe the interface could use static typing as well (http://mypy-lang.org). Even if it's too much for now, at least make the transition easy with enhanced docstrings.

klayoutmatthias commented 6 years ago

Thanks for the feedback so far. I'm sorry the branch is in a quite early stage and it only builds on Linux so far - I'm updating it frequently and I hope I can provide a functional binary distribution for trying. Specifically Windows is an issue.

I haven't seen a static type system inside the Python C API, so I wonder how to support it. But that's for later.

Is anything wrong with "pykl"? I can still change it - I just thought the name should be short. "klayout" is feasible of course.

Thanks,

Matthias

lukasc-ubc commented 6 years ago

I also prefer “import klayout". Why use a different name? It is much more clear from the branding and code readability perspective.

klayoutmatthias commented 6 years ago

I see: two votes for "klayout" :-)

amccaugh commented 6 years ago

Thirded for "klayout", or even "pyklayout". The trouble with "pykl" is it sounds (at least in my head) like "pickle", which I don't think you want it to be confused with when you're talking to colleagues

ejprinz commented 6 years ago

I think "klayout" would be best. This way when people google for the module they end up at klayout.de which is where they should be going. Also, can you let us know what environment actually works for building (e.g. Ubuntu 18.04 / Qt 5)? It didn't build for me in Ubuntu 16.04 / Qt 4), the install failed. Also, I would focus on the non-Qt stuff first since I don't see a use case for starting the GUI under Python, and Python Qt bindings already exist. My use case is to build the layout with PyCharm => Python => KLayout => gds and use the GUI Klayout for checking (after generating new gds it auto-updates the gds which is very nice :-) ).

klayoutmatthias commented 6 years ago

Hi all,

thanks for basically a 100% votes for "klayout" :-)

I have changed the name of the module and updated the Wiki page accordingly. The branch status is more and more stable. I think I will soon be able to provide pre-built version for download on Windows and Ubuntu/CentOS.

Matthias

TianyangGai commented 6 years ago

Dear Matthias: I've built it, but only the files in the klayout folder are not imported correctly. Is there a problem with my build? aeolus@aeolus-Lenovo:~/Desktop/klayout-pymod/bin-release/pymod/klayout$ ls db.so QtDesigner.so QtPrintSupport.so QtXmlPatterns.so __init__.py QtGui.so QtSql.so QtXml.so lay.so QtMultimedia.so QtSvg.so rdb.so QtCore.so QtNetwork.so QtWidgets.so tl.so ;-)

ejprinz commented 6 years ago

Hi Tianyang Gai, can you share your setup? I don't even get that far. I am getting some symbol errors. It would be best to all agree on one setup first before trying others. Is everyone else running on Ubuntu 16.04 / Qt5? I would like to get to the point where I can actually contribute and for this I need a fully functional build. I am willing to install any Linux OS needed (I am not on 18.04 yet, due to quality requirements from my end customers at home).

thomaslima commented 6 years ago

The build in my forked branch based on pymod (30aff1e) succeeded but I cannot start klayout. I am pasting my build and packaging logs. Will test pymod nonetheless. qt5.build.macos-HighSierra-release-30aff1e.log qt5.pkg.macos-HighSierra-release-30aff1e.log

This is the error output:

$ klayout.app/Contents/MacOS/klayout
Warning: Unable to find GSI class binding for: N2Qt15HitTestAccuracyE
Warning: Unable to find GSI class binding for: 5QPairIiiE
Warning: Unable to find GSI class binding for: N2Qt16ApplicationStateE
Warning: Unable to find GSI class binding for: 6QFlagsIN2Qt19ImageConversionFlagEE
Warning: Unable to find GSI class binding for: N2Qt10BrushStyleE
Warning: Unable to find GSI class binding for: N2Qt11GlobalColorE
Warning: Unable to find GSI class binding for: 6QFlagsIN2Qt16KeyboardModifierEE
Warning: Unable to find GSI class binding for: N2Qt11CursorShapeE
Warning: Unable to find GSI class binding for: N2Qt10DropActionE
Warning: Unable to find GSI class binding for: 6QFlagsIN2Qt10DropActionEE
Warning: Unable to find GSI class binding for: 6QFlagsIN2Qt11MouseButtonEE
Warning: Unable to find GSI class binding for: 6QFlagsIN9QIODevice12OpenModeFlagEE
Warning: Unable to find GSI class binding for: N2Qt11FocusReasonE
Warning: Unable to find GSI class binding for: N2Qt13TextElideModeE
Warning: Unable to find GSI class binding for: N2Qt15LayoutDirectionE
Warning: Unable to find GSI class binding for: 6QFlagsIN2Qt13AlignmentFlagEE
Warning: Unable to find GSI class binding for: N2Qt8MaskModeE
Warning: Unable to find GSI class binding for: N2Qt15AspectRatioModeE
Warning: Unable to find GSI class binding for: N2Qt18TransformationModeE
Warning: Unable to find GSI class binding for: 6QFlagsIN2Qt16InputMethodQueryEE
Warning: Unable to find GSI class binding for: N2Qt16InputMethodQueryE
Warning: Unable to find GSI class binding for: N2Qt11MouseButtonE
Warning: Unable to find GSI class binding for: 6QFlagsIN2Qt14MouseEventFlagEE
Warning: Unable to find GSI class binding for: N2Qt16MouseEventSourceE
Warning: Unable to find GSI class binding for: N2Qt17NativeGestureTypeE
Warning: Unable to find GSI class binding for: 9QMarginsF
Warning: Unable to find GSI class binding for: 8QMargins
Warning: Unable to find GSI class binding for: N2Qt17ScreenOrientationE
Warning: Unable to find GSI class binding for: N2Qt14WindowModalityE
Warning: Unable to find GSI class binding for: N2Qt11WindowStateE
Warning: Unable to find GSI class binding for: N2Qt6BGModeE
Warning: Unable to find GSI class binding for: N2Qt13ClipOperationE
Warning: Unable to find GSI class binding for: N2Qt8FillRuleE
Warning: Unable to find GSI class binding for: N2Qt8SizeModeE
Warning: Unable to find GSI class binding for: N2Qt8PenStyleE
Warning: Unable to find GSI class binding for: N2Qt11PenCapStyleE
Warning: Unable to find GSI class binding for: N2Qt12PenJoinStyleE
Warning: Unable to find GSI class binding for: 6QFlagsIN2Qt17ScreenOrientationEE
Warning: Unable to find GSI class binding for: N2Qt10CheckStateE
Warning: Unable to find GSI class binding for: 6QFlagsIN2Qt8ItemFlagEE
Warning: Unable to find GSI class binding for: N2Qt9SortOrderE
Warning: Unable to find GSI class binding for: 6QFlagsIN2Qt9MatchFlagEE
Warning: Unable to find GSI class binding for: N2Qt11OrientationE
Warning: Unable to find GSI class binding for: N2Qt10TextFormatE
Warning: Unable to find GSI class binding for: N2Qt16TabFocusBehaviorE
Warning: Unable to find GSI class binding for: N2Qt15CursorMoveStyleE
Warning: Unable to find GSI class binding for: 6QFlagsIN2Qt15TouchPointStateEE
Warning: Unable to find GSI class binding for: N2Qt15TouchPointStateE
Warning: Unable to find GSI class binding for: N2Qt4AxisE
Warning: Unable to find GSI class binding for: N2Qt11ScrollPhaseE
Warning: Unable to find GSI class binding for: 6QFlagsIN2Qt10WindowTypeEE
Warning: Unable to find GSI class binding for: N2Qt10WindowTypeE
Warning: Unable to find GSI class binding for: 6QFlagsIN2Qt11WindowStateEE
Warning: Unable to find GSI class binding for: N2Qt16InputMethodQueryE
Warning: Unable to find GSI class binding for: N2Qt17ItemSelectionModeE
Warning: Unable to find GSI class binding for: N2Qt13TextElideModeE
Warning: Unable to find GSI class binding for: N2Qt10DropActionE
Warning: Unable to find GSI class binding for: 6QFlagsIN2Qt16KeyboardModifierEE
Warning: Unable to find GSI class binding for: 6QFlagsIN19QItemSelectionModel13SelectionFlagEE
Warning: Unable to find GSI class binding for: 8QMargins
Warning: Unable to find GSI class binding for: 6QFlagsIN2Qt10DropActionEE
Warning: Unable to find GSI class binding for: 6QFlagsIN2Qt13AlignmentFlagEE
Warning: Unable to find GSI class binding for: N2Qt15ScrollBarPolicyE
Warning: Unable to find GSI class binding for: N2Qt11OrientationE
Warning: Unable to find GSI class binding for: 6QFlagsIN11QAccessible12RelationFlagEE
Warning: Unable to find GSI class binding for: 5QPairIP20QAccessibleInterface6QFlagsIN11QAccessible12RelationFlagEEE
Warning: Unable to find GSI class binding for: N2Qt15ShortcutContextE
Warning: Unable to find GSI class binding for: N2Qt16ApplicationStateE
Warning: Unable to find GSI class binding for: N2Qt15LayoutDirectionE
Warning: Unable to find GSI class binding for: N2Qt8UIEffectE
Warning: Unable to find GSI class binding for: 6QFlagsIN2Qt11OrientationEE
Warning: Unable to find GSI class binding for: N2Qt9DayOfWeekE
Warning: Unable to find GSI class binding for: N2Qt10CheckStateE
Warning: Unable to find GSI class binding for: N2Qt15CaseSensitivityE
Warning: Unable to find GSI class binding for: 6QFlagsIN2Qt9MatchFlagEE
Warning: Unable to find GSI class binding for: N2Qt8TimeSpecE
Warning: Unable to find GSI class binding for: 6QFlagsIN2Qt10WindowTypeEE
Warning: Unable to find GSI class binding for: 6QFlagsIN4QDir6FilterEE
Warning: Unable to find GSI class binding for: 6QFlagsIN2Qt8ItemFlagEE
Warning: Unable to find GSI class binding for: 6QFlagsIN4QDir8SortFlagEE
Warning: Unable to find GSI class binding for: N2Qt9SortOrderE
Warning: Unable to find GSI class binding for: 6QFlagsIN2Qt14DockWidgetAreaEE
Warning: Unable to find GSI class binding for: N2Qt14DockWidgetAreaE
Warning: Unable to find GSI class binding for: 6QFlagsIN11QFileDevice10PermissionEE
Warning: Unable to find GSI class binding for: N2Qt11GestureTypeE
Warning: Unable to find GSI class binding for: N2Qt12GestureStateE
Warning: Unable to find GSI class binding for: N2Qt11AnchorPointE
Warning: Unable to find GSI class binding for: N2Qt6CornerE
Warning: Unable to find GSI class binding for: N2Qt8SizeHintE
Warning: Unable to find GSI class binding for: N2Qt16CoordinateSystemE
Warning: Unable to find GSI class binding for: 6QFlagsIN2Qt11MouseButtonEE
Warning: Unable to find GSI class binding for: 6QFlagsIN2Qt15InputMethodHintEE
Warning: Unable to find GSI class binding for: N2Qt11FocusReasonE
Warning: Unable to find GSI class binding for: 5QPairId7QPointFE
Warning: Unable to find GSI class binding for: 5QPairIddE
Warning: Unable to find GSI class binding for: 6QFlagsIN2Qt11GestureFlagEE
Warning: Unable to find GSI class binding for: N2Qt18TransformationModeE
Warning: Unable to find GSI class binding for: N2Qt8FillRuleE
Warning: Unable to find GSI class binding for: N2Qt18WindowFrameSectionE
Warning: Unable to find GSI class binding for: N2Qt4AxisE
Warning: Unable to find GSI class binding for: N2Qt15AspectRatioModeE
Warning: Unable to find GSI class binding for: N2Qt22ItemSelectionOperationE
Warning: Unable to find GSI class binding for: N2Qt11MouseButtonE
Warning: Unable to find GSI class binding for: 6QFlagsIN2Qt14MouseEventFlagEE
Warning: Unable to find GSI class binding for: N2Qt16MouseEventSourceE
Warning: Unable to find GSI class binding for: 6QFlagsIN2Qt19TextInteractionFlagEE
Warning: Unable to find GSI class binding for: 6QFlagsIN8QPainter10RenderHintEE
Warning: Unable to find GSI class binding for: N2Qt11FocusPolicyE
Warning: Unable to find GSI class binding for: N2Qt15WidgetAttributeE
Warning: Unable to find GSI class binding for: N2Qt10WindowTypeE
Warning: Unable to find GSI class binding for: N2Qt10TextFormatE
Warning: Unable to find GSI class binding for: 6QFlagsIN2Qt15FindChildOptionEE
Warning: Unable to find GSI class binding for: N2Qt15CursorMoveStyleE
Warning: Unable to find GSI class binding for: N2Qt11ToolBarAreaE
Warning: Unable to find GSI class binding for: N2Qt15ToolButtonStyleE
Warning: Unable to find GSI class binding for: 6QFlagsIN2Qt11WindowStateEE
Warning: Unable to find GSI class binding for: N2Qt14WindowModalityE
Warning: Unable to find GSI class binding for: N2Qt15HitTestAccuracyE
Warning: Unable to find GSI class binding for: 6QFlagsIN13QTextDocument8FindFlagEE
Warning: Unable to find GSI class binding for: N2Qt8PenStyleE
Warning: Unable to find GSI class binding for: 6QFlagsIN2Qt11ToolBarAreaEE
Warning: Unable to find GSI class binding for: N2Qt9ArrowTypeE
Warning: Unable to find GSI class binding for: N2Qt17ContextMenuPolicyE
Warning: Unable to find GSI class binding for: 6QFlagsIN9QIODevice12OpenModeFlagEE
Warning: Unable to find GSI class binding for: 5QPairI10QByteArrayS0_E
Warning: Unable to find GSI class binding for: N2Qt11OrientationE
Warning: Unable to find GSI class binding for: N2Qt10DropActionE
Warning: Unable to find GSI class binding for: 6QFlagsIN2Qt8ItemFlagEE
Warning: Unable to find GSI class binding for: 6QFlagsIN2Qt9MatchFlagEE
Warning: Unable to find GSI class binding for: N2Qt9SortOrderE
Warning: Unable to find GSI class binding for: 6QFlagsIN2Qt10DropActionEE
Warning: Unable to find GSI class binding for: N2Qt15AspectRatioModeE
Warning: Unable to find GSI class binding for: N2Qt17ItemSelectionModeE
Warning: Unable to find GSI class binding for: N2Qt16InputMethodQueryE
Warning: Unable to find GSI class binding for: 5QPairIiiE
Warning: Unable to find GSI class binding for: N2Qt16InputMethodQueryE
Warning: Unable to find GSI class binding for: 6QFlagsIN2Qt10WindowTypeEE
Warning: Unable to find GSI class binding for: 9QMarginsF
Warning: Unable to find GSI class binding for: 5QPairI7QString6QSizeFE
Warning: Unable to find GSI class binding for: N2Qt17ItemSelectionModeE
Warning: Unable to find GSI class binding for: N2Qt16InputMethodQueryE
Warning: Unable to find GSI class binding for: 9QtMsgType
ERROR: std::bad_cast

thomaslima commented 6 years ago

I was able to import the klayout module. (the way the build works on mac, you have to be in the binary folder, the one that contains libklayout_*.dylib.

The tutorial script opened a blank canvas. I was not able to read from a very simple gds file.

$ PYTHONPATH=$(realpath pymod/) python3
Python 3.6.5 (default, Jun 22 2018, 12:43:45)
[GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from klayout.QtCore import *
Warning: Unable to find GSI class binding for: N2Qt15HitTestAccuracyE
Warning: Unable to find GSI class binding for: 5QPairIiiE
Warning: Unable to find GSI class binding for: N2Qt16ApplicationStateE
Warning: Unable to find GSI class binding for: 6QFlagsIN2Qt19ImageConversionFlagEE
Warning: Unable to find GSI class binding for: N2Qt10BrushStyleE
Warning: Unable to find GSI class binding for: N2Qt11GlobalColorE
Warning: Unable to find GSI class binding for: 6QFlagsIN2Qt16KeyboardModifierEE
Warning: Unable to find GSI class binding for: N2Qt11CursorShapeE
Warning: Unable to find GSI class binding for: N2Qt10DropActionE
Warning: Unable to find GSI class binding for: 6QFlagsIN2Qt10DropActionEE
Warning: Unable to find GSI class binding for: 6QFlagsIN2Qt11MouseButtonEE
Warning: Unable to find GSI class binding for: 6QFlagsIN9QIODevice12OpenModeFlagEE
Warning: Unable to find GSI class binding for: N2Qt11FocusReasonE
Warning: Unable to find GSI class binding for: N2Qt13TextElideModeE
Warning: Unable to find GSI class binding for: N2Qt15LayoutDirectionE
Warning: Unable to find GSI class binding for: 6QFlagsIN2Qt13AlignmentFlagEE
Warning: Unable to find GSI class binding for: N2Qt8MaskModeE
Warning: Unable to find GSI class binding for: N2Qt15AspectRatioModeE
Warning: Unable to find GSI class binding for: N2Qt18TransformationModeE
Warning: Unable to find GSI class binding for: 6QFlagsIN2Qt16InputMethodQueryEE
Warning: Unable to find GSI class binding for: N2Qt16InputMethodQueryE
Warning: Unable to find GSI class binding for: N2Qt11MouseButtonE
Warning: Unable to find GSI class binding for: 6QFlagsIN2Qt14MouseEventFlagEE
Warning: Unable to find GSI class binding for: N2Qt16MouseEventSourceE
Warning: Unable to find GSI class binding for: N2Qt17NativeGestureTypeE
Warning: Unable to find GSI class binding for: 9QMarginsF
Warning: Unable to find GSI class binding for: 8QMargins
Warning: Unable to find GSI class binding for: N2Qt17ScreenOrientationE
Warning: Unable to find GSI class binding for: N2Qt14WindowModalityE
Warning: Unable to find GSI class binding for: N2Qt11WindowStateE
Warning: Unable to find GSI class binding for: N2Qt6BGModeE
Warning: Unable to find GSI class binding for: N2Qt13ClipOperationE
Warning: Unable to find GSI class binding for: N2Qt8FillRuleE
Warning: Unable to find GSI class binding for: N2Qt8SizeModeE
Warning: Unable to find GSI class binding for: N2Qt8PenStyleE
Warning: Unable to find GSI class binding for: N2Qt11PenCapStyleE
Warning: Unable to find GSI class binding for: N2Qt12PenJoinStyleE
Warning: Unable to find GSI class binding for: 6QFlagsIN2Qt17ScreenOrientationEE
Warning: Unable to find GSI class binding for: N2Qt10CheckStateE
Warning: Unable to find GSI class binding for: 6QFlagsIN2Qt8ItemFlagEE
Warning: Unable to find GSI class binding for: N2Qt9SortOrderE
Warning: Unable to find GSI class binding for: 6QFlagsIN2Qt9MatchFlagEE
Warning: Unable to find GSI class binding for: N2Qt11OrientationE
Warning: Unable to find GSI class binding for: N2Qt10TextFormatE
Warning: Unable to find GSI class binding for: N2Qt16TabFocusBehaviorE
Warning: Unable to find GSI class binding for: N2Qt15CursorMoveStyleE
Warning: Unable to find GSI class binding for: 6QFlagsIN2Qt15TouchPointStateEE
Warning: Unable to find GSI class binding for: N2Qt15TouchPointStateE
Warning: Unable to find GSI class binding for: N2Qt4AxisE
Warning: Unable to find GSI class binding for: N2Qt11ScrollPhaseE
Warning: Unable to find GSI class binding for: 6QFlagsIN2Qt10WindowTypeEE
Warning: Unable to find GSI class binding for: N2Qt10WindowTypeE
Warning: Unable to find GSI class binding for: 6QFlagsIN2Qt11WindowStateEE
Warning: Unable to find GSI class binding for: N2Qt16InputMethodQueryE
Warning: Unable to find GSI class binding for: N2Qt17ItemSelectionModeE
Warning: Unable to find GSI class binding for: N2Qt13TextElideModeE
Warning: Unable to find GSI class binding for: N2Qt10DropActionE
Warning: Unable to find GSI class binding for: 6QFlagsIN2Qt16KeyboardModifierEE
Warning: Unable to find GSI class binding for: 6QFlagsIN19QItemSelectionModel13SelectionFlagEE
Warning: Unable to find GSI class binding for: 8QMargins
Warning: Unable to find GSI class binding for: 6QFlagsIN2Qt10DropActionEE
Warning: Unable to find GSI class binding for: 6QFlagsIN2Qt13AlignmentFlagEE
Warning: Unable to find GSI class binding for: N2Qt15ScrollBarPolicyE
Warning: Unable to find GSI class binding for: N2Qt11OrientationE
Warning: Unable to find GSI class binding for: 6QFlagsIN11QAccessible12RelationFlagEE
Warning: Unable to find GSI class binding for: 5QPairIP20QAccessibleInterface6QFlagsIN11QAccessible12RelationFlagEEE
Warning: Unable to find GSI class binding for: N2Qt15ShortcutContextE
Warning: Unable to find GSI class binding for: N2Qt16ApplicationStateE
Warning: Unable to find GSI class binding for: N2Qt15LayoutDirectionE
Warning: Unable to find GSI class binding for: N2Qt8UIEffectE
Warning: Unable to find GSI class binding for: 6QFlagsIN2Qt11OrientationEE
Warning: Unable to find GSI class binding for: N2Qt9DayOfWeekE
Warning: Unable to find GSI class binding for: N2Qt10CheckStateE
Warning: Unable to find GSI class binding for: N2Qt15CaseSensitivityE
Warning: Unable to find GSI class binding for: 6QFlagsIN2Qt9MatchFlagEE
Warning: Unable to find GSI class binding for: N2Qt8TimeSpecE
Warning: Unable to find GSI class binding for: 6QFlagsIN2Qt10WindowTypeEE
Warning: Unable to find GSI class binding for: 6QFlagsIN4QDir6FilterEE
Warning: Unable to find GSI class binding for: 6QFlagsIN2Qt8ItemFlagEE
Warning: Unable to find GSI class binding for: 6QFlagsIN4QDir8SortFlagEE
Warning: Unable to find GSI class binding for: N2Qt9SortOrderE
Warning: Unable to find GSI class binding for: 6QFlagsIN2Qt14DockWidgetAreaEE
Warning: Unable to find GSI class binding for: N2Qt14DockWidgetAreaE
Warning: Unable to find GSI class binding for: 6QFlagsIN11QFileDevice10PermissionEE
Warning: Unable to find GSI class binding for: N2Qt11GestureTypeE
Warning: Unable to find GSI class binding for: N2Qt12GestureStateE
Warning: Unable to find GSI class binding for: N2Qt11AnchorPointE
Warning: Unable to find GSI class binding for: N2Qt6CornerE
Warning: Unable to find GSI class binding for: N2Qt8SizeHintE
Warning: Unable to find GSI class binding for: N2Qt16CoordinateSystemE
Warning: Unable to find GSI class binding for: 6QFlagsIN2Qt11MouseButtonEE
Warning: Unable to find GSI class binding for: 6QFlagsIN2Qt15InputMethodHintEE
Warning: Unable to find GSI class binding for: N2Qt11FocusReasonE
Warning: Unable to find GSI class binding for: 5QPairId7QPointFE
Warning: Unable to find GSI class binding for: 5QPairIddE
Warning: Unable to find GSI class binding for: 6QFlagsIN2Qt11GestureFlagEE
Warning: Unable to find GSI class binding for: N2Qt18TransformationModeE
Warning: Unable to find GSI class binding for: N2Qt8FillRuleE
Warning: Unable to find GSI class binding for: N2Qt18WindowFrameSectionE
Warning: Unable to find GSI class binding for: N2Qt4AxisE
Warning: Unable to find GSI class binding for: N2Qt15AspectRatioModeE
Warning: Unable to find GSI class binding for: N2Qt22ItemSelectionOperationE
Warning: Unable to find GSI class binding for: N2Qt11MouseButtonE
Warning: Unable to find GSI class binding for: 6QFlagsIN2Qt14MouseEventFlagEE
Warning: Unable to find GSI class binding for: N2Qt16MouseEventSourceE
Warning: Unable to find GSI class binding for: 6QFlagsIN2Qt19TextInteractionFlagEE
Warning: Unable to find GSI class binding for: 6QFlagsIN8QPainter10RenderHintEE
Warning: Unable to find GSI class binding for: N2Qt11FocusPolicyE
Warning: Unable to find GSI class binding for: N2Qt15WidgetAttributeE
Warning: Unable to find GSI class binding for: N2Qt10WindowTypeE
Warning: Unable to find GSI class binding for: N2Qt10TextFormatE
Warning: Unable to find GSI class binding for: 6QFlagsIN2Qt15FindChildOptionEE
Warning: Unable to find GSI class binding for: N2Qt15CursorMoveStyleE
Warning: Unable to find GSI class binding for: N2Qt11ToolBarAreaE
Warning: Unable to find GSI class binding for: N2Qt15ToolButtonStyleE
Warning: Unable to find GSI class binding for: 6QFlagsIN2Qt11WindowStateEE
Warning: Unable to find GSI class binding for: N2Qt14WindowModalityE
Warning: Unable to find GSI class binding for: N2Qt15HitTestAccuracyE
Warning: Unable to find GSI class binding for: 6QFlagsIN13QTextDocument8FindFlagEE
Warning: Unable to find GSI class binding for: N2Qt8PenStyleE
Warning: Unable to find GSI class binding for: 6QFlagsIN2Qt11ToolBarAreaEE
Warning: Unable to find GSI class binding for: N2Qt9ArrowTypeE
Warning: Unable to find GSI class binding for: N2Qt17ContextMenuPolicyE
>>> from klayout.QtGui import *
>>> from klayout.QtWidgets import *
>>> from klayout.lay import *
>>>
>>> app = QApplication([ "my_app" ])
>>>
>>> w = LayoutView()
>>> w.load_layout("myfile.gds", False)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: Stream has unknown format: myfile.gds in LayoutView.load_layout
>>> w.max_hier()
>>> w.show()
>>>
>>> app.exec_()
0
>>> # ensures that w is destroyed before app.
... # Otherwise Qt segfaults.
... w = None
>>>

ejprinz commented 6 years ago

OK, I would say try to create one shape first and save it as gds file. If that works, read it back in. Also, is your fork on github and if so how can I get it? (I am relatively new to github). Thx.

thomaslima commented 6 years ago

@ejprinz that is what I did. I created a file in klayout manually with a box in layer 1/0. Then saved to myfile.gds. Then tried to load the layout, which is when I got the error.

Here is the link to the commit cited above: https://github.com/lightwave-lab/klayout/commit/30aff1ef60e4349f48c063cfbb08dc4a16edc4c2

If you want to see all forks of the project, you just have to go here: https://github.com/klayoutmatthias/klayout/network

TianyangGai commented 6 years ago

@ejprinz This is my build log, although it told me that the build is successful in the end, I still can't import correctly.

klayout-pymod-build.log

aeolus@aeolus-Lenovo:~/Desktop/klayout-pymod$ python3
Python 3.5.2 (default, Nov 23 2017, 16:37:01) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import klayout
>>> klayout
<module 'klayout' from '/home/aeolus/Desktop/klayout-pymod/bin-release/pymod/klayout/__init__.py'>
>>> from klayout.QtCore import *
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: libklayout_tl.so.0: cannot open shared object file: No such file or directory
>>> from klayout.lay import *
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: libklayout_tl.so.0: cannot open shared object file: No such file or directory

thomaslima commented 6 years ago

@TianyangGai I think I know what is going on. You gotta start python3 from the directory that contains the libklayout_to.so file, which in your case would be bin-release. But at the same time, your PYTHONPATH should point to pymod, otherwise klayout won't be imported. Check how I started my python interpreter above:

PYTHONPATH=$(realpath pymod/) python3

klayoutmatthias commented 6 years ago

Hi all,

thanks a lot for your efforts to get things running.

Configuration and installation is easier when you specify an installation target. For example:

build.sh -bin ~/klayout-inst ...

This will set the RPATH inside the libraries. With this, you just need to set the PYTHONPATH to the corresponding package directory. In the example above:

export PYTHONPATH=~/klayout-inst/pymod

I have set up an automated build for the pymod branch. Currently for some Linux flavours. In effect, I can provide rpm and deb packages for CentOS6, 7 and Ubunut 16, 18. These packages will not just install KLayout, but also Python packages for the default Python installation.

Please find the download links and documentation links here: http://www.klayout.org/downloads/pymod/

Matthias

atait commented 6 years ago

Here is a script to simplify the linking process post-build. I haven't tried running the solution just posted. Seems much better since it works during build and actually uses RPATH properly, but I'll post this script anyways.

https://gist.github.com/atait/faeb785038e8f6888e40702582126a86

ejprinz commented 6 years ago

OK, I was able to compile the pymod branch after setting a -bin target and providing a correct gcc executable (this is basically on RHEL 6, with gcc 5.4.0. I get the same error on reading a gds as mentioned before, which is great progress, so I'll setup PyCharm and start debugging .... I would be surprised if all the options needed by the reader from LoadLayoutOptions are by default working, I assume this gets set in the normal klayout program with the preferences etc. I'll write some test cases...

atait commented 6 years ago

I made some headway on packaging a standalone distribution of the python package. It is on the lightwave-lab fork at lightwave-lab/klayout@42b5a6.

After building, you run python pymod-distribute.py to create a wheel file, which is basically a zip file with some metadata and a self-contained python package. The wheel file can be installed with pip directly. It can also be uploaded to pypi using twine. The wheel I made on macos HighSierra with Brew python 3.6 and Brew Qt5 is available here.

A wheel made on macos will not work on Linux; however, pymod-distribute should theoretically produce something that works on the same machine. A nice feature of pypi, is that you can upload wheels for a bunch of OS's and pip will figure out which one to pull down based on your OS. The end goal here is "pip install klayout".

Technical notes The binary shared objects produced by build.sh are linked to dynamic libraries such as

libklayout_tl.0.dylib (compatibility version 0.26.0, current version 0.26.0)

The search path is relative to the current directory which is bad. Currently, when the klayout application is packaged, it comes with Frameworks which are referenced relatively by changing this link to

@executable_path/../Frameworks/libklayout_tl.0.dylib (compatibility ...)

The issue with a python package is that there is no executable. It is more like a plugin, which needs something like

@loader_path/.dylibs/libklayout_tl.0.dylib (compatibility ...)

This is worked out by delocate (see stackoverflow). I'm afraid delocate in its current form won't port to windows, maybe not even Linux, although it could be modified to do so. An alternative approach would be to affix these things somewhere in the qmake build, specifically in src/pymod/pymod.pri.

thomaslima commented 6 years ago

Another thought in complement to @atait's post:

If some part of klayout is made to be lightweight enough, it can be compiled directly by distutils https://docs.python.org/3/extending/building.html

atait commented 6 years ago

The Extensions feature is doing something similar to gcc make. So I think there are two options: pymod-specific gcc options in pymod.pri, so that it links properly, or: distutils Extension with dependencies on the rest of klayout.

The benefit of option 1 is that this flow is already in place. The change might (hopefully) be a matter of a well-placed compiler tag. Benefit of option 2 is that distutils is going to be used in the following packaging step – more likely to produce something it understands.

klayoutmatthias commented 6 years ago

Hi all,

thanks for these comments. My personal goal is to reduce the dependency on Qt. I think the core libraries can basically made compile without Qt. There are very few more dependencies, so I think this will achieve the lightweightness Thomas mentioned.

Thanks,

Matthias

thomaslima commented 6 years ago

This is great. After this is done, all I'll have to do is change from pya to klayout.db in my scripts. :)

klayoutmatthias commented 6 years ago

I made some progress - the pymod branch now can basically be built without Qt offering the very core features as Python modules. So a distutils-based build is coming closer.

Qt-less build is enabled with

build.sh -without-qt -libexpat ...

Essentially this will only build the buddy tools and the python modules without the higher-level modules.

In order to emulate a few features from Qt I depend on two (common) third-party libraries:

expat for XML parsing
pthread for threads

Both apparently are available on most platforms including Windows.

Matthias

atait commented 6 years ago

Thanks for advancing this Matthias! A distutils build would be the ideal. I gave the new pymod build a shot today and got some errors. I fixed them to the extent that building worked (with qt; I think -without-qt also works, but haven't characterized as well).

Had to add this line to tlFileUtils.cc
```
#include <libproc.h>
```

MacOS does not have pthread_timedjoin_np, so I followed the solution here, dropping this block into tlThreads.cc


#ifdef __APPLE__
#ifndef ETIMEDOUT
#define ETIMEDOUT 60
#endif
struct args {
int joined;
pthread_t td;
pthread_mutex_t mtx;
pthread_cond_t cond;
void **res;
};

static void waiter(void ap) { struct args *args = ap; pthread_join(args->td, args->res); pthread_mutex_lock(&args->mtx); args->joined = 1; pthread_mutex_unlock(&args->mtx); pthread_cond_signal(&args->cond); return 0; }

int pthread_timedjoin_np(pthread_t td, void *res, struct timespec ts) { pthread_t tmp; int ret; struct args args = { .td = td, .res = res };

pthread_mutex_init(&args.mtx, 0);
pthread_cond_init(&args.cond, 0);
pthread_mutex_lock(&args.mtx);

ret = pthread_create(&tmp, 0, waiter, &args);
if (ret) goto done;

do ret = pthread_cond_timedwait(&args.cond, &args.mtx, ts);
while (!args.joined && ret != ETIMEDOUT);

pthread_mutex_unlock(&args.mtx);

pthread_cancel(tmp);
pthread_join(tmp, 0);

pthread_cond_destroy(&args.cond);
pthread_mutex_destroy(&args.mtx);

return args.joined ? 0 : ret;

}

endif


3. Then I got an error related to gerber. Log is attached below. I just deleted that plugin from src to see what would happen.

[GerberError.log](https://github.com/klayoutmatthias/klayout/files/2194505/GerberError.log)

### Build and deploy finished
As before, the produced klayout.app does not open. Also as before, the klayout python module needs to be delocated or added to the PYTHONPATH and DYLD_LIBRARY_PATH. klayout.db imports fine and behaves as expected except for read/write. When I do
```python
layout2.write(gdsname)
ipc.load(gdsname)

I get

ERROR: for_pya.py:36: Cannot determine format from filename in Layout.write

Changing that to

conf = pya.SaveLayoutOptions()
conf.format = 'GDS2'
layout.write(gdsname, conf)
ipc.load(gdsname)

yields

ERROR: for_pya.py:36: Unknown stream format: GDS2 in Layout.write

Without writing, it is hard to see the behavior of klayout.db in more detail. Does it work for anybody else?

Here is the log of my successful build success-withQt.log

klayoutmatthias commented 6 years ago

Hi atait and all others,

I have provided a first version of setup.py in the pymod branch.

It builds on MacOS (a virtual machine with HighSierra) simply by using:

sudo python3 setup.py install

The issues you mentioned should be solved at least for this configuration. Thanks for this analysis.

On Linux, setup.py works out of the box. Windows support is coming next.

Thanks and best regards,

Matthias

thomaslima commented 6 years ago

@klayoutmatthias thank you for this monumental work!

It works! Here's what I did on mac:

brew install expat (I already had curl)
made virtual environment for klayout and activated it
python setup.py install
python test-klayoutdb.py

Contents of test-klayoutdb.py:

import klayout.db as kdb

layout = kdb.Layout()

layer1 = layout.layer(kdb.LayerInfo('1/0'))

TOP = layout.create_cell("TOP")

box = kdb.DBox(kdb.DPoint(-4500, -4500), kdb.DPoint(4500, 4500))

TOP.shapes(layer1).insert(box)

layout.write('test.gds')

The gds file was written properly. Hope it helps other mac users for initial testing.

I have caught a segfault happening when I call help on the objects (it works on klayout, though).

>>> import klayout.db as kdb
>>> help(kdb)  # works, but simply shows klayout.db = <module 'klayout.db' from '/Users/tlima/Envs/kla.../site-packages/klayout/db.cpython-37m-darwin.so'>

>>> help(kdb.anything)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'klayout.db' has no attribute 'anything'
>>> help(kdb.Box)
[1]    32131 segmentation fault  python

thomaslima commented 6 years ago

I have detected another issue here. After install, the libraries had hard-coded dependencies, which means that it would only work if python was run in klayout's source directory. I am not 100% certain because after my initial test I was playing around with setuptools vs. distutils (I recommend migration to setuptools). This is related to @atait 's post above about distributing a wheel.

Posting another useful link here: https://stackoverflow.com/questions/47380150/dealing-with-dylibs-when-distributing-python-packages

otool -L ~/Envs/klayout/lib/python3.7/site-packages/klayout/db.cpython-37m-darwin.so
/Users/tlima/Envs/klayout/lib/python3.7/site-packages/klayout/db.cpython-37m-darwin.so:
    build/lib.macosx-10.13-x86_64-3.7/klayout/_db.cpython-37m-darwin.so (compatibility version 0.0.0, current version 0.0.0)
    build/lib.macosx-10.13-x86_64-3.7/klayout/_tl.cpython-37m-darwin.so (compatibility version 0.0.0, current version 0.0.0)
    build/lib.macosx-10.13-x86_64-3.7/klayout/_gsi.cpython-37m-darwin.so (compatibility version 0.0.0, current version 0.0.0)
    build/lib.macosx-10.13-x86_64-3.7/klayout/_pya.cpython-37m-darwin.so (compatibility version 0.0.0, current version 0.0.0)
    /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 400.9.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.50.4)

Edit: Fixed this with #143 . With this, it is likely that we won't even need to use delocate to distribute the wheel.

atait commented 6 years ago

I reproduced @thomaslima's behavior. Startup is super fast! For simple scripts, like the one above, it takes about 10% as long as running in application batch mode.

The klayout python package can be used anywhere by delocating its libraries after build, before distribution. First, you need to pip install delocate. Then

python3 setup.py build
libdir=$(ls build/lib.*)/klayout
delocate-path -L . $libdir
python3 setup.py install

This klayout package can be distributed to other machines most easily as a wheel. To make that, first, you need to import setuptools somewhere in setup.py. Then (after delocating),

python setup.py bdist_wheel
wheelfile=$(ls dist/*.whl)
pip install $wheelfile

The wheel only works on the same OS, same architecture, same version of python. A typical approach is to build multiple versions and upload them all to the package manager (like PyPI); however, the delocate package is OSX only. There is a similar package for Linux called auditwheel. Windows is a mystery to me, but linking should follow the same idea.

atait commented 6 years ago

Here are wheels for python 3.6 and 3.7 on dropbox.

thomaslima commented 6 years ago

Hey @atait,

Do you still need delocate after the fix I proposed in #143?

Best

Thomas

On 16 Jul 2018, at 03:54, Alex Tait notifications@github.com<mailto:notifications@github.com> wrote:

Here are wheels for python [3.6] and 3.7https://www.dropbox.com/s/n8czqinht2c6trk/klayout-0.26-cp37-cp37m-macosx_10_13_x86_64.whl?dl=0 on dropbox.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/klayoutmatthias/klayout/issues/130#issuecomment-405171996, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ABCck3vbNQjXq1r5xed2ALWR4m6O8WG0ks5uHEa9gaJpZM4Uh2kH.

thomaslima commented 6 years ago

I have confirmed, @atait and others, that #143 can be distributed as a wheel without delocation. All you need is python setup.py bdist_wheel.

@klayoutmatthias could you provide examples of python code that would need expat and/or curl so I could test?

Second question: how can we make these python namespaces (klayout.db) available inside klayout? I think aliases to pya would be sufficient. It would be nice to be able to run the same scripts from python and from klayout.

atait commented 6 years ago

Yup, confirmed here too. Very nice @thomaslima. I think using @rpath instead of @loaderpath is closer to, if not already, linux-compatible.

For the time being, the quick workaround to your second question is

try:
    import pya
except ImportError:
    import klayout.db as pya

klayoutmatthias commented 6 years ago

Hi Thomas, hi Alex,

thanks for this testing and the contributions.

I'm on Linux where the relocation isn't required. setup.py will simply install where it belongs.

Is it possible to use Travis for continuous deployment of the wheels to PyPI? I think that's the smartest way to handling the deployment topic.

I'll take a look into the segfault issue on help. I noticed similar issues when using klayout.db.__dict__. Solving this shouldnt be a big deal as soon as I'm able to reproduce it.

@thomaslima regarding the expat and curl usable: expat is used on some occasions where XML is involved. This is the KLayout RDB format, technology files and the PCB project files. If you don't need this, you don't need expat. But as expat is the most popular XML library, I think it should basically be available on all platforms. curl is required if you want direct access to layout files through "http" URLs. This feature is not absolutely required for the basic functionality. It's just handy.

Regarding the Naming of the Module: I see some chance of making the KLayout application accept Python scripts with "klayout.db" too. But first priority for me is to get the standalone package up.

I'm currently fighting with the Windows build. I was able to adapt the setup.py script (mainly through heavy use of os.path.join instead of "x/y" paths). But for Windows building (specifically with Anaconda) I need 3rd party libraries - beside expat and curl (which are optional) this is pthread and zlib. Does anyone know what is the best way to include them for easy building on Windows? In contrast to MacOS/Linux I cannot assume they are installed somewhere on Windows. I could package their sources with KLayout's sources, but that feels a bit like stealing code. Any hints?

Thanks and best regards,

Matthias

klayoutmatthias commented 6 years ago

The equivalent of @loader_path on Linux is $ORIGIN. I have updated setup.py as it wasn't working with setuptools on Linux.

The original rpath scheme is obsolete so I removed it. Hope it's still working on MacOS.

Is there a Travis instance where I can check this?

Thanks,

Matthias

thomaslima commented 6 years ago

Yes. Travis can upload to pypi. I will try to add a template for you which you can fill with your credentials. Relocation might be a big deal even in linux because of virtual environments and different installation paths. I think we should fix that.
For x/y paths I use pathlib. It's nice and handy but has two different imports for python2/python3.
klayout.db.__dict__ also segfaults over here. Good catch!

thomaslima commented 6 years ago

I just pushed your commit to lightwave-lab/klayout, you can check the travis instance here: https://travis-ci.org/lightwave-lab/klayout/builds/404670379

I suggest you make an account in https://travis-ci.org (org, not com) to get it building for your repo as well. the travis.yml is already setup. Klayout normal build will error due to their build time limits, but if you email them you can ask for more. they granted me extra time limit for builds in lightwave-lab/klayout.

atait commented 6 years ago

The $ORIGIN approach is working for me on an Ubuntu 18.04 machine. Setuptools platlib was making assumptions about where the package would end up (/usr/local/lib/....). Now, it is possible to do a user install such as in ~/.local/lib/..... and is definitely more portable. Thanks for the update @klayoutmatthias.

It seems that $ORIGIN denotes the directory of the .so file, while @loaderpath denotes the root directory of the installed package/plugin. That was an issue for db_plugins like GDS writing, etc. Inserting these lines fixed that

if not '_dbpi' in mod:
    loader_path = '$ORIGIN'
else:
    loader_path = '$ORIGIN/..'
args += ['-Wl,-rpath,' + loader_path]

A more robust solution would be

def link_args(self, mod, loaderpath_relative_to_origin=''):
    ...
    args += ['-Wl,-rpath,$ORIGIN' + loaderpath_relative_to_origin]

thomaslima commented 6 years ago

@klayoutmatthias to include automatic deploy in travis (https://docs.travis-ci.com/user/deployment/pypi/):

deploy:
  provider: pypi
  user: klayoutmatthias
  password: ...
  distributions: "sdist bdist_wheel"
  on:
    tags: true
    branch: master

be careful with the version in setup.py. You can only upload a version once, which is really annoying. So I would start by a manual test of twine which is the pypi helper tool, but make sure you get your version to look something like 0.26-alpha0. Together with the matrix feature, that would allow you to build and upload one wheel for each major distribution.

klayoutmatthias commented 6 years ago

@atai You're right, I missed that. Thanks.

ejprinz commented 6 years ago

For what it's worth, I generated the Python module "klayout" using the setup.py file on RHEL6, generated a 40MB test vehicle (compressed gds), and did XOR against the same Python script run within KLayout (as batch file). The XOR was clean. So this is pretty amazing, since I can now use PyCharm with the built-in default Python and use the klayout Python module (and any other module). Great job !!

thomaslima commented 6 years ago

@klayoutmatthias : a request regarding the python modules.

Right now the setup.py script compiles all db c++ libs into _db.*.so (I'm using db as example). Then you make a c++ python extension in db.*.so which links the original _db. Is it possible to include the python module inside _db.so? so we can import as klayout._db. The purpose for that is to create a db.py module with extensions that looks like this:

from klayout._db import *

__all__ = []
__all__ += _db.__all__

# this allows a numpy array of Points to be multipliable by a factor 
# (this is a hack, there is a better way to do this)
PointLike = (pya.Point, pya.DPoint, pya.DVector, pya.Vector)

def pyaPoint__mul__(self, factor):
    """ This implements P * factor """
    if isinstance(factor, Number):
        return self.__class__(self.x * factor, self.y * factor)
    elif MODULE_NUMPY and isinstance(factor, np.ndarray):  # Numpy can multiply any object
        return factor.__mul__(self)
    elif isinstance(factor, PointLike):
        return self.x * factor.x + self.y * factor.y
    else:
        return NotImplemented

for klass in PointLike:
    klass.__mul__ = pyaPoint__mul__

This example was taken from https://github.com/lightwave-lab/SiEPIC-Tools/blob/lightwave/klayout_dot_config/python/SiEPIC/extend.py

What do you think?

atait commented 6 years ago

Any updates on this?

One application could be using klayout.db in a Docker container to perform XOR unit tests or chip builds in the cloud. Of course, you can't exactly build klayout on a docker, so this type of thing would be enabled by a "pip install klayout" for the general public.

@klayoutmatthias please post if there is anything we could potentially help with. If it's Windows holding it up, would you consider releasing a Linux/OSX compatible package for the time being?

thomaslima commented 5 years ago

Update: I think all of the issues in this thread have been fixed. python setup.py install should work in all platforms out of the box, except windows, which needs a few pre-combiled klayout-bits loaded in the library path (cf. https://github.com/klayoutmatthias/klayout/issues/202#issuecomment-442618888 , also look at azure-pipelines.yml). As of now, there is a formula for continuous deployment to pypi, so users should expect pip install klayout to work in linux, mac and windows. Available wheels are posted here: https://pypi.org/project/klayout/#files

KLayout / klayout

Please give feedback on standalone Python module #130

endif