pyside2-uic missing

[mottosso]

There might be something missing, or I'm doing it wrong. :(

Stack trace

$ pyside2-uic 
Traceback (most recent call last):
  File "/usr/local/bin/pyside2-uic", line 7, in <module>
    from PySide2.scripts.uic import main
  File "/usr/local/lib/python2.7/dist-packages/PySide2/scripts/uic.py", line 28, in <module>
    from pyside2uic.driver import Driver
  File "/usr/local/lib/python2.7/dist-packages/pyside2uic/__init__.py", line 27, in <module>
    from pyside2uic.Compiler import indenter, compiler
ImportError: No module named Compiler

Full reproducible

$ apt-get install wget python-pip python-dev software-properties-common
$ add-apt-repository ppa:beineri/opt-qt561-trusty
$ apt-get update
$ apt-get install qt56-meta-full
$ . /opt/qt56/bin/qt56-env.sh
$ wget https://bintray.com/fredrikaverpil/pyside2-wheels/download_file?file_path=ubuntu14.04%2FPySide2-2.0.0.dev0-cp27-none-linux_x86_64.whl -O PySide2-2.0.0.dev0-cp27-none-linux_x86_64.whl
$ pip install PySide2-2.0.0.dev0-cp27-none-linux_x86_64.whl
$ pyside2-uic

[fredrikaverpil]

Hm, could it be that we need to compile pyside-tools manually? https://codereview.qt-project.org/gitweb?p=pyside/pyside-tools.git;a=tree

[mottosso]

I can't say for sure, but it kind of looks like that is a pure-Python package. I'll give it a try.

[fredrikaverpil]

Some steps to actually be able to execute pyside2-uic, and this is on OS X:

1. Build the pyside-setup project (pyside2uic is part of the submodules).
2. Copy the built pyside2uic package from pyside-setup/pyside_package/pyside2uic into your local site-packages location.
3. Copy the binary from pyside_setup/pyside_install/py2.7-qt5.6.1-64bit-release/bin/pyside2-uic to your Python's bin directory (called "Scripts" on Windows).

[mottosso]

Would it be possible to include these in the wheel too?

[fredrikaverpil]

Most likely. Right off the bat, I'm not sure how to do it. I would have to read up on wheel building.

[fredrikaverpil]

I've noticed that on my system I actually did get pyside2uic installed when pip installing the PySide2 wheel. It's just that the Compiler module isn't included. Can you verify this @mottosso ?

You can do a find / -name pyside2uic to perform a search.

[mottosso]

Yes, I got that too.

[fredrikaverpil]

@mottosso What OS is that on? I'm seeing this on OS X.

[mottosso]

Ubuntu 14, from the wheel.

[mottosso]

The full reproducible is in the original post.

[fredrikaverpil]

https://bugreports.qt.io/browse/PYSIDE-357

[davidherran]

hello everyone. Sorry for my english ;(

I had a error when I try compile UI from pysideUIC on OS X:

Unknown Qt widget: QFont Unknown Qt widget: QIcon ... Unknown Qt widget: ....

I followed your recomendation...

Some steps to actually be able to execute pyside2-uic, and this is on OS X: Build the pyside-setup project (pyside2uic is part of the submodules). Copy the built pyside2uic package from pyside-setup/pyside_package/pyside2uic into your local site->packages location. Copy the binary from pyside_setup/pyside_install/py2.7-qt5.6.1-64bit-release/bin/pyside2-uic to your >Python's bin directory (called "Scripts" on Windows).

Finally, remplace all pyside2uic installation from site-packagesdir for https://github.com/pyside/pyside2-tools and works for me.

[fredrikaverpil]

Wow, the reported bug is still open/unresolved. 😞

[bjquinn]

Same here, I tried the workaround from davidherran, but still get the error:

ImportError: cannot import name 'compileUi'

[fredrikaverpil]

@davidherran @bjquinn it would be much better to poke QtC about this. The issue seems to be on their end. You can make a comment of this here for example: https://bugreports.qt.io/browse/PYSIDE-357

[bjquinn]

Yep, I tried to do that there as well, but my registration confirmation from them seems to get lost in the ether, so I'm not able to complete registration. I'll try again later.

[leycec]

I just hit this long-standing issue and am best unpleased. Unfortunately, the fault would indeed appear to lie on our end – not QtC's. Actually, after closer examination of the pyside-setup repository, it's all QtC's fault after all. Cue sad cat faces. :crying_cat_face:

The good news is that the upstream PYSIDE-357 issue can now be safely closed, because we're mostly to blame. Actually, upstream PYSIDE-357 issue should be left open, as QtC is absolutely to blame. The bad news is that our installation scripts will require substantial edits. As you yourself contemplated above, @fredrikaverpil:

Hm, could it be that we need to compile pyside-tools manually?

Yes, that is exactly what it could be. We have to do that. Because we didn't do that, the pyside2uic package remains effectively empty and hence broken.

Yes, this is arguably poor package installation design – and that at least is not our fault. A new upstream issue requesting that the pyside-setup project not install a stupidly broken pyside2uic package at all should probably be opened, but I am lazy and overtired and increasingly grumpy and shall not be doing that. ^{It's Friday night. Weep at the empty husk of my life.}

For inscrutable reasons that remain unclear to me, QtC elected not to merge the pyside-tools repository into the pyside-setup repository when they merged the PySide2 and Shiboken2 repositories together. They probably should have. But they didn't. In particular, PySide2's May 2017 developer notes proudly state that:

- merge of psyide repositories -> no objections against the plan in the community
  -> merge will happen in the next few days (pyside.git, pyside-setup.git and shiboken.git become one repo)
  -> examples, wiki and tools will remain as they are

this is awful ↗

Likewise, QtC performed literally no work on PySide2 during August because:

10. August 2017
- holiday time is hampering project at the moment -  not much progress

17. August 2017
- Almost everyone is still on holiday - not much progress

this is awful ↗

Which means that, even if QtC technically could help us, they won't. Which means that:

• All existing Python 3.x-specific build scripts need to be refactored to explicitly clone, compile, and install the 5.6 branch of the separate pyside-tools repository.
• All existing Python 3.x-specific wheels need to be rebuilt and reuploaded.

Your time appears limited, @fredrikaverpil. I can appreciate the pained expression presumably now sprouting across your face. My face is similarly distraught. Nonetheless, this is a significant blocker for numerous users (notably, me). It's also well within our capacity to fix.

Would you like a bit of assistance in doing so? My intuition is that the Ubuntu 16.04 + Python 3.5 Dockerfile would be a reasonable place for us to start hammering out a working solution. For reasons that should become clear shortly, the pyside-tools repository is not currently buildable for Python 2.7 – at least not on Gentoo. Why? Because this repository's top-level CMakeLists.txt makefile requires explicitly declaring the version of Python to be built against via a Python 3.x-specific extension tag (e.g., .cpython-34m for Python 3.4). You see the problem, I trust.

Yes, a new upstream issue requesting that this be clarified and/or corrected should probably be opened. Again, I am lazy and overtired and increasingly grumpy and shall not be doing that.

I'm afraid your Docker expertise exceeds my own by a generous margin. While completely untested, amending the aforementioned Dockerfile as follows should nudge us closer to where we want to be:

• Replace the trailing ENTRYPOINT with RUN. Maybe? I'm somewhat unclear as to the implications of ENTRYPOINT, but assume it behaves similar to that of exec in a traditional shell script.
• Appending some variant of the following lines:

RUN git clone --recursive --branch 5.6 https://codereview.qt-project.org/pyside/pyside-tools
WORKDIR pyside-tools

#FIXME: For Python 2-specific Dockerfiles, this should of course be changed to:
#RUN rm -rf pyside2uic/port_v3
# Remove Python 2-specific paths.
RUN rm -rf pyside2uic/port_v2

#FIXME: For different Python versions (e.g., Python 3.6), extension tags unique
#to those versions will need to be explicitly specified below.
# Configure "pyside-tools". For disambiguation, the top-level "CMakeLists.txt"
# makefile must be notified of where to find the previously installed
# "Shiboken2Config.*.cmake" and "PySide2Config.*.cmake" files specific to this
# Python version. Note that:
#
# * These variables accept a Python 3.x-specific extension tag unique to the
#   current Python 3.x version (e.g., ".cpython-34m" for CPython 3.4).
# * There currently appears to be no means of doing so for Python 2.7, due to
#   Python 2.7 providing no equivalent concept of a Python 3.x-specific
#   extension tag.
RUN /opt/cmake-3.5.2-Linux-x86_64/bin/cmake \
    -DPYTHON_EXTENSION_SUFFIX=".cpython-35m" \
    -DPYTHON_BASENAME=".cpython-35m"

#FIXME: Do we really need to end Dockerfiles on "ENTRYPOINT"? I have no idea.
# Compile and install "pyside-tools".
RUN make
ENTRYPOINT make install

The above instructions derive entirely from our working Gentoo ebuild for pyside-tools, which successfully installs the 5.6 branch of this repository (and thus the pyside2-uic command and pyside2uic package) for all supported Python versions. This includes Python 3.4, 3.5, and 3.6. It verifiably works for us. It should verifiably work for you, too. Or your unpaid money back!

I leave this to your capable expertise, Team PySide2-Wheels. Because I am tired and penniless.

[mottosso]

This gave me a good chuckle, thanks for the well articulated and humorous, late-night write-up @leycec. XD

[leycec]

Thank you! I'll be performing all evening for pennies and toothless grimaces. :grimacing:

Actually, I'm about to collapse. I couldn't help but investigate this a bit further, however, as the setup.py script for the pyside-setup repository is blatantly defective with respect to the pyside-tools Git submodule. It's also unreadable spaghetti code.

From what little I can tell of that vast morass of illegible "wizardry," the underlying issue might be that their setup.py script fails to pass the -DPYTHON_EXTENSION_SUFFIX and -DPYTHON_BASENAME options to CMake, as detailed in my Dockerfile snippet above. Or it might be something altogether more hideous.

Since QtC have been out-to-lunch for over a year on this issue, we can't really depend on a timely fix from their end. Instead, it would probably behove us to attempt to fix this on our end by manually installing pyside-tools ourselves – the right way.

[fredrikaverpil]

Hm. I was under the impression that all that was missing was the Compiler module which resides here. I thought somehow this didn't get copied into the wheel for some reason (probably because of setup.py). But you're saying we need to compile all of pyside2-tools prior to building PySide2 itself? To me, it seems this is done automatically when compiling the pyside-setup project.

• All existing Python 3.x-specific build scripts need to be refactored to explicitly clone, compile, and install the 5.6 branch of the separate pyside-tools repository.

Do you mean that cloning the 5.6 branch of PySide2 is not enough?

Here is the pyside-setup's tree and if you go into the sources dir, the pyside-tools folder is surprisingly non-clickable for some reason... but if you clone the repo, it's there:

$ git clone --recursive --branch 5.6 https://codereview.qt-project.org/pyside/pyside-setup
Cloning into 'pyside-setup'...
remote: Counting objects: 38108, done
remote: Finding sources: 100% (38108/38108)
remote: Total 38108 (delta 28036), reused 37909 (delta 28036)
Receiving objects: 100% (38108/38108), 13.00 MiB | 305.00 KiB/s, done.
Resolving deltas: 100% (28036/28036), done.
Submodule 'sources/pyside2-examples' (https://codereview.qt-project.org/pyside/examples.git) registered for path 'sources/pyside2-examples'
Submodule 'sources/pyside2-tools' (https://codereview.qt-project.org/pyside/pyside-tools.git) registered for path 'sources/pyside2-tools'
Submodule 'wiki' (https://github.com/PySide/pyside2.wiki.git) registered for path 'wiki'
Cloning into '/Users/fredrik/code/repos/pyside-setup/sources/pyside2-examples'...
remote: Counting objects: 3814, done        
remote: Finding sources: 100% (3814/3814)           
remote: Total 3814 (delta 1918), reused 3803 (delta 1918)        
Receiving objects: 100% (3814/3814), 9.89 MiB | 2.22 MiB/s, done.
Resolving deltas: 100% (1918/1918), done.
Cloning into '/Users/fredrik/code/repos/pyside-setup/sources/pyside2-tools'...
remote: Counting objects: 586, done        
remote: Finding sources: 100% (586/586)           
remote: Total 586 (delta 379), reused 585 (delta 379)        
Receiving objects: 100% (586/586), 181.38 KiB | 1.23 MiB/s, done.
Resolving deltas: 100% (379/379), done.
Cloning into '/Users/fredrik/code/repos/pyside-setup/wiki'...
remote: Counting objects: 241, done.        
remote: Total 241 (delta 0), reused 0 (delta 0), pack-reused 241        
Receiving objects: 100% (241/241), 138.59 KiB | 403.00 KiB/s, done.
Resolving deltas: 100% (106/106), done.
Submodule path 'sources/pyside2-examples': checked out '8df6dccecc5165f7c3ec5896c9be8baceda7161f'
Submodule path 'sources/pyside2-tools': checked out '7fe32567c75b6b9985b9efbecf74477c9e829fb2'
Submodule path 'wiki': checked out 'ac1b69fabc1a9f81da585fe1a1aa4188862ced66'

$ ll pyside-setup/sources/pyside2-tools
total 144
drwxr-xr-x  16 fredrik  staff   544B Aug 19 21:43 ./
drwxr-xr-x   7 fredrik  staff   238B Aug 19 21:43 ../
-rw-r--r--   1 fredrik  staff    49B Aug 19 21:43 .git
-rw-r--r--   1 fredrik  staff    61B Aug 19 21:43 .gitattributes
-rw-r--r--   1 fredrik  staff    12B Aug 19 21:43 .gitignore
-rw-r--r--   1 fredrik  staff   168B Aug 19 21:43 AUTHORS
-rw-r--r--   1 fredrik  staff   3.2K Aug 19 21:43 CMakeLists.txt
-rw-r--r--   1 fredrik  staff    18K Aug 19 21:43 LICENSE-rcc
-rw-r--r--   1 fredrik  staff    19K Aug 19 21:43 LICENSE-uic
-rw-r--r--   1 fredrik  staff   817B Aug 19 21:43 README.md
-rw-r--r--   1 fredrik  staff   940B Aug 19 21:43 cmake_uninstall.cmake
drwxr-xr-x  17 fredrik  staff   578B Aug 19 21:43 pylupdate/
drwxr-xr-x   7 fredrik  staff   238B Aug 19 21:43 pyrcc/
-rw-r--r--   1 fredrik  staff   2.7K Aug 19 21:43 pyside2-uic
drwxr-xr-x  14 fredrik  staff   476B Aug 19 21:43 pyside2uic/  <------------------ !
drwxr-xr-x   5 fredrik  staff   170B Aug 19 21:43 tests/

Because this repository's top-level CMakeLists.txt makefile requires explicitly declaring the version of Python to be built against via a Python 3.x-specific extension tag (e.g., .cpython-34m for Python 3.4). You see the problem, I trust.

What the... are you sure? Although pyside2-tools in the tree of the PySide2 5.6 branch is not clickable, after cloning I can see that it's at commit SHA 7fe32567c75b6b9985b9efbecf74477c9e829fb2. When viewing the pyside2-tools tree at this commit, you get this. When viewing CMakeLists.txt, I'm not sure what you mean by having to define a Python 3 tag. Where do you see this?

Your time appears limited, @fredrikaverpil.

Yes, unfortunately.

Would you like a bit of assistance in doing so?

Absolutely 🙂

My intuition is that the Ubuntu 16.04 + Python 3.5 Dockerfile would be a reasonable place for us to start hammering out a working solution

Works for me!

a new upstream issue requesting that this be clarified and/or corrected should probably be opened. Again, I am lazy and overtired and increasingly grumpy and shall not be doing that.

It seems you have already investigated this quite thoroughly, possibly more than QtC has done, so I really think you should report this when you have the time. Perhaps use the already opened issue and append your findings: https://bugreports.qt.io/browse/PYSIDE-357

The above instructions derive entirely from our working Gentoo ebuild for pyside-tools

But what you do there is actually installing pyside2-tools onto the system. You're not bundling it with the PySide2 wheel, which is what I'm looking to do.

You can just ignore ENTRYPOINT, I guess for now. The Dockerfiles don't really need to use that. We can stick to RUN. Actually, the thing with ENTRYPOINT could be useful to keep. It's not executed on docker build but on docker run. You cannot map volumes with docker build, which is how I get the wheel out of the container.

Could you set up a PR where you update the Xenial Python 3.5 Dockerfile where you run the cmake command to set those tags just prior to building the wheel? – It's possible that you need to do it within the same ENTRYPOINT procedure, like so:

ENTRYPOINT \

    cd /pyside-setup/sources/pyside2-tools && \

    # Delete the port_v2 also?

    /opt/cmake-3.5.2-Linux-x86_64/bin/cmake \
        -DPYTHON_EXTENSION_SUFFIX=".cpython-35m" \
        -DPYTHON_BASENAME=".cpython-35m" && \

    cd /pyside-setup && \

    python3 setup.py \
        bdist_wheel \
        --ignore-git \
        --qmake=/opt/qt56/bin/qmake \
        --cmake=/opt/cmake-3.5.2-Linux-x86_64/bin/cmake

Then;

# Build image
docker build . -f Dockerfile-ubuntu16.04-qt5.6-py3.5 -t fredrikaverpil/pyside2-ubuntu16.04-qt5.6-py3.5 .

# Build PySide, generate wheel
docker run --rm -v $(pwd):/pyside-setup/dist fredrikaverpil/pyside2-ubuntu16.04-qt5.6-py3.5

[fredrikaverpil]

By the way, when I build PySide2 with the Xenial/Python3.5 Dockerfile, I see this: PYTHON_EXTENSION_SUFFIX: .cpython-35m-x86_64-linux-gnu

[fredrikaverpil]

So, if you build PySide2 right now using the Xenial/Python 3.5 Dockerfile, this is what happens during the build phase when it reaches pyside2-tools:

Building module pyside2-tools...
Creating module build folder /pyside-setup/pyside3_build/py3.5-qt5.6.2-64bit-release/pyside2-tools...
Configuring module pyside2-tools (/pyside-setup/sources/pyside2-tools)...
Running process in /pyside-setup/pyside3_build/py3.5-qt5.6.2-64bit-release/pyside2-tools: /opt/cmake-3.5.2-Linux-x86_64/bin/cmake -G "Unix Makefiles" -DQT_QMAKE_EXECUTABLE='/opt/qt56/bin/qmake' -DBUILD_TESTS=False -DQt5Help_DIR=/opt/qt56/doc -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/pyside-setup/pyside3_install/py3.5-qt5.6.2-64bit-release /pyside-setup/sources/pyside2-tools -DPYTHON_EXECUTABLE=/usr/bin/python3 -DPYTHON_INCLUDE_DIR=/usr/include/python3.5m -DPYTHON_LIBRARY=/usr/lib/x86_64-linux-gnu/libpython3.5m.so -DQT_SRC_DIR=/opt/qt56
/opt/cmake-3.5.2-Linux-x86_64/bin/cmake -G Unix Makefiles -DQT_QMAKE_EXECUTABLE='/opt/qt56/bin/qmake' -DBUILD_TESTS=False -DQt5Help_DIR=/opt/qt56/doc -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/pyside-setup/pyside3_install/py3.5-qt5.6.2-64bit-release /pyside-setup/sources/pyside2-tools -DPYTHON_EXECUTABLE=/usr/bin/python3 -DPYTHON_INCLUDE_DIR=/usr/include/python3.5m -DPYTHON_LIBRARY=/usr/lib/x86_64-linux-gnu/libpython3.5m.so -DQT_SRC_DIR=/opt/qt56
-- The C compiler identification is GNU 5.4.0
-- The CXX compiler identification is GNU 5.4.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Shiboken2Config: Using default python: .cpython-35m-x86_64-linux-gnu
-- libshiboken built for Release
-- Configuring done
-- Generating done
CMake Warning:
  Manually-specified variables were not used by the project:

    PYTHON_EXECUTABLE
    PYTHON_INCLUDE_DIR
    PYTHON_LIBRARY
    QT_QMAKE_EXECUTABLE
    QT_SRC_DIR
    Qt5Help_DIR

-- Build files have been written to: /pyside-setup/pyside3_build/py3.5-qt5.6.2-64bit-release/pyside2-tools
Compiling module pyside2-tools...
Running process in /pyside-setup/pyside3_build/py3.5-qt5.6.2-64bit-release/pyside2-tools: /usr/bin/make
/usr/bin/make
Scanning dependencies of target pyside2-rcc
[  6%] Building CXX object pyrcc/CMakeFiles/pyside2-rcc.dir/main.cpp.o
[ 13%] Building CXX object pyrcc/CMakeFiles/pyside2-rcc.dir/rcc.cpp.o
[ 20%] Linking CXX executable pyside2-rcc
[ 20%] Built target pyside2-rcc
[ 26%] Generating moc_translator.cpp
Scanning dependencies of target pyside2-lupdate
[ 33%] Building CXX object pylupdate/CMakeFiles/pyside2-lupdate.dir/fetchtr.cpp.o
[ 40%] Building CXX object pylupdate/CMakeFiles/pyside2-lupdate.dir/main.cpp.o
[ 46%] Building CXX object pylupdate/CMakeFiles/pyside2-lupdate.dir/merge.cpp.o
[ 53%] Building CXX object pylupdate/CMakeFiles/pyside2-lupdate.dir/metatranslator.cpp.o
[ 60%] Building CXX object pylupdate/CMakeFiles/pyside2-lupdate.dir/numberh.cpp.o
[ 66%] Building CXX object pylupdate/CMakeFiles/pyside2-lupdate.dir/proparser.cpp.o
[ 73%] Building CXX object pylupdate/CMakeFiles/pyside2-lupdate.dir/sametexth.cpp.o
[ 80%] Building CXX object pylupdate/CMakeFiles/pyside2-lupdate.dir/simtexth.cpp.o
[ 86%] Building CXX object pylupdate/CMakeFiles/pyside2-lupdate.dir/translator.cpp.o
[ 93%] Building CXX object pylupdate/CMakeFiles/pyside2-lupdate.dir/moc_translator.cpp.o
[100%] Linking CXX executable pyside2-lupdate
[100%] Built target pyside2-lupdate
Installing module pyside2-tools...
Running process in /pyside-setup/pyside3_build/py3.5-qt5.6.2-64bit-release/pyside2-tools: /usr/bin/make install/fast
/usr/bin/make install/fast
Install the project...
-- Install configuration: "Release"
-- Installing: /pyside-setup/pyside3_install/py3.5-qt5.6.2-64bit-release/bin/pyside2-uic
-- Installing: /pyside-setup/pyside3_install/py3.5-qt5.6.2-64bit-release/lib/python3.5/site-packages/pyside2uic/__init__.py
-- Up-to-date: /pyside-setup/pyside3_install/py3.5-qt5.6.2-64bit-release/lib/python3.5/site-packages/pyside2uic
-- Installing: /pyside-setup/pyside3_install/py3.5-qt5.6.2-64bit-release/lib/python3.5/site-packages/pyside2uic/properties.py
-- Installing: /pyside-setup/pyside3_install/py3.5-qt5.6.2-64bit-release/lib/python3.5/site-packages/pyside2uic/port_v2
-- Installing: /pyside-setup/pyside3_install/py3.5-qt5.6.2-64bit-release/lib/python3.5/site-packages/pyside2uic/port_v2/__init__.py
-- Installing: /pyside-setup/pyside3_install/py3.5-qt5.6.2-64bit-release/lib/python3.5/site-packages/pyside2uic/port_v2/load_plugin.py
-- Installing: /pyside-setup/pyside3_install/py3.5-qt5.6.2-64bit-release/lib/python3.5/site-packages/pyside2uic/port_v2/proxy_base.py
-- Installing: /pyside-setup/pyside3_install/py3.5-qt5.6.2-64bit-release/lib/python3.5/site-packages/pyside2uic/port_v2/invoke.py
-- Installing: /pyside-setup/pyside3_install/py3.5-qt5.6.2-64bit-release/lib/python3.5/site-packages/pyside2uic/port_v2/ascii_upper.py
-- Installing: /pyside-setup/pyside3_install/py3.5-qt5.6.2-64bit-release/lib/python3.5/site-packages/pyside2uic/port_v2/as_string.py
-- Installing: /pyside-setup/pyside3_install/py3.5-qt5.6.2-64bit-release/lib/python3.5/site-packages/pyside2uic/port_v2/string_io.py
-- Installing: /pyside-setup/pyside3_install/py3.5-qt5.6.2-64bit-release/lib/python3.5/site-packages/pyside2uic/port_v3
-- Installing: /pyside-setup/pyside3_install/py3.5-qt5.6.2-64bit-release/lib/python3.5/site-packages/pyside2uic/port_v3/__init__.py
-- Installing: /pyside-setup/pyside3_install/py3.5-qt5.6.2-64bit-release/lib/python3.5/site-packages/pyside2uic/port_v3/load_plugin.py
-- Installing: /pyside-setup/pyside3_install/py3.5-qt5.6.2-64bit-release/lib/python3.5/site-packages/pyside2uic/port_v3/proxy_base.py
-- Installing: /pyside-setup/pyside3_install/py3.5-qt5.6.2-64bit-release/lib/python3.5/site-packages/pyside2uic/port_v3/invoke.py
-- Installing: /pyside-setup/pyside3_install/py3.5-qt5.6.2-64bit-release/lib/python3.5/site-packages/pyside2uic/port_v3/ascii_upper.py
-- Installing: /pyside-setup/pyside3_install/py3.5-qt5.6.2-64bit-release/lib/python3.5/site-packages/pyside2uic/port_v3/as_string.py
-- Installing: /pyside-setup/pyside3_install/py3.5-qt5.6.2-64bit-release/lib/python3.5/site-packages/pyside2uic/port_v3/string_io.py
-- Installing: /pyside-setup/pyside3_install/py3.5-qt5.6.2-64bit-release/lib/python3.5/site-packages/pyside2uic/icon_cache.py
-- Installing: /pyside-setup/pyside3_install/py3.5-qt5.6.2-64bit-release/lib/python3.5/site-packages/pyside2uic/objcreator.py
-- Installing: /pyside-setup/pyside3_install/py3.5-qt5.6.2-64bit-release/lib/python3.5/site-packages/pyside2uic/exceptions.py
-- Installing: /pyside-setup/pyside3_install/py3.5-qt5.6.2-64bit-release/lib/python3.5/site-packages/pyside2uic/Compiler
-- Installing: /pyside-setup/pyside3_install/py3.5-qt5.6.2-64bit-release/lib/python3.5/site-packages/pyside2uic/Compiler/__init__.py
-- Installing: /pyside-setup/pyside3_install/py3.5-qt5.6.2-64bit-release/lib/python3.5/site-packages/pyside2uic/Compiler/misc.py
-- Installing: /pyside-setup/pyside3_install/py3.5-qt5.6.2-64bit-release/lib/python3.5/site-packages/pyside2uic/Compiler/indenter.py
-- Installing: /pyside-setup/pyside3_install/py3.5-qt5.6.2-64bit-release/lib/python3.5/site-packages/pyside2uic/Compiler/qtproxies.py
-- Installing: /pyside-setup/pyside3_install/py3.5-qt5.6.2-64bit-release/lib/python3.5/site-packages/pyside2uic/Compiler/proxy_type.py
-- Installing: /pyside-setup/pyside3_install/py3.5-qt5.6.2-64bit-release/lib/python3.5/site-packages/pyside2uic/Compiler/qobjectcreator.py
-- Installing: /pyside-setup/pyside3_install/py3.5-qt5.6.2-64bit-release/lib/python3.5/site-packages/pyside2uic/Compiler/compiler.py
-- Installing: /pyside-setup/pyside3_install/py3.5-qt5.6.2-64bit-release/lib/python3.5/site-packages/pyside2uic/uiparser.py
-- Installing: /pyside-setup/pyside3_install/py3.5-qt5.6.2-64bit-release/lib/python3.5/site-packages/pyside2uic/widget-plugins
-- Installing: /pyside-setup/pyside3_install/py3.5-qt5.6.2-64bit-release/lib/python3.5/site-packages/pyside2uic/widget-plugins/qtwebkit.py
-- Installing: /pyside-setup/pyside3_install/py3.5-qt5.6.2-64bit-release/lib/python3.5/site-packages/pyside2uic/widget-plugins/qtdeclarative.py
-- Installing: /pyside-setup/pyside3_install/py3.5-qt5.6.2-64bit-release/lib/python3.5/site-packages/pyside2uic/driver.py
-- Installing: /pyside-setup/pyside3_install/py3.5-qt5.6.2-64bit-release/share/man/man1/pyside2-uic.1
-- Installing: /pyside-setup/pyside3_install/py3.5-qt5.6.2-64bit-release/bin/pyside2-rcc
-- Set runtime path of "/pyside-setup/pyside3_install/py3.5-qt5.6.2-64bit-release/bin/pyside2-rcc" to ""
-- Installing: /pyside-setup/pyside3_install/py3.5-qt5.6.2-64bit-release/share/man/man1/pyside2-rcc.1
-- Installing: /pyside-setup/pyside3_install/py3.5-qt5.6.2-64bit-release/bin/pyside2-lupdate
-- Set runtime path of "/pyside-setup/pyside3_install/py3.5-qt5.6.2-64bit-release/bin/pyside2-lupdate" to ""
-- Installing: /pyside-setup/pyside3_install/py3.5-qt5.6.2-64bit-release/share/man/man1/pyside2-lupdate.1

[fredrikaverpil]

But even so, the Compiler module is missing when you try to run pyside2uic:

$ docker run --rm --interactive --tty -v $(pwd):/pyside-setup/dist --entrypoint=bash fredrikaverpil/pyside2-ubuntu16.04-qt5.6-py3.5

root@ea6cfb5cadf1:/# pip3 install pyside-setup/dist/PySide2-5.6-cp35-cp35m-linux_x86_64.whl 
Processing /pyside-setup/dist/PySide2-5.6-cp35-cp35m-linux_x86_64.whl
Installing collected packages: PySide2
Successfully installed PySide2-5.6

root@ea6cfb5cadf1:/# /usr/local/bin/pyside2-uic 
Traceback (most recent call last):
  File "/usr/local/bin/pyside2-uic", line 7, in <module>
    from PySide2.scripts.uic import main
  File "/usr/local/lib/python3.5/dist-packages/PySide2/scripts/uic.py", line 28, in <module>
    from pyside2uic.driver import Driver
  File "/usr/local/lib/python3.5/dist-packages/pyside2uic/__init__.py", line 27, in <module>
    from pyside2uic.Compiler import indenter, compiler
ImportError: No module named 'pyside2uic.Compiler'

And this is because the Compiler module was never moved here:

root@ea6cfb5cadf1:/# ll /usr/local/lib/python3.5/dist-packages/pyside2uic
total 88
drwxr-sr-x 3 root staff  4096 Aug 19 22:03 ./
drwxrwsr-x 1 root staff  4096 Aug 19 22:03 ../
-rw-r--r-- 1 root staff  5000 Aug 19 22:03 __init__.py
drwxr-sr-x 2 root staff  4096 Aug 19 22:03 __pycache__/
-rw-r--r-- 1 root staff  4039 Aug 19 22:03 driver.py
-rw-r--r-- 1 root staff  1101 Aug 19 22:03 exceptions.py
-rw-r--r-- 1 root staff  4533 Aug 19 22:03 icon_cache.py
-rw-r--r-- 1 root staff  3986 Aug 19 22:03 objcreator.py
-rw-r--r-- 1 root staff 16295 Aug 19 22:03 properties.py
-rw-r--r-- 1 root staff 32268 Aug 19 22:03 uiparser.py

But if we move it there (along with port_v3)..

root@ea6cfb5cadf1:/# cp -r /pyside-setup/sources/pyside2-tools/pyside2uic/Compiler /usr/local/lib/python3.5/dist-packages/pyside2uic/
root@ea6cfb5cadf1:/# cp -r /pyside-setup/sources/pyside2-tools/pyside2uic/port_v3 
root@ea6cfb5cadf1:/# /usr/local/bin/pyside2-uic 
Error: one input ui-file must be specified
root@ea6cfb5cadf1:/# /usr/local/bin/pyside2-uic --help
Usage: pyside2-uic [options] <ui-file>

Options:
  --version             show program's version number and exit
  -h, --help            show this help message and exit
  -p, --preview         show a preview of the UI instead of generating code
  -o FILE, --output=FILE
                        write generated code to FILE instead of stdout
  -x, --execute         generate extra code to test and display the class
  -d, --debug           show debug output
  -i N, --indent=N      set indent width to N spaces, tab if N is 0 (default:
                        4)

  Code generation options:
    --from-imports      generate imports relative to '.'

Now why was Compiler and port_v3 not copied into /usr/local/lib/python3.5/dist-packages/pyside2uic/ when pip installing the wheel? – I have no idea!

[leycec]

Thanks for investigating this deeper, Fred. You are a scholar and a bearded gentleman.

Hm. I was under the impression that all that was missing was the Compiler module which resides here.

All pyside2uic submodules (e.g., pyside2uic.Compiler, pyside2uic.port_v3) are missing, presumably because the setup.py script of the pyside2-setup project is only non-recursively copying files rather than recursively copying subdirectories when it installs from the source to target pyside2uic directory.

Which brings us to setup.py. WHAT. WERE. THEY. THINKING.

No, really. This script is inane, insane, redundant, and patently ridiculous. Someone – Nokia, QtC, I have no idea who – thought that reinventing the entire makefile toolchain by implementing an ad-hoc (and plainly broken) CMake and GNU make meta-build wrapper in PyPI-compatible setup.py format was a good idea.

They were dead wrong. The result is an unreadable, undocumented, and unmaintainable swamp of detritus that should never be. Just remembering it makes my scalp psoriasis itch with bloody pustules.

I thought somehow this didn't get copied into the wheel for some reason (probably because of setup.py).

Exactly. Because apparently copying directories is hard when you try to do it in setup.py.

But you're saying we need to compile all of pyside2-tools prior to building PySide2 itself?

EDIT: None of the following work, for the obvious reason that they fail to modify the binary wheel – exactly as you note. I too now know shame.

The opposite, rather. I no longer trust the pyside2-setup project to correctly build anything. So, the original idea was to manually reclone, recompile, and reinstall pyside2-tools after pyside2-setup – thus replacing the broken pyside2uic package installed by pyside2-setup with the working pyside2uic package installed by itself.

But that's probably overkill. There's a fine line between mistrust and tinfoil-hat paranoia – and tinfoil hats no longer appear to be in fashion. As you cleverly suggest, recursively copying from the source to target pyside2uic directory should suffice for us.

Unrelatedly, here's a poetic Git emoticon. I call it Blood Moon over Bridge over Troubled Water. 🌉

To me, it seems this is done automatically when compiling the pyside-setup project.

...if pyside2-setup weren't painfully broken. Yes. Yes, it would.

When viewing CMakeLists.txt, I'm not sure what you mean by having to define a Python 3 tag. Where do you see this?

Right. Let's drop the code bomb, because things are about to get heavyweight. When compiling pyside2-tools, the PYTHON_EXTENSION_SUFFIX and PYTHON_BASENAME environment variables must be explicitly defined to the extension tag specific to the desired Python version. Failure to do so typically results in pyside2-tools building itself against the wrong Python version.

To see why this is, locate the system-wide CMake directory containing PySide2-specific .cmake-suffixed files. On Gentoo, they reside in the /usr/lib64/cmake/PySide2-2.0.0/ directory:

$ ls /usr/lib64/cmake/PySide2-2.0.0/
PySide2Config.cmake
PySide2Config.cpython-34m.cmake
PySide2ConfigVersion.cmake

These files are the hot glue integrating the CMakeLists.txt file of the pyside2 Git submodule with the CMakeLists.txt file of the pyside2-tools Git submodule. Specifically, these files notify pyside2-tools of the system paths to which PySide2 and shiboken2 were previously installed to.

The contents of PySide2Config.cmake resemble:

if (NOT PYTHON_BASENAME)
    message(STATUS "PySide2Config: Using default python: .cpython-34m")
    SET(PYTHON_BASENAME .cpython-34m)
endif()
include(/usr/lib64/cmake/PySide2-2.0.0/PySide2Config${PYTHON_BASENAME}.cmake)

Thus, the need to explicitly specify these environment variables. </sigh>

By the way, when I build PySide2 with the Xenial/Python3.5 Dockerfile, I see this: > PYTHON_EXTENSION_SUFFIX: .cpython-35m-x86_64-linux-gnu

Excellent catch. That would be the one... for the Python 3.5 Dockerfile, anyway. Presumably, each platform will have its own unique extension tag.

Because. Just because.

But what you do there is actually installing pyside2-tools onto the system. You're not bundling it with the PySide2 wheel, which is what I'm looking to do.

In my defense, I claim self-inflicted insanity. You're... absolutely right. I neglected the critically important bdist_wheel subcommand passed to setuptools, without which the world no longer makes any sense.

Very well, cruel fate. The proper solution is to fix QtC's setup.py mess, so that is what we shall do. When I have a working setup.py patch, I intend to:

• Post that patch here. If it's only a few lines that need munging (please god please), we can inline these changes as sed statements to be run before running "python3 pyside-setup/setup.py bdist_wheel" in our build scripts; else, we'll need to apply a unified diff... somehow.
• Post that patch as an upstream Gerrit-style pull request to QtC, officially murdering this fetid beast.

Pray for my burning soul. I tread into dark waters.

[fredrikaverpil]

@leycec hehe, thanks for clarifying all of this. Okay, so I propose something like this:

Let's find something in setup.py, like for example perhaps this line:

# <install>/lib/site-packages/pyside2uic/* -> <setup>/pyside2uic

And replace that with injected custom code which'll copy the files so that it will be included in the wheel. Kind of what I did here to work around PYSIDE-552 (or PySide2.QtUiTools will not be built):

RUN sed -i.bak $'s/if(Qt5Designer_FOUND)/find_package(Qt5Designer)\\\nif(Qt5Designer_FOUND)/g' pyside-setup/sources/pyside2/CMakeLists.txt

Then we'll add this fix prior to building, in all Dockerfile files. Should work, I think?

When we know it works, you or I can submit our solution as a patch to QtC.

By the way. I know QtC is aware of the state of setup.py and that it needs to be rewritten. I certainly hope it'll be a top priority soon.

[fredrikaverpil]

One idea to achieve this is to create a pyside2uicfix.py which takes care of the file copying. We copy this file next to setup.py and then we inject it somewhere in setup.py:

# Replace
# # <install>/lib/site-packages/pyside2uic/* -> <setup>/pyside2uic
# with...
import pyside2uicfix
pyside2uicfix.apply_fix()

[leycec]

Let's find something in setup.py, like for example perhaps this line:

Exactly. Brilliant minds think brilliantly.

This is the line I've been inspecting for the past hour. For reasons unclear to me, "they" decided to reinvent the stdlib shutil.copytree() function with their own ad-hoc spaghetti code alternative named utils.copydir()... because. Although this is fundamentally insane and therefore bad, I haven't yet teased out the underlying issue.

It's there. Somewhere. Deep within the metastasised tumour of setup.py. My gnawing suspicion is that the bdist_wheel subcommand expects to be internally passed a sequence of all source paths to be bundled into the current wheel – and that this sequence currently omits all pyside2uic subdirectories.

Further meditation and scowling into the darkness are needed.

By the way. I know QtC is aware of the state of setup.py...

I vomit inside every time I rifle through this file. Every time.

One idea to achieve this is to create a pyside2uicfix.py which takes care of the file copying.

Nice. Let's rummage a bit deeper and see where we can slot that into.

Completely unrelatedly, were you aware of the --standalone option? I wasn't, but it looks pretty swag. If I'm perusing the code correctly, ^{i'm prolly not} passing this additional option to python3 setup.py bdist_wheel should bundle all requisite Qt libraries with this wheel – just like with the PyQt5 wheels. Since it's too good to be true, my working assumption is that this option is critically broken.

This option only appears to be supported under Linux, but sumthin's better'n nuthin'.

[leycec]

f!ck f!!cK f!!!CK f@CK F@@CK

Did a three year-old toddler with an adolescent neckbeard author this setup.py script? Because I just discovered the underlying issue – and it paints a grim picture of corporate competency.

It's the package keyword passed to the setup() function at the tail end of this 1,200-line undocumented, unreadable, unmaintainable script. There is a reason that almost everybody defers to the setuptools.find_packages() utility function when specifying this keyword's value. This is that reason.

The package keyword expects to be passed a sequence (in arbitrary order) of the fully-qualified names of all top-level packages and lower-level subpackages comprising this application. Since explicitly listing all lower-level subpackages is guaranteed to fail (e.g., due to an existing subpackage being renamed or removed or a new subpackage added), you're never supposed to manually attempt to list these (sub)package names. Instead, you're supposed to defer to setuptools.find_packages(). Yes, this is an odd and arguably awful design decision – but we're stuck with setuptools and it is stuck with is.

In our PySide-based multiphysics biology simulator, for example, we define this keyword as follows:

setup(
    ...

    # List of all Python packages (i.e., directories containing zero or more
    # Python modules) to be installed. Currently, this includes the "betse"
    # package and all subpackages of this package excluding:
    #
    # * The top-level test package and all subpackages of this package, test
    #   functionality *NOT* intended to be installed with this application.
    # * The top-level setup package and all subpackages of this package,
    #   setuptools functionality required only for application installation.
    # * "build", caching both setuptools metadata and a complete copy of this
    #   package, required only by a prior application installation.
    # * "freeze", providing PyInstaller-specific functionality required only for
    #   application freezing (i.e., conversion into an executable binary).
    'packages': setuptools.find_packages(
        exclude = [
            metadata.PACKAGE_NAME + '_test',
            metadata.PACKAGE_NAME + '_test.*',
            metadata.PACKAGE_NAME + '_setup',
            metadata.PACKAGE_NAME + '_setup.*',
            'build',
            'freeze',
        ],
    ),
    ...
)

We leverage a blacklist-style approach, packaging everything except the six specified top-level directories. Because it's dynamic, it's fully future-proofed against unexpected codebase changes. It just works. Excellent.

Prepare to have your blood curdled, because this is how pyside2-setup defines this keyword:

setup(
    ...
    packages = ['PySide2', 'pyside2uic'],
    ...
)

Note the omission of a call to setuptools.find_packages() or listing of the pyside2uic.Compiler, pyside2uic.port_v2, and pyside2uic.port_v3 subpackages – guaranteeing these subpackages to be neither installed nor bundled with any wheels. Like, you know, ours.

In this case, QtC leveraged neither a blacklist- nor whitelist-style approach. They did it manually. Which you're never supposed to do. Predictably, they failed spectacularly. Because it's static, it's guaranteed to catastrophically blow up on the first codebase change. Which it did. Not so excellent.

One year, people. This issue was open for one year because somebody neglected to learn about packaging standards and how to implement sane setup.py scripts.

Coddling toddlers is not currently listed in my job description. This was my Sunday evening and I frankly had better things to do. Like shoot vengeance into murderous Russian war criminals as Nathan Drake in the critically acclaimed Uncharted 2: Among Thieves remake bundled with the Uncharted Collection for the best-selling PlayStation 4! ^{Only $59.95, while limited supplies last.}

Fortunately, the fix is trivial – slowly transforming my dour frown into a grim, shark-toothed smile. For us at least, the fix is just to roll our eyes and manually patch their setup.py with a single sed statement appending the requisite subpackages to the package keyword: e.g.,

RUN sed -i -e 's~\b\(packages\b.*\)],~\1, "pyside2uic.Compiler", "pyside2uic.port_v2", "pyside2uic.port_v3"],~' pyside-setup/setup.py

That's it. Done.

[fredrikaverpil]

Oh. My. God.

[fredrikaverpil]

I got an error during PySide2 wheel building on CentOS 7 (Python 3.6) when applying your sed hack.

error: package directory 'pyside_package/pyside2uic/port_2' does not exist.

Tracked down the issue by glancing through setup.py:

        if sys.version_info[0] > 2:
            rmtree("{dist_dir}/pyside2uic/port_v2".format(**vars))
        else:
            rmtree("{dist_dir}/pyside2uic/port_v3".format(**vars))

Meaning, your sed hack needs either port_v2 or port_v3 (not both), depending on which Python major version you're on. Figured it was worth mentioning here.

So;

# Python 2
RUN sed -i -e 's~\b\(packages\b.*\)],~\1, "pyside2uic.Compiler", "pyside2uic.port_v2"],~' pyside-setup/setup.py

# Python 3
RUN sed -i -e 's~\b\(packages\b.*\)],~\1, "pyside2uic.Compiler", "pyside2uic.port_v3"],~' pyside-setup/setup.py

[fredrikaverpil]

Although it would've been much better with a setuptools.find_packages() hack, I ended up using this sed here for the PySide2 wheels, so I don't have to use different hacks depending on Python version:

RUN sed -i -e "s~\b\(packages\b.*\)],~\1, 'pyside2uic.Compiler', 'pyside2uic.port_v' + str(sys.version_info[0])],~" pyside-setup/setup.py

Will be updating all Dockerfile files and merging this shortly. Thanks @leycec 👍 for narrowing it down to a single sed.

[bjquinn]

You guys rock. Very excited about this!

[fredrikaverpil]

Okay, I believe this is now fixed on all platforms. Wheels are currently uploading for Windows and should be done within the next couple of hours.

[leycec]

Meaning, your sed hack needs either port_v2 or port_v3 (not both), depending on which Python major version you're on. Figured it was worth mentioning here.

Absolutely. I knew you'd be onto that like a hot potato on day-old pizza. I meant to mention it, but felt too livid with red-eyed rage to form coherent sentences or valid sed constructs.

Thankfully, you picked up the muddy ball and carried us into the home stretch. 🏈

I ended up using this sed here for the PySide2 wheels, so I don't have to use different hacks depending on Python version:

That's... clever. I like clever.

Okay, I believe this is now fixed on all platforms.

And you are amazing. Thanks for devoting scarce time and sanity to finalizing this. We're all exceptionally grateful! We owe you several buckets worth of organic free-range craft microbrew, delivered to the discrete back alley stairwell of your discretion.

[fredrikaverpil]

Haha. No, thank you @leycec for all this digging. It really helps. 👍

[bjquinn]

Thank you! Started using it today, works perfectly so far!