danschef
commented
1 year ago @jakimowb
Closed danschef closed 11 months ago
danschef
commented
1 year ago @jakimowb
SrNetoChan
commented
1 year ago So... I have tried 3.28.8 on linux and I was able to replicate the issue.
Then I have installed 3.28.3 (which is about the same time as 3.30.0 ) and it worked. 3.22 latest version also works fine.
So, something has happened after 3.30.0 and 3.28.3
jakimowb
commented
1 year ago I don't know why, but calling the constructor with an additional "None"
QgsSingleSymbolRenderer(symbol, None) will let the code run through.
SrNetoChan
commented
1 year ago Still, the api only expects one argument
SrNetoChan
commented
1 year ago My guess is that has something to do with pyqt or sip versions
danschef
commented
1 year ago 3.22 latest version also works fine.
Note that in my case above, it did not work for QGIS 3.32 (py311h66b4bf9_0). If it worked for you, it might be worth to compare the rest of the environment (conda list output above). In the above environment, pyqt 5.15.7 (py311ha74522f_3) and sip 6.7.9 (py311hb755f60_0) are installed.
SrNetoChan
commented
1 year ago @danschef It works on 3.22, not on 3.32
danschef
commented
1 year ago Sorry, I was confused because you wrote "3.22 latest version" but the latest version is actually 3.32.
SrNetoChan
commented
1 year ago I mean the last 3.22.x patch
SrNetoChan
commented
1 year ago So, I kept digging in this issue. I believe the problems go deeper than this snippet. I am getting another error and crash doing another task (adding a input in the model builder)
So, the snippet works fine in 3.28.4, and the installation between 3.28.3 and 3.28.4 was just an update on the qgis package.
Then I have installed 3.28.5, and it forced lots of updates in all dependencies (see below).
Now, the script gives the error, and I also starts getting the error with the model builder. So, whatever happened started in this version.
Looking for: ['qgis=3.28.5']
conda-forge/linux-64 Using cache
conda-forge/noarch Using cache
Pinned packages:
- python 3.11.*
Transaction
Prefix: /home/aneto/mambaforge/envs/qgis3.8.3
Updating specs:
- qgis=3.28.5
- ca-certificates
- certifi
- openssl
Package Version Build Channel Size
─────────────────────────────────────────────────────────────────────────────────
Install:
─────────────────────────────────────────────────────────────────────────────────
+ libllvm16 16.0.3 hbf9e925_1 conda-forge/linux-64 Cached
Change:
─────────────────────────────────────────────────────────────────────────────────
- boost-cpp 1.78.0 h5adbc97_2 conda-forge
+ boost-cpp 1.78.0 h6582d0a_3 conda-forge/linux-64 Cached
- cairo 1.16.0 ha61ee94_1014 conda-forge
+ cairo 1.16.0 h35add3b_1015 conda-forge/linux-64 Cached
- geotiff 1.7.1 hb963b44_7 conda-forge
+ geotiff 1.7.1 h480ec47_8 conda-forge/linux-64 Cached
- harfbuzz 6.0.0 h8e241bc_0 conda-forge
+ harfbuzz 6.0.0 h3ff4399_1 conda-forge/linux-64 Cached
- librttopo 1.1.0 ha49c73b_12 conda-forge
+ librttopo 1.1.0 h0d5128d_13 conda-forge/linux-64 Cached
- libspatialite 5.0.1 h221c8f1_23 conda-forge
+ libspatialite 5.0.1 h7d1ca68_25 conda-forge/linux-64 Cached
- qt-main 5.15.8 h67dfc38_7 conda-forge
+ qt-main 5.15.8 h5c52f38_10 conda-forge/linux-64 Cached
- qtwebkit 5.212 he984d69_9 conda-forge
+ qtwebkit 5.212 h7b3bce5_10 conda-forge/linux-64 Cached
- xerces-c 3.2.4 h55805fa_1 conda-forge
+ xerces-c 3.2.4 h8d71039_2 conda-forge/linux-64 Cached
Upgrade:
─────────────────────────────────────────────────────────────────────────────────
- gdal 3.6.3 py311hadb6153_0 conda-forge
+ gdal 3.6.4 py311h6122507_2 conda-forge/linux-64 Cached
- geos 3.11.1 h27087fc_0 conda-forge
+ geos 3.11.2 hcb278e6_0 conda-forge/linux-64 Cached
- hdf5 1.12.2 nompi_h4df4325_101 conda-forge
+ hdf5 1.14.0 nompi_hb72d44e_103 conda-forge/linux-64 Cached
- icu 70.1 h27087fc_0 conda-forge
+ icu 72.1 hcb278e6_0 conda-forge/linux-64 Cached
- kealib 1.5.0 ha7026e8_0 conda-forge
+ kealib 1.5.1 h3845be2_3 conda-forge/linux-64 Cached
- libclang 15.0.7 default_h7634d5b_2 conda-forge
+ libclang 16.0.3 default_h1cdf331_2 conda-forge/linux-64 21kB
- libclang13 15.0.7 default_h9986a30_2 conda-forge
+ libclang13 16.0.3 default_h4d60ac6_2 conda-forge/linux-64 10MB
- libdeflate 1.17 h0b41bf4_0 conda-forge
+ libdeflate 1.18 h0b41bf4_0 conda-forge/linux-64 Cached
- libgdal 3.6.3 hb7af45b_0 conda-forge
+ libgdal 3.6.4 hada8d5e_2 conda-forge/linux-64 Cached
- libnetcdf 4.9.1 nompi_hd2e9713_102 conda-forge
+ libnetcdf 4.9.2 nompi_hdf9a29f_104 conda-forge/linux-64 Cached
- libtiff 4.5.0 hddfeb54_5 conda-forge
+ libtiff 4.5.1 h8b53f26_0 conda-forge/linux-64 Cached
- libxml2 2.10.3 hca2bb57_4 conda-forge
+ libxml2 2.10.4 hfdac1af_0 conda-forge/linux-64 Cached
- pdal 2.5.2 h0ea1e05_1 conda-forge
+ pdal 2.5.3 h09aa857_0 conda-forge/linux-64 Cached
- poppler 23.03.0 hf052cbe_1 conda-forge
+ poppler 23.05.0 hd18248d_1 conda-forge/linux-64 Cached
- proj 9.1.1 h8ffa02c_2 conda-forge
+ proj 9.2.0 h8ffa02c_0 conda-forge/linux-64 Cached
- pyproj 3.5.0 py311h945b3ca_0 conda-forge
+ pyproj 3.6.0 py311h331fe15_0 conda-forge/linux-64 Cached
- qgis 3.28.4 py311hc4a2dbe_0 conda-forge
+ qgis 3.28.5 py311hc7474de_2 conda-forge/linux-64 88MB
Summary:
Install: 1 packages
Change: 9 packages
Upgrade: 17 packages
Total download: 98MB@gillins any idea?
gillins
commented
1 year ago Sorry, no. My instinct tells me there may be a subtle memory corruption type error that's causing some weird behaviour. Perhaps worth running a script that crashes through valgrind but you are likely to get a lot of info to sift through...
Do the qgis.org packages show the same error?
SrNetoChan
commented
1 year ago @gillins no they don't. Using the installers prepared by qgis.org you are not able to replicate the error.
The change happened from 3.28.4 to 3.28.5. today I will hunt down which build cause it, because build 0 was just a bump in version.
Then I will try to rebuild pining some dependencies to match what qgis.org is using.
SrNetoChan
commented
1 year ago Ok, I kept investigating.
version 3.28.5 had several builds. I have tried installing build_0 and it works fine, both the code snippet and the issue I mentioned earlier.
Then I installed the next build qgis==3.28.5=py39hc0d3e04_1 and that's when the bug appears for the first time. In terms of dependencies, from build 0 to build 1 there were this diferences:
Updating specs:
openssl
Package Version Build Channel Size ───────────────────────────────────────────────────────────────────────────── Change: ─────────────────────────────────────────────────────────────────────────────
Upgrade: ─────────────────────────────────────────────────────────────────────────────
So the only actual change in dependecies was geos, which I don't think is related to this.
Talking with @alexbruy he mentioned that the thing that could cause this kind of problems could be SIP, so I went to see what SIP version was used in each build, but according to what I saw, both used sip 6.6.2 ...
SrNetoChan
commented
1 year ago This seem to have been the one causing the problem:
https://github.com/conda-forge/qgis-feedstock/pull/317/files (Rebuilt for geos3.11.2) and it actually only seems to have changed the GEOS version from 3.11.1 to 3.11.2. QGIS 3.28.8 on Windows from osgeo4w also uses GEOS 3.11.2 without the issue... So... another deadend.
jakimowb
commented
1 year ago Some more examples that fail on conda installations (QGIS 3.30) but not OSGeo4W (python 3.9) or QGIS dev on ubuntu (20.04, python 3.10)
from qgis.core import *
from qgis.gui import *
QgsFillSymbol()
QgsMarkerSymbol()
QgsFillSymbol()
QgsLineSymbol()
QgsExpressionContext([])
QgsRendererCategory(1, QgsFillSymbol(), 'name', render=False)Can you install Python 3.11 with OSGeo4W or is it fixed on 3.9? I have seen weird things with sip and more recent Python...
jakimowb
commented
1 year ago OSGeo4W is still on 3.9., but the errors [updated] do not occur on Ubuntu 20.04 with python 3.10 as well (I updated my previous comment)
SrNetoChan
commented
1 year ago Indeed osgeo4w is fixed on 3.9, but I have tested also with 3.9 in conda, unless it's a specific version of 3.9.
But notice that the first build of qgis 3.28.5 didn't have the bug, and I have tested it with the same python.
Same source, just an update of geos, and some build updates of a few dependencies.
Something have changed meanwhile, but it's very hard to say what.
gillins
commented
1 year ago Do we know what versions of pyqt and pyqt5-sip qgis.org use?
Nothing useful with valgrind I'm afraid.
SrNetoChan
commented
1 year ago @gillins said:
Do we know what versions of
pyqtandpyqt5-sipqgis.org use? Nothing useful withvalgrindI'm afraid.
In QGIS 3.28.8 installed from OSGeo4W on windows, this is what we have.
Qt version: 5.15.2 SIP version: 5.4.0 PyQt version: 5.15.4
Nevertheless, both builds 0 and 1 of qgis=3.28.5 in conda-forge use SIP 6.6.2 (and I believe that the same version of Qt and PyQt)
gillins
commented
1 year ago OK this is very weird... I can't explain the GEOS thing.
But, I do notice that every class that has the problem also uses SIP_TRANSFER in the C++ constructor definition. For some reason I suspect that a) SIP_TRANSFER is causing the code to expect another parameter and b) it isn't actually causing the C++ ownership to be transferred hence the random crashes.
@alexbruy do you have any idea what could be causing this? I see sipify.pl should do some replacement before compilation.
SrNetoChan
commented
1 year ago @gillins yesterday I have built locally (using docker). But then I was not able to install it. How can I do that? Get the built package from the docker machine?
alexbruy
commented
1 year ago SIP_TRANSFER is just an annotation needed for SIP to generate correct bindings. Also there is no need to run sipify.pl before the build, as sip files are kept updated.
gillins
commented
1 year ago @SrNetoChan see https://conda-forge.org/docs/maintainer/updating_pkgs.html#testing-changes-locally: Once built, you can find the finished package in the build_artifacts directory in your feedstock, which can be used as a channel. To create a new environment my-new-env using conda, and which will contain the new built package my-package, run: conda create -n my-new-env -c "file://${PWD}/build_artifacts" my-package
gillins
commented
1 year ago Thanks @alexbruy, any pointers on where we should look for the source of this problem?
gillins
commented
12 months ago Hi guys, I'm going to have to leave this for now. I suspect this is being caused by the rather large difference in versions between pyqt5-sip (used at runtime - uses sip version 6.6.2) and sip (used when building the bindings - version 6.7.9), but cannot prove this. Unfortunately we cannot build qgis easily with an older sip and there seems to be some problem building more recent versions of the PyQt5 stack (https://github.com/conda-forge/pyqt-feedstock/pulls)....
SrNetoChan
commented
11 months ago @gillins is there any path to fix this? This really screws our efforts to make our builds viable to large usage. What can we try to do?
gillins
commented
11 months ago I did mean to mention that I found this: https://riverbankcomputing.com/news/SIP_v6.7.9_Released (we are on sip 6.7.9) which does mention "API v12.12.1" and I'm guessing this corresponds to PyQt5-sip 12.12.1 (https://pypi.org/project/PyQt5-sip/). Since conda-forge's pyqt5-sip is still stuck on 12.11.0 I think this is the problem.
sip is updated in conda-forge very often but pyqt5/pyqt5-sip is not.
When I get some time I'll try bumping the version and see what happens, but this will mean fixing the mysterious build problem with more recent pyqt5 versions which may be hard.
SrNetoChan
commented
11 months ago @gillins I decided to give it a go (not for you to feel any pressure) I used 12.12.2 because it seems to be the last version available on pypi... but maybe I should have used 12.12.1 Let's see how it goes.
gillins
commented
11 months ago 12.12.2 corresponds to sip 6.7.10 (https://riverbankcomputing.com/news/SIP_v6.7.10_Released) which has just been merged into conda-forge (https://github.com/conda-forge/sip-feedstock/pull/70) so I think we are all good...
SrNetoChan
commented
11 months ago Do we know what versions of
pyqtandpyqt5-sipqgis.org use? Nothing useful withvalgrindI'm afraid.
@gillins I have asked and it seems that OSGeo4w uses 12.8.1, while ubuntu is providing 12.9.1 for pyqt-sip. I wonder is the problem is conda-forge being jumping ahead.
gillins
commented
11 months ago Yeah sounds like everybody is using older sip and pyqt5-sip. conda-forge always has the latest of everything, but things fell apart when sip was updated but pyqt5-sip wasn't. Hopefully that will now be fixed and we can try again... Always a bit risky using recent versions but if there is a problem with new sip would be good to feed that back to qgis.
gillins
commented
11 months ago Fixed by #358
SrNetoChan
commented
11 months ago I have tested 3.32.1 with the new build and it's fixed. I will do the same in the LTR branch.
SrNetoChan
commented
11 months ago Tested the new build of QGIS 3.28.9 and it's fixed! Thanks @gillins !!
danschef
commented
11 months ago @gillins @SrNetoChan
Many thanks for your work here, I can confirm that our downstream issue is fixed with the new builds.
Solution to issue cannot be found in the documentation.
Issue
I am not entirely sure if that is related to the conda-forge builds of QGIS but I would guess so, so I report it here:
We are currently facing issues in the EnMAP-Box (https://github.com/EnMAP-Box/enmap-box/issues/488) that we narrowed down to the following code snippet:
This fails fails with a TypeError:
We tested this with the following QGIS versions:
Just use
mamba create -n test "qgis=3.32.0" "python=3.11"to create an environment to reproduce it.Installed packages
Environment info