Open m-kuhn opened 1 year ago
joaoponceleao
commented
1 year ago This is possible, but I would add a step before (1): fix bugs and dependencies. The advantage of a conda based installer would be that, hopefully, the same or very similar process could apply to Linux, Mac, and Windows.
m-kuhn
commented
1 year ago This is possible, but I would add a step before (1): fix bugs and dependencies.
This is currently planned as step 2) mostly because this will make it easier also for a wider range of testers and it's harder to agree on a "definition of done".
The advantage of a conda based installer would be that, hopefully, the same or very similar process could apply to Linux, Mac, and Windows.
Indeed, this is another interesting feature of Conda, but I'd keep that separate of this QEP.
mbernasocchi
commented
12 months ago This was not accepted in the grant voting, so the PSC decided to have it financed from a separate budget. https://lists.osgeo.org/pipermail/qgis-psc/2023-July/010086.html
tsmcgrath
commented
11 months ago Thanks to @agiudiceandrea for the pointer to this. I see that this proposal remains unassigned. Can we make @justinbb a contributor and assign it to him? Is there a breakdown of tickets identifying the work that relate to this proposal?
dawgermany
commented
11 months ago Thanks to @agiudiceandrea for the pointer to this. I see that this proposal remains unassigned. Can we make @justinbb a contributor and assign it to him? Is there a breakdown of tickets identifying the work that relate to this proposal?
This is a QEP of @m-kuhn which has been granted as per
This was not accepted in the grant voting, so the PSC decided to have it financed from a separate budget. https://lists.osgeo.org/pipermail/qgis-psc/2023-July/010086.html
So as the financing is now cleared Matthias will develop a solution for this QEP as I understand it...
jctull
commented
8 months ago Any movement on this? TIA
m-kuhn
commented
8 months ago Yes, there has been movement.
As defined in the grant proposal, I have been working on a proof of concept installer based on Conda.
The repository for this can be found here: https://github.com/opengisch/qgis-conda-builder. It automatically creates an installer for the latest release and adds it to the release page.
Notes:
.pkg instead of a .dmg, this is imposed by the nature of conda, that wants to be installed to a specific path (so it needs a scripted installer) and does not have relocatable packages, that can be copied somewhere (as a .dmg does)constructor tool to be able to create a macOS .app bundle. Normally conda is meant mostly for installing into a conda environment.While working on this, I was wondering if I should re-evaluate an approach using vcpkg, which we are using successfully for QField for a while to build a macos .dmg and on top of which recently some amazing webassembly work has been done. I was originally afraid of python and arm based mac systems, but after some tests done in the last couple of days, this seems not as difficult as originally estimated.
joaoponceleao
commented
8 months ago This is great news @m-kuhn. Congratulations and thanks for your hard work. I would recommend adding some minor information on the repo readme, particularly since it is a .pkg file. I am reluctant to install it without knowing how it affects my own conda environments. I am assuming it creates a custom conda environment somewhere (conda envs, even multiple base envs, can be installed anywhere), and the pkg simply adds scripts to activate it? Could you advise?
m-kuhn
commented
8 months ago @joaoponceleao thanks for the feedback and advice, I added a minimal README.md with the information you requested, please don't hesitate to ask for more information directly in a ticket in the repository.
nirvn
commented
8 months ago Big +1 to consider using vcpkg to build/serve our macos installer. It's a mature and vibrant open source cross platform framework that is surprisingly easy to manage (their overlay system is so, sooo good).
I can see quite a few benefits from QGIS beyond macos in the future too.
boardend
commented
8 months ago My Apple days are more than 10 years gone (for good), and I did make mixed experiences with Conda in the past. So I'm probably not the right person to judge on this specific issue and maybe my comment is a bit off-topic, but I also want to give a big +1 to adopt vcpkg more for building QGIS in general.
Background: We are using vcpkg in qgis-js. Most of the QGIS dependencies were just working with the custom wasm32-emscripten-qt-threads triplet. And for the ones who didn't, the integration of a custom port was straight forward. So was the integration of QGIS itself (besides the heavy patching needed to get the whole thing working on the web platform 😅).
Actually @wonder-sk brought vcpkg into the project and did it's integration 🙏 Maybe he has some more insights to share?
jctull
commented
8 months ago Thank you for the update and the continued work!
jctull
commented
8 months ago I downloaded and tested the new installer today. Limited testing, but very nice and well integrated! Thank you!!
eflaten
commented
8 months ago I tried the installer and it worked fine. I used the select install place and put qgis in the app folder. This is good work and a promissing way to keep qgis updated.
wonder-sk
commented
8 months ago As for vcpkg vs conda - I do not have enough experience with conda to compare, but I have to say I was surprised how things went generally smoothly with vcpkg for the qgis-webassembly project (based heavily on @m-kuhn 's work on vcpkg for qfield + adjustments from @PeterPetrik for merginmaps mobile) even when targeting a custom platform ("triplet"). vcpkg certainly feels lighter than conda, given that it's "just a bunch of cmake scripts" :slightly_smiling_face:
Maybe also worth discussing with @hobu in Vienna, he has been using conda a lot for packaging, and proposed having QGIS packaged with conda on multiple platforms.
It is also worth discussing in QGIS community what macOS users actually want - I don't know enough about .pkg and .dmg - is one superior over the other? I vaguely recall @PeterPetrik has spent a considerable amount of time on the existing mac packager to patch all the libraries/executables to make them relocatable as users wanted to have multiple QGIS installations side by side :slightly_smiling_face:
hobu
commented
8 months ago Maybe also worth discussing with @hobu in Vienna, he has been using conda a lot for packaging, and proposed having QGIS packaged with conda on multiple platforms.
Unfortunately I didn't make it to Vienna 😦
I don't have any experience of using vcpkg for executable building, but we do have some experience using PyInstaller to wrap up Conda Forge environments. It has worked well for us on windows, but we don't have any specific OSX experience. I hadn't realized that Conda Constructor is now active again, which is good to see.
The only significant caveat I would make about vcpkg is it has a policy of de-vendoring packages (see https://github.com/microsoft/vcpkg/pull/32093#issuecomment-1602764780 for what happened with PDAL). A number of geospatial libraries have vendored things (like GDAL, PDAL, etc), and this policy can have unintended effects. Not a problem per se, but something to be aware of.
tsmcgrath
commented
7 months ago I've tested the installer. Works well. Thank you mucho!
tsmcgrath
commented
5 months ago I've been using this on a Mac M1 since it rolled out and it's working great. It doesn't crash. It does throw this error at startup every time. I haven't been able to identify the problem: An error occurred during execution of following code: faulthandler.enable(file=qgis.utils.__qgis_fault_handler_file)
Traceback (most recent call last): File "", line 1, in PermissionError: [Errno 1] Operation not permitted
Python version: 3.11.6 | packaged by conda-forge | (main, Oct 3 2023, 10:39:39) [Clang 15.0.7 ]
QGIS version: 3.34.1-Prizren 'Prizren', exported
Python path: ['/Applications/QGIS-3.34.app/Contents/MacOS/../share/qgis/python', '/Users/tim/Library/Application Support/QGIS/QGIS3/profiles/default/python', '/Users/tim/Library/Application Support/QGIS/QGIS3/profiles/default/python/plugins', '/Applications/QGIS-3.34.app/Contents/MacOS/../share/qgis/python/plugins', '/Applications/QGIS-3.34.app/Contents/lib/python311.zip', '/Applications/QGIS-3.34.app/Contents/lib/python3.11', '/Applications/QGIS-3.34.app/Contents/lib/python3.11/lib-dynload', '/Applications/QGIS-3.34.app/Contents/lib/python3.11/site-packages']
tsmcgrath
commented
5 months ago Huh. First issue with this build. QGIS hung and I had to "force quit". Here's the crash dump: qgiscrash_20240115a.txt
dawgermany
commented
5 months ago Issues regarding the m1 conda qgis installer are supposed to go here: https://github.com/opengisch/qgis-conda-builder/issues
m-kuhn
commented
5 months ago I am still investigating a solution based on vcpkg, this will need a bit more time, but expect some updates to appear here
eflaten
commented
5 months ago Issues regarding the m1 conda qgis installer are supposed to go here: https://github.com/opengisch/qgis-conda-builder/issues
Thanks. Moved it there.
tsmcgrath
commented
3 months ago @m-kuhn Hi Matthias, in this build (3.34 for mac m1) I am getting this error from a flatgeobuf line file that I am copy/pasting other lines into.
Commit errors: Could not commit changes to layer APT2024_v1.0
Layer APT2024_v1.0: wrong data type for attribute 0 of feature -326: 10
I see this was already an issue here but was resolved:
https://github.com/qgis/QGIS/pull/52771
Do you think the problem is specific to this build? Should I re-open and add it to #52771?
When I try to save my edits by exporting to a new file I get this error:
Export to vector file failed.
Error: Creation of layer failed (OGR error: Failed to create APT2024_v1.1.fgb:
Read-only file system)
agiudiceandrea
commented
3 months ago @tsmcgrath your issue seems similar to the one reported in the issue report https://github.com/qgis/QGIS/issues/55517, not to the ones fixed by #52771. I think you should open an issue report with all the requested info, providing sample project and layer(s) and exact steps to replicate the issue.
tsmcgrath
commented
3 months ago @agiudiceandrea Maybe, but I'm not sure #55517 is a fit. I do have boolean attributes in the dataset but did not update any in the edit operations preceding the error. It's a linestring data set and the new lines I was adding were just the geometry and a name string. I'll open a new issue though and point to #55517 in case it is related. New issue opened: #57010
aufdenkampe
commented
1 month ago +1 for using conda for multi-platform packaging.
I use it on a Mac every day for environmental data science. The biggest strength of conda, which hasn't been mentioned in this thread, is that it creates a sandboxed virtual environment for all dependancies. So it won't affect any other software, including the system-wide Python installation that is part of MacOS and Linux. This virtual environment also installs in the user directory, so doesn't require admin privileges to install or update.
Also, geospatial libraries for all platforms are now very actively maintained on condo-forge, as has been mentioned in this thread.
joaoponceleao
commented
1 month ago Also, geospatial libraries for all platforms are now very actively maintained on condo-forge, as has been mentioned in this thread.
Except for Grass and co.
But yes, conda would be useful for multi-platform and should be a priority on MacOS as it is the closest QGIS version to run properly. It would benefit from a simple drag and drop or gui installer though which is what opengis is trying to do with the conda-builder but that needs a little more maintenance (at least a current ltr release).
tsmcgrath
commented
3 weeks ago Installing 3.36.3 for Mac M1 (MacOS 14.4.1) ran successfully but on first open it threw the error below but then launched successfully. But it launched and it opens 3.34 QGIS projects fine.
An error has occurred while executing Python code:
TypeError: QgsProcessingParameterFile(): arguments did not match any overloaded call: overload 1: argument 3 has unexpected type 'int' overload 2: argument 1 has unexpected type 'str'
Traceback (most recent call last):
File "/Users/tim/Library/Application Support/QGIS/QGIS3/profiles/default/python/plugins/GpxSegmentImporter/processing/gpx_segment_importer_algorithm.py", line 91, in initAlgorithm
self.addParameter(QgsProcessingParameterFile(self.INPUT, self.tr('Input gpx file'),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: QgsProcessingParameterFile(): arguments did not match any overloaded call:
overload 1: argument 3 has unexpected type 'int'
overload 2: argument 1 has unexpected type 'str'
Python version: 3.11.9 | packaged by conda-forge | (main, Apr 19 2024, 18:38:37) [Clang 16.0.6 ]
QGIS version: 3.36.3-Maidenhead Maidenhead, exported
Python Path:
/Users/tim/Applications/QGIS-3.36.app/Contents/MacOS/../share/qgis/python
/Users/tim/Library/Application Support/QGIS/QGIS3/profiles/default/python
/Users/tim/Library/Application Support/QGIS/QGIS3/profiles/default/python/plugins
/Users/tim/Applications/QGIS-3.36.app/Contents/MacOS/../share/qgis/python/plugins
/Users/tim/Applications/QGIS-3.36.app/Contents/lib/python311.zip
/Users/tim/Applications/QGIS-3.36.app/Contents/lib/python3.11
/Users/tim/Applications/QGIS-3.36.app/Contents/lib/python3.11/lib-dynload
/Users/tim/Applications/QGIS-3.36.app/Contents/lib/python3.11/site-packages
/Users/tim/Library/Application Support/QGIS/QGIS3/profiles/default/python@tsmcgrath, the reported issue is due to an external third-party plugin installed in your QGIS user profile.
QGIS Enhancement: Conda Installer for macOS
Date 2023/05/02
Author Matthias Kuhn (@m-kuhn)
Contact matthias@opengis.ch
maintainer @m-kuhn
Version QGIS 3.X
Summary
Packages for macOS are a unique selling point for QGIS that make it stand out among other GIS applications. An easy to use installer for macOS is therefore required. The existing mac installer contains old dependencies and the maintenance effort is quite high and currently underpowered. Conda provides a way to install QGIS with up to date dependencies and even provides builds for Mac M1. This QEP proposes to build an installer on top of conda builds and with this on an ecosystem and community that helps to keep dependencies alive and up to date.
Why conda and not [X]
There are many systems around that deal with dependency management and each of them has their own strengths and weaknesses. Some points in favor of Conda: it has been around for a long time, QGIS builds from Conda are used in production and with that a lot of the basework is done, the community is active and healthy and providing a continuous stream of updates to libraries.
Proposed Solution
Note for grant voting
This QEP outlines the big picture. The grant proposal is for creating an installer based on the conda installation and identifying missing dependencies. It consists of the first steps towards a more sustainable future. Notarization and adding missing dependencies is not part of the grant proposal.
Backwards Compatibility
Fully backwards compatible, but missing dependencies need to be tackled one by one
Issue Tracking ID(s)
(optional)
Votes
(required)