search

geosolutions-it / C179-DBIAIT

DBIAIT project for C179
0 stars 1 forks source link

Test QGIS headless installation on SUSE #12

Closed randomorder closed 3 years ago

randomorder commented 3 years ago

We want to try to install QGIS and run QGIS in headless mode on SUSE and add the setup to our Ansible scripts for PA

randomorder commented 3 years ago

Target Machie for is this one on Geos7: https://docs.google.com/document/d/1WlzGioZnkLAdmeKB6csaCuxiPwzqgseipodgHge2ErI/edit

Please take a snapshot before messing with the machine so you can revert and test the ansible role on a pristine machine

lpasquali commented 3 years ago

snapshot taken "before QGIS"

lpasquali commented 3 years ago

@randomorder I documented the "not so few" steps for gathering all dependecies needed by qgis and successfully installed, side note: as far as I could see I had to use different repositories from PA as we are running opensuse, following the installation log without changing anything will result in failure on PA with ansible, becaue qgis is packaged inside a repository directly offered by suse, meaning it is built differently for SLE and opensuse.

lpasquali commented 3 years ago

unfortunately customer runs a specific SLE 15 SP1 version "SUSE Linux Enterprise Server for SAP Applications" which lacks registration privileges for the correct repositories for qgis in SLE, so this workarount was made to achieve all dependencies for latest qgis in Suse:


# Activate this repositories
SUSEConnect --product sle-module-development-tools/15.1/x86_64
SUSEConnect --product PackageHub/15.1/x86_64
zypper ar https://download.opensuse.org/repositories/science/SLE_15_SP1/science.repo
zypper ar https://download.opensuse.org/repositories/home:rhabacker:branches:Application:Geo/openSUSE_Leap_15.1/home:rhabacker:branches:Application:Geo.repo
zypper ar -f https://download.opensuse.org/repositories/Application:/Geo/SLE_15_SP1_Backports/ qgis
zypper ar -f https://download.opensuse.org/repositories/Application:/Geo/SLE_15_SP1_Backports_debug/ qgis-dev
# Install qgis
zypper --gpg-auto-import-keys -n install qgis
randomorder commented 3 years ago

So We need to use 2 opensuse repositories?

zypper ar https://download.opensuse.org/repositories/science/openSUSE_Leap_15.2/science.repo
zypper ar https://download.opensuse.org/repositories/home:rhabacker:branches:Application:Geo/openSUSE_Leap_15.1/home:rhabacker:branches:Application:Geo.repo

What packages are we fetching from those repos?

lpasquali commented 3 years ago

nope only one first it was a typo I corrected above:

zypper ar https://download.opensuse.org/repositories/home:rhabacker:branches:Application:Geo/openSUSE_Leap_15.1/home:rhabacker:branches:Application:Geo.repo needed for python3-GDAL2

lpasquali commented 3 years ago

@randomorder for python3-GDAL2 there is no other way: https://software.opensuse.org/package/python3-GDAL2

randomorder commented 3 years ago

Please try the following:

$ export PYTHONPATH=/usr/share/qgis/python/:/usr/share/qgis/python/plugins:/usr/lib/python3/dist-packages/qgis:/usr/share/qgis/python/qgis

$ python3

from qgis.core import *
from qgis.analysis import QgsNativeAlgorithms
qgis_path = "/usr/share/qgis"
QgsApplication.setPrefixPath(qgis_path, True)
qgs = QgsApplication([], True)
qgs.initQgis()
sys.path.append(qgis_path + r'\python\plugins')
import processing
from processing.core.Processing import Processing
from processing.tools import postgis
Processing.initialize()
QgsApplication.processingRegistry().addProvider(QgsNativeAlgorithms())
lpasquali commented 3 years ago

unfortunately package qgis and its python bindings look totally broken to me:

webgis-template:~ # export PYTHONPATH=/usr/share/qgis/python/:/usr/share/qgis/python/plugins:/usr/lib/python3/dist-packages/qgis:/usr/share/qgis/python/qgis
webgis-template:~ # python3
Python 3.6.10 (default, Dec 19 2019, 15:48:40) [GCC] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from qgis.core import *
>>> from qgis.analysis import QgsNativeAlgorithms
>>> qgis_path = "/usr/share/qgis"
>>> QgsApplication.setPrefixPath(qgis_path, True)
>>> qgs = QgsApplication([], True)
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'

src/core/qgsapplication.cpp:160 : (QgsApplication) [0ms] loading of qgis translation failed /usr/share/qgis/share/qgis/i18n//qgis_
src/core/qgsapplication.cpp:175 : (QgsApplication) [0ms] loading of qt translation failed /usr/share/qt5/translations/qt_
>>> 
>>> 
>>> qgs.initQgis()
src/core/qgsproviderregistry.cpp:133 : (init) [17193ms] Checking /usr/share/qgis/lib64/qgis for provider plugins
src/core/qgsproviderregistry.cpp:137 : (init) [0ms] No dynamic QGIS data provider plugins found in:
/usr/share/qgis/lib64/qgis

src/core/qgsproviderregistry.cpp:215 : (init) [0ms] Loaded 4 providers (gdal;memory;mesh_memory;ogr) 
src/core/qgsapplication.cpp:1685 : (applyGdalSkippedDrivers) [0ms] Gdal Skipped driver list set to:
src/core/qgsapplication.cpp:1686 : (applyGdalSkippedDrivers) [0ms] 
src/core/providers/ogr/qgsogrprovider.cpp:3287 : (createFilters) [4ms] Unknown driver VICAR for file filters.
src/core/providers/ogr/qgsogrprovider.cpp:3287 : (createFilters) [0ms] Unknown driver EEDA for file filters.
src/core/providers/ogr/qgsogrprovider.cpp:3287 : (createFilters) [0ms] Unknown driver OGR_SDTS for file filters.
src/core/providers/ogr/qgsogrprovider.cpp:3287 : (createFilters) [0ms] Unknown driver Memory for file filters.
src/core/providers/ogr/qgsogrprovider.cpp:3287 : (createFilters) [1ms] Unknown driver GPSBabel for file filters.
src/core/providers/ogr/qgsogrprovider.cpp:3287 : (createFilters) [0ms] Unknown driver OGR_PDS for file filters.
src/core/providers/ogr/qgsogrprovider.cpp:3287 : (createFilters) [0ms] Unknown driver WFS for file filters.
src/core/providers/ogr/qgsogrprovider.cpp:3287 : (createFilters) [0ms] Unknown driver OAPIF for file filters.
src/core/providers/ogr/qgsogrprovider.cpp:3287 : (createFilters) [0ms] Unknown driver AeronavFAA for file filters.
src/core/providers/ogr/qgsogrprovider.cpp:3287 : (createFilters) [0ms] Unknown driver Cloudant for file filters.
src/core/providers/ogr/qgsogrprovider.cpp:3287 : (createFilters) [0ms] Unknown driver Elasticsearch for file filters.
src/core/providers/ogr/qgsogrprovider.cpp:3287 : (createFilters) [0ms] Unknown driver Walk for file filters.
src/core/providers/ogr/qgsogrprovider.cpp:3287 : (createFilters) [0ms] Unknown driver Carto for file filters.
src/core/providers/ogr/qgsogrprovider.cpp:3287 : (createFilters) [0ms] Unknown driver AmigoCloud for file filters.
src/core/providers/ogr/qgsogrprovider.cpp:3287 : (createFilters) [0ms] Unknown driver Selafin for file filters.
src/core/providers/ogr/qgsogrprovider.cpp:3287 : (createFilters) [0ms] Unknown driver PLSCENES for file filters.
src/core/providers/ogr/qgsogrprovider.cpp:3287 : (createFilters) [0ms] Unknown driver CSW for file filters.
src/core/providers/ogr/qgsogrprovider.cpp:3287 : (createFilters) [0ms] Unknown driver NGW for file filters.
src/core/providers/ogr/qgsogrprovider.cpp:3287 : (createFilters) [0ms] Unknown driver MapML for file filters.
src/core/providers/ogr/qgsogrprovider.cpp:3287 : (createFilters) [0ms] Unknown driver HTTP for file filters.
src/core/auth/qgsauthmanager.cpp:176 : (init) [0ms] Initializing QCA...
src/core/auth/qgsauthmanager.cpp:179 : (init) [0ms] QCA initialized.
src/core/auth/qgsauthmanager.cpp:182 : (init) [2ms] QCA Plugin Diagnostics Context: Checking Qt static plugins:
  (none)
Checking Qt Library Path: /usr/lib64/qt5/plugins
  libqca-cyrus-sasl.so: (class: saslPlugin) loaded as qca-cyrus-sasl
  libqca-gcrypt.so: (class: gcryptPlugin) loaded as qca-gcrypt
  libqca-gnupg.so: (class: gnupgPlugin) loaded as qca-gnupg
  libqca-logger.so: (class: loggerPlugin) loaded as qca-logger
  libqca-nss.so: (class: nssPlugin) loaded as qca-nss
  libqca-ossl.so: (class: opensslPlugin) loaded as qca-ossl
  libqca-softstore.so: (class: softstorePlugin) loaded as qca-softstore
Checking Qt Library Path: /usr/bin
  (No 'crypto' subdirectory)

src/core/auth/qgsauthmanager.cpp:186 : (init) [0ms] QCA supports: random,md5,sha1,keystorelist,sasl,md4,ripemd160,sha256,sha384,sha512,hmac(md5),hmac(sha1),hmac(sha256),hmac(sha384),hmac(sha512),hmac(ripemd160),aes128-ecb,aes128-cfb,aes128-cbc,aes192-ecb,aes192-cfb,aes192-cbc,aes256-ecb,aes256-cfb,aes256-cbc,blowfish-ecb,blowfish-cbc,blowfish-cfb,tripledes-ecb,des-ecb,des-cbc,des-cfb,aes128-ofb,aes192-ofb,aes256-ofb,des-ofb,tripledes-ofb,blowfish-ofb,pbkdf1(sha1),pbkdf2(sha1),pgpkey,openpgp,log,md2,des-cbc-pkcs7,sha224,hmac(sha224),aes128-cbc-pkcs7,aes128-ctr,aes192-cbc-pkcs7,aes192-ctr,aes256-cbc-pkcs7,aes256-ctr,blowfish-cbc-pkcs7,tripledes-cbc,des-ecb-pkcs7,cast5-ecb,cast5-cbc,cast5-cbc-pkcs7,cast5-cfb,cast5-ofb,pkey,dlgroup,rsa,dsa,dh,cert,csr,crl,certcollection,pkcs12,tls,cms,ca
src/core/auth/qgsauthmanager.cpp:196 : (init) [0ms] Prioritizing qca-ossl over all other QCA providers...
src/core/auth/qgsauthmanager.cpp:210 : (init) [0ms] QCA provider priorities: qca-cyrus-sasl:1, qca-gcrypt:1, qca-gnupg:1, qca-logger:1, qca-nss:1, qca-ossl:0, qca-softstore:1
src/core/auth/qgsauthmanager.cpp:212 : (init) [0ms] Populating auth method registry
src/core/auth/qgsauthmethodregistry.cpp:66 : (QgsAuthMethodRegistry) [0ms] Checking for auth method plugins in: /usr/share/qgis/lib64/qgis
src/core/qgsmessagelog.cpp:29 : (logMessage) [0ms] 2020-10-16T17:23:14 No Authentication Methods[1] No QGIS auth method plugins found in:
/usr/share/qgis/lib64/qgis
No authentication methods can be used. Check your QGIS installation
src/core/auth/qgsauthmanager.cpp:217 : (init) [0ms] Authentication methods found: 
src/core/auth/qgsauthmanager.cpp:479 : (isDisabled) [0ms] Authentication system DISABLED: QCA's qca-ossl (OpenSSL) plugin is missing
>>> sys.path.append(qgis_path + r'\python\plugins')
>>> import processing
>>> from processing.core.Processing import Processing
>>> from processing.tools import postgis
>>> Processing.initialize()
src/core/qgsmessagelog.cpp:29 : (logMessage) [17171ms] 2020-10-16T17:23:31 Processing[2] Problem with SAGA installation: SAGA was not found or is not correctly installed
>>> QgsApplication.processingRegistry().addProvider(QgsNativeAlgorithms())
proj_create_from_database: Cannot find proj.db
src/core/qgscoordinatereferencesystem.cpp:607 : (loadFromDatabase) [19002ms] failed : /usr/share/qgis/share/qgis/resources/srs.db does not exist!
True
>>>
lpasquali commented 3 years ago

I would advise to do one last try with only opensuse repositories for everything of qgis, this combination was not tested against python code above

randomorder commented 3 years ago

ok, go ahead

lpasquali commented 3 years ago

I tried on a local suse installation of mine (SLE 15 SP1 Desktop) to install a debian chroot and run it as a container with systemd-nspawn, besides that qgis 3.14 compiled for debian stabled but qgis.org lacks the postgis module in processing from processing.tools import postgis (it was removed completely) and it works (note the absence of errors, oddly in debian I had to import sys):

root@linux-ztqz:~# python3
Python 3.7.3 (default, Jul 25 2020, 13:03:44) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from qgis.core import *
>>> from qgis.analysis import QgsNativeAlgorithms
>>> qgis_path = "/usr/share/qgis"
>>> QgsApplication.setPrefixPath(qgis_path, True)
>>> qgs = QgsApplication([], True)
>>> qgs.initQgis()
>>> import sys
>>> import processing
/usr/lib/python3/dist-packages/qgis/utils.py:794: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  mod = _builtin_import(name, globals, locals, fromlist, level)
>>> from processing.core.Processing import Processing
>>> Processing.initialize()
>>> QgsApplication.processingRegistry().addProvider(QgsNativeAlgorithms())
True
>>>

in parallel I compiled the rpm from source rpm but I got same results about saga not being installed correctly. making a debian chroot and using it with systemd-nspawn and machinectl could be a solution to let them use suse...

lpasquali commented 3 years ago

@randomorder putting into blocked until we meet at 16:00 today

randomorder commented 3 years ago

@chpicone please check this: https://github.com/qgis/QGIS/issues/35213 looks like a module in use has been deprecated

chpicone commented 3 years ago

@randomorder I am using the version QGIS 3.12 (like in PA). Let me make some check installing this version and updating this script. If I can use an alternative to that library it is ok for me.

randomorder commented 3 years ago

@lpasquali please test the dockerized version of QGIS 3.12

lpasquali commented 3 years ago

@randomorder @chpicone I tested latest stable 3.12 version of QGIS for docker which comes into two versions final-3.12.3-focal and final-3.12.3 (note, to make test work without X11 at all I used os.environ["QT_QPA_PLATFORM"] = "offscreen" variable to configure QgsApplication headless) I think we should use final-3.12.3 as it gives less deprecation warnings:

# final-3.12.3-focal results
root@c96c0e27c800:/# python /app/script.py
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'
/usr/lib/python3/dist-packages/qgis/utils.py:743: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  mod = _builtin_import(name, globals, locals, fromlist, level)
/usr/share/qgis/python/plugins/processing/algs/qgis/ImportIntoSpatialite.py:103: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if uri.database() is '':
/usr/share/qgis/python/plugins/processing/algs/qgis/SpatialiteExecuteSQL.py:69: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if uri.database() is '':
/usr/share/qgis/python/plugins/processing/algs/grass7/Grass7Utils.py:192: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if path is '':
root@c96c0e27c800:/# cat /app/script.py
import os,sys
os.environ["QT_QPA_PLATFORM"] = "offscreen"
from qgis.core import *
from qgis.analysis import QgsNativeAlgorithms
qgis_path = "/usr/share/qgis"
QgsApplication.setPrefixPath(qgis_path, True)
qgs = QgsApplication([], True)
qgs.initQgis()
sys.path.append(qgis_path + r'\python\plugins')
import processing
from processing.core.Processing import Processing
from processing.tools import postgis
Processing.initialize()
QgsApplication.processingRegistry().addProvider(QgsNativeAlgorithms())
root@c96c0e27c800:/# python3
Python 3.8.2 (default, Apr 27 2020, 15:53:34) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os,sys
>>> os.environ["QT_QPA_PLATFORM"] = "offscreen"
>>> from qgis.core import *
>>> from qgis.analysis import QgsNativeAlgorithms
>>> qgis_path = "/usr/share/qgis"
>>> QgsApplication.setPrefixPath(qgis_path, True)
>>> qgs = QgsApplication([], True)
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'
>>> qgs.initQgis()
>>> sys.path.append(qgis_path + r'\python\plugins')
>>> import processing
/usr/lib/python3/dist-packages/qgis/utils.py:743: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  mod = _builtin_import(name, globals, locals, fromlist, level)
>>> from processing.core.Processing import Processing
>>> from processing.tools import postgis
>>> Processing.initialize()
>>> QgsApplication.processingRegistry().addProvider(QgsNativeAlgorithms())
True
>>> 
# final-3.12.3 
geosolutions@webgis-template:~> docker run -it --rm -v /home/geosolutions/script.py:/app/script.py qgis/qgis:final-3_12_3_focal bash
root@c2dc81202fc1:/# python /app/script.py
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'
/usr/lib/python3/dist-packages/qgis/utils.py:743: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  mod = _builtin_import(name, globals, locals, fromlist, level)
/usr/share/qgis/python/plugins/processing/algs/qgis/ImportIntoSpatialite.py:103: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if uri.database() is '':
/usr/share/qgis/python/plugins/processing/algs/qgis/SpatialiteExecuteSQL.py:69: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if uri.database() is '':
/usr/share/qgis/python/plugins/processing/algs/grass7/Grass7Utils.py:192: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if path is '':
root@c2dc81202fc1:/# cat /app/script.py
import os,sys
os.environ["QT_QPA_PLATFORM"] = "offscreen"
from qgis.core import *
from qgis.analysis import QgsNativeAlgorithms
qgis_path = "/usr/share/qgis"
QgsApplication.setPrefixPath(qgis_path, True)
qgs = QgsApplication([], True)
qgs.initQgis()
sys.path.append(qgis_path + r'\python\plugins')
import processing
from processing.core.Processing import Processing
from processing.tools import postgis
Processing.initialize()
QgsApplication.processingRegistry().addProvider(QgsNativeAlgorithms())
root@c2dc81202fc1:/# python 
Python 3.8.2 (default, Apr 27 2020, 15:53:34) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os,sys
>>> os.environ["QT_QPA_PLATFORM"] = "offscreen"
>>> from qgis.core import *
>>> from qgis.analysis import QgsNativeAlgorithms
>>> qgis_path = "/usr/share/qgis"
>>> QgsApplication.setPrefixPath(qgis_path, True)
>>> qgs = QgsApplication([], True)
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'
>>> qgs.initQgis()
>>> sys.path.append(qgis_path + r'\python\plugins')
>>> import processing
/usr/lib/python3/dist-packages/qgis/utils.py:743: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  mod = _builtin_import(name, globals, locals, fromlist, level)
>>> from processing.core.Processing import Processing
>>> from processing.tools import postgis
>>> Processing.initialize()
>>> QgsApplication.processingRegistry().addProvider(QgsNativeAlgorithms())
True
>>>
randomorder commented 3 years ago

Finally, some good news

Can you please let us know how to test it further? @chpicone I want to make sure we get it right before we report to PA

Also we'll have to ingrate the webapp into this docker image. We need to understand what that is about..

lpasquali commented 3 years ago

Documentation modified for docker, added notes on the official way for PA specific suse version