Open das-g opened 5 years ago
future
I would expect to be quite essential, as you've demonstrated. plotly
I would be a bit more hesitant to. Anyway, it should be straightforward to extend it with optional Python packages.
By the way, I suggest always cc'ing the maintainer of the expression.
By the way, I suggest always cc'ing the maintainer of the expression.
@lsix, thus?
I can't quite wrap my head around the description in https://nixos.org/nixpkgs/manual/#ssec-stdenv-dependencies. Which of these variables aught to be used for runtime-only dependencies? (As they're obviously not required for building QGIS, that's what the additional Python libraries would be, wouldn't they?)
future
I would expect to be quite essential, as you've demonstrated.plotly
I would be a bit more hesitant to. Anyway, it should be straightforward to extend it with optional Python packages.
QGIS, depending on QT, GDAL/OGR, GRASS GIS, NumPy and whatnot is already a very "fat" software, so I feel some python packages give-or-take shouldn't matter too much. Thus I'm tempted to add all of these python packages to its dependencies.
We can always make them optional later through an option, if the need for a (slightly) slimmer QGIS arises. (Or is there something particularly expensive about adding plotly
, @FRidh?)
Hi,
I have added a extraPythonPackages
to the qgis
derivation to cover those use cases. If find that some required dependencies are missing, I will add them. If this is something that you need for custom plugins, I would recommend using a custom derivation:
$ nix-shell -p 'qgis.override { extraPythonPackages = (ps: [ ps.numpy ps.future ]);}'
We can for sure add a heavy version of qgis in all-packages.nix that will bring all the extra packages you would expect:
qgis = …;
qgisWithDeps = qgis.override { extraPythonPackages = (ps: [ ps.… ]); };
This will add nearly no load to hydra servers since qgis itsefl does not need to be rebuilt.
What do you mean by "custom plugins", @lsix?
I'd expect that if I install QGIS, I can install all plugins it offers me in its built-in plugin manager from the official QGIS plugin repository (https://plugins.qgis.org/plugins/), just like I can install arbitrary compatible Firefox plug-ins from addons.mozilla.org in Firefox.
The community-provided plugins are an integral part of the QGIS ecosystem and QGIS isn't a dev tool where we can expect the users to craft a dedicated shell.nix
for every separate project, so having the variant with these packages in NixPkgs seems prudent. I do like your idea of using your extraPythonPackages
mechanism for that, but I think the qgis
attribute should then point to the variant with those "default extra packages", e.g.:
{
#…
qgisMinimal = …;
qgis = qgisMinimal.override { extraPythonPackages = (ps: [ ps.… ]); };
#…
}
Unless there's a strong reason not to go that way, of course.
(I'm not quite happy with the name qgisMinimal
, but it seems at least better than the qgisWithoutDeps
I first thought of. Any suggestions for alternatives?)
qgisLight
could be what you are looking for.
I have absolutely no reason to keep qgis as it is now with little dependencies. It is enough for my needs (or if not, I just use the extraPythonPackages
), but I admin I do not rely on the plugins from QGIS.
I’ll make the modification probably somewhere next week (I’ll be mostly away from my computer for the long week end to come).
Cool, no hurry.
I may give it a try myself meanwhile.
I dunno yet whether what corresponds to
python3-pyqt5.qsci
python3-pyqt5.qtsql
python3-pyqt5.qtsvg
python3-pyqt5.qtwebkit
is already included in
python3Packages.pyqt5
.
Python modules PyQt5.Qsci
, PyQt5.QtSql
and PyQt5.QtSvg
are already importable in QGIS and seem to be provided by /nix/store/8jkwfdn6hyw7sv464n1a6y5ddzjvzl15-python3.7-qscintilla-2.9.4
.
I don't know yet whether Python module PyQt5.QtWebKit
is available in NixPkgs at all.
I'm not quite sure how reliable my approach of looking at the dependencies of python3-qgis
is for finding out what Python modules officially (for whatever value of "official") should be available to QGIS plugins. Thus, I figured I'd just ask on the QGIS-Developer mailing list:
Sources mentioned so far in that mailing list thread:
python3-qgis
https://github.com/qgis/QGIS/blob/master/debian/control.in#L370setup.hint
files of OSGeo4W's qgis-full
: http://download.osgeo.org/osgeo4w/x86_64/release/qgis/Even taking into account that some of the latter might be build-only dependencies and that packaging granularity might be different, there seems to be quite some discrepancy:
Maybe the list for the Mac OS installer includes transitive dependencies, too?
I don't know yet whether Python module
PyQt5.QtWebKit
is available in NixPkgs at all.
Looks like pyqt5_with_webkit
would provide that, but I'm not sure how to use that, yet.
Thank you for your contributions.
This has been automatically marked as stale because it has had no activity for 180 days.
If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.
Here are suggestions that might help resolve this more quickly:
If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me".
still important to me :hugs:
I marked this as stale due to inactivity. → More info
Can we get to some conclusion
nixpkgs
attributes with and without that set of packages, respectively?
I'd suggest to go with the python3-qgis
dependencies, i.e.
nixpkgs
attributesAs already mentioned, for the one including that Python package set, I'd go with just qgis
.
Suggestions for the attribute for bare QGIS without that package set have been:
qgisMinimal
qgisWithoutDeps
qgisLight
I feel none of them really get to the essence of what that derivation is (or, more importantly, what it isn't or hasn't). Maybe qgisWithJustBarePython
would convey that? But that name would be kinda unwieldy.
@das-g I ran into this problem as well. It would be good to execute on something, My preference is as yours, that plugins should "just work".
Adding packages is "easy", removing them is not. With Nix the whole approach is to declare your dependencies. If you start using third-party package/plugin managers that have undeclared dependencies, then where lies the issue? One can create a QGIS plugin that relies on the whole of PyPI. Should we then include the whole of PyPI? And what when you have plugins that require different versions of Python packages?
qtwebkit
is not what you by default would want to rely on. Often there are security issues, which, if taken into account directly, would result in an unbuildable QGIS.
I suggest having a minimal one and another one that takes an approach like is done with home-manager: declaring for plugins what deps they have. That way we at least keep track of why deps are added. Ideally the plugin itself would also be managed by Nix then.
I marked this as stale due to inactivity. → More info
declaring for plugins what deps they have. That way we at least keep track of why deps are added. Ideally the plugin itself would also be managed by Nix then.
While that might look like a cleaner approach, would that entail mirroring all of https://plugins.qgis.org/plugins/ (currently 1483 entries) in nixpkgs, augmented by our own assessment of what plugin needs what pip
-packages? With the low number of people maintaining QGIS related stuff in nixpkgs, this doesn't seem very feasible.
But maybe we can just wait for QGIS/QGIS-Enhancement-Proposals#179 and/or QGIS/QGIS-Enhancement-Proposals#202 to become reality, which would at least put the burden of figuring out / declaring dependencies onto the plugin authors rather than nixpkgs maintainers.
I marked this as stale due to inactivity. → More info
I have also run into this (particularly due to plotly
being missing). I agree with others who have said that plugins should just work.
If you are new to nixos and looking for a working example of how to incorporate new python dependencies into your flake or configuration.nix without recompiling QGIS yourself, maybe this snippet will help you:
{
config,
pkgs,
...
}: {
environment.systemPackages = with pkgs; [
(qgis.override {
extraPythonPackages = ps:
with ps; [
numpy
requests
debugpy
future
matplotlib
pandas
geopandas
plotly
pyqt5_with_qtwebkit
pyqtgraph
rasterio
sqlalchemy
];
})
# other packages
];
}
TL;DR:
QGIS from NixPkgs seems to be missing the following Python packages, which may be relied on by QGIS plugins:
future
httplib2
markupsafe
plotly
pyproj
Issue description
For non-Nix(OS) systems, QGIS usually ships with a base set of Python packages. Some of them are required by QGIS itself, while others aren't. But QGIS plugins or Python scripts of the user may still rely on them being accessible from within QGIS.
The QGIS in NixPkgs currently doesn't ship with the complete set of them:
QGIS 3.4 from NixOS release-19.03 currently has these
python3Packages
in itsbuildInputs
:chardet
dateutil
gdal
jinja2
numpy
owslib
psycopg2
pygments
pyqt5
pytz
pyyaml
qscintilla-qt5
requests
sip
six
urllib3
The Ubuntu package
python3-qgis
(from https://qgis.org/ubuntu-ltr) currently depends on these packages:python3-dateutil
python3-future
python3-httplib2
python3-jinja2
python3-markupsafe
python3-owslib
python3-plotly
python3-psycopg2
python3-pygments
python3-pyproj
python3-pyqt5
python3-pyqt5.qsci
python3-pyqt5.qtsql
python3-pyqt5.qtsvg
python3-pyqt5.qtwebkit
python3-requests
python3-sip
python3-six
python3-tz
python3-yaml
So assuming that
python3Packages.pytz
corresponds topython3-tz
andpython3Packages.pyyaml
topython3-yaml
, the ones missing in NixOS QGIS are:future
httplib2
markupsafe
plotly
pyproj
I dunno yet whether what corresponds to
python3-pyqt5.qsci
python3-pyqt5.qtsql
python3-pyqt5.qtsvg
python3-pyqt5.qtwebkit
is already included in
python3Packages.pyqt5
.Steps to reproduce
To see that some of these are indeed missing:
Expected
module is successfully imported (no output)
Observed
To see that some of these are indeed needed by some Plugins:
Expected
Plugin is successfully installed
Observed
Lengthy error message:
"ModuleNotFoundError: No module named 'future'" (click to expand)
screenshot (click to expand)
![Bildschirmfoto von 2019-04-18 18-05-34](https://user-images.githubusercontent.com/97746/56375004-b48c5e00-6204-11e9-8471-b390b34387e6.png)Technical details
"x86_64-linux"
Linux 4.19.34, NixOS, 19.03.172205.ea497998e4b (Koi)
yes
yes
nix-env (Nix) 2.2
"nixos-19.03.172231.7b36963e7a7"
""
/nix/var/nix/profiles/per-user/root/channels/nixos