Open das-g opened 5 years ago
FRidh
commented
5 years ago future I would expect to be quite essential, as you've demonstrated. plotly I would be a bit more hesitant to. Anyway, it should be straightforward to extend it with optional Python packages.
FRidh
commented
5 years ago By the way, I suggest always cc'ing the maintainer of the expression.
das-g
commented
5 years ago By the way, I suggest always cc'ing the maintainer of the expression.
@lsix, thus?
das-g
commented
5 years ago I can't quite wrap my head around the description in https://nixos.org/nixpkgs/manual/#ssec-stdenv-dependencies. Which of these variables aught to be used for runtime-only dependencies? (As they're obviously not required for building QGIS, that's what the additional Python libraries would be, wouldn't they?)
das-g
commented
5 years ago
futureI would expect to be quite essential, as you've demonstrated.plotlyI would be a bit more hesitant to. Anyway, it should be straightforward to extend it with optional Python packages.
QGIS, depending on QT, GDAL/OGR, GRASS GIS, NumPy and whatnot is already a very "fat" software, so I feel some python packages give-or-take shouldn't matter too much. Thus I'm tempted to add all of these python packages to its dependencies.
We can always make them optional later through an option, if the need for a (slightly) slimmer QGIS arises. (Or is there something particularly expensive about adding plotly, @FRidh?)
lsix
commented
5 years ago Hi,
I have added a extraPythonPackages to the qgis derivation to cover those use cases. If find that some required dependencies are missing, I will add them. If this is something that you need for custom plugins, I would recommend using a custom derivation:
$ nix-shell -p 'qgis.override { extraPythonPackages = (ps: [ ps.numpy ps.future ]);}'We can for sure add a heavy version of qgis in all-packages.nix that will bring all the extra packages you would expect:
qgis = …;
qgisWithDeps = qgis.override { extraPythonPackages = (ps: [ ps.… ]); };This will add nearly no load to hydra servers since qgis itsefl does not need to be rebuilt.
das-g
commented
5 years ago What do you mean by "custom plugins", @lsix?
I'd expect that if I install QGIS, I can install all plugins it offers me in its built-in plugin manager from the official QGIS plugin repository (https://plugins.qgis.org/plugins/), just like I can install arbitrary compatible Firefox plug-ins from addons.mozilla.org in Firefox.
The community-provided plugins are an integral part of the QGIS ecosystem and QGIS isn't a dev tool where we can expect the users to craft a dedicated shell.nix for every separate project, so having the variant with these packages in NixPkgs seems prudent. I do like your idea of using your extraPythonPackages mechanism for that, but I think the qgis attribute should then point to the variant with those "default extra packages", e.g.:
{
#…
qgisMinimal = …;
qgis = qgisMinimal.override { extraPythonPackages = (ps: [ ps.… ]); };
#…
}Unless there's a strong reason not to go that way, of course.
(I'm not quite happy with the name qgisMinimal, but it seems at least better than the qgisWithoutDeps I first thought of. Any suggestions for alternatives?)
lsix
commented
5 years ago qgisLight could be what you are looking for.
I have absolutely no reason to keep qgis as it is now with little dependencies. It is enough for my needs (or if not, I just use the extraPythonPackages), but I admin I do not rely on the plugins from QGIS.
I’ll make the modification probably somewhere next week (I’ll be mostly away from my computer for the long week end to come).
das-g
commented
5 years ago Cool, no hurry.
I may give it a try myself meanwhile.
das-g
commented
5 years ago I dunno yet whether what corresponds to
python3-pyqt5.qscipython3-pyqt5.qtsqlpython3-pyqt5.qtsvgpython3-pyqt5.qtwebkitis already included in
python3Packages.pyqt5.
Python modules PyQt5.Qsci, PyQt5.QtSql and PyQt5.QtSvg are already importable in QGIS and seem to be provided by /nix/store/8jkwfdn6hyw7sv464n1a6y5ddzjvzl15-python3.7-qscintilla-2.9.4.
I don't know yet whether Python module PyQt5.QtWebKit is available in NixPkgs at all.
das-g
commented
5 years ago I'm not quite sure how reliable my approach of looking at the dependencies of python3-qgis is for finding out what Python modules officially (for whatever value of "official") should be available to QGIS plugins. Thus, I figured I'd just ask on the QGIS-Developer mailing list:
das-g
commented
5 years ago Sources mentioned so far in that mailing list thread:
python3-qgis https://github.com/qgis/QGIS/blob/master/debian/control.in#L370setup.hint files of OSGeo4W's qgis-full: http://download.osgeo.org/osgeo4w/x86_64/release/qgis/Even taking into account that some of the latter might be build-only dependencies and that packaging granularity might be different, there seems to be quite some discrepancy:
Maybe the list for the Mac OS installer includes transitive dependencies, too?
das-g
commented
5 years ago I don't know yet whether Python module
PyQt5.QtWebKitis available in NixPkgs at all.
Looks like pyqt5_with_webkit would provide that, but I'm not sure how to use that, yet.
stale[bot]
commented
4 years ago Thank you for your contributions.
This has been automatically marked as stale because it has had no activity for 180 days.
If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.
Here are suggestions that might help resolve this more quickly:
das-g
commented
4 years ago If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me".
still important to me :hugs:
stale[bot]
commented
3 years ago I marked this as stale due to inactivity. → More info
das-g
commented
3 years ago Can we get to some conclusion
nixpkgs attributes with and without that set of packages, respectively?
I'd suggest to go with the python3-qgis dependencies, i.e.
nixpkgs attributesAs already mentioned, for the one including that Python package set, I'd go with just qgis.
Suggestions for the attribute for bare QGIS without that package set have been:
qgisMinimalqgisWithoutDepsqgisLightI feel none of them really get to the essence of what that derivation is (or, more importantly, what it isn't or hasn't). Maybe qgisWithJustBarePython would convey that? But that name would be kinda unwieldy.
mpickering
commented
3 years ago @das-g I ran into this problem as well. It would be good to execute on something, My preference is as yours, that plugins should "just work".
FRidh
commented
3 years ago Adding packages is "easy", removing them is not. With Nix the whole approach is to declare your dependencies. If you start using third-party package/plugin managers that have undeclared dependencies, then where lies the issue? One can create a QGIS plugin that relies on the whole of PyPI. Should we then include the whole of PyPI? And what when you have plugins that require different versions of Python packages?
qtwebkit is not what you by default would want to rely on. Often there are security issues, which, if taken into account directly, would result in an unbuildable QGIS.
I suggest having a minimal one and another one that takes an approach like is done with home-manager: declaring for plugins what deps they have. That way we at least keep track of why deps are added. Ideally the plugin itself would also be managed by Nix then.
stale[bot]
commented
3 years ago I marked this as stale due to inactivity. → More info
das-g
commented
3 years ago declaring for plugins what deps they have. That way we at least keep track of why deps are added. Ideally the plugin itself would also be managed by Nix then.
While that might look like a cleaner approach, would that entail mirroring all of https://plugins.qgis.org/plugins/ (currently 1483 entries) in nixpkgs, augmented by our own assessment of what plugin needs what pip-packages? With the low number of people maintaining QGIS related stuff in nixpkgs, this doesn't seem very feasible.
But maybe we can just wait for QGIS/QGIS-Enhancement-Proposals#179 and/or QGIS/QGIS-Enhancement-Proposals#202 to become reality, which would at least put the burden of figuring out / declaring dependencies onto the plugin authors rather than nixpkgs maintainers.
stale[bot]
commented
2 years ago I marked this as stale due to inactivity. → More info
bgamari
commented
2 years ago I have also run into this (particularly due to plotly being missing). I agree with others who have said that plugins should just work.
timlinux
commented
2 months ago If you are new to nixos and looking for a working example of how to incorporate new python dependencies into your flake or configuration.nix without recompiling QGIS yourself, maybe this snippet will help you:
{
config,
pkgs,
...
}: {
environment.systemPackages = with pkgs; [
(qgis.override {
extraPythonPackages = ps:
with ps; [
numpy
requests
debugpy
future
matplotlib
pandas
geopandas
plotly
pyqt5_with_qtwebkit
pyqtgraph
rasterio
sqlalchemy
];
})
# other packages
];
}
TL;DR:
QGIS from NixPkgs seems to be missing the following Python packages, which may be relied on by QGIS plugins:
futurehttplib2markupsafeplotlypyprojIssue description
For non-Nix(OS) systems, QGIS usually ships with a base set of Python packages. Some of them are required by QGIS itself, while others aren't. But QGIS plugins or Python scripts of the user may still rely on them being accessible from within QGIS.
The QGIS in NixPkgs currently doesn't ship with the complete set of them:
QGIS 3.4 from NixOS release-19.03 currently has these
python3Packagesin itsbuildInputs:chardetdateutilgdaljinja2numpyowslibpsycopg2pygmentspyqt5pytzpyyamlqscintilla-qt5requestssipsixurllib3The Ubuntu package
python3-qgis(from https://qgis.org/ubuntu-ltr) currently depends on these packages:python3-dateutilpython3-futurepython3-httplib2python3-jinja2python3-markupsafepython3-owslibpython3-plotlypython3-psycopg2python3-pygmentspython3-pyprojpython3-pyqt5python3-pyqt5.qscipython3-pyqt5.qtsqlpython3-pyqt5.qtsvgpython3-pyqt5.qtwebkitpython3-requestspython3-sippython3-sixpython3-tzpython3-yamlSo assuming that
python3Packages.pytzcorresponds topython3-tzandpython3Packages.pyyamltopython3-yaml, the ones missing in NixOS QGIS are:futurehttplib2markupsafeplotlypyprojI dunno yet whether what corresponds to
python3-pyqt5.qscipython3-pyqt5.qtsqlpython3-pyqt5.qtsvgpython3-pyqt5.qtwebkitis already included in
python3Packages.pyqt5.Steps to reproduce
To see that some of these are indeed missing:
Expected
module is successfully imported (no output)
Observed
To see that some of these are indeed needed by some Plugins:
Expected
Plugin is successfully installed
Observed
Lengthy error message:
"ModuleNotFoundError: No module named 'future'" (click to expand)
screenshot (click to expand)
Technical details
"x86_64-linux"Linux 4.19.34, NixOS, 19.03.172205.ea497998e4b (Koi)yesyesnix-env (Nix) 2.2"nixos-19.03.172231.7b36963e7a7"""/nix/var/nix/profiles/per-user/root/channels/nixos