search

nsidc / qgreenland

Source code for generating the QGreenland package hosted at https://qgreenland.org/
https://qgreenland.readthedocs.io
Other
36 stars 9 forks source link

Linux: "Too many open files" #225

Closed MartinLuethi closed 2 years ago

MartinLuethi commented 3 years ago

QGreenland_v1.0.1 has quite a few issues. I downloaded the zip from the webpage, unzipped it, and then opened the qgreenland.qgs

  1. there are 45 layers that are not available (dialog box)
  2. startup is extremely slow
  3. Critical: Could not create AF_NETLINK socket (Too many open files)
  4. Warning 4: Failed to open /home/data/QGreenland_v1.0.1/Places/Towns and settlements/populated_places.gpkg: Too many open files.

lots of the above two error messages occur (3./4.).

QGIS 3.18.1-Zürich Arch Linux as of today very performant laptop with lots of RAM (32G) of which almost nothing is used.

MartinLuethi commented 3 years ago

Solved by setting the number of open files to a very high number (basically unlimited)

ulimit -n  20000

Currently, qgis with QGreenland loaded, qgis has some 19000 open files

lsof | grep qgis | wc

Maybe you could make the whole project modular. Or pull in layers on demand with git-annex or Datalad

qgreenland commented 3 years ago

Hi Martin -

Thank you for getting in touch. I’m glad that you found this solution to your original email. We have run into this problem with Linux. Since this is v1.0, we are very interested to hear user experiences and appreciate your thoughts about modular design. We are exploring options, though like so much science we are time and people limited. Some advances may come more quickly, others slowly, and some may have to remain on the wish list until a future grant.

Do get in touch anytime with questions and feedback.

Thank you, Twila

On Apr 18, 2021, at 6:38 AM, MartinLuethi @.***> wrote:

Solved by setting the number of open files to a very high number (basically unlimited) ulimit -n 20000 Currently, qgis with QGreenland loaded, qgis has some 19000 open files lsof | grep qgis | wc

Maybe you could make the whole project modular. Or pull in layers on demand with git-annex or Datalad

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/nsidc/qgreenland/issues/225#issuecomment-821985432, or unsubscribe https://github.com/notifications/unsubscribe-auth/ASTMUOKN7EEJYXBR7YGEZHLTJLHENANCNFSM43EFFQFA.

— QGreenland: A free GIS environment for Greenland www.QGreenland.org

MattF-NSIDC commented 3 years ago

I think using lsof in the way shown above may produce an inflated count of open files by listing duplicates, I'm not 100% sure. On my machine with QGIS 3.10 LTS I measured 830 files opened by QGreenland like so:

First grab the PID of QGIS (2900713 on my machine)

$ ps -e | grep qgis
2900713 pts/3    00:00:18 qgis.bin

Then use that PID to list open file descriptors and count them:

$ ls -l /proc/2900713/fd | wc -l
874
$ ls -l /proc/2900713/fd | grep QGreenland | wc -l
830

This is telling me that QGIS has 830 QGreenland files open, and 44 unrelated files open. If your operating system's ulimit is low, e.g. 1024, having any other applications open at the same time as QGreenland may cause you to experience this issue.

MattF-NSIDC commented 2 years ago

This is now part of our documentation:

https://github.com/nsidc/qgreenland/blob/main/doc/TROUBLESHOOTING.md