UserWarning: Valid PROJ data directory not found

[DOptimusPrime]

I am running this code but for, I realized that the FileNotFound Error is happening because it is looking for a file on another old computer? how do I solve that issue? Is it related to the Pyproj warning:

C:\Users\hdela\.conda\envs\chur2\python.exe D:\Chur\Manager\main.py batch_compute_precipitation 
running return period: 500
running return period: 500
executing 2 processes
['python', 'D:/Chur/Engine/main.py', 'precipitation', 'D:/Chur/Manager/precipitation_definitions/54ac8997-efe7-4877-922c-f10e68f4dc84']
['python', 'D:/Chur/Engine/main.py', 'precipitation', 'D:/Chur/Manager/precipitation_definitions/f563a296-bc30-4474-ac18-7f272fd58e14']
C:\Users\hdela\.conda\envs\chur2\lib\site-packages\pyproj\__init__.py:91: UserWarning: Valid PROJ data directory not found. Either set the path using the environmental variable PROJ_LIB or with `pyproj.datadir.set_data_dir`.
  warnings.warn(str(err))
C:\Users\hdela\.conda\envs\chur2\lib\site-packages\pyproj\__init__.py:91: UserWarning: Valid PROJ data directory not found. Either set the path using the environmental variable PROJ_LIB or with `pyproj.datadir.set_data_dir`.
  warnings.warn(str(err))
Traceback (most recent call last):
  File "D:\Chur\Engine\main.py", line 1676, in <module>
    main(sys.argv[1:])
  File "D:\Chur\Engine\main.py", line 606, in main
    engine.precipitationSimulator.initialize()
  File "D:\Chur\Engine\Models\Precipitation\precipitation.py", line 46, in initialize
    self.generator.initialize()
  File "D:\Chur\Engine\Models\Precipitation\precipitation.py", line 154, in initialize
    self.fh = Dataset(self.ncfile, mode='r')
  File "src\netCDF4\_netCDF4.pyx", line 2353, in netCDF4._netCDF4.Dataset.__init__
  File "src\netCDF4\_netCDF4.pyx", line 1963, in netCDF4._netCDF4._ensure_nc_success
FileNotFoundError: [Errno 2] No such file or directory: b'C:/Users/dhoussou/chur/Chur/Engine/Data/SourceData/precipitation/MeteoSwiss/RdisaggH_ch01r.swisscors_200703010100_200704010000.nc'
Traceback (most recent call last):
  File "D:\Chur\Engine\main.py", line 1676, in <module>
    main(sys.argv[1:])
  File "D:\Chur\Engine\main.py", line 606, in main
    engine.precipitationSimulator.initialize()
  File "D:\Chur\Engine\Models\Precipitation\precipitation.py", line 46, in initialize
    self.generator.initialize()
  File "D:\Chur\Engine\Models\Precipitation\precipitation.py", line 154, in initialize
    self.fh = Dataset(self.ncfile, mode='r')
  File "src\netCDF4\_netCDF4.pyx", line 2353, in netCDF4._netCDF4.Dataset.__init__
  File "src\netCDF4\_netCDF4.pyx", line 1963, in netCDF4._netCDF4._ensure_nc_success
FileNotFoundError: [Errno 2] No such file or directory: b'C:/Users/dhoussou/chur/Chur/Engine/Data/SourceData/precipitation/MeteoSwiss/RdisaggH_ch01r.swisscors_200710010100_200711010000.nc'
D:/Chur/Manager/precipitation_definitions/f563a296-bc30-4474-ac18-7f272fd58e14
2023-10-15 23:44:33-- engine: creating output directories at D:/Chur/Engine/Data/SimulationData//Precipitation/f563a296-bc30-4474-ac18-7f272fd58e14/ --
D:/Chur/Manager/precipitation_definitions/54ac8997-efe7-4877-922c-f10e68f4dc84
2023-10-15 23:44:33-- engine: creating output directories at D:/Chur/Engine/Data/SimulationData//Precipitation/54ac8997-efe7-4877-922c-f10e68f4dc84/ --
executed 2 processes

Process finished with exit code 0

My environment variables: C:\Users\hdela.conda\envs\chur2\Lib\site-packages\pyproj C:\Users\hdela.conda\envs\chur2\Library\share\proj

[djhoese]

Why did you delete all of the questions on the issue template? They are there for a reason. That reason is that we need that information to give you any useful kind of answer. Please answer these questions and then we'll see if we can help:

#### Installation method/steps
 - Installation method (conda, pip wheel, from source, etc...)
 - How did you install PROJ? Where is it installed?
 - Please provide all commands/steps you used to install pyproj and PROJ.

#### Environment Information
 - pyproj version you are attempting to install
 - PROJ version (Execute `proj` command and give version here.)
 - Python version (`python -c "import sys; print(sys.version.replace('\n', ' '))"`)
 - Operation System Information (`python -c "import platform; print(platform.platform())"`)

[DOptimusPrime]

Rel. 8.1.1, September 1st, 2021 usage: PROJ [-bdeEfiIlmorsStTvVwW [args]] [+opt[=arg] ...] [file ...]

This is my environment information:

3.9.15 | packaged by conda-forge | (main, Nov 22 2022, 08:39:05) [MSC v.1929 64 bit (AMD64)]
Windows-10-10.0.22621-SP0

I installed it through conda-forge:

These are all the package.:

``` Name Version Build Channel attrs 23.1.0 pyh71513ae_1 conda-forge boost-cpp 1.74.0 h9f4b32c_8 conda-forge brotli 1.1.0 hcfcfb64_1 conda-forge brotli-bin 1.1.0 hcfcfb64_1 conda-forge brotli-python 1.1.0 py39h99910a6_1 conda-forge bzip2 1.0.8 h8ffe710_4 conda-forge ca-certificates 2023.7.22 h56e8100_0 conda-forge cairo 1.16.0 hb19e0ff_1008 conda-forge certifi 2023.7.22 pyhd8ed1ab_0 conda-forge cfitsio 3.470 h0af3d06_7 conda-forge cftime 1.6.2 py39hc266a54_1 conda-forge charset-normalizer 3.3.0 pyhd8ed1ab_0 conda-forge click 8.1.7 win_pyh7428d3b_0 conda-forge click-plugins 1.1.1 py_0 conda-forge cligj 0.7.2 pyhd8ed1ab_1 conda-forge colorama 0.4.6 pyhd8ed1ab_0 conda-forge contourpy 1.1.1 py39h1f6ef14_1 conda-forge curl 7.86.0 heaf79c2_1 conda-forge cycler 0.12.0 pyhd8ed1ab_0 conda-forge decorator 4.4.2 py_0 conda-forge expat 2.5.0 h63175ca_1 conda-forge fiona 1.8.20 py39hea8b339_2 conda-forge font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge font-ttf-inconsolata 3.000 h77eed37_0 conda-forge font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge font-ttf-ubuntu 0.83 hab24e00_0 conda-forge fontconfig 2.14.2 hbde0cde_0 conda-forge fonts-conda-ecosystem 1 0 conda-forge fonts-conda-forge 1 0 conda-forge fonttools 4.43.0 py39ha55989b_0 conda-forge freetype 2.12.1 hdaf720e_2 conda-forge freexl 1.0.6 h67ca5e6_1 conda-forge gdal 3.3.3 py39h3f5efd6_0 conda-forge geopandas 0.9.0 pyhd8ed1ab_1 conda-forge geopandas-base 0.9.0 pyhd8ed1ab_1 conda-forge geos 3.9.1 h39d44d4_2 conda-forge geotiff 1.7.0 h350af67_3 conda-forge gettext 0.21.1 h5728263_0 conda-forge hdf4 4.2.15 h1b1b6ef_5 conda-forge hdf5 1.12.1 nompi_h2a0e4a3_104 conda-forge icu 68.2 h0e60522_0 conda-forge idna 3.4 pyhd8ed1ab_0 conda-forge importlib-resources 6.1.0 pyhd8ed1ab_0 conda-forge importlib_resources 6.1.0 pyhd8ed1ab_0 conda-forge intel-openmp 2023.2.0 h57928b3_50496 conda-forge joblib 1.3.2 pyhd8ed1ab_0 conda-forge jpeg 9e hcfcfb64_3 conda-forge kealib 1.4.15 h8995ca9_0 conda-forge kiwisolver 1.4.5 py39h1f6ef14_1 conda-forge krb5 1.19.3 h1176d77_0 conda-forge lcms2 2.14 h90d422f_0 conda-forge lerc 4.0.0 h63175ca_0 conda-forge libblas 3.9.0 18_win64_mkl conda-forge libbrotlicommon 1.1.0 hcfcfb64_1 conda-forge libbrotlidec 1.1.0 hcfcfb64_1 conda-forge libbrotlienc 1.1.0 hcfcfb64_1 conda-forge libcblas 3.9.0 18_win64_mkl conda-forge libcurl 7.86.0 heaf79c2_1 conda-forge libdeflate 1.14 hcfcfb64_0 conda-forge libexpat 2.5.0 h63175ca_1 conda-forge libffi 3.4.2 h8ffe710_5 conda-forge libgdal 3.3.3 h08edb8f_0 conda-forge libglib 2.78.0 he8f3873_0 conda-forge libhwloc 2.9.1 h51c2c0f_0 conda-forge libiconv 1.17 h8ffe710_0 conda-forge libkml 1.3.0 hd45a9bc_1016 conda-forge liblapack 3.9.0 18_win64_mkl conda-forge libnetcdf 4.8.1 nompi_h1cc8e9d_102 conda-forge libpng 1.6.39 h19919ed_0 conda-forge libpq 13.8 h04fd8be_0 conda-forge librttopo 1.1.0 hb340de5_6 conda-forge libspatialindex 1.9.3 h39d44d4_4 conda-forge libspatialite 5.0.1 h6a90f17_9 conda-forge libsqlite 3.43.0 hcfcfb64_0 conda-forge libssh2 1.10.0 h680486a_3 conda-forge libtiff 4.4.0 hc4f729c_5 conda-forge libwebp-base 1.3.2 hcfcfb64_0 conda-forge libxcb 1.13 hcd874cb_1004 conda-forge libxml2 2.10.4 hc3477c8_0 conda-forge libzip 1.9.2 hfed4ece_1 conda-forge libzlib 1.2.13 hcfcfb64_5 conda-forge lz4-c 1.9.3 h8ffe710_1 conda-forge m2w64-gcc-libgfortran 5.3.0 6 conda-forge m2w64-gcc-libs 5.3.0 7 conda-forge m2w64-gcc-libs-core 5.3.0 7 conda-forge m2w64-gmp 6.1.0 2 conda-forge m2w64-libwinpthread-git 5.0.0.4634.697f757 2 conda-forge mapclassify 2.5.0 pyhd8ed1ab_1 conda-forge matplotlib-base 3.7.1 py39haf65ace_0 conda-forge mkl 2022.1.0 h6a75c08_874 conda-forge msys2-conda-epoch 20160418 1 conda-forge munch 4.0.0 pyhd8ed1ab_0 conda-forge munkres 1.1.4 pyh9f0ad1d_0 conda-forge netcdf4 1.6.0 nompi_py39hf113b1f_100 conda-forge networkx 2.6.2 pyhd8ed1ab_0 conda-forge numpy 1.20.3 py39h6635163_2 conda-forge openjpeg 2.5.0 hc9384bd_1 conda-forge openssl 1.1.1w hcfcfb64_0 conda-forge packaging 23.2 pyhd8ed1ab_0 conda-forge pandas 1.5.3 py39h2ba5b7c_1 conda-forge pcre 8.45 h0e60522_0 conda-forge pcre2 10.40 h17e33f8_0 conda-forge pillow 9.2.0 py39h595c93f_3 conda-forge pip 23.2.1 pyhd8ed1ab_0 conda-forge pixman 0.42.2 h63175ca_0 conda-forge platformdirs 3.11.0 pyhd8ed1ab_0 conda-forge pooch 1.7.0 pyhd8ed1ab_4 conda-forge poppler 21.09.0 h24fffdf_3 conda-forge poppler-data 0.4.12 hd8ed1ab_0 conda-forge postgresql 13.8 h392df6f_0 conda-forge proj 8.1.1 h1cfcee9_2 conda-forge pthread-stubs 0.4 hcd874cb_1001 conda-forge pthreads-win32 2.9.1 hfa6e2cd_3 conda-forge pyparsing 3.1.1 pyhd8ed1ab_0 conda-forge pyproj 3.2.1 py39h39b2389_2 conda-forge pysocks 1.7.1 pyh0701188_6 conda-forge python 3.9.15 h0269646_0_cpython conda-forge python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge python_abi 3.9 4_cp39 conda-forge pytz 2023.3.post1 pyhd8ed1ab_0 conda-forge requests 2.31.0 pyhd8ed1ab_0 conda-forge rtree 1.0.1 py39h09fdee3_3 conda-forge scikit-learn 1.2.2 py39hb6325b5_1 conda-forge scipy 1.10.1 py39hfbf2dce_0 conda-forge setuptools 68.2.2 pyhd8ed1ab_0 conda-forge shapely 1.7.1 py39haadaec5_5 conda-forge six 1.16.0 pyh6c4a22f_0 conda-forge sqlite 3.43.0 hcfcfb64_0 conda-forge tbb 2021.9.0 h91493d7_0 conda-forge threadpoolctl 3.2.0 pyha21a80b_0 conda-forge tiledb 2.3.4 h78dabda_0 conda-forge tk 8.6.13 hcfcfb64_0 conda-forge typing-extensions 4.8.0 hd8ed1ab_0 conda-forge typing_extensions 4.8.0 pyha770c72_0 conda-forge tzdata 2023c h71feb2d_0 conda-forge ucrt 10.0.22621.0 h57928b3_0 conda-forge unicodedata2 15.1.0 py39ha55989b_0 conda-forge uriparser 0.9.7 h1537add_1 conda-forge urllib3 2.0.6 pyhd8ed1ab_0 conda-forge vc 14.3 h64f974e_17 conda-forge vc14_runtime 14.36.32532 hdcecf7f_17 conda-forge vs2015_runtime 14.36.32532 h05e6639_17 conda-forge wheel 0.41.2 pyhd8ed1ab_0 conda-forge win_inet_pton 1.1.0 pyhd8ed1ab_6 conda-forge xarray 2023.1.0 pyhd8ed1ab_0 conda-forge xerces-c 3.2.4 h63175ca_3 conda-forge xorg-libxau 1.0.11 hcd874cb_0 conda-forge xorg-libxdmcp 1.1.3 hcd874cb_0 conda-forge xz 5.2.6 h8d14728_0 conda-forge zipp 3.17.0 pyhd8ed1ab_0 conda-forge zlib 1.2.13 hcfcfb64_5 conda-forge zstd 1.5.5 h12be248_0 conda-forge ```

[djhoese]

Thank you. Are you activating your conda environment before using it?

[djhoese]

Side note: I'm a little surprised you're getting such an old version of pyproj, although that's not the problem here most likely. When you created your environment did you specify any version limits for any of your dependencies?

[DOptimusPrime]

Yes. This is what I needed to run my code so pyproj came as a dependency: networkx== 2.6.2 shapely== 1.7.1 geopandas== 0.9.0 numpy==1.20.3 That is why my version of pyproj is so old.

[djhoese]

Is that "Yes" to the activation question? The various PROJ directory environment variables are meant to be set on environment activation.

[DOptimusPrime]

Yes. I think because I installed an older version version of geopandas it came with an older version of proj. I don't know if I answered your question correctly

[djhoese]

Ok, so at some point you are running conda activate chur2? Ok. There is a small-ish chance that something has been fixed since this older version of pyproj in the conda-forge builds, but it is hard to say. I didn't see anything obvious in the commit history on conda-forge. Let me see if I can reproduce the warnings on my linux box. It could also be that I'm just completely missing something so @snowman2 could probably answer this too.

[DOptimusPrime]

I run the following: conda activate chur2 conda install -c conda-forge networkx== 2.6.2 shapely== 1.7.1 geopandas== 0.9.0 numpy== 1.20.3 fiona netcdf4 That was how I isntalled my packages. Thank you for your help

[djhoese]

If you run python -c "import pyproj" do you see the warning?

Edit: I'm trying to figure out the smallest amount of code to reproduce the warning. I'm not seeing it on linux with these package versions, but it wouldn't surprise me that Windows builds have an issue that linux doesn't.

[DOptimusPrime]

It works fine: (chur2) C:\Users\hdela>python -c "import pyproj"

(chur2) C:\Users\hdela>python Python 3.9.15 | packaged by conda-forge | (main, Nov 22 2022, 08:39:05) [MSC v.1929 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. import pyproj

[djhoese]

Well the warning you were getting is supposed to show up in pyproj/__init__.py when you import it so I'm not sure why you would stop getting it now. You could do:

python -c "import warnings; warnings.simplefilter('error'); import pyproj"

but I wouldn't guess it would be a problem. So perhaps one of the other packages is interfering with pyproj? What order do you import things in your script? Also, it looks like you might be using some multiprocessing, is that right?

[DOptimusPrime]

Yes I am using some multiprocessing. These are the other import statement: import sys

import numpy as np import pandas as pd import csv import pickle import math import os import json import uuid import shutil import threading

from scipy.stats import norm from time import strftime, time import time import cProfile, pstats, io from pstats import SortKey

from osgeo.gdalconst import * from osgeo import osr, ogr, gdal import geopandas as gpd import subprocess import matplotlib.pyplot as plt import matplotlib.cm as cm from mpl_toolkits.axes_grid1.inset_locator import inset_axes

import matplotlib as mpl

from fiona import collection, open as fiona_open from shapely.geometry import mapping, shape, Point, LineString

[DOptimusPrime]

Do you think is it that warning that is making python to look for that folder on my old machine. The last line of the error message where FileNotFoundError: [Errno 2] No such file or directory: b'C:/Users/dhoussou/chur/Chur/Engine/Data/SourceData/precipitation/MeteoSwiss/RdisaggH_ch01r.swisscors_200703010100_200704010000.nc' That's the path on my old machine. I do not think but I am using a new machine so I am confused why that is? Do you think it has something to do with PROJ. I have looked through my code and I don't think I provided any path to my old machine so I am a bit confused.

[djhoese]

Did you copy your old machine's environment to your new machine?

No I don't think that warning has anything to do with your incorrect netcdf path being used besides that both issues could be pointing to the same problem: something got copied or configured incorrectly when switching to the new machine.

[djhoese]

Note: Even with that set of imports in a python file and with:

import warnings
warnings.simplefilter("always")

I don't see a pyproj warning on my linux environment.

[snowman2]

Providing a Minimal Reproducible Example may help to debug the issue: https://stackoverflow.com/help/minimal-reproducible-example