Closed jreiberkyle closed 4 years ago
Based on the PROJ6.3.0 FAQ, this is a common issue.
Updated the Dockerfile to use the latest build of jupyter/minimal-notebook. same failure in using gdalwarp
in crop-temporal notebook.
Trying to reduce this to the minimal code to test but getting surprising results.
rasterio test succeeds, this has been known to fail (https://github.com/mapbox/rasterio/issues/1850)
from rasterio import crs
a = crs.CRS.from_epsg(3005)
a.is_epsg_code
using gdalwarp
in classify-cart-l8-ps actually fails if run as a bash command:
gdalwarp -te 475500.0 4727500.0 500500.0 4752500.0 -ts 8000 8000 \
-r near -overwrite \
data/cart/LC80260302016245LGN00/LC80260302016245LGN00_BQA.TIF \
data/cart/210879_1558814_2016-07-25_0e16/L8/LC80260302016245LGN00_BQA.TIF
ERROR 1: PROJ: pj_obj_create: Open of /opt/conda/share/proj failed
ERROR 1: PROJ: createGeodeticReferenceFrame: Open of /opt/conda/share/proj failed
ERROR 1: PROJ: proj_as_wkt: Open of /opt/conda/share/proj failed
ERROR 1: PROJ: createGeodeticReferenceFrame: Open of /opt/conda/share/proj failed
ERROR 1: PROJ: pj_obj_create: Open of /opt/conda/share/proj failed
ERROR 1: PROJ: proj_as_wkt: Open of /opt/conda/share/proj failed
ERROR 1: PROJ: proj_create_from_wkt: Open of /opt/conda/share/proj failed
ERROR 1: PROJ: proj_create_from_wkt: Open of /opt/conda/share/proj failed
ERROR 1: PROJ: pj_obj_create: Open of /opt/conda/share/proj failed
ERROR 1: PROJ: proj_as_wkt: Open of /opt/conda/share/proj failed
ERROR 1: PROJ: proj_as_wkt: Open of /opt/conda/share/proj failed
ERROR 1: PROJ: proj_create_from_wkt: Open of /opt/conda/share/proj failed
ERROR 1: PROJ: proj_create_from_database: Open of /opt/conda/share/proj failed
Creating output file that is 8000P x 8000L.
Processing data/cart/LC80260302016245LGN00/LC80260302016245LGN00_BQA.TIF [1/1] : 0...10...20...30...40...50...60...70...80...90...100 - done.
It looks like the issue is that the environmental variable PROJ_LIB
is not being set. It appears this is because we are not creating/activating a conda environment, just using the base environment (ref). This even though all of the jupyter-notebook docker images are installing packages into the conda base environment (e.g. datascience notebook Dockerfile). So, next thing to try is creating/activating a conda environment in the Docker image. This is a little tricky, but this article demonstrates how.
I have tried a few things with no luck so far.
This works when I run the docker image in interactive mode using the bash shell, but not when I run the docker image as a notebook:
FROM jupyter/minimal-notebook:7a0c7325e470
ARG conda_env=notebooks
COPY environment.yml .
RUN conda env create --quiet -f environment.yml && \
conda clean --all -f -y
RUN conda init bash
RUN echo "conda activate ${conda_env}" >> ~/.bashrc
WORKDIR work
This actually activates the conda environment in the notebooks, and the necessary environmental variable is set, but the code fails with: ERROR 1: PROJ: pj_obj_create: Open of /opt/conda/envs/notebooks/share/proj failed
FROM jupyter/minimal-notebook:7a0c7325e470
ARG conda_env=notebooks
COPY environment.yml .
RUN conda env create --quiet -f environment.yml && \
conda clean --all -f -y
ENV PATH $CONDA_DIR/envs/${conda_env}/bin:$PATH
ENV CONDA_DEFAULT_ENV ${conda_env}
# this appears to do nothing but was worth a try in case it was a permissions issue
RUN fix-permissions $CONDA_DIR
WORKDIR work
asking for help on the jupyter discourse site: post
The current access issue seems to be specific to notebooks.
Running the below code works when running the docker image in bash:
gdalwarp --debug on -te 475500.0 4727500.0 500500.0 4752500.0 -ts 8000 8000 \
-r near -overwrite data/cart/LC80260302016245LGN00/LC80260302016245LGN00_BQA.TIF \
data/test/bash.TIF
GDAL: GDALOpen(data/cart/LC80260302016245LGN00/LC80260302016245LGN00_BQA.TIF, this=0x55e4c3f53340) succeeds as GTiff.
GDAL: GDALOpen(data/test/bash.TIF, this=0x55e4c3f65be0) succeeds as GTiff.
GDAL: GDALClose(data/test/bash.TIF, this=0x55e4c3f65be0)
GDAL: Using GTiff driver
GDAL: Computing area of interest: -93.9157, 42.0991, -91.0002, 44.2434
Creating output file that is 8000P x 8000L.
GDAL: QuietDelete(data/test/bash.TIF) invoking Delete()
GDAL: GDALOpen(data/test/bash.TIF, this=0x55e4c3f6b300) succeeds as GTiff.
GDAL: GDALDefaultOverviews::OverviewScan()
MDReaderPleiades: Not a Pleiades product
MDReaderPleiades: Not a Pleiades product
GDAL: GDALClose(data/test/bash.TIF, this=0x55e4c3f6b300)
GDAL: GDALDriver::Create(GTiff,data/test/bash.TIF,8000,8000,1,UInt16,(nil))
Processing data/cart/LC80260302016245LGN00/LC80260302016245LGN00_BQA.TIF [1/1] : 0WARP: Copying metadata from first source to destination dataset
MDReaderPleiades: Not a Pleiades product
MDReaderPleiades: Not a Pleiades product
GDAL: GDALDefaultOverviews::OverviewScan()
GDALWARP: Defining SKIP_NOSOURCE=YES
GDAL: GDAL_CACHEMAX = 599 MB
GDAL: GDALWarpKernel()::GWKNearestNoMasksOrDstDensityOnlyShort() Src=1620,4893,834x418 Dst=0,0,8000x4000
...10...20...30...40...50GDAL: GDALWarpKernel()::GWKNearestNoMasksOrDstDensityOnlyShort() Src=1620,5310,834x418 Dst=0,4000,8000x4000
...60...70...80...90...100 - done.
GDAL: Flushing dirty blocks: 0...10...20...30...40...50...60...70...80...90...100 - done.
GDAL: GDALClose(data/test/bash.TIF, this=0x55e4c3f67d40)
GDAL: GDALClose(data/cart/LC80260302016245LGN00/LC80260302016245LGN00_BQA.TIF, this=0x55e4c3f53340)
Running above code gets this failure message when run as a bash command in the notebook:
GDAL: GDALOpen(data/cart/LC80260302016245LGN00/LC80260302016245LGN00_BQA.TIF, this=0x555f71c843f0) succeeds as GTiff.
GDAL: GDALOpen(data/test/notebook.TIF, this=0x555f71c96c90) succeeds as GTiff.
GDAL: GDALClose(data/test/notebook.TIF, this=0x555f71c96c90)
GDAL: Using GTiff driver
PROJ: pj_obj_create: Open of /opt/conda/envs/notebooks/share/proj failed
PROJ: createGeodeticReferenceFrame: Open of /opt/conda/envs/notebooks/share/proj failed
PROJ: proj_as_wkt: Open of /opt/conda/envs/notebooks/share/proj failed
PROJ: createGeodeticReferenceFrame: Open of /opt/conda/envs/notebooks/share/proj failed
PROJ: pj_obj_create: Open of /opt/conda/envs/notebooks/share/proj failed
PROJ: proj_as_wkt: Open of /opt/conda/envs/notebooks/share/proj failed
PROJ: proj_create_from_wkt: Open of /opt/conda/envs/notebooks/share/proj failed
PROJ: proj_create_from_wkt: Open of /opt/conda/envs/notebooks/share/proj failed
PROJ: pj_obj_create: Open of /opt/conda/envs/notebooks/share/proj failed
PROJ: proj_as_wkt: Open of /opt/conda/envs/notebooks/share/proj failed
PROJ: proj_as_wkt: Open of /opt/conda/envs/notebooks/share/proj failed
PROJ: proj_as_wkt: Open of /opt/conda/envs/notebooks/share/proj failed
PROJ: proj_as_wkt: Open of /opt/conda/envs/notebooks/share/proj failed
PROJ: proj_create_from_wkt: Open of /opt/conda/envs/notebooks/share/proj failed
PROJ: proj_as_wkt: Open of /opt/conda/envs/notebooks/share/proj failed
PROJ: proj_as_wkt: Open of /opt/conda/envs/notebooks/share/proj failed
PROJ: proj_as_proj_string: Open of /opt/conda/envs/notebooks/share/proj failed
PROJ: proj_as_proj_string: Open of /opt/conda/envs/notebooks/share/proj failed
ERROR 1: PROJ: proj_create_from_database: Open of /opt/conda/envs/notebooks/share/proj failed
PROJ: proj_as_wkt: Open of /opt/conda/envs/notebooks/share/proj failed
PROJ: proj_as_wkt: Open of /opt/conda/envs/notebooks/share/proj failed
PROJ: proj_create: Open of /opt/conda/envs/notebooks/share/proj failed
PROJ: proj_create: Open of /opt/conda/envs/notebooks/share/proj failed
PROJ: proj_create_operation_factory_context: Open of /opt/conda/envs/notebooks/share/proj failed
PROJ: pj_obj_create: Open of /opt/conda/envs/notebooks/share/proj failed
GDAL: Computing area of interest: -93.9157, 42.0991, -91.0002, 44.2434
Creating output file that is 8000P x 8000L.
GDAL: QuietDelete(data/test/notebook.TIF) invoking Delete()
GDAL: GDALOpen(data/test/notebook.TIF, this=0x555f71c876a0) succeeds as GTiff.
GDAL: GDALDefaultOverviews::OverviewScan()
MDReaderPleiades: Not a Pleiades product
MDReaderPleiades: Not a Pleiades product
GDAL: GDALClose(data/test/notebook.TIF, this=0x555f71c876a0)
GDAL: GDALDriver::Create(GTiff,data/test/notebook.TIF,8000,8000,1,UInt16,(nil))
PROJ: proj_create_from_wkt: Open of /opt/conda/envs/notebooks/share/proj failed
Processing data/cart/LC80260302016245LGN00/LC80260302016245LGN00_BQA.TIF [1/1] : 0WARP: Copying metadata from first source to destination dataset
MDReaderPleiades: Not a Pleiades product
MDReaderPleiades: Not a Pleiades product
GDAL: GDALDefaultOverviews::OverviewScan()
PROJ: proj_as_wkt: Open of /opt/conda/envs/notebooks/share/proj failed
GDALWARP: Defining SKIP_NOSOURCE=YES
GDAL: GDAL_CACHEMAX = 599 MB
GDAL: GDALWarpKernel()::GWKNearestNoMasksOrDstDensityOnlyShort() Src=1620,4893,834x418 Dst=0,0,8000x4000
...10...20...30...40...50GDAL: GDALWarpKernel()::GWKNearestNoMasksOrDstDensityOnlyShort() Src=1620,5310,834x418 Dst=0,4000,8000x4000
...60...70...80...90...100 - done.
GDAL: Flushing dirty blocks: 0PROJ: proj_as_wkt: Open of /opt/conda/envs/notebooks/share/proj failed
PROJ: proj_create_from_wkt: Open of /opt/conda/envs/notebooks/share/proj failed
PROJ: proj_as_wkt: Open of /opt/conda/envs/notebooks/share/proj failed
...10...20...30...40...50...60...70...80...90...100 - done.
GDAL: GDALClose(data/test/notebook.TIF, this=0x555f71ca3280)
GDAL: GDALClose(data/cart/LC80260302016245LGN00/LC80260302016245LGN00_BQA.TIF, this=0x555f71c843f0)
Trying to fix permissions the way they are fixed in the jupyter notebook stacks isn't helping.
Tried:
# 2019-12-02 build
FROM jupyter/minimal-notebook:7a0c7325e470
ARG conda_env=notebooks
# try this to see if additional permissions fix helps notebooks
ADD environment.yml /tmp/environment.yml
RUN conda env create --quiet -f /tmp/environment.yml && \
conda clean --all -f -y && \
fix-permissions $CONDA_DIR && \
fix-permissions /home/$NB_USER
RUN conda init bash
RUN echo "conda activate ${conda_env}" >> ~/.bashrc
ENV PATH $CONDA_DIR/envs/${conda_env}/bin:$PATH
ENV CONDA_DEFAULT_ENV ${conda_env}
WORKDIR work
Running in notebook I get the same access issues as above.
ok, I don't know if this is related to my permissions issue, but I just figured out that creating and activating a new environment using conda env create --quiet -f /tmp/environment.yml
results in me losing all of the conda packages that were installed in the base image because those packages were installed in the base conda environment. I am going to try activating the base conda environment instead of making a new environment. This isn't recommended by conda, but it's what the jupyter notebook docker stacks are doing.
So, just to document before we go in another direction (using the base environment), I don't think the error Open of /opt/conda/envs/notebooks/share/proj failed
was caused by a permissions issue. I ran fix-permissions
on /opt/conda/envs/notebooks/share/proj
and root directories and nothing changed and ls -l
showed that the notebook user and group have access to read and write the proj.db
file inside that folder.
Trying to install gdal
in the root environment gives a different error.
FROM jupyter/minimal-notebook:7a0c7325e470
RUN conda install --quiet -y -c conda-forge gdal
WORKDIR work
Installs gdal version 3.0.2 (same version installed when I install using an environment and it all works)
when I run the container in bash and try gdalinfo
I get
gdalinfo: error while loading shared libraries: libpoppler.so.76: cannot open shared object file: No such file or directory
For some reason, removing the --quiet
flag fixed this. I built the container with the --no-cache
flag so it shouldn't be a docker build issue. I'm going to avoid the --quiet
flag from now on.
So now that gdal is installed properly in the base environment, I get the following when running from shell:
gdalwarp --debug on -te 475500.0 4727500.0 500500.0 4752500.0 -ts 8000 8000 \
-r near -overwrite data/cart/LC80260302016245LGN00/LC80260302016245LGN00_BQA.TIF \
data/test/bash.TIF
...
PROJ: pj_obj_create: Open of /opt/conda/share/proj failed
PROJ: createGeodeticReferenceFrame: Open of /opt/conda/share/proj failed
...
So, it knows where proj.db is (it is in that folder), but it can't access it
Someone else has had this issue and just set the PROJ_LIB and GDAL_DATA environmental variables - https://github.com/conda/conda/issues/9152
Following advice from https://github.com/conda-forge/geopandas-feedstock/issues/63#issuecomment-526145391, I set PROJ_LIB=/opt/conda/share/proj
in the bash shell and get no errors (this is without activating a conda env)
gdalwarp --debug on -te 475500.0 4727500.0 500500.0 4752500.0 -ts 8000 8000 \
-r near -overwrite data/cart/LC80260302016245LGN00/LC80260302016245LGN00_BQA.TIF \
data/test/bash.TIF
GDAL: GDALOpen(data/cart/LC80260302016245LGN00/LC80260302016245LGN00_BQA.TIF, this=0x55a3465f03e0) succeeds as GTiff.
GDAL: GDALOpen(data/test/bash.TIF, this=0x55a346602c80) succeeds as GTiff.
GDAL: GDALClose(data/test/bash.TIF, this=0x55a346602c80)
GDAL: Using GTiff driver
GDAL: Computing area of interest: -93.9157, 42.0991, -91.0002, 44.2434
Creating output file that is 8000P x 8000L.
GDAL: QuietDelete(data/test/bash.TIF) invoking Delete()
GDAL: GDALOpen(data/test/bash.TIF, this=0x55a3466082f0) succeeds as GTiff.
GDAL: GDALDefaultOverviews::OverviewScan()
MDReaderPleiades: Not a Pleiades product
MDReaderPleiades: Not a Pleiades product
GDAL: GDALClose(data/test/bash.TIF, this=0x55a3466082f0)
GDAL: GDALDriver::Create(GTiff,data/test/bash.TIF,8000,8000,1,UInt16,(nil))
Processing data/cart/LC80260302016245LGN00/LC80260302016245LGN00_BQA.TIF [1/1] : 0WARP: Copying metadata from first source to destination dataset
MDReaderPleiades: Not a Pleiades product
MDReaderPleiades: Not a Pleiades product
GDAL: GDALDefaultOverviews::OverviewScan()
GDALWARP: Defining SKIP_NOSOURCE=YES
GDAL: GDAL_CACHEMAX = 599 MB
GDAL: GDALWarpKernel()::GWKNearestNoMasksOrDstDensityOnlyShort() Src=1620,4893,834x418 Dst=0,0,8000x4000
...10...20...30...40...50GDAL: GDALWarpKernel()::GWKNearestNoMasksOrDstDensityOnlyShort() Src=1620,5310,834x418 Dst=0,4000,8000x4000
...60...70...80...90...100 - done.
GDAL: Flushing dirty blocks: 0...10...20...30...40...50...60...70...80...90...100 - done.
GDAL: GDALClose(data/test/bash.TIF, this=0x55a3466082f0)
GDAL: GDALClose(data/cart/LC80260302016245LGN00/LC80260302016245LGN00_BQA.TIF, this=0x55a3465f03e0)
ok, just to benchmark, we can get gdal and proj working again using this Dockerfile:
FROM jupyter/minimal-notebook:7a0c7325e470
RUN conda install -y -c conda-forge gdal
ENV PROJ_LIB=/opt/conda/share/proj
WORKDIR work
Is this good enough? Should we just add the environmental variable and get on with our lives? At this point I am inclined to say yes.
But I will try one more thing: recreating the notebook kernel with the environment activated. Following https://github.com/jupyter/docker-stacks/blob/master/docs/using/recipes.md#add-a-python-3x-environment. But overwriting the original kernel.
Why? Because I have come so far and I have this working in the shell, I just need jupyter to play along.
Okay, it appears that trying to overwrite the original kernel doesn't work. I have tried activating the base environment before installing gdal (and therefore proj) and installing a new kernel that activates the base environment, but the PROJ_LIB
variable still is not set. I am going to give up on using the approach conda suggests and just store the PROJ_LIB
var in the Dockerfile. What a mess.
Dockerfile:
# 2019-12-02 build
FROM jupyter/minimal-notebook:7a0c7325e470
# activate so we can store some vars in here
RUN conda init bash
# new shells activate base
RUN echo "conda activate base" >> ~/.bashrc
SHELL ["/bin/bash", "-c"]
RUN source ~/.bashrc && conda install -y -c conda-forge gdal && \
python -m ipykernel install --user && \
fix-permissions $CONDA_DIR && \
fix-permissions /home/$NB_USER
WORKDIR work
After 2 days of working through gdal errors in my Dockerfile, I found your post! Thank you for documenting this!
In the crop-temporal notebook, in cell 23, gdalwarp vsicurl is called to download a portion of a geotiff. In the current docker image, this process fails with the following message:
The current installed version of gdal is 3.0.1. In the past, the version was pinned to 2.4.0.
This may be related to the switch to gdal 3. (ref and possible solutions: https://github.com/PDAL/PDAL/issues/2544)