filters.python and readers.numpy not available on Windows Conda install

[pjhartzell]

Describe the bug filters.python and readers.numpy are not available when installing python-pdal via Conda into a fresh Conda environment on a Windows machine.

conda create -n python-pdal-test conda activate python-pdal-test conda install -c conda-forge python-pdal pdal --drivers

============================ ===================================================
Name                         Description
============================ ===================================================
filters.approximatecoplanar  Estimates the planarity of a neighborhood of
                             points using eigenvalues.
filters.assign               Assign values for a dimension range to a specified
                             value.
filters.chipper              Organize points into spatially contiguous,
                             squarish, and non-overlapping chips.
filters.cluster              Extract and label clusters using Euclidean
                             distance.
filters.colorinterp          Assigns RGB colors based on a dimension and a ramp
filters.colorization         Fetch and assign RGB color information from a
                             GDAL-readable datasource.
filters.covariancefeatures   Filter that calculates local features based on the
                             covariance matrix of a point's neighborhood.
filters.crop                 Filter points inside or outside a bounding box or
                             a polygon
filters.csf                  Cloth Simulation Filter (Zhang et al., 2016)
filters.dbscan               DBSCAN Clustering.
filters.decimation           Rank decimation filter. Keep every Nth point
filters.delaunay             Perform Delaunay triangulation of a pointcloud
filters.dem                  Filter points about an elevation surface
filters.divider              Divide points into approximately equal sized
                             groups based on a simple scheme
filters.eigenvalues          Returns the eigenvalues for a given point, based
                             on its k-nearest neighbors.
filters.elm                  Marks low points as noise.
filters.estimaterank         Computes the rank of a neighborhood of points.
filters.ferry                Copy data from one dimension to another.
filters.fps                  Farthest point sampling filter
filters.greedyprojection     Greedy Triangulation filter
filters.groupby              Split data categorically by dimension.
filters.hag                  Computes height above ground using
                             ground-classified returns.
filters.hag_delaunay         Computes height above ground using delaunay
                             interpolation of ground returns.
filters.hag_dem              Computes height above ground using a DEM raster.
filters.hag_nn               Computes height above ground using
                             nearest-neighbor ground-classified returns.
filters.head                 Return N points from beginning of the point cloud.
filters.hexbin               Tessellate the point's X/Y domain and determine
                             point density and/or point boundary.
filters.icp                  Iterative Closest Point (ICP) registration.
filters.info                 Gather basic info about points.
filters.iqr                  Interquartile Range Filter
filters.locate               Return a single point with min/max value in the
                             named dimension.
filters.lof                  LOF Filter
filters.mad                  Median Absolute Deviation Filter
filters.merge                Merge data from two different readers into a
                             single stream.
filters.miniball             Miniball (Kutz et al., 2003)
filters.mongo                Pass only points that pass a logic filter.
filters.mortonorder          Morton or z-order sorting of points. See
                             http://en.wikipedia.org/wiki/Z-order_curve for
                             more detail.
filters.neighborclassifier   Re-assign some point attributes based KNN voting
filters.nndistance           NN-Distance Filter
filters.normal               Normal Filter
filters.outlier              Outlier removal
filters.overlay              Assign values to a dimension based on the extent
                             of an OGR-readable data source or an OGR SQL
                             query.
filters.planefit             Plane Fit (Kutz et al., 2003)
filters.pmf                  Progressive morphological filter
filters.poisson              Poisson Surface Reconstruction Filter
filters.projpipeline         Transform coordinates using Proj pipeline string,
                             WKT2 coordinate operations or URN definition
filters.radialdensity        RadialDensity Filter
filters.randomize            Randomize points in a view.
filters.range                Pass only points given a dimension/range.
filters.reciprocity          Returns the percentage of neighbors that do NOT
                             have the query point as a neighbor
filters.reprojection         Reproject data using GDAL from one coordinate
                             system to another.
filters.returns              Split data by return order
filters.sample               Subsampling filter
filters.separatescanline     Split data by scan line.
filters.shell                Execute a shell operation inline with PDAL
                             pipeline steps
filters.sort                 Sort data based on a given dimension.
filters.splitter             Split data based on a X/Y box length.
filters.stats                Compute statistics about each dimension (mean,
                             min, max, etc.)
filters.streamcallback       Provide a hook for a simple point-by-point
filters.tail                 Return N points from end of the point cloud.
                             matrix
filters.voxelcentroidnearestneighbor Voxel Centroid Nearest Neighbor Filter
filters.voxeldownsize        First Entry Voxel Filter
                             a simple DoD and research format that is used by
readers.faux                 Faux Reader
readers.gdal                 Read GDAL rasters as point clouds.
readers.ilvis2               ILVIS2 Reader
readers.las                  ASPRS LAS 1.0 - 1.4 read support. LASzip support
                             is also enabled through this driver if LASzip was
                             found during compilation.
readers.memoryview           Memory View Reader
readers.optech               Optech reader support.
readers.pcd                  Read data in the Point Cloud Library (PCL) format.
readers.ply                  Read ply files.
readers.pts                  Pts Reader
readers.qfit                 QFIT Reader
readers.sbet                 SBET Reader
readers.terrasolid           TerraSolid Reader
readers.text                 Text Reader
readers.tindex               TileIndex Reader
writers.bpf                  "Binary Point Format" (BPF) writer support. BPF is
                             a simple DoD and research format that is used by
                             some sensor and processing chains.
writers.ept_addon            EPT Writer
writers.gdal                 Write a point cloud as a GDAL raster.
writers.gltf                 Gltf Writer
writers.las                  ASPRS LAS 1.0 - 1.4 writer. LASzip support is also
                             available if enabled at compile-time. Note that
                             LAZ does not provide LAS 1.4 support at this time.
writers.null                 Null writer. Provides a sink for points in a
                             pipeline. It's the same as sending pipeline output
                             to /dev/null.
writers.ogr                  Write a point cloud as a set of OGR
                             points/multipoints
writers.pcd                  Write data in the Point Cloud Library (PCL)
                             format.
writers.ply                  ply writer
writers.sbet                 SBET Writer
writers.text                 Text Writer
============================ ===================================================

Expected behavior filters.python and readers.numpy would be available without having to manually set the PDAL_DRIVER_PATH environment variable. In my case, conda env config vars set PDAL_DRIVER_PATH="C:\Users\prest\Miniconda3\envs\python-pdal-test\Lib\site-packages\bin" fixed the issue. But this is not user friendly.

System/installation information: pdal --version

--------------------------------------------------------------------------------
pdal 2.1.0 (git-version: Release)
--------------------------------------------------------------------------------

$Properties = 'Caption', 'CSName', 'Version', 'BuildType', 'OSArchitecture' Get-CimInstance Win32_OperatingSystem | Select-Object $Properties | Format-Table -AutoSize

Caption                  CSName          Version    BuildType           OSArchitecture
-------                  ------          -------    ---------           --------------
Microsoft Windows 10 Pro DESKTOP-982BCV2 10.0.18362 Multiprocessor Free 64-bit

[hobu]

hmm, I wonder if the install paths for the DLLs is different per-python version.

[pjhartzell]

Not 100% sure what you're getting at, but the install is using Python 3.8.3. I can experiment with different Python versions if you think there is something there.

[hobu]

https://github.com/hobu/python-pdal-feedstock/blob/master/recipe/scripts/activate.bat is how this gets set. The issue is %PREFIX%\Lib\site-packages\bin might no longer be the place where pip is installing the plugin DLLs

[pjhartzell]

The plugins are installed at the location specified in the bat file (%CONDA_PREFIX%\lib\site-packages\bin) on my machine.

(python-pdal-test3) PS C:\dev\python\lidar-error-prop\error-prop> $env:CONDA_PREFIX
C:\Users\prest\Miniconda3\envs\python-pdal-test3
(python-pdal-test3) PS C:\dev\python\lidar-error-prop\error-prop> $env:PDAL_DRIVER_PATH
C:\Users\prest\Miniconda3\envs\python-pdal-test\lib\site-packages\bin

But PDAL_DRIVER_PATH is not being set during the install. After unsetting my manually set PDAL_DRIVER_PATH and then installing python-pdal into a fresh Conda environment, the PDAL_DRIVER_PATH environment variable does not exist (was not created):

(python-pdal-test3) PS C:\dev\python\lidar-error-prop\error-prop> $env:PDAL_DRIVER_PATH
(python-pdal-test3) PS C:\dev\python\lidar-error-prop\error-prop> conda env config vars list

[pjhartzell]

(python-pdal-test3) PS C:\dev\python\lidar-error-prop\error-prop> cd "C:\Users\prest\Miniconda3\envs\python-pdal-test\lib\site-packages\bin"
(python-pdal-test3) PS C:\Users\prest\Miniconda3\envs\python-pdal-test\lib\site-packages\bin> ls

    Directory: C:\Users\prest\Miniconda3\envs\python-pdal-test\lib\site-packages\bin

Mode                LastWriteTime         Length Name
----                -------------         ------ ----
-a----        4/17/2020   6:51 AM         388608 libpdal_plugin_filter_python.dll
-a----        4/17/2020   6:51 AM         321024 libpdal_plugin_reader_numpy.dll

(python-pdal-test3) PS C:\Users\prest\Miniconda3\envs\python-pdal-test\lib\site-packages\bin> 

[pjhartzell]

Per our brief discussion on jitsi, if I use the standard command prompt rather than powershell, it works fine. Both filters.python and readers.numpy are available in the pdal --drivers listing and PDAL_DRIVER_PATH is set correctly.

(lidar-error) C:\dev\python\lidar-error-prop\error-prop>echo %CONDA_PREFIX%
C:\Users\prest\Miniconda3\envs\lidar-error

(lidar-error) C:\dev\python\lidar-error-prop\error-prop>echo %PDAL_DRIVER_PATH% 
C:\Users\prest\Miniconda3\envs\lidar-error\Library\bin;C:\Users\prest\Miniconda3\envs\lidar-error\lib\site-packages\bin

[hobu]

Per our brief discussion on jitsi, if I use the standard command prompt rather than powershell, it works fine.

Ok, we need to add activate.ps and deactivate.ps scripts to the pdal-python-feedstock.

[hobu]

Addressed in https://github.com/conda-forge/pdal-feedstock/pull/112 Reopen there if there are any future issues with it once the packages are pushed out.