Check if rgdal supports streaming data directly from data source

[jlehtoma]

rwfs already has class WFSStreamingClient() which reads the data directly from the source. However, there seems to a lot of issues with geometry types and there may be no way of doing ogr2ogr -explodecollections with readOGR(). Will need to look in more detail to this.

[jlehtoma]

Streaming both layers and features directly from data source seems to work, using following set up:

rgdal: version: 1.1-10, (SVN revision 622)
 Geospatial Data Abstraction Library extensions to R successfully loaded
 Loaded GDAL runtime: GDAL 2.1.1, released 2016/07/07
 Path to GDAL shared files: /usr/share/gdal
 Loaded PROJ.4 runtime: Rel. 4.9.2, 08 September 2015, [PJ_VERSION: 492]
 Path to PROJ.4 shared files: (autodetected)
 Linking to sp version: 1.2-3 

Briefly tested with WFS versions 1.0.0, 1.1.0 and 2.0.0. Needs more testing, though.

[jlehtoma]

From version 0.2.0 onward, WFSStreamingRequest and WFSStreamingClient can work directly with the data source without the need to download the intermediate file and using ogr2ogr on it.

This worked all along. Previously there were issues with WFS 2.0.0 data sources.

[jlehtoma]

Perhaps I should've read the README more closely...

For streaming, the WFSStreamingRequest abstract class is required to be inherited to a subclass that implements the abstract method getDataSource(), which provides a data access reference.

I now implemented getDataSource() in the WFSStreamingRequest abstract class directly. Shouldn't break anything, but needs to be reverted. I so hate OO :(

[jlehtoma]

On a more positive note, streaming from a WFS 2.0.0 data source (the FMI) seems to work, at least for the simple cases.

[torsti]

So what's the status in terms of things still needing to be done? I'm a bit confused. (Also, I'm sure it's just OO in R, that is the problem.)

[jlehtoma]

So what's the status in terms of things still needing to be done? I'm a bit confused. (Also, I'm sure it's just OO in R, that is the problem.)

Sorry for being unclear! It's been a bit of an uphill struggle to understand the class structures in rwfs, and reading the documentation would've helped. As far as I can tell, streaming directly from the data source should work, also with WFS 2.0.0 (which is what FMI has). At least when rgdal is using GDAL >= 2.0.0. I've only tested with GDAL 2.1.1, so I'm not 100% sure. On Windows, the latest rgdal (1.1-10) binary comes with GDAL 2.0.1, so I'm hopeful that WFS 2.0.0 is supported out of the box.

Here's what is needed:

1. In rwfs: Nothing except perhaps improving caching. I'd be very happy if you have any comments on that.
2. In fmi: To create a FMIWFSStreamingRequest class that inherits from rwfs::WFSStreamingRequest and implements the abstract method getDataSource(). However, this needs a bit of refactoring in terms of the existing classes, so I'll set up another issue in fmi. Streaming from the FMI API also needs to be tested.

[jlehtoma]

Oh boy, test are still failing on Windows machines the following error in WFSClient:

 > rgdal::ogrListLayers(dsn = dataSource)
Error in rgdal::ogrListLayers(dsn = dataSource) : Cannot open data source

Here's the rgdal-setup on Windows 10 (64 bit) and R 3.3.2 :

> library(rgdal)
rgdal: version: 1.2-5, (SVN revision 648)
 Geospatial Data Abstraction Library extensions to R successfully loaded
 Loaded GDAL runtime: GDAL 2.0.1, released 2015/09/15
 Path to GDAL shared files: C:/Users/jlehtoma/Documents/R/win-library/3.3/rgdal/gdal
 Loaded PROJ.4 runtime: Rel. 4.9.2, 08 September 2015, [PJ_VERSION: 492]
 Path to PROJ.4 shared files: C:/Users/jlehtoma/Documents/R/win-library/3.3/rgdal/proj
 Linking to sp version: 1.2-4 

Little more detective work lead me to this post on SO. It turns out that the Windows binary for rgdal does not seem to come with the WFS driver enabled. This can be tested be running:

"WFS" %in% ogrDrivers()$name

Opening this issue as it seems unlikely that data can be streamed without ogr2ogr unless the binay comes with the WFS driver.

[jlehtoma]

Missing WFS driver also an issue on the following setup:

> sessionInfo()
R version 3.3.2 (2016-10-31)
Platform: i386-w64-mingw32/i386 (32-bit)
Running under: Windows 7 (build 7601) Service Pack 1

locale:
[1] LC_COLLATE=Finnish_Finland.1252  LC_CTYPE=Finnish_Finland.1252    LC_MONETARY=Finnish_Finland.1252
[4] LC_NUMERIC=C                     LC_TIME=Finnish_Finland.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] rgdal_1.2-5 sp_1.2-4    rwfs_0.1.15 R6_2.2.0   

loaded via a namespace (and not attached):
[1] tools_3.3.2     grid_3.3.2      digest_0.6.11   lattice_0.20-34

>library(rgdal)
Loading required package: sp
rgdal: version: 1.2-5, (SVN revision 648)
 Geospatial Data Abstraction Library extensions to R successfully loaded
 Loaded GDAL runtime: GDAL 2.0.1, released 2015/09/15
 Path to GDAL shared files: C:/Program Files/R/R-3.3.2/library/rgdal/gdal
 Loaded PROJ.4 runtime: Rel. 4.9.2, 08 September 2015, [PJ_VERSION: 492]
 Path to PROJ.4 shared files: C:/Program Files/R/R-3.3.2/library/rgdal/proj
 Linking to sp version: 1.2-4 

[jlehtoma]

As reported in fmi issue #23, on OS X (Yosemite) rgdal (@2.1.2_1+expat) iinstalled via MacPorts does have WFS drivers enabled:

> library(rgdal)
Loading required package: sp
rgdal: version: 1.2-5, (SVN revision 648)
 Geospatial Data Abstraction Library extensions to R successfully loaded
 Loaded GDAL runtime: GDAL 2.1.2, released 2016/10/24
 Path to GDAL shared files: 
 Loaded PROJ.4 runtime: Rel. 4.9.1, 04 March 2015, [PJ_VERSION: 491]
 Path to PROJ.4 shared files: (autodetected)
WARNING: no proj_defs.dat in PROJ.4 shared files
 Linking to sp version: 1.2-3 

>ogrDrivers()$name

 [1] "AeronavFAA"     "AmigoCloud"     "ARCGEN"         "AVCBin"        
 [5] "AVCE00"         "BNA"            "Carto"          "Cloudant"      
 [9] "CouchDB"        "CSV"            "CSW"            "DGN"           
[13] "DXF"            "EDIGEO"         "ElasticSearch"  "ESRI Shapefile"
[17] "Geoconcept"     "GeoJSON"        "GeoRSS"         "GFT"           
[21] "GML"            "GPKG"           "GPSBabel"       "GPSTrackMaker" 
[25] "GPX"            "HTF"            "HTTP"           "Idrisi"        
[29] "JML"            "KML"            "MapInfo File"   "Memory"        
[33] "netCDF"         "ODS"            "OGR_GMT"        "OGR_PDS"       
[37] "OGR_SDTS"       "OGR_VRT"        "OpenAir"        "OpenFileGDB"   
[41] "OSM"            "PCIDSK"         "PDF"            "PGDUMP"        
[45] "PLSCENES"       "REC"            "S57"            "SEGUKOOA"      
[49] "SEGY"           "Selafin"        "SQLite"         "SUA"           
[53] "SVG"            "SXF"            "TIGER"          "UK .NTF"       
[57] "VDV"            "VFK"            "WAsP"           "WFS"           
[61] "XLSX"           "XPlane"     

>sessionInfo()
R version 3.3.2 (2016-10-31)
Platform: x86_64-apple-darwin13.4.0 (64-bit)
Running under: OS X Yosemite 10.10.5

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] rgdal_1.2-5 sp_1.2-4   

loaded via a namespace (and not attached):
 [1] Rcpp_0.12.9          lattice_0.20-34      codetools_0.2-15     listenv_0.6.0        future_1.2.0         assertthat_0.1       digest_0.6.11        grid_3.3.2           plyr_1.8.4          
[10] gtable_0.2.0         stats4_3.3.2         StanHeaders_2.14.0-1 scales_0.4.1         ggplot2_2.2.1        lazyeval_0.2.0       tools_3.3.2          munsell_0.4.3        rstan_2.14.1        
[19] parallel_3.3.2       inline_0.3.14        colorspace_1.3-2     globals_0.8.0        tibble_1.2    

[jlehtoma]

Issue #14 opened specifically for getting the WFS driver to Windows rgdal binary, closing this issue.