sf plot problems with LIST layers

[mdsumner]

• [ ] find a way for this to be automatable, need plot.sf to just plot what's there
• [ ] see if shp different to tab/gdb in error/invalid status (used shp for speed here)

Script to read all statewide layers (binding municipalities as needed).

library(raadfiles)

thelist_files()

## to get statewide sets, find the individual groups first and pick one
grps <- raadfiles:::thelist_groups()
print(grps)
read_all <- function(pattern) {
 files <- thelist_files(format = "shp", pattern = pattern)
 do.call(rbind, lapply(files$fullname,  sf::read_sf))
}

ix <- sample(grps)
print(ix)
for (i in seq_along(ix)) {
    print(ix[i])
    try({x <- read_all(ix[i])
    plot(x[1], border = NA)
})
}

Script output

R version 3.5.1 (2018-07-02) -- "Feather Spray"
Copyright (C) 2018 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> library(raadfiles)
global option 'raadfiles.data.roots' set:
'/rdsi/PRIVATE/raad/data
 /rdsi/PRIVATE/raad/data_local
 /rdsi/PRIVATE/raad/data_staging
 /rdsi/PRIVATE/raad/data_deprecated
 /rdsi/PUBLIC/raad/data'
Uploading raad file cache as at 2019-01-11 04:13:16 (858493 files listed) 

> 
> thelist_files()
[90m# A tibble: 831 x 2[39m
   fullname                                                    root             
   [3m[90m<chr>[39m[23m                                                       [3m[90m<chr>[39m[23m            
[90m 1[39m /rdsi/PUBLIC/raad/data/listdata.thelist.tas.gov.au/opendat… /rdsi/PUBLIC/raa…
[90m 2[39m /rdsi/PUBLIC/raad/data/listdata.thelist.tas.gov.au/opendat… /rdsi/PUBLIC/raa…
[90m 3[39m /rdsi/PUBLIC/raad/data/listdata.thelist.tas.gov.au/opendat… /rdsi/PUBLIC/raa…
[90m 4[39m /rdsi/PUBLIC/raad/data/listdata.thelist.tas.gov.au/opendat… /rdsi/PUBLIC/raa…
[90m 5[39m /rdsi/PUBLIC/raad/data/listdata.thelist.tas.gov.au/opendat… /rdsi/PUBLIC/raa…
[90m 6[39m /rdsi/PUBLIC/raad/data/listdata.thelist.tas.gov.au/opendat… /rdsi/PUBLIC/raa…
[90m 7[39m /rdsi/PUBLIC/raad/data/listdata.thelist.tas.gov.au/opendat… /rdsi/PUBLIC/raa…
[90m 8[39m /rdsi/PUBLIC/raad/data/listdata.thelist.tas.gov.au/opendat… /rdsi/PUBLIC/raa…
[90m 9[39m /rdsi/PUBLIC/raad/data/listdata.thelist.tas.gov.au/opendat… /rdsi/PUBLIC/raa…
[90m10[39m /rdsi/PUBLIC/raad/data/listdata.thelist.tas.gov.au/opendat… /rdsi/PUBLIC/raa…
[90m# … with 821 more rows[39m
> 
> ## to get statewide sets, find the individual groups first and pick one
> grps <- raadfiles:::thelist_groups()
> print(grps)
 [1] "10m_contours"                                        
 [2] "2d_building_polys"                                   
 [3] "5m_contours"                                         
 [4] "acid_sulfate_soil_sites_statewide"                   
 [5] "address_points_statewide"                            
 [6] "aerial_photo_centres_statewide"                      
 [7] "aerial_photo_coverages_statewide"                    
 [8] "authority_land"                                      
 [9] "boundary_segs"                                       
[10] "building_pts"                                        
[11] "coastal_vulnerability_clayey_shores_statewide"       
[12] "coastal_vulnerability_rocky_shores_statewide"        
[13] "coastal_vulnerability_soft_shores_statewide"         
[14] "coastal_vulnerability_unclassified_statewide"        
[15] "coastline_statewide"                                 
[16] "community_facilities"                                
[17] "correlated_reconnaissance_soils_soil_types_statewide"
[18] "crown_leases_statewide"                              
[19] "crown_licences_statewide"                            
[20] "dhhs_ambulance_station_locations_statewide"          
[21] "easements"                                           
[22] "electoral_divs_statewide"                            
[23] "hydarea"                                             
[24] "hydline"                                             
[25] "hydpnt"                                              
[26] "irrigationia_dams_statewide"                         
[27] "irrigationia_hydropower_statewide"                   
[28] "irrigationia_irrigation_districts_statewide"         
[29] "irrigationia_pipeline_statewide"                     
[30] "irrigationia_pumpstation_statewide"                  
[31] "irrigationia_trade_zone_statewide"                   
[32] "irrigationia_transmission_line_statewide"            
[33] "land capability_statewide"                           
[34] "land_districts_statewide"                            
[35] "land_tenure_statewide"                               
[36] "land_use_2002_BRS_statewide"                         
[37] "land_use_2009_2010_BRS"                              
[38] "land_use_2009_2010_BRS_statewide"                    
[39] "land_use_2013_BRS"                                   
[40] "land_use_2013_BRS_statewide"                         
[41] "land_use_2015_BRS"                                   
[42] "land_use_2015_BRS_statewide"                         
[43] "legislative_council_divs_statewide"                  
[44] "liquor_licences_statewide"                           
[45] "listian_reserve_estate_statewide"                    
[46] "local_govt_areas_statewide"                          
[47] "local_govt_reserves_statewide"                       
[48] "locality_postcode"                                   
[49] "marine_farming_zones_statewide"                      
[50] "marine_leases_statewide"                             
[51] "marine_nature_reserves_statewide"                    
[52] "named_feature_extent_statewide"                      
[53] "nomenclature_csv_statewide"                          
[54] "nomenclature_statewide"                              
[55] "parcels"                                             
[56] "parish_town_bdys_statewide"                          
[57] "private_reserves_statewide"                          
[58] "private_timber_reserves_statewide"                   
[59] "public_land_class"                                   
[60] "ramsar_wetlands_statewide"                           
[61] "reconnaissance_soils_map_extents_100k_statewide"     
[62] "scalefish_blocks_statewide"                          
[63] "soil_boundaries_statewide"                           
[64] "soil_sample_sites_statewide"                         
[65] "spot_heights_statewide"                              
[66] "survey_control_statewide"                            
[67] "tasveg_20"                                           
[68] "tasveg_20_statewide"                                 
[69] "tasveg_30"                                           
[70] "tasveg_30_groups"                                    
[71] "tasveg_30_groups_statewide"                          
[72] "tasveg_30_statewide"                                 
[73] "threatened_native_vegetation_communities_statewide"  
[74] "transport_nodes"                                     
[75] "transport_segments"                                  
[76] "water_districts_statewide"                           
[77] "water_management_area_statewide"                     
> read_all <- function(pattern) {
+  files <- thelist_files(format = "shp", pattern = pattern)
+  do.call(rbind, lapply(files$fullname,  sf::read_sf))
+ }
> 
> ix <- sample(grps)
> print(ix)
 [1] "nomenclature_statewide"                              
 [2] "land_use_2013_BRS"                                   
 [3] "building_pts"                                        
 [4] "locality_postcode"                                   
 [5] "private_timber_reserves_statewide"                   
 [6] "coastline_statewide"                                 
 [7] "named_feature_extent_statewide"                      
 [8] "irrigationia_pumpstation_statewide"                  
 [9] "2d_building_polys"                                   
[10] "transport_nodes"                                     
[11] "marine_nature_reserves_statewide"                    
[12] "aerial_photo_coverages_statewide"                    
[13] "spot_heights_statewide"                              
[14] "land_districts_statewide"                            
[15] "public_land_class"                                   
[16] "ramsar_wetlands_statewide"                           
[17] "liquor_licences_statewide"                           
[18] "easements"                                           
[19] "water_districts_statewide"                           
[20] "hydarea"                                             
[21] "dhhs_ambulance_station_locations_statewide"          
[22] "marine_leases_statewide"                             
[23] "local_govt_reserves_statewide"                       
[24] "address_points_statewide"                            
[25] "irrigationia_hydropower_statewide"                   
[26] "crown_licences_statewide"                            
[27] "coastal_vulnerability_unclassified_statewide"        
[28] "irrigationia_transmission_line_statewide"            
[29] "tasveg_30"                                           
[30] "acid_sulfate_soil_sites_statewide"                   
[31] "tasveg_20_statewide"                                 
[32] "irrigationia_dams_statewide"                         
[33] "land_use_2015_BRS_statewide"                         
[34] "nomenclature_csv_statewide"                          
[35] "local_govt_areas_statewide"                          
[36] "land_use_2009_2010_BRS"                              
[37] "land_use_2015_BRS"                                   
[38] "authority_land"                                      
[39] "soil_sample_sites_statewide"                         
[40] "legislative_council_divs_statewide"                  
[41] "crown_leases_statewide"                              
[42] "land_use_2002_BRS_statewide"                         
[43] "boundary_segs"                                       
[44] "coastal_vulnerability_soft_shores_statewide"         
[45] "electoral_divs_statewide"                            
[46] "water_management_area_statewide"                     
[47] "soil_boundaries_statewide"                           
[48] "coastal_vulnerability_clayey_shores_statewide"       
[49] "5m_contours"                                         
[50] "transport_segments"                                  
[51] "land_use_2013_BRS_statewide"                         
[52] "irrigationia_irrigation_districts_statewide"         
[53] "irrigationia_pipeline_statewide"                     
[54] "irrigationia_trade_zone_statewide"                   
[55] "threatened_native_vegetation_communities_statewide"  
[56] "tasveg_30_groups_statewide"                          
[57] "parish_town_bdys_statewide"                          
[58] "marine_farming_zones_statewide"                      
[59] "reconnaissance_soils_map_extents_100k_statewide"     
[60] "land capability_statewide"                           
[61] "hydline"                                             
[62] "correlated_reconnaissance_soils_soil_types_statewide"
[63] "land_use_2009_2010_BRS_statewide"                    
[64] "parcels"                                             
[65] "listian_reserve_estate_statewide"                    
[66] "hydpnt"                                              
[67] "community_facilities"                                
[68] "survey_control_statewide"                            
[69] "coastal_vulnerability_rocky_shores_statewide"        
[70] "scalefish_blocks_statewide"                          
[71] "tasveg_30_statewide"                                 
[72] "aerial_photo_centres_statewide"                      
[73] "land_tenure_statewide"                               
[74] "private_reserves_statewide"                          
[75] "tasveg_30_groups"                                    
[76] "tasveg_20"                                           
[77] "10m_contours"                                        
> for (i in seq_along(ix)) {
+   print(ix[i])
+   try({x <- read_all(ix[i])
+   plot(x[1], border = NA)
+ })
+ }
[1] "nomenclature_statewide"
[1] "land_use_2013_BRS"
Error in CPL_geos_is_empty(st_geometry(x)) : 
  Evaluation error: IllegalArgumentException: Invalid number of points in LinearRing found 3 - must be 0 or >= 4.
In addition: Warning messages:
1: In title(...) : "border" is not a graphical parameter
2: In CPL_read_ogr(dsn, layer, query, as.character(options), quiet,  :
  GDAL Message 1: organizePolygons() received an unexpected geometry.  Either a polygon with interior rings, or a polygon with less than 4 points, or a non-Polygon geometry.  Return arguments as a collection.
3: In CPL_read_ogr(dsn, layer, query, as.character(options), quiet,  :
  GDAL Message 1: organizePolygons() received an unexpected geometry.  Either a polygon with interior rings, or a polygon with less than 4 points, or a non-Polygon geometry.  Return arguments as a collection.
[1] "building_pts"
[1] "locality_postcode"
[1] "private_timber_reserves_statewide"
[1] "coastline_statewide"
[1] "named_feature_extent_statewide"
[1] "irrigationia_pumpstation_statewide"
Error in thelist_files(format = "shp", pattern = pattern) : 
  no files found
In addition: Warning messages:
1: In title(...) : "border" is not a graphical parameter
2: In title(...) : "border" is not a graphical parameter
[1] "2d_building_polys"
[1] "transport_nodes"
[1] "marine_nature_reserves_statewide"
[1] "aerial_photo_coverages_statewide"
[1] "spot_heights_statewide"
[1] "land_districts_statewide"
[1] "public_land_class"
[1] "ramsar_wetlands_statewide"
[1] "liquor_licences_statewide"
[1] "easements"
[1] "water_districts_statewide"
[1] "hydarea"
[1] "dhhs_ambulance_station_locations_statewide"
[1] "marine_leases_statewide"
[1] "local_govt_reserves_statewide"
[1] "address_points_statewide"
[1] "irrigationia_hydropower_statewide"
Error in thelist_files(format = "shp", pattern = pattern) : 
  no files found
In addition: Warning messages:
1: In title(...) : "border" is not a graphical parameter
2: In title(...) : "border" is not a graphical parameter
3: In title(...) : "border" is not a graphical parameter
4: In title(...) : "border" is not a graphical parameter
5: In title(...) : "border" is not a graphical parameter
[1] "crown_licences_statewide"
[1] "coastal_vulnerability_unclassified_statewide"
[1] "irrigationia_transmission_line_statewide"
Error in thelist_files(format = "shp", pattern = pattern) : 
  no files found
In addition: Warning message:
In title(...) : "border" is not a graphical parameter
[1] "tasveg_30"
Error in rbind.data.frame(...) : 
  numbers of columns of arguments do not match
[1] "acid_sulfate_soil_sites_statewide"
[1] "tasveg_20_statewide"
[1] "irrigationia_dams_statewide"
Error in thelist_files(format = "shp", pattern = pattern) : 
  no files found
In addition: Warning message:
In title(...) : "border" is not a graphical parameter
[1] "land_use_2015_BRS_statewide"
Error in CPL_geos_is_empty(st_geometry(x)) : 
  Evaluation error: IllegalArgumentException: Invalid number of points in LinearRing found 3 - must be 0 or >= 4.
[1] "nomenclature_csv_statewide"
[1] "local_govt_areas_statewide"
[1] "land_use_2009_2010_BRS"
[1] "land_use_2015_BRS"
Error in rbind.data.frame(...) : 
  numbers of columns of arguments do not match
In addition: Warning message:
In title(...) : "border" is not a graphical parameter
[1] "authority_land"
[1] "soil_sample_sites_statewide"
[1] "legislative_council_divs_statewide"
[1] "crown_leases_statewide"
[1] "land_use_2002_BRS_statewide"
[1] "boundary_segs"
Raad file cache is up to date as at 2019-01-11 04:13:16 (858493 files listed) 

[mdsumner]

Issue 1.

The problem here is that the validation checks return the bad geometries in a sfc_GEOMETRY, and plot.sfc_GEOMETRY doesn't have capability to ignore empties (I think).

Example 1

[1] "land_use_2013_BRS"
Error in CPL_geos_is_empty(st_geometry(x)) : 
  Evaluation error: IllegalArgumentException: Invalid number of points in LinearRing found 3 - must be 0 or >= 4.
In addition: Warning messages:
1: In title(...) : "border" is not a graphical parameter
2: In CPL_read_ogr(dsn, layer, query, as.character(options), quiet,  :
  GDAL Message 1: organizePolygons() received an unexpected geometry.  Either a polygon with interior rings, or a polygon with less than 4 points, or a non-Polygon geometry.  Return arguments as a collection.
3: In CPL_read_ogr(dsn, layer, query, as.character(options), quiet,  :
  GDAL Message 1: organizePolygons() received an unexpected geometry.  Either a polygon with interior rings, or a polygon with less than 4 points, or a non-Polygon geometry.  Return arguments as a collection.

Example 2

[1] "land_use_2015_BRS_statewide"
Error in CPL_geos_is_empty(st_geometry(x)) : 
  Evaluation error: IllegalArgumentException: Invalid number of points in LinearRing found 3 - must be 0 or >= 4.

[mdsumner]

Issue 2

(Very likely) local logic to separate layers is not sufficient here, also for "land_use_2015_BRS"

[1] "tasveg_30"
Error in rbind.data.frame(...) : 
  numbers of columns of arguments do not match

Issue 3

Some patterns are finding no files. ??

[mdsumner]

Something like this will do it, after "border" handling and before "st_is_empty" in plot.sfc_MULTIPOLYGON:

    bad = st_is_valid(x)

    if (any(is.na(bad))) {
        x = x[!is.na(bad)]
        if (length(x) < 1) {
            warning("no valid geometries to plot in sfc_MULTIPOLYGON")
            return(invisible(NULL))
        }
    }

Need to test other functions, and compare for the other cases above where they are GEOMETRY type.

[mdsumner]

BUT, it is just the shitty shapefiles - use the MapInfo TAB and all is well.

[mdsumner]

use basf to solve the annoying sf plot