Jobs in multiple batches aren't processed correctly

aaronkrusniak commented 4 months ago

Did some more stress testing today and came across this bug— it appears that jobs exceeding the max batch size for a geocoder will fail in one of two ways:

If the job size exceeds max batch size, the geocoder will return Error in data.frame(..., check.names = FALSE) : arguments imply differing number of rows: {max batch size}, {number of rows provided}
If the job exceeds max batch size BUT is an even multiple of that batch size, the job will run but will return a data frame of (empty) duplicates, the size of which will be n * the original number of input rows, where n = input rows / max batch size. All the data will be either zero or null. E.g. if max batch size = 1000 and you provide a job with 4000 records, you will receive a data frame of 16000 empty records.

Here's an illustration for this one:

library(arcgis)
library(arcgisbinding)
library(arcgisgeocode)
library(tidyverse)

arc.check_portal()
set_arc_token(auth_binding())

# Some dummy data:
music_venues <- tribble(
  ~Name,              ~Address,
  "Aragon Ballroom",  "1106 W. Lawrence Ave.",
  "House of Blues",   "329 N. Dearborn St.",
  "Bottom Lounge",    "1375 W. Lake St.",
  "The Vic",          "3145 N. Sheffield Ave.",
  "Park West",        "322 W. Armitage Ave.",
  "Thalia Hall",      "1807 S. Allport St.",
  "Lincoln Hall",     "2424 N. Lincoln Ave.",
  "Schubas Tavern",   "3159 N. Southport Ave."
)

# Set up to use our geocoder:
service <- "https://maps.chicago.gov/arcgis/rest/services/Chicago_Addresses/GeocodeServer"
chicago_geocoder <- geocode_server(service)

# Check max batch size:
message(paste0("Max batch size is: ",
               chicago_geocoder[["locatorProperties"]][["MaxBatchSize"]]))

#> Max batch size is: 1000

# Create datasets of varying sizes...

# Just under max batch size (992 rows):
short <- music_venues %>% slice(rep(1:n(), each = 124))

# Exactly at max batch size (1000 rows):
exact <- music_venues %>% slice(rep(1:n(), each = 125))

# Exceeding max batch size (1008 rows):
long <- music_venues %>% slice(rep(1:n(), each = 126))

# Way over max batch size, but an even multiple of 1000 (8000 rows):
mega <- music_venues %>% slice(rep(1:n(), each = 1000))

# Try to geocode each size:
#####-----------------------------------------#####

results.short <- geocode_addresses(single_line = short$Address,
                                   geocoder = chicago_geocoder)
results.short
#> Simple feature collection with 992 features and 61 fields...

#####-----------------------------------------#####

results.exact <- geocode_addresses(single_line = exact$Address,
                                   geocoder = chicago_geocoder)
results.exact
#> Simple feature collection with 1000 features and 61 fields...

#####-----------------------------------------#####

results.long  <- geocode_addresses(single_line = long$Address,
                                   geocoder = chicago_geocoder)
#> Error in data.frame(..., check.names = FALSE) : 
#>   arguments imply differing number of rows: 1000, 1008

#####-----------------------------------------#####

results.mega  <- geocode_addresses(single_line = mega$Address,
                                   geocoder = chicago_geocoder)
#> Warning messages:                                          
#>   1: In data.frame(..., check.names = FALSE) :
#>   row names were found from a short variable and have been discarded
#> 2: In data.frame(..., check.names = FALSE) :
#>   row names were found from a short variable and have been discarded
#> 3: In data.frame(..., check.names = FALSE) :
#>   row names were found from a short variable and have been discarded
#> 4: In data.frame(..., check.names = FALSE) :
#>   row names were found from a short variable and have been discarded
#> 5: In data.frame(..., check.names = FALSE) :
#>   row names were found from a short variable and have been discarded
#> 6: In data.frame(..., check.names = FALSE) :
#>   row names were found from a short variable and have been discarded
#> 7: In data.frame(..., check.names = FALSE) :
#>   row names were found from a short variable and have been discarded
#> 8: In data.frame(..., check.names = FALSE) :
#>   row names were found from a short variable and have been discarded

results.mega
#> Simple feature collection with 64000 features and 61 fields...
# (note: all rows are empty in this result)

Additionally, sometimes I'm getting that Warning message: In data.frame(... check.names=FALSE): row names were found from a short variable and have been discarded on any job, even ones that are smaller than the max batch size. Whenever this happens, the whole data frame returns with empty results. I can't seem to create a reprex for that particular issue; I've noticed that resetting arc.check_portal() and set_arc_token(auth_binding()) seems to resolve it, so maybe my portal authorization is just timing out? But sometimes it seems like I can go 20 minutes without running into a problem, and other times it seems like I can't even go 2 minutes before it happens. Not sure what's going on, but happy to drop an issue in the arcgisbinding repo if it ends up being better suited there.

Thanks!