How to debug connection error?

[Nowosad]

Hi @kadyb -- I tried to use ortho_request() today, but without any success. Now, the question is -- is it my local problem, rgugik issue, or GUGiK's servers problem?

library(rgugik)
library(sf)
#> Linking to GEOS 3.9.2, GDAL 3.3.2, PROJ 8.2.0
polygon_path = system.file("datasets/search_area.gpkg", package = "rgugik")
polygon = read_sf(polygon_path)
req_df = ortho_request(polygon)
req_df
#> [1] "connection error"

This issues is partially related to https://github.com/kadyb/rgugik/issues/81.

CC: @krystiano200

[kadyb]

I just tested it on Windows (R 4.0.5) and it worked. To debug you can use debug().

These lines are crucial: https://github.com/kadyb/rgugik/blob/e276ac0b20c026543674cf15be3b0c532611957b/R/ortho_request.R#L72-L75

You can check URL from prepared_URL variable in web browser to exclude a local problem.

[Nowosad]

Thanks @kadyb.

The prepared url is "https://mapy.geoportal.gov.pl/gprest/services/SkorowidzeFOTOMF/MapServer/0/query?geometry={'xmin':360207.759479622, 'ymin':512866.601609941, 'xmax':360298.871256537, 'ymax':512957.713386855, 'spatialReference':{'wkid':2180}}&geometryType=esriGeometryEnvelope&spatialRel=esriSpatialRelIntersects&outFields=godlo,akt_rok,piksel,kolor,zrDanych,ukladXY,czy_ark_wypelniony,url_do_pobrania,idSerie,sha1,akt_data,nazwa_pliku&returnGeometry=false&f=json" and it works in the browser.

However, when I try to run the next line of code, I got:

> jsonlite::fromJSON(prepared_URL)
Error in open.connection(con, "rb") : 
  URL using bad/illegal format or missing URL

[kadyb]

I tested on Ubuntu (R 4.1.2) and it works too. Does the code below work for you?

URL = "https://mapy.geoportal.gov.pl/gprest/services/SkorowidzeFOTOMF/MapServer/0/query?geometry={'xmin':360207.759479622, 'ymin':512866.601609941, 'xmax':360298.871256537, 'ymax':512957.713386855, 'spatialReference':{'wkid':2180}}&geometryType=esriGeometryEnvelope&spatialRel=esriSpatialRelIntersects&outFields=godlo,akt_rok,piksel,kolor,zrDanych,ukladXY,czy_ark_wypelniony,url_do_pobrania,idSerie,sha1,akt_data,nazwa_pliku&returnGeometry=false&f=json"
output = jsonlite::fromJSON(URL)

[Nowosad]

URL = "https://mapy.geoportal.gov.pl/gprest/services/SkorowidzeFOTOMF/MapServer/0/query?geometry={'xmin':360207.759479622, 'ymin':512866.601609941, 'xmax':360298.871256537, 'ymax':512957.713386855, 'spatialReference':{'wkid':2180}}&geometryType=esriGeometryEnvelope&spatialRel=esriSpatialRelIntersects&outFields=godlo,akt_rok,piksel,kolor,zrDanych,ukladXY,czy_ark_wypelniony,url_do_pobrania,idSerie,sha1,akt_data,nazwa_pliku&returnGeometry=false&f=json"
output = jsonlite::fromJSON(URL)
#> Error in open.connection(con, "rb"): URL using bad/illegal format or missing URL

^{Created on 2021-12-09 by the reprex package (v2.0.1)}

Session info

``` r sessioninfo::session_info() #> ─ Session info ────────────────────────────────────────────────────────────── #> hash: locked, fast down button, ice cream #> #> setting value #> version R version 4.1.2 (2021-11-01) #> os Fedora Linux 35 (Thirty Five) #> system x86_64, linux-gnu #> ui X11 #> language (EN) #> collate en_US.UTF-8 #> ctype en_US.UTF-8 #> tz Europe/Warsaw #> date 2021-12-09 #> pandoc 2.14.0.3 @ /usr/libexec/rstudio/bin/pandoc/ (via rmarkdown) #> #> ─ Packages ─────────────────────────────────────────────────────────────────── #> package * version date (UTC) lib source #> backports 1.4.0 2021-11-23 [2] CRAN (R 4.1.2) #> cli 3.1.0 2021-10-27 [2] CRAN (R 4.1.1) #> crayon 1.4.2 2021-10-29 [2] CRAN (R 4.1.1) #> curl 4.3.2 2021-06-23 [2] CRAN (R 4.1.1) #> digest 0.6.28 2021-09-23 [2] CRAN (R 4.1.1) #> ellipsis 0.3.2 2021-04-29 [2] CRAN (R 4.1.1) #> evaluate 0.14 2019-05-28 [2] CRAN (R 4.1.1) #> fansi 0.5.0 2021-05-25 [2] CRAN (R 4.1.1) #> fastmap 1.1.0 2021-01-25 [2] CRAN (R 4.1.1) #> fs 1.5.1 2021-11-30 [2] CRAN (R 4.1.2) #> glue 1.5.1 2021-11-30 [2] CRAN (R 4.1.2) #> highr 0.9 2021-04-16 [2] CRAN (R 4.1.1) #> htmltools 0.5.2 2021-08-25 [2] CRAN (R 4.1.1) #> jsonlite 1.7.2 2020-12-09 [2] CRAN (R 4.1.1) #> knitr 1.36 2021-09-29 [2] CRAN (R 4.1.2) #> lifecycle 1.0.1 2021-09-24 [2] CRAN (R 4.1.1) #> magrittr 2.0.1 2020-11-17 [2] CRAN (R 4.1.1) #> pillar 1.6.4 2021-10-18 [2] CRAN (R 4.1.1) #> pkgconfig 2.0.3 2019-09-22 [2] CRAN (R 4.1.1) #> purrr 0.3.4 2020-04-17 [2] CRAN (R 4.1.1) #> R.cache 0.15.0 2021-04-30 [2] CRAN (R 4.1.1) #> R.methodsS3 1.8.1 2020-08-26 [2] CRAN (R 4.1.1) #> R.oo 1.24.0 2020-08-26 [2] CRAN (R 4.1.1) #> R.utils 2.11.0 2021-09-26 [2] CRAN (R 4.1.1) #> reprex 2.0.1 2021-08-05 [2] CRAN (R 4.1.1) #> rlang 0.4.12 2021-10-18 [2] CRAN (R 4.1.1) #> rmarkdown 2.11 2021-09-14 [2] CRAN (R 4.1.1) #> rstudioapi 0.13 2020-11-12 [2] CRAN (R 4.1.1) #> sessioninfo 1.2.1 2021-11-02 [2] CRAN (R 4.1.1) #> stringi 1.7.5 2021-10-04 [2] CRAN (R 4.1.2) #> stringr 1.4.0 2019-02-10 [2] CRAN (R 4.1.1) #> styler 1.6.2 2021-09-23 [2] CRAN (R 4.1.1) #> tibble 3.1.6 2021-11-07 [2] CRAN (R 4.1.1) #> utf8 1.2.2 2021-07-24 [2] CRAN (R 4.1.1) #> vctrs 0.3.8 2021-04-29 [2] CRAN (R 4.1.1) #> withr 2.4.3 2021-11-30 [2] CRAN (R 4.1.2) #> xfun 0.28 2021-11-04 [2] CRAN (R 4.1.2) #> yaml 2.2.1 2020-02-01 [2] CRAN (R 4.1.1) #> #> [1] /home/jn/R/x86_64-redhat-linux-gnu-library/4.1 #> [2] /usr/local/lib/R/library #> [3] /usr/lib64/R/library #> [4] /usr/share/R/library #> #> ────────────────────────────────────────────────────────────────────────────── ```

[Nowosad]

Interestingly, the URL works when spaces were removed:

URL = "https://mapy.geoportal.gov.pl/gprest/services/SkorowidzeFOTOMF/MapServer/0/query?geometry={'xmin':360207.759479622,'ymin':512866.601609941,'xmax':360298.871256537,'ymax':512957.713386855,'spatialReference':{'wkid':2180}}&geometryType=esriGeometryEnvelope&spatialRel=esriSpatialRelIntersects&outFields=godlo,akt_rok,piksel,kolor,zrDanych,ukladXY,czy_ark_wypelniony,url_do_pobrania,idSerie,sha1,akt_data,nazwa_pliku&returnGeometry=false&f=json"
output = jsonlite::fromJSON(URL)

^{Created on 2021-12-09 by the reprex package (v2.0.1)}

Session info

``` r sessioninfo::session_info() #> ─ Session info ────────────────────────────────────────────────────────────── #> hash: locked, fast down button, ice cream #> #> setting value #> version R version 4.1.2 (2021-11-01) #> os Fedora Linux 35 (Thirty Five) #> system x86_64, linux-gnu #> ui X11 #> language (EN) #> collate en_US.UTF-8 #> ctype en_US.UTF-8 #> tz Europe/Warsaw #> date 2021-12-09 #> pandoc 2.14.0.3 @ /usr/libexec/rstudio/bin/pandoc/ (via rmarkdown) #> #> ─ Packages ─────────────────────────────────────────────────────────────────── #> package * version date (UTC) lib source #> backports 1.4.0 2021-11-23 [2] CRAN (R 4.1.2) #> cli 3.1.0 2021-10-27 [2] CRAN (R 4.1.1) #> crayon 1.4.2 2021-10-29 [2] CRAN (R 4.1.1) #> curl 4.3.2 2021-06-23 [2] CRAN (R 4.1.1) #> digest 0.6.28 2021-09-23 [2] CRAN (R 4.1.1) #> ellipsis 0.3.2 2021-04-29 [2] CRAN (R 4.1.1) #> evaluate 0.14 2019-05-28 [2] CRAN (R 4.1.1) #> fansi 0.5.0 2021-05-25 [2] CRAN (R 4.1.1) #> fastmap 1.1.0 2021-01-25 [2] CRAN (R 4.1.1) #> fs 1.5.1 2021-11-30 [2] CRAN (R 4.1.2) #> glue 1.5.1 2021-11-30 [2] CRAN (R 4.1.2) #> highr 0.9 2021-04-16 [2] CRAN (R 4.1.1) #> htmltools 0.5.2 2021-08-25 [2] CRAN (R 4.1.1) #> jsonlite 1.7.2 2020-12-09 [2] CRAN (R 4.1.1) #> knitr 1.36 2021-09-29 [2] CRAN (R 4.1.2) #> lifecycle 1.0.1 2021-09-24 [2] CRAN (R 4.1.1) #> magrittr 2.0.1 2020-11-17 [2] CRAN (R 4.1.1) #> pillar 1.6.4 2021-10-18 [2] CRAN (R 4.1.1) #> pkgconfig 2.0.3 2019-09-22 [2] CRAN (R 4.1.1) #> purrr 0.3.4 2020-04-17 [2] CRAN (R 4.1.1) #> R.cache 0.15.0 2021-04-30 [2] CRAN (R 4.1.1) #> R.methodsS3 1.8.1 2020-08-26 [2] CRAN (R 4.1.1) #> R.oo 1.24.0 2020-08-26 [2] CRAN (R 4.1.1) #> R.utils 2.11.0 2021-09-26 [2] CRAN (R 4.1.1) #> reprex 2.0.1 2021-08-05 [2] CRAN (R 4.1.1) #> rlang 0.4.12 2021-10-18 [2] CRAN (R 4.1.1) #> rmarkdown 2.11 2021-09-14 [2] CRAN (R 4.1.1) #> rstudioapi 0.13 2020-11-12 [2] CRAN (R 4.1.1) #> sessioninfo 1.2.1 2021-11-02 [2] CRAN (R 4.1.1) #> stringi 1.7.5 2021-10-04 [2] CRAN (R 4.1.2) #> stringr 1.4.0 2019-02-10 [2] CRAN (R 4.1.1) #> styler 1.6.2 2021-09-23 [2] CRAN (R 4.1.1) #> tibble 3.1.6 2021-11-07 [2] CRAN (R 4.1.1) #> utf8 1.2.2 2021-07-24 [2] CRAN (R 4.1.1) #> vctrs 0.3.8 2021-04-29 [2] CRAN (R 4.1.1) #> withr 2.4.3 2021-11-30 [2] CRAN (R 4.1.2) #> xfun 0.28 2021-11-04 [2] CRAN (R 4.1.2) #> yaml 2.2.1 2020-02-01 [2] CRAN (R 4.1.1) #> #> [1] /home/jn/R/x86_64-redhat-linux-gnu-library/4.1 #> [2] /usr/local/lib/R/library #> [3] /usr/lib64/R/library #> [4] /usr/share/R/library #> #> ────────────────────────────────────────────────────────────────────────────── ```

[kadyb]

This error is from curl and is unhappy about the link. Maybe try adding "/" at the end? Or maybe space encoding will help: URL = gsub(" ", "%20", URL)? Or remove these spaces at all?

[kadyb]

We have nice post timing :smile:

[Nowosad]

So the question now is -- can we remove these (and similar) spaces from the package's code without making issues for other platforms?

[kadyb]

This issue is very similar to this one #10. I see Geoportal is using + to encode spaces. I will try to remove these spaces at all and test next. If it doesn't work, I encode this as %20 or +.

[Nowosad]

Great. Let me know when it is ready and I will test it on my machine.

[kadyb]

The patch (#84) for this issue can be installed in this way:

remotes::install_github("kadyb/rgugik")

[Nowosad]

Hi @kadyb -- now I have a different problem. The new code (tested on the reprex in the first post) does not return any result -- it just keeps the process working... I also tried the url that had previously worked (https://github.com/kadyb/rgugik/issues/83#issuecomment-990048092) -- it does not work now. My browser keeps trying to access the website for several minutes, and at the end returns {"error":{"code":500,"message":"Error performing query operation","details":[]}}

Is this an issue with the gugik servers or could it be my local issue?

[Nowosad]

Update: R code gave the proper result after maaaany minutes...

[kadyb]

Error 500 means there are some problems with the server. Maybe the server is overloaded. It is certain that searching for orthophotomaps takes longer than for DEMs (this is independent of R, just the server).

library(rgugik)
library(sf)
polygon_path = system.file("datasets/search_area.gpkg", package = "rgugik")
polygon = read_sf(polygon_path)

system.time({req_df = ortho_request(polygon)})
#> user  system elapsed 
#> 0.06    0.00  105.44

system.time({req_df = DEM_request(polygon)})
#> user  system elapsed 
#> 0.01    0.00    0.47

[Nowosad]

Thanks for the explanation, Krzysztof.