Closed KornTob closed 2 months ago
goergen95
commented
2 months ago I am really sorry your are still experiencing this issue. Any chance you are running on Windows? If so, could you please try with this branch installed?
remotes::install_github("mapme-initiative/mapme.biodiversity", ref = "fix-worldpop")` 
KornTob
commented
2 months ago Hi Darius,
thanks for the quick response! I tried it again after installing the developer version, but get the same(?) error:
Error in map2(.x, .y, .f, ..., .progress = .progress) :
ℹ In index: 1.
Caused by error in `sf::gdal_utils()`:
! gdal_utils translate: an error occured
Zusätzlich: Warnmeldungen:
1: In CPL_gdaltranslate(source, destination, options, oo, config_options, :
GDAL Message 1: C:\Users\Korn\AppData\Local\Temp\Rtmp2nsQ55\file4584440d48fc.tif: TIFFFetchNormalTag:ASCII value for tag "GDALMetadata" contains null byte in value; value incorrectly truncated during reading due to implementation limitations
2: In CPL_gdaltranslate(source, destination, options, oo, config_options, :
GDAL Error 1: TIFFFetchDirectory:Sanity check on directory count failed, this is probably not a valid IFD offset
3: In CPL_gdaltranslate(source, destination, options, oo, config_options, :
GDAL Error 1: TIFFReadDirectory:Failed to read directory at offset 867355646
4: In CPL_gdaltranslate(source, destination, options, oo, config_options, :
GDAL Message 1: TIFFFetchNormalTag:ASCII value for tag "GDALMetadata" contains null byte in value; value incorrectly truncated during reading due to implementation limitations
5: In CPL_gdaltranslate(source, destination, options, oo, config_options, :
GDAL Message 1: C:\Users\Korn\AppData\Local\Temp\Rtmp2nsQ55\file4584440d48fc.tif: GeoTIFF tags apparently corrupt, they are being ignored.
6: In CPL_gdaltranslate(source, destination, options, oo, config_options, :
GDAL Error 1: TIFFFillTile:Read error at row 4294967295, col 4294967295, tile 0; got 0 bytes, expected 1865
7: In CPL_gdaltranslate(source, destination, options, oo, config_options, :
GDAL Error 1: TIFFReadEncodedTile() failed.
8: In CPL_gdaltranslate(source, destination, options, oo, config_options, :
GDAL Error 1: C:\Users\Korn\AppData\Local\Temp\Rtmp2nsQ55\file4584440d48fc.tif, band 1: IReadBlock failed at X offset 0, Y offset 0: TIFFReadEncodedTile() failed.
Fehler in .check_footprints(resource, resource_name) :
Download for resource worldpop failed.
Returning unmodified portfolio.
I also tried running my script on another tower computer with stable internet. Weirdly enough, I get a new error there. I it seems to fail already before downloading any resources:
CRS of x is not EPSG:4326. Attempting to transform.
Fehler in if (is_http) { : Argument hat Länge 0
I’m running the same code as I posted on github on both computers, so I can’t explain where this is coming from. Any idea?
Best,
Tobias
Von: Darius Görgen @.> Gesendet: Freitag, 2. August 2024 09:02 An: mapme-initiative/mapme.biodiversity @.> Cc: KornTob @.>; Author @.> Betreff: [extern] Re: [mapme-initiative/mapme.biodiversity] worldpop indicator (Issue #319)
I am really sorry your are still experiencing this issue. Any chance you are running on Windows? If so, could you please try with this branch installed?
remotes::install_github("mapme-initiative/mapme.biodiversity", ref = "fix-worldpop")`
— Reply to this email directly, view it on GitHub https://github.com/mapme-initiative/mapme.biodiversity/issues/319#issuecomment-2264704585 , or unsubscribe https://github.com/notifications/unsubscribe-auth/BHDICNAOBZSFYYRH3EWPH2DZPMVE7AVCNFSM6AAAAABL32GJESVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENRUG4YDINJYGU . You are receiving this because you authored the thread.Message ID: @.***>
goergen95
commented
2 months ago I do not have access to your data. You could either share your data or use a geometry we all have access to in order to reproduce, e.g.:
read_sf(system.file("extdata", "sierra_de_neiba_478140_2.gpkg", package = "mapme.biodiversity"))Could you also please report versions of GDAL, GEOS, PROJ?
Either via library(sf) in a new R session or via sf::sf_extSoftVersion()
goergen95
commented
2 months ago Please run the following code (you might need to install the {reprex} and {remotes} packages) and post your output here:
reprex::reprex({
remotes::install_github("mapme-initiative/mapme.biodiversity", ref = "fix-worldpop")
library(mapme.biodiversity)
library(magrittr)
library(sf)
sf_extSoftVersion()
options(timeout = 1000)
x <- system.file("extdata", "sierra_de_neiba_478140_2.gpkg", package = "mapme.biodiversity") %>%
read_sf() %>%
get_resources(get_worldpop(years = 2000)) %>%
calc_indicators(calc_population_count(engine = "extract", stats = "sum"))
x$population_count
}, session_info = TRUE)Thank you. Here is the Output from running the code you suggested:
remotes::install_github("mapme-initiative/mapme.biodiversity", ref = "fix-worldpop")
library(mapme.biodiversity)
library(magrittr)
library(sf)
sf_extSoftVersion()
options(timeout = 1000)
x <- system.file("extdata", "sierra_de_neiba_478140_2.gpkg", package = "mapme.biodiversity") %>%
read_sf() %>%
get_resources(get_worldpop(years = 2000)) %>%
calc_indicators(calc_population_count(engine = "extract", stats = "sum"))x$population_count
Created on 2024-08-02 with reprex v2.1.1
Session info
sessioninfo::session_info()
Von: Darius Görgen @.> Gesendet: Freitag, 2. August 2024 11:38 An: mapme-initiative/mapme.biodiversity @.> Cc: KornTob @.>; Author @.> Betreff: [extern] Re: [mapme-initiative/mapme.biodiversity] worldpop indicator (Issue #319)
Please run the following code (you might need to install the {reprex} and {remotes} packages) and post your output here:
reprex::reprex({
remotes::install_github("mapme-initiative/mapme.biodiversity", ref = "fix-worldpop") library(mapme.biodiversity) library(magrittr) library(sf) sf_extSoftVersion()
options(timeout = 1000) x <- system.file("extdata", "sierra_de_neiba_478140_2.gpkg", package = "mapme.biodiversity") %>% read_sf() %>% get_resources(get_worldpop(years = 2000)) %>% calc_indicators(calc_population_count(engine = "extract", stats = "sum"))
x$population_count
}, session_info = TRUE)
— Reply to this email directly, view it on GitHub https://github.com/mapme-initiative/mapme.biodiversity/issues/319#issuecomment-2264976013 , or unsubscribe https://github.com/notifications/unsubscribe-auth/BHDICNDWELWQ7PQNL5V74DLZPNHOBAVCNFSM6AAAAABL32GJESVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENRUHE3TMMBRGM . You are receiving this because you authored the thread.Message ID: @.***>
goergen95
commented
2 months ago Thanks!
There seem to be some artifacts of failed package installations in your library. If you wanted to update your packages you can head over to C:\Users\Korn\AppData\Local\R\win-library\4.4\ and manually delete directories starting with 00LOCK.
But, this does not seem the cause of the issue, since the resource download and indicator calculation runs successfully for the example code. In your real code, can you make sure you have write permissions to the value set by mapme_options(outdir="")?
At least the get_resource() step for worldpop should not directly be related to the features in x (e.g. invalid geometries). We currently assume EPSG:4326 for portfolios, so you might have to transform beforehand.
You said your real data is a grid on Madagscar? I adapted the example code, can you please run and share the output?
reprex::reprex({
remotes::install_github("mapme-initiative/mapme.biodiversity", ref = "fix-worldpop", upgrade = FALSE)
library(mapme.biodiversity)
library(geodata)
library(magrittr)
library(sf)
sf_extSoftVersion()
world <- st_as_sf(geodata::world(path=tempdir()))
mdg <- subset(world, NAME_0 == "Madagascar")
mdg <- st_make_valid(mdg)
grid <- st_sf(geometry = st_make_grid(mdg, n = c(4, 4)))
grid <- st_intersection(grid, mdg)
options(timeout = 1000)
grid <- grid %>%
get_resources(get_worldpop(years = 2000)) %>%
calc_indicators(calc_population_count(engine = "exactextract", stats = "sum"))
head(grid$population_count, 5)
}, session_info = TRUE)Hi Darius,
thanks for checking. Yes, I have writing permission for the directory I specify in outdir. I also tried setting EPSG = 4326 before using the get_resources() function, but this also didn’t help it. But the get_resources() function returns the message that it is converting my grid to 4326, so should be fine either way.
That is right, I use a Madagascar grid. Thanks for the new code example, the output is below. It throws an error that apparently “mixed-type geometries” are not supported. Maybe this brings us closer to the issue? Note that in a first iteration, it returned (after running for hours) an error that the exactextractr package is not installed. Maybe this is also relevant – the output below comes from a second try after in installed the exactextractr package:
remotes::install_github("mapme-initiative/mapme.biodiversity", ref = "fix-worldpop", upgrade = FALSE)
library(mapme.biodiversity)
library(geodata)
library(magrittr)
library(sf)
sf_extSoftVersion()
world <- st_as_sf(geodata::world(path=tempdir()))
mdg <- subset(world, NAME_0 == "Madagascar")
mdg <- st_make_valid(mdg)
grid <- st_sf(geometry = st_make_grid(mdg, n = c(4, 4)))
grid <- st_intersection(grid, mdg)
options(timeout = 1000)
grid <- grid %>%
get_resources(get_worldpop(years = 2000)) %>%
calc_indicators(calc_population_count(engine = "exactextract", stats = "sum"))head(grid$population_count, 5)
Created on 2024-08-09 with https://reprex.tidyverse.org/ reprex v2.1.1
Session info
Von: Darius Görgen @.> Gesendet: Mittwoch, 7. August 2024 12:39 An: mapme-initiative/mapme.biodiversity @.> Cc: KornTob @.>; Author @.> Betreff: [extern] Re: [mapme-initiative/mapme.biodiversity] worldpop indicator (Issue #319)
Thanks!
There seem to be some artifacts of failed package installations in your library. If you wanted to update your packages you can head over to C:\Users\Korn\AppData\Local\R\win-library\4.4\ and manually delete directories starting with 00LOCK.
But, this does not seem the cause of the issue, since the resource download and indicator calculation runs successfully for the example code. In your real code, can you make sure you have write permissions to the value set by mapme_options(outdir="")?
At least the get_resource() step for worldpop should not directly be related to the features in x (e.g. invalid geometries). We currently assume EPSG:4326 for portfolios, so you might have to transform beforehand.
You said your real data is a grid on Madagscar? I adapted the example code, can you please run and share the output?
reprex::reprex({
remotes::install_github("mapme-initiative/mapme.biodiversity", ref = "fix-worldpop", upgrade = FALSE) library(mapme.biodiversity) library(geodata) library(magrittr) library(sf) sf_extSoftVersion()
world <- st_as_sf(geodata::world(path=tempdir())) mdg <- subset(world, NAME_0 == "Madagascar") mdg <- st_make_valid(mdg) grid <- st_sf(geometry = st_make_grid(mdg, n = c(4, 4))) grid <- st_intersection(grid, mdg)
options(timeout = 1000) grid <- grid %>% get_resources(get_worldpop(years = 2000)) %>% calc_indicators(calc_population_count(engine = "exactextract", stats = "sum"))
head(grid$population_count, 5)
}, session_info = TRUE)
— Reply to this email directly, view it on GitHub https://github.com/mapme-initiative/mapme.biodiversity/issues/319#issuecomment-2273164078 , or unsubscribe https://github.com/notifications/unsubscribe-auth/BHDICNCMPHCOEXCGLLOYEADZQH2NRAVCNFSM6AAAAABL32GJESVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENZTGE3DIMBXHA . You are receiving this because you authored the thread.Message ID: @.***>
goergen95
commented
2 months ago Note that in a first iteration, it returned (after running for hours) an error that the exactextractr package is not installed.
Yes, my bad. We do not fail early in the case the package is not installed.
It is warnings you encounter for chunks of the input data which return invalid indicators. Consider increasing the chunk_size in mapme_options() (see details below).
Since you seem not to be able to reproduce your original issue, and in the light that it seems that you are now actually able to successfully get the data and calculate the indicator, I will close this issue now. Feel free to re-open this or another issue with a reproducible example, though.
A few general recommendations:
chunk_size to a reasonable value which fits in your RAM to speed up processing, e.g. something along the following lines:chunk_size <- max(as.numeric(st_area(grid))) / 10000 # in ha
mapme_options(chunk_size = chunk_size)
Hi all, thanks for the mapme update! I'm unfortunately still running into problems when calculating the worldpop indicator. I get this error message:
This is the code I am using:
The object "mdg_grid" is a grid-shapefile for madagascar.
Am I doing something wrong here? The code worked great for computing other resources.
Thanks!