search

chrisvwn / Rnightlights

R package to extract data from satellite nightlights.
GNU General Public License v3.0
47 stars 14 forks source link

Error in `proj4string<-`(`*tmp*`, value = crs) : Geographical CRS given to non-conformant data: -1640.8982 2197.5138 #60

Closed firatgundem closed 4 years ago

firatgundem commented 4 years ago

Hi Chris,

I could not have any proceeding after corresponded with you. I am trying to have nightlight statistics for Turkey on a daily bases VIIRS data. Here are my codes:

packageurl <- "https://cran.r-project.org/src/contrib/Archive/Rnightlights/Rnightlights_0.2.4.tar.gz"

install.packages(packageurl, repos=NULL, type="source")

library(Rnightlights) library(lubridate) library(reshape2)

(Optional performance enhancement if you have aria2c and gdal installed)

pkgOptions(downloadMethod = "auto", cropMaskMethod = "gdal", extractMethod = "gdal", deleteTiles = TRUE)

Optional performance enhancement. If extractMethod="rast" you can specify the number of

CPU cores to use in parallel

pkgOptions(extractMethod = "rast", numCores=4)

ctry <- "TUR" #replace to run for any other country

download and process monthly VIIRS stats at the highest admin level

highestAdmLevelStats <- getCtryNlData(ctryCode = ctry, admLevel = "highest", nlType = "VIIRS.D", nlPeriods = nlRange("20200317","20200508","VIIRS.D"), nlStats = list("sum",na.rm=TRUE), ignoreMissing=FALSE)

Then I got this:

OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_Algeria_1", layer: "Flares_Algeria_1" with 11 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_Angola_1", layer: "Flares_Angola_1" with 4 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_Argentina_1", layer: "Flares_Argentina_1" with 18 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_Australia_1", layer: "Flares_Australia_1" with 10 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_Azerbaijan_1", layer: "Flares_Azerbaijan_1" with 3 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_Belgium_1", layer: "Flares_Belgium_1" with 1 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_Bolivia_1", layer: "Flares_Bolivia_1" with 7 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_Brazil_1", layer: "Flares_Brazil_1" with 12 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_Brunei_1", layer: "Flares_Brunei_1" with 4 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_Cameroon_1", layer: "Flares_Cameroon_1_point" with 2 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_Canada_1", layer: "Flares_Canada_1" with 18 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_Chad_1", layer: "Flares_Chad_1" with 1 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_Chile_1", layer: "Flares_Chile_1" with 2 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_China_1", layer: "Flares_China_1" with 58 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_Colombia_1", layer: "Flares_Colombia_1" with 12 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_Congo_1", layer: "Flares_Congo_1" with 2 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_Cote_dIvoire_1", layer: "Flares_Cote_dIvoire_1" with 1 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_Dem_Rep_Congo_1", layer: "Flares_Dem_Rep_Congo_1" with 1 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_Denmark_1", layer: "Flares_Denmark_1" with 1 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_East_Timor_1", layer: "Flares_East_Timor_1" with 1 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_Ecuador_1", layer: "Flares_Ecuador_1" with 2 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_Egypt_1", layer: "Flares_Egypt_1" with 17 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_Eq_Guinea_1", layer: "Flares_Eq_Guinea_1" with 2 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_Gabon_1", layer: "Flares_Gabon_1" with 7 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_Germany_1", layer: "Flares_Germany_1" with 1 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_Ghana_1", layer: "Flares_Ghana_1" with 2 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_India_1", layer: "Flares_India_1" with 3 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_Indonesia_1", layer: "Flares_Indonesia_1" with 64 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_Iran_1", layer: "Flares_Iran_1" with 28 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_Iraq_1", layer: "Flares_Iraq_1" with 5 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_Kazakhstan_1", layer: "Flares_Kazakhstan_1" with 22 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_Kuwait_1", layer: "Flares_Kuwait_1" with 2 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_Libya_1", layer: "Flares_Libya_1" with 40 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_Malaysia_1", layer: "Flares_Malaysia_1" with 20 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_Mauritania_1", layer: "Flares_Mauritania_1" with 1 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_Mexico_1", layer: "Flares_Mexico_1" with 1 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_Myanmar_1", layer: "Flares_Myanmar_1" with 2 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_Netherlands_1", layer: "Flares_Netherlands_1" with 1 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_Nigeria_1", layer: "Flares_Nigeria_1" with 1 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_Norway_1", layer: "Flares_Norway_1" with 7 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_Oman_1", layer: "Flares_Oman_1" with 19 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_Peru_1", layer: "Flares_Peru_1" with 2 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_Philippines_1", layer: "Flares_Philippines_1" with 4 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_PNG_1", layer: "Flares_PNG_1" with 2 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_Qatar_1", layer: "Flares_Qatar_1" with 5 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_Romania_1", layer: "Flares_Romania_1" with 1 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_Russia_KM_1", layer: "Flares_Russia_KM_1" with 19 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_Russia_not_KM_1", layer: "Flares_Russia_not_KM_1" with 69 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_Saudi_Arabia_1", layer: "Flares_Saudi_Arabia_1" with 14 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_South_Africa_1", layer: "Flares_South_Africa_1" with 6 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_Sudan_1", layer: "Flares_Sudan_1" with 8 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_Syria_1", layer: "Flares_Syria_1" with 11 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_Thailand_1", layer: "Flares_Thailand_1" with 11 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_Trinidad_1", layer: "Flares_Trinidad_1" with 2 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_Tunisia_1", layer: "Flares_Tunisia_1" with 7 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_Turkmenistan_1", layer: "Flares_Turkmenistan_1" with 10 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_UAE_1", layer: "Flares_UAE_1" with 5 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_UK_1", layer: "Flares_UK_1" with 10 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_USA_Alaska_1", layer: "Flares_USA_Alaska_1" with 3 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_USA_CONUS_1", layer: "Flares_USA_CONUS_1" with 27 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_Uzbekistan_1", layer: "Flares_Uzbekistan_1" with 7 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_Venezuela_1_point", layer: "Flares_Venezuela_1_point" with 45 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_Vietnam_1", layer: "Flares_Vietnam_1" with 6 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_Yemen_1", layer: "Flares_Yemen_1" with 6 features It has 1 fields OGR data source with driver: ESRI Shapefile Source: "C:\Users\firat.Rnightlights\gasflares\Flares_New_Zealand", layer: "Flares_New_Zealand" with 1 features It has 1 fields 2020-05-10 04:03:15: mosaicing Error in proj4string<-(*tmp*, value = crs) : Geographical CRS given to non-conformant data: -1640.8982 2197.5138 In addition: Warning messages: 1: In proj4string<-(*tmp*, value = crs) : A new CRS was assigned to an object with an existing CRS: +proj=longlat +datum=WGS84 +no_defs +ellps=WGS84 +towgs84=0,0,0 without reprojecting. For reprojection, use function spTransform 2: In proj4string<-(*tmp*, value = crs) : A new CRS was assigned to an object with an existing CRS: +proj=longlat +datum=WGS84 +no_defs +ellps=WGS84 +towgs84=0,0,0 without reprojecting. For reprojection, use function spTransform

First of all, I did not understand why it did download all of the countries' gas flares. Secondly, why am I always getting the same error "Error in proj4string<-(*tmp*, value = crs) : "

I tried several things in order to fix this which is included reading all the issue topics of even the ones that were closed by you and tried some small changes in the codes but I could not achieve any proceeding.

This is really so important to me. I need to have daily nightlight data for Turkey between 03/17/2020 and 05/08/2020. Could you please help me?

chrisvwn commented 4 years ago

Hi @firatgundem ,

The gasflares requires all countries to be downloaded. It is a one-off operation and will not happen again.

I am not sure what is going on as I cannot replicate the problem you are facing.

Please try using the github package. First uninstall the package and delete the data directory. Then run:

install.packages("devtools")
devtools::install_github("chrisvwn/Rnightlights")

Then retry the code.

Let me know how this goes.

chrisvwn commented 4 years ago

Also note that daily nightlight data is very limited. It is usually only available for the immediate previous 2 months or so.

firatgundem commented 4 years ago

Hi Chris, Thank you for your help. I followed your recommendations and run this:

install.packages("devtools") devtools::install_github("chrisvwn/Rnightlights")

library(Rnightlights) library(lubridate) library(reshape2) library(sf) library(sp)

pkgOptions(downloadMethod = "auto", cropMaskMethod = "gdal", extractMethod = "gdal", deleteTiles = TRUE)

ctry <- "TUR" #replace to run for any other country

pkgOptions(deleteTiles=TRUE) pkgOptions("ntLtsIndexUrlVIIRS.D" = "https://eogdata.mines.edu/pages/download_dnb_composites_iframe.html")

download and process monthly VIIRS stats at the highest admin level

highestAdmLevelStats <- getCtryNlData(ctryCode = ctry, admLevel = "highest", nlType = "VIIRS.D", nlPeriods = nlRange("20200320","20200510","VIIRS.D"), nlStats = list("sum",na.rm=TRUE), ignoreMissing=FALSE)

And then I got this error:

2020-05-13 01:16:26: Downloading country polygons ... DONE 2020-05-13 01:16:26: PROCESSING nlType:VIIRS.D | configName: VCMCFG | nlPeriod:20200320 2020-05-13 01:16:26: Checking tiles required for VIIRS.D 20200320 2020-05-13 01:16:27: TUR: Stats missing. Adding tiles 2020-05-13 01:16:27: numTiles: 1, Required tiles: 75N060W 2020-05-13 01:16:27: Downloading tile: 202003202 2020-05-13 01:16:27: Tile not available on the NOAA page. Please manually check for the 20200320 tile at 'https://eogdata.mines.edu/pages/download_dnb_composites_iframe.html'. If it exists please report this as a bug 2020-05-13 01:16:27: Something went wrong with the tile downloads. Aborting ... 2020-05-13 01:16:27: COMPLETED PROCESSING :ctryCodes=TUR, admLevels=gadm36_TUR_1, nlTypes=VIIRS.D, configNames=VCMCFG, multiTileStrategy=all, multiTileMergeFun=mean, removeGasFlares=TRUE, nlPeriods=c("20200320", "20200321", "20200322", "20200323", "20200324", "20200325", "20200326", "20200327", "20200328", "20200329", "20200330", "20200331", "20200401", "20200402", "20200403", "20200404", "20200405", "20200406", "20200407", "20200408", "20200409", "20200410", "20200411", "20200412", "20200413", "20200414", "20200415", "20200416", "20200417", "20200418", "20200419", "20200420", "20200421", "20200422", "20200423", "20200424", "20200425", "20200426", "20200427", "20200428", "20200429", "20200430", "20200501", "20200502", "20200503", "20200504", "20200505", "20200506", "20200507", "20200508", "20200509", "20200510"), nlStats=list(list("sum", na.rm = TRUE)), custPolyPath=NULL, gadmVersion=3.6, gadmPolyType=shpZip, downloadMethod=auto, cropMaskMethod=gdal, extractMethod=gdal Error in data.table::fread(getCtryNlDataFnamePath(ctryCode = ctryCode, : File 'C:/Users/firat/.Rnightlights/data/NL_DATA_TUR_ADM1_GADM-3.6-SHPZIP.csv' does not exist or is non-readable. getwd()=='C:/Users/firat/Desktop'

It says there might be a bug if the tile that I input exists whereas the code is not running! What should I do now? Thanks.

firatgundem commented 4 years ago

Any help please?

firatgundem commented 4 years ago

If I am not mistaken, the problem is that I am trying to download daily data which is not available at this address: https://eogdata.mines.edu/pages/download_dnb_composites_iframe.html I have used this in order to reach out daily composites: pkgOptions("ntLtsIndexUrlVIIRS.D" = "https://ngdc.noaa.gov/eog/viirs/download_ut_mos.html") But still it does not work. After running this: highestAdmLevelStats <- getCtryNlData(ctryCode = ctry, admLevel = "highest", nlType = "VIIRS.D", nlPeriods = nlRange("20200320","20200516","VIIRS.D"), nlStats = list("sum",na.rm=TRUE), ignoreMissing=FALSE) I got this: 2020-05-17 02:17:24: Tile not available on the NOAA page. Please manually check for the 20200320 tile at 'https://ngdc.noaa.gov/eog/viirs/download_dnb_composites.html'. If it exists please report this as a bug 2020-05-17 02:17:24: Something went wrong with the tile downloads. Aborting ... Error in data.table::fread(getCtryNlDataFnamePath(ctryCode = ctryCode, : File 'C:/Users/firat/.Rnightlights/data/NL_DATA_TUR_ADM1_GADM-3.6-SHPZIP.csv' does not exist or is non-readable. getwd()=='C:/Users/firat/Desktop'

Could you please help me?

chrisvwn commented 4 years ago

Hi @firatgundem It seems like that is the problem. The tile for the date you are looking for is not available on the page. Could you try a range of dates that is available?

chrisvwn commented 4 years ago

Closing this issue. If you would like to pursue it further please reopen it.