Using custPolyPath to download radiance using different administrative divisions

[nreguera]

Hi Chris.

I am trying to download the radiances using the following geographical layers instead of the preset GADM:

• Myanmar National Boundary MIMU v9.2.1
• Myanmar State and Region Boundaries MIMU v9.2.2
• Myanmar Township Boundaries MIMU v9.2.4

The formats of files available for the layers are the following ones:

I choose the shapefile format. This format contains the following files (same for each division admin0, admin1 and admin3, in the next screenshot showing the ones in the admin3 level):

In R (version 4.0.2) with Rstudio (version 1.4.904) and Rnightlights (version 0.2.4.5), I run this code to download the nightlights data:

having the custPolyPath file needed to crop the rasters in that folder that you can see in the previous screenshot (and in the same folder where all the other admin levels files are). This compressed file contains all the files I showed you previously, I haven´t modified it.

After running it, this is the ouptut and the error that I get:

After the previous process finishes, there are these new files in a local folder created by the process:

and inside the POLY_MMR_CUST-mmr_polbnda_250k_adm3_mimu.zip-SHPZIP folder there are these files:

It seems that it fails to create the CSV file structure to store the radiances right?

Thanks.

[chrisvwn]

Hi Natxo,

Thank you for trying out the custPolyPath feature and for the detail in this issue.

Yes, it seems to have gone most of the way but is unable to create the CSV file structure.

Let me try it here and get back to you.

[chrisvwn]

Seems the package was able to import it. I used getCtryNlData() so the error at the end is because I used the wrong admLevel which is the same layer name since it is difficult to determine the actual name.

See below:

Verifying the structure has been imported:

> Rnightlights:::readCtryStruct("MMR", custPolyPath = "mmr_polbnda_250k_adm3_mimu.zip")
     0_mmr_polbnda_250k_adm3_mimu area_sq_km
  1:                       Ramree   1170.124
  2:                      Toungup   4792.945
  3:                     Kyaukpyu   1168.149
  4:                      Labutta   2492.103
  5:                     Ngapudaw   3210.414
 ---                                        
326:                       Laihka   2797.747
327:                    Mongkaing   3779.654
328:                        Hseni   1241.766
329:                      Kunlong   1174.341
330:                         Muse   1981.160

Looking at the error again, it seems it is a failure to write the file due to some filesystem issue- either it was unable to create the file or the directory did not exist. Not sure what is going on there. Could you try again? If it still doesn't work try deleting the NL_STRUCT_MMR_CUST-mmr_polbnda_250k_adm3_mimu.zip-SHPZIP.csv file and try again.

[nreguera]

So a couple of things here:

• Just realizing that after installing Rnightlights package it didn´t ask me for the path.
• Looking at the folder that the process used, it only contains these folders:

• After manually launching the process setupDataPath, I got these ones:

Maybe the error came from that reason?

• I relaunched the process, and I get this error:

After this, I do have the CSV file structre created in the data folder, and in the polygons folder I have the same files I told you previously.

[chrisvwn]

Yes, that sounds like the problem. For some reason the data directory was not created first time round at installation. I am not sure what might have happened there but glad it worked when you run setupDataPath manually. If this happens again let me know so I can try to track down the problem.

For the admLevel you can run the following to list all admLevels in the polygon:

searchAdmLevel(ctryCode = "MMR", custPolyPath = "mmr_polbnda_250k_adm3_mimu.zip")

Note that to access this particular polygon you must specify the ctryCode and custPolyPath together. For custPolyPath we only look at the filename so you don't have to specify the full path subsequently. This also means that even if you use a custPolyPath with a different path but same filename it will think this is the same one as before.

[nreguera]

So I run that code only once before getCtryNlData?

When you said that I must specify the ctryCode and custPolyPath together, do you mean in the getCtryNlData function?

So I tried again with this code:

searchAdmLevel(ctryCode = "MMR", 
               custPolyPath = "mmr_polbnda_250k_adm3_mimu.zip")

# Download rasters (tiff) and radiance readings (csv)
nl_source <- getCtryNlData(
  ctryCode = ctry, # country
  admLevel = "admin3", # admin level
  custPolyPath = "mmr_polbnda_250k_adm3_mimu.zip",
  nlType = "VIIRS.M", # monthly values
  nlPeriods = nlRange("201204", "201212"),
  ignoreMissing = FALSE, # not ignoring missing values
  nlStats = list("mean", na.rm=FALSE) # aggregation: average 
)

And I got this error:

If I set the full path of in the CustPolyPath parameter I get the same error.

Maybe should I remove all the files and try it again?

[chrisvwn]

What I mean is whenever you are referencing this map e.g. getCtryNlData, searchAdmLevel, and any other function that has ctryCode and custPolyPath parameters you need to specify both otherwise it will consider it a different request and you may see it try to re-import the polygon or fail to recognize it.

So searchAdmLevel will give you a list of valid admLevels. Unfortunately, for custPolyPath we cannot determine the real name at import so we will find it has a different admLevel name. When I run it I get:

> Rnightlights::searchAdmLevel(ctryCode = "MMR", custPolyPath = "mmr_polbnda_250k_adm3_mimu.zip")
$mmr_polbnda_250k_adm3_mimu.zip
$mmr_polbnda_250k_adm3_mimu.zip$MMR
[1] "0_mmr_polbnda_250k_adm3_mimu"

This shows there is only 1 admLevel and it is assigned the name "0_mmr_polbnda_250k_adm3_mimu". So this is the admLevel you use in getCtryNlData, not "admin3".

[nreguera]

Does searchAdmLevel does create the CSV as well? And when using custPolyPath we still need to set the admLevelright?

I have run this code:

nl_source <- getCtryNlData(
  ctryCode = ctry, # country
  custPolyPath = "mmr_polbnda_250k_adm3_mimu.zip",
  admLevel = "0_mmr_polbnda_250k_adm3_mimu", # admin level
  nlType = "VIIRS.M", # monthly values
  nlPeriods = nlRange("201204", "201212"),
  ignoreMissing = FALSE, # not ignoring missing values
  nlStats = list("mean", na.rm=FALSE) # aggregation: average 
)

and I get this output with an error:

I do have the _NL_STRUCT_MMR_CUST-mmr_polbnda_250k_adm3mimu.zip-SHPZIP.csv file in E:/R/.Rnightlights/data and I can open it. It looks fine for me. I attach it here.

NL_STRUCT_MMR_CUST-mmr_polbnda_250k_adm3_mimu.zip-SHPZIP.zip

[chrisvwn]

So the main error here I think is something went wrong with tile downloads.

It seems the download site now requires registration and login. https://payneinstitute.mines.edu/eog-2/transition-to-secured-data-access/

The second error is because the data file has not been created yet due to the first error. Note the file it is complaining about is not NL_STRUCT_... it is NL_DATA_....

Let me see if I can quickly add the auth required.

[chrisvwn]

Hi @nreguera ,

I have been able to add a preliminary authorization bit. Could you give it a go?

You can update the package and try again.

[nreguera]

Hi Chris,

This is what I get when trying to download them:

Should we create our own credentials or it will be managed through the package?

[chrisvwn]

Hi,

You need to register individually at https://eogauth.mines.edu/auth/realms/master/account/ then enter the credentials in the package. The credentials will be saved in the .Rnightlights folder for future use so you only need to do this once.

[nreguera]

OK, done, but I got this error:

[chrisvwn]

Not sure what happened there. Could you try again now that the credentials are saved?

[nreguera]

Just tried again, same error, but it doesn´t ask for the credentials

[nreguera]

ok, wait, I just restarted the session and it looks working. hold on because i forgot to install aria2 :) I will write you later

[nreguera]

I think it's working now 👍 Let´s close this issue and if I see any other thing when using them we can reopen it again.

Thanks for your support Chris 🙏🏻

[chrisvwn]

You're welcome