search

chrisvwn / Rnightlights

R package to extract data from satellite nightlights.
GNU General Public License v3.0
47 stars 14 forks source link

Using (1) Custom shapefiles and (2) custom downloaded tarballs #6

Closed pjpaulpj closed 6 years ago

pjpaulpj commented 6 years ago

Hi Chris,

Thanks for a great package.

I have two queries relating to the extensibility of the package.

  1. Is it possible to use a shapefile different from the one downloaded from GADM? If so, how would I go about it-- just replace the existing shapefile in the .Rnightlights folder?

  2. I noticed that downloading the tarballs from here containing luminosity data via your package was slower than the download using aria2 directly. I now have all the tarballs saved in a different folder. How can I make the package use these files instead of fresh downloads?

chrisvwn commented 6 years ago

Hi,

Thanks for the compliment!

  1. It is currently not possible to use shapefiles not sourced from GADM. The GADM shapefiles have a specific format for the admin level name data which we query. That is a feature I am hoping to introduce soon.

  2. The page you reference is the same one that the package downloads from. I have by default limited aria2 to use only 2 simultaneous connections to avoid overloading the NOAA servers. I plan to add the option to increase the speed or eliminate restrictions after checking with the NOAA admins to avoid the possibility of having the package blacklisted.

    [a] To use the tarballs you can rename them using the Rnightlights tile naming format and copy them to your Rnightlights tiles folder. You can get the filename by running the unexported function:

    Rnightlights:::getNlTileTifLclNamePathVIIRS(nlPeriod ,tileNum, nlType)

    For monthly VIIRS data:

    • nlPeriod is for example "201801",
    • tileNum can be retrieved by running e.g. Rnightlights:::tileName2Idx("00N060W") and
    • nlType is e.g. "VIIRS.M"

    [b] Another somewhat more complex method but which could potentially be faster for many tiles would be to create an index file with URLs to the tiles e.g.:

    <html>
  <body>
    <ul>
      <li><a href="file:///path/to/tiles/SVDNB_npp_20171201-20171231_75N180W_vcmcfg_v10_c201801021747.tgz"></a></li>
      ...
    </ul>
  </body>
</html>

    and make the package "download" the tiles by changing the VIIRS url e.g.:

    pkgOptions(ntLtsIndexUrlVIIRS.M = "[http|file]://...")

pjpaulpj commented 6 years ago

Thanks for the quick response.

I do have a text file with all the tarball urls, so I'll use the second method you propose. Could you make your suggestion on changing the url a bit more explicit?

Let's say that my tarballs are saved in ~/Home/VIIRS_Tiles. In the same folder, I also have an html file url_list.html.txt which contains the urls.

Your suggestion is to use :

pkgOptions(ntLtsIndexUrlVIIRS.M = "file://~/Home/VIIRS_Tiles/url_list.html")

Do I have that right? Btw, is my html file attached above with .txt extension structured correctly?

A ton of thanks for all this help. I have a lot to learn in terms of general coding etc-- really appreciate your patience.

chrisvwn commented 6 years ago

Note edit on file format

Note that this is experimental. I haven't tried it but it should work in theory :) So might take a couple of iterations.

For the file ensure you change the urls to point to the directory. So you would be using http:// urls if you're using a webserver or file:/// urls to "download" from the filesystem. I think in this case based on the file path you gave earlier they should look like:

<a href="file:///~/Home/VIIRS_Tiles/SVDNB_npp_20180301-20180331_75N060E_vcmcfg_v10_c201804022005.tgz"

For the url, I think a forward slash is missing in the file url. Try:

pkgOptions(ntLtsIndexUrlVIIRS.M = "file:///~/Home/VIIRS_Tiles/url_list.html")

Glad to help.

chrisvwn commented 6 years ago

Update regarding your first query: Rnightlights v0.2.2 now has some support for custom shapefiles. Use the custPolyPath parameter to provide a path to the shapefile in zip format. This will override the GADM shapefile parameters. This is still in development so is expected to be buggy. Do open an issue for any errors you encounter.

pjpaulpj commented 6 years ago

This is great. Will test it out and send an update.

chrisvwn commented 6 years ago

Thanks. By the way did your second query work out? Were you able to point to your local tile repo as discussed above? Will be helpful in case someone else wants to try this.