Identify ARCO datasets for socioeconomic area of V0 climate dashboard

[abarciauskas-bgse]

Direction given in Trilateral Dataset list

• [ ] Propose initial dataset(s) for thematic area climate dashboard
• [ ] Check with Manil and team
• [ ] Identify data experts who will validate COG for those datasets

[abarciauskas-bgse]

Notes from SEDAC Tag-up January 12 are here: https://docs.google.com/document/d/1KOajDTRXO8apW5RicvmlgGtFqO21T_C3yFkLEjoSeaM/edit

For v0, I think we want to use services for Gridded Population of the World (GPW), v4

For latter versions we will want to include the U.S. Social Vulnerability Index Grids, v1 (2000, 2010, 2014, 2016, 2018)

Questions:

• Do we still want to include facebook population data?
• Just to double check - the popgrid viewer has "GPWv4.11 UN Adjusted 2020 Count" as a layer, assuming that is the same as "Gridded Population of the World (GPW), v4"

[abarciauskas-bgse]

Updates based on discussions with Manil and Leo today:

Manil

• we want to bring in demographic data, we think this exists with SEDAC datasets
• would also like to include vulnerability index

Leo on Facebook Population data

(copy/paste from slack)

• facebook used satellite imagery to estimate residential settlement density and with census population HRSL https://dataforgood.facebook.com/dfg/tools s3://dataforgood-fb-data there's a VRT file, titiler can open up higher zoom level each tile does not have overviews
• The facebook population density data (s3://covid-eo-data/dataforgood-fb-population-density/cog.tif) has no temporal dimension, however it does get semi-regular updates. The dataset is split up into COGs/tiles covering 10x10 degree blocks, with world wide coverage.
• I have a processing script that builds a single COG from all of these 10x10 degree blocks to build a single world-wide layer, that I run on an ad-hoc basis (whenever I notice that the data has been updated in the bucket). It usually takes several hours to run the script.
• I haven't found a better solution than this due to the problem of overviews. The dataset has a .vrt file you can point TiTiler to, but since each tile only covers a 10x10 degree block, each tile's overviews are only defined up to level 5 (see rio cogeo info output below). Meaning at any zoom level greater than 5 (zoom 6 - 12) TiTiler pointing to the .vrt file will just comparably fast to loading from a single COG. However at any zoom level less than 5 (zoom 0 - 4) TiTiler has to load the entirety of the COGs at zoom level 5, which becomes impossibly slow! (Took over a minute to load tiles at level 3 three if I recall correctly).
• Downloading all the 10x10 degree COGs to my machine and building a single world-wide layer using gdal_translate (or rio cogeo create ) ensures that overviews will be created all the way to zoom level 0, so that the map can be loaded as very low zoom levels.
• gdal_translate to a single COG

  gdalbuildvrt cog_population_global.vrt ./*.tif
  # I tried running:
  gdal_translate cog_global_population_2019-07-01.vrt dataforgood-fb-hrsl-population-2019_07_01.tif
  # But it failed telling me I needed 42 Terabytes of free space on my disk :grimacing, That's when @vincentsarago came to the rescue with:
  rio cogeo create ./cog_population_global.vrt  cog.tif --allow-intermediate-compression --blocksize 512 --overview-blocksize 512
  # Which took many hours but eventually produced a global layer of the population density dataset

• Also noticed in this facebook data bucket there is a subpath RWI which is a Relative Wealth Index

SEDAC data

• Most recent details are available in Jan 12 tagup notes
• Leo also did research and compiled a list of other interesting datasets in the trilateral dataset spreadsheet

Questions

• According to this site https://dataforgood.facebook.com/dfg/docs/methodology-high-resolution-population-density-maps facebook data is available at 30m resolution but the cogs we are using are 10 x 10 degree (much lower resolution). Do we want to try and find and create higher resolution maps?
  • SEDAC population estimates are at 1km resolution "GPWv4 is gridded with an output resolution of 30 arc-seconds (approximately 1 km at the equator)." https://sedac.ciesin.columbia.edu/data/collection/gpw-v4
• Where to find demographic data from facebook, presumably it's in the bucket
• Can vulnerability index wait for v1?

[abarciauskas-bgse]

I started looking at the SEDAC datasets more closely, since they provide gridded (1km) estimates of different demographics I think we are interested in and took notes here: https://docs.google.com/spreadsheets/d/1WvwifHCN44fntlUKPDBMkfzpvrGQcbyMsBCFtn7Pu1M/edit#gid=1882070188

so far I can't find anything on gridded demographic data from the census itself.

Questions:

1. Is demographic data just for the U.S. ok?
2. Which demographics do we want specifically? There are many:

3. These values are provided in absolute numbers - presumably we want some sort of proportion or percentage of the total population? I have tested doing that for the "population in households" variable https://delta-staging.surge.sh/explore/global?map=-118.36%2C47.1488%2C5.63&layers=usgrid-2010-hhp&lState= but not sure about the rescale and colormap values. (0.95,1 and reds), created here https://github.com/NASA-IMPACT/delta-dashboard-datasets/blob/main/datasets/usgrid-2010-hhp.json#L14
4. Do we want to use the social vulnerability index? Need to look more at what that entails.
5. If we're using this SEDAC dataset do we also want to include SEDAC's gridded population dataset of the world or stick with Facebook or both?
6. Are there any comparable products which are worldwide?

[abarciauskas-bgse]

Closing this for now as I believe we have identified HLS, nightlights, bluetarp, GRDI, SVI and facebook population data for V0