Update Nightlights data

olafveerman commented 3 years ago

New Nightlight data through Dec 1 was added to the S3 bucket s3://covid-eo-blackmarble/

COVID_Dashboard_BMHD_Jan_Nov_2020.tar.gz
COVID_Dashboard_VNP46A2_Jan_Nov_2020.tar.gz

leothomas commented 3 years ago

@olafveerman update on re-processing of the nightlight's dataset:

BMHD (30m monthly dataset) is updated and in a staging folder in S3 (s3://covid-eo-data/bmhd_30m_monthly_staging)
VNP46A2 (500m daily dataset) is almost entirely updated in a staging folder in S3 (s3://covid-eo-data/bm_500m_daily_staging). There were some image archives (.img.tar.gz) that I was unable to extract (see below). It looks like those files are each only 45 bytes. I would love a second pair of eyes on this see if there was a processing error somewhere along the chain.

Size, Last Updated, Filepath (relative)
45B, Dec  4 01:07, ./h06v05/2020/012/near_nadir/VNP46A2_V011.h06v05.A2020012.5000.22.22_68.73_V37.img.tar.gz
45B, Dec  5 03:00, ./h18v03/2020/007/near_nadir/VNP46A2_V011.h18v03.A2020007.5000.36.85_51.77_V37.img.tar.gz
45B, Dec  5 03:01, ./h18v03/2020/009/near_nadir/VNP46A2_V011.h18v03.A2020009.5000.20.21_30.76_V37.img.tar.gz
45B, Dec  5 03:01, ./h18v03/2020/008/near_nadir/VNP46A2_V011.h18v03.A2020008.5000.23.84_58.21_V37.img.tar.gz
45B, Dec  5 02:59, ./h18v03/2020/006/near_nadir/VNP46A2_V011.h18v03.A2020006.5000.48.58_45.54_V37.img.tar.gz
45B, Dec  5 03:03, ./h18v03/2020/011/near_nadir/VNP46A2_V011.h18v03.A2020011.5000.15.44_52.94_V37.img.tar.gz
45B, Dec  5 03:02, ./h18v03/2020/010/near_nadir/VNP46A2_V011.h18v03.A2020010.5000.2.25_31.58_V37.img.tar.gz
45B, Dec  4 03:52, ./h29v04/2020/011/near_nadir/VNP46A2_V011.h29v04.A2020011.5000.13.60_69.20_V37.img.tar.gz
45B, Dec  4 03:52, ./h29v04/2020/011/near_nadir/VNP46A2_V011.h29v04.A2020011.5000.13.60_69.20_V37.img.tar.gz
45B, Dec  4 01:09, ./h10v04/2020/012/near_nadir/VNP46A2_V011.h10v04.A2020012.5000.20.29_67.99_V37.img.tar.gz
45B, Dec  4 01:09, ./h10v04/2020/011/near_nadir/VNP46A2_V011.h10v04.A2020011.5000.6.92_65.39_V37.img.tar.gz
45B, Dec  4 00:28, ./h05v05/2020/012/near_nadir/VNP46A2_V011.h05v05.A2020012.5000.22.22_68.73_V37.img.tar.gz
45B, Dec  4 01:07, ./h06v05/2020/012/near_nadir/VNP46A2_V011.h06v05.A2020012.5000.22.22_68.73_V37.img.tar.gz
45B, Dec  5 03:00, ./h18v03/2020/007/near_nadir/VNP46A2_V011.h18v03.A2020007.5000.36.85_51.77_V37.img.tar.gz
45B, Dec  5 03:01, ./h18v03/2020/009/near_nadir/VNP46A2_V011.h18v03.A2020009.5000.20.21_30.76_V37.img.tar.gz
45B, Dec  5 03:01, ./h18v03/2020/008/near_nadir/VNP46A2_V011.h18v03.A2020008.5000.23.84_58.21_V37.img.tar.gz
45B, Dec  5 02:59, ./h18v03/2020/006/near_nadir/VNP46A2_V011.h18v03.A2020006.5000.48.58_45.54_V37.img.tar.gz
45B, Dec  5 03:03, ./h18v03/2020/011/near_nadir/VNP46A2_V011.h18v03.A2020011.5000.15.44_52.94_V37.img.tar.gz
45B, Dec  5 03:02, ./h18v03/2020/010/near_nadir/VNP46A2_V011.h18v03.A2020010.5000.2.25_31.58_V37.img.tar.gz
45B, Dec  4 03:52, ./h29v04/2020/011/near_nadir/VNP46A2_V011.h29v04.A2020011.5000.13.60_69.20_V37.img.tar.gz
45B, Dec  4 03:52, ./h29v04/2020/011/near_nadir/VNP46A2_V011.h29v04.A2020011.5000.13.60_69.20_V37.img.tar.gz
45B, Dec  4 01:09, ./h10v04/2020/012/near_nadir/VNP46A2_V011.h10v04.A2020012.5000.20.29_67.99_V37.img.tar.gz
45B, Dec  4 01:09, ./h10v04/2020/011/near_nadir/VNP46A2_V011.h10v04.A2020011.5000.6.92_65.39_V37.img.tar.gz
45B, Dec  4 00:28, ./h05v05/2020/012/near_nadir/VNP46A2_V011.h05v05.A2020012.5000.22.22_68.73_V37.img.tar.gz
45B, Dec  4 01:07, ./h06v05/2020/012/near_nadir/VNP46A2_V011.h06v05.A2020012.5000.22.22_68.73_V37.img.tar.gz
45B, Dec  5 03:00, ./h18v03/2020/007/near_nadir/VNP46A2_V011.h18v03.A2020007.5000.36.85_51.77_V37.img.tar.gz
45B, Dec  5 03:01, ./h18v03/2020/009/near_nadir/VNP46A2_V011.h18v03.A2020009.5000.20.21_30.76_V37.img.tar.gz
45B, Dec  5 03:01, ./h18v03/2020/008/near_nadir/VNP46A2_V011.h18v03.A2020008.5000.23.84_58.21_V37.img.tar.gz
45B, Dec  5 02:59, ./h18v03/2020/006/near_nadir/VNP46A2_V011.h18v03.A2020006.5000.48.58_45.54_V37.img.tar.gz
45B, Dec  5 03:03, ./h18v03/2020/011/near_nadir/VNP46A2_V011.h18v03.A2020011.5000.15.44_52.94_V37.img.tar.gz
45B, Dec  5 03:02, ./h18v03/2020/010/near_nadir/VNP46A2_V011.h18v03.A2020010.5000.2.25_31.58_V37.img.tar.gz
45B, Dec  4 03:52, ./h29v04/2020/011/near_nadir/VNP46A2_V011.h29v04.A2020011.5000.13.60_69.20_V37.img.tar.gz
45B, Dec  4 03:52, ./h29v04/2020/011/near_nadir/VNP46A2_V011.h29v04.A2020011.5000.13.60_69.20_V37.img.tar.gz
45B, Dec  4 01:09, ./h10v04/2020/012/near_nadir/VNP46A2_V011.h10v04.A2020012.5000.20.29_67.99_V37.img.tar.gz
45B, Dec  4 01:09, ./h10v04/2020/011/near_nadir/VNP46A2_V011.h10v04.A2020011.5000.6.92_65.39_V37.img.tar.gz
45B, Dec  4 00:28, ./h05v05/2020/012/near_nadir/VNP46A2_V011.h05v05.A2020012.5000.22.22_68.73_V37.img.tar.gz

olafveerman commented 3 years ago

Nightlight data is updated. Since the missing data is not breaking the frontend and it will be added in a future release, we can close this issue.

olafveerman commented 3 years ago

Issue with missing data should be solved in the latest data drop.

leothomas commented 3 years ago

Re-opening this issue:

Last data drop included tiles: h30v05 , h31v05 and h32v05 (all three tiles were associated with the Tokyo spotlight but cover the area between Tokyo and Seoul, with the tileh30v05 being over Seoul)

The updated data drop doesn't contain the h30v05 tile anymore. I can process without this tile with no problem, but I wanted to flag this in case it's a technical issue to solve.

Also, there are no longer any header (.hdr) files with each tile's daily tiff. Again I can process without the header, but I wanted to raise it in case it's an issue.

The processed data (without header files and h30v05 tiles) can be found at s3://covid-eo-data/bm_500m_daily_staging

NASA-IMPACT / covid-dashboard

Update Nightlights data #535