search

MAAP-Project / Community

Issue for MAAP (Zenhub)
2 stars 1 forks source link

[Data]: Upgrade to Version 5 of ATL03/08 #504

Closed wildintellect closed 2 years ago

wildintellect commented 2 years ago

Dataset Name ATL08 & ATL03 version 5

Dataset Description This is an upgrade of the existing data, L3A Land and Vegetation Height and L2A Global Geolocated Photon Data products.

Requestor name/affiliation Laura Duncanson @lauraduncanson , MAAP UWG, UMD Paul Montesano @pahbs

Platform/Instrument/Method ICESAT-2/ATLAS

URL or DOI to dataset description https://nsidc.org/data/atl08

URL to download or access data LPDAAC EarthData Cloud

Data License Attribution: Neumann, T. A., A. Brenner, D. Hancock, J. Robbins, J. Saba, K. Harbeck, A. Gibbons, J. Lee, S. B. Luthcke, T. Rebold, et al. 2021. ATLAS/ICESat-2 L2A Global Geolocated Photon Data, Version 4. [Indicate subset used]. Boulder, Colorado USA. NASA National Snow and Ice Data Center Distributed Active Archive Center. doi: https://doi.org/10.5067/ATLAS/ATL03.004. [Date Accessed].

Intended science use case Above ground biomass estimated, core MAAP dataset.

Optional

Approximate size of data 250 TB

Additional This data is required as it contains additional dates after 2021-07-15 and has new quality flags relevant to Biomass.

spa0002 commented 2 years ago

Metadata created for ATL08 v005 data

slesaad commented 2 years ago

ATL08 v005: One granule has been successfully published (linked here). Waiting for the metadata to be verified before going ahead and publishing the rest.

ATL03 v005: Waiting for the collection metadata to be published to be able to ingest; everything else is ready.

slesaad commented 2 years ago

ATL08 v005:

Kicked off the ingest.

Since the number of granules is huge (172,869), doing it in chucks of ~40k data each.

Feb 2nd: Temporal range: 2018-10-03T00:00:00.000Z, 2019-07-30T23:59:59.000Z (total 41,992 granules)

As of 2:30 CST Feb 3rd, 10959 granules have been published. This can be monitored in this link.

slesaad commented 2 years ago

I ran ingest for the ATL08 v005 granules, temporal[]": "2018-10-03T00:00:00.000Z, 2019-07-30T23:59:59.000Z which in nasa cmr has 41,992 granules. All the ingests ran successfully, no failed executions. The backgroundJobQueue has 0 messages available or in flight. But the number of granules published to the maap cmr is just 24,697 . The only weird thing I see is 14 messages available in the cleanExecutionsDeadLetterQueue, but i don’t know what it represents.

slesaad commented 2 years ago

On running ingest on single granules (from the granules that were published from the last ingest run), they seem to be working fine individually. Wondering if the silent failure was a one time thing and continuing ingest for the next batch.

slesaad commented 2 years ago

Upped the desired_count to 20 based on the instruction from the document by Phil

Started new ingest for "temporal[]": "2021-03-15T00:00:00.000Z, 2021-11-15T23:59:59.000Z" on 8th Feb, 12:36PM. Total granules: 38,288

slesaad commented 2 years ago

All granules have been ingested and published!