Open karanjeets opened 8 years ago
@kacasey : Can you please help with the questions?
@nithinkrishna : Please review and add more information and/or questions.
Thanks for the proposed pipeline and questions.
1) No, extracting latitude longitude and keeping data classified by position is recommended.
2) Correct. Keeping the separate products in their native temporal acquisition is preferred. (i.e. MERRA-2 will be kept in monthly format, CALIOP in monthly format, MODIS in 8-day format, Landsat in daily format. We will compare to similar time frames (e.g. MERRA-2, CALIOP month Jan corresponds to MODIS first 4 8-day periods).
3) All variables are/will be shortly updated in the README file.
4) Landsat data will be reviewed.
@kacasey : Thanks for the answers. Currently, I am developing a script to extract all the variables (mentioned in the README file) and index it into the ElasticSearch index. The index schema will be shared soon.
I want to run this complete pipeline for one region first. This will help us to fail fast and provide an opportunity to work through most of the issues before we reach there.
So, from the list of 19 regions, please pick one that you are most interested in and could provide good Aerosol data.
@nithinkrishna : Let's sit tomorrow evening to discuss the schema.
@r4space : Do you know where can we host the ElasticSearch index? What systems are we getting from XSEDE? Sorry, I have a couple of them from my current project and I am not able to differentiate the new ones. Is it Comet and Gordon Compute Cluster?
@karanjeets Great! Thanks for all the progress, including script development.
For the testing region, I propose the Himalayas. Although one mountain range, it is broken down into 3 RGI regions (regions 13, 14, 15) -- denoted by yellow box in below image. There is a great mix of aerosol species which impact the region (dust, black carbon, sulfates).
@kacasey Thanks for the regions. We are working on them.
Yesterday, I and Nithin discussed the pipeline in depth and came up with some questions.
(a) Will the Latitudes and Longitudes [Lat, Lon] in RGI be different from the one mentioned in MODIS, MERRA-2 and other sources? Our assumption is yes and if we are correct, we need to map them using the corresponding area field in RGI.
(b) Do we have multiple region-specific Lat Lon in RGI based on time? The reason why we are asking this is there may be a Lat Lon which was part of the region in 2000 but not now due to the melting of ice or other climatic changes.
(c) Will the (Lat, Lon) be same across time in one source? For eg: MERRA-2 files from January 2000 will have same (Lat, Lon) as files from February 2000 or March 2001?
Hi @karanjeets & @nithinkrishna, great that you were able to discuss yesterday.
(a) Latitude/Longitude values are reported from RGI and the different products in slightly different ways. All Latitude/Longitude values should be used for mapping on one central grid. I can help define where/how latitude/longitude is saved for each file type if that is helpful. I just added this info to CALIPSO data and Landsat data in the Readme file.
(b) and (c) Yes, you are correct in thinking that climate change alters landscapes, melts ice, etc. However, the latitude/longitude points remain constant over time. Yes MERRA-2 files from January 2000 will have the same latitude/longitude range, points, resolution as files from March 2010.
I hope this is helpful. Please let me know if you have other questions.
Pipeline
Questions