search

NOAA-EMC / WW3-tools

20 stars 16 forks source link

Altimeter #36

Closed Ghazal-Mohammadpour closed 1 year ago

Ghazal-Mohammadpour commented 1 year ago

Pull Request Summary

Create satellite information by time and optionally with time averaging.

Description

This pull request represents the initial phase towards achieving our overarching goal. The primary objective is to process previously downloaded satellite data organized by latitude/longitude buckets and create files containing satellite data grouped by specified time intervals (e.g., day, month, year).

Please also include the following information:

  1. Reviewers are encouraged to assess the code for efficiency and accuracy in handling the specified inputs and producing the expected outputs.

  2. Verify that the script correctly handles different time bounds (e.g., day, month, year) and optional quality control features.

  3. Ensure that the documentation is clear and comprehensive, providing sufficient guidance on using the script and its inputs.

Issue(s) addressed

Commit Message

feat: Introduce script for satellite data organization and NetCDF file creation

This commit adds a new script to process previously downloaded satellite data from AODN, organizing it by specified time intervals (e.g., day, month, year), and creating NetCDF files with observation time, location, significant wave height (HS), wind speed, and optional satellite ID and quality control information.

- Script handles input directory location, satellite selection, and time bounds.
- Future extension to support time-averaged data creation.

See detailed PR description for more information.

Check list

Testing

These new features tested with test runs using several satellite data.

JessicaMeixner-NOAA commented 1 year ago

@Ghazal-Mohammadpour thank you so much for creating this PR! It looks like you unintentionally added a second under the pycache folder. Can you remove that if it was unintentional?

Ghazal-Mohammadpour commented 1 year ago

Hi Jessica,

I will remove it. Thanks for letting me know.

On Mon, Sep 18, 2023 at 10:08 AM Jessica Meixner @.***> wrote:

@Ghazal-Mohammadpour https://github.com/Ghazal-Mohammadpour thank you so much for creating this PR! It looks like you unintentionally added a second under the pycache folder. Can you remove that if it was unintentional?

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/WW3-tools/pull/36#issuecomment-1724018847, or unsubscribe https://github.com/notifications/unsubscribe-auth/A5W2POF5P3IFTT6NLZ4UUI3X3B5ZBANCNFSM6AAAAAA45A7I7A . You are receiving this because you were mentioned.Message ID: @.***>

JessicaMeixner-NOAA commented 1 year ago

@Ghazal-Mohammadpour I tested your code this morning however I ran into an issue where most files did not exist: File /scratch2/NCEPDEV/marine/Matthew.Masarik/dat/sat/AODN/altimeter/JASON2/IMOS_SRS-Surface-Waves_MW_JASON-2 _FV02_090N-360E-DM00.nc does not exist

I believe this is because sname had a space at the end of it. I'm testing a minor update which you can find here: https://github.com/JessicaMeixner-NOAA/WW3-tools/tree/update2altimeter this is now running. I'll be running a few more tests throughout the day, but I was curious if you had run into this issue or not.

GhazalMohammadpour-NOAA commented 1 year ago

It reads all the longitudes and latitudes. And it will skip the ones that it does not have for the track of measurement. So if there is a track with that specific lon/lat it will read, but if it does not, it will skip.

On Fri, Oct 13, 2023 at 10:33 AM Jessica Meixner @.***> wrote:

@Ghazal-Mohammadpour https://github.com/Ghazal-Mohammadpour I tested your code this morning however I ran into an issue where most files did not exist: File /scratch2/NCEPDEV/marine/Matthew.Masarik/dat/sat/AODN/altimeter/JASON2/IMOS_SRS-Surface-Waves_MW_JASON-2 _FV02_090N-360E-DM00.nc does not exist

I believe this is because sname had a space at the end of it. I'm testing a minor update which you can find here: https://github.com/JessicaMeixner-NOAA/WW3-tools/tree/update2altimeter this is now running. I'll be running a few more tests throughout the day, but I was curious if you had run into this issue or not.

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/WW3-tools/pull/36#issuecomment-1761624381, or unsubscribe https://github.com/notifications/unsubscribe-auth/BA6FDZMTJAZSGHEJ5Q6VYSDX7FGMNANCNFSM6AAAAAA45A7I7A . You are receiving this because you were mentioned.Message ID: @.***>

JessicaMeixner-NOAA commented 1 year ago

@GhazalMohammadpour-NOAA The issue was that it found no valid data at all, even though there should have been, see for example the output last several lines: 

File /scratch2/NCEPDEV/marine/Matthew.Masarik/dat/sat/AODN/altimeter/JASON2/IMOS_SRS-Surface-Waves_MW_JASON-2 _FV02_090N-354E-DM00.nc does not exist
File /scratch2/NCEPDEV/marine/Matthew.Masarik/dat/sat/AODN/altimeter/JASON2/IMOS_SRS-Surface-Waves_MW_JASON-2 _FV02_090N-355E-DM00.nc does not exist
File /scratch2/NCEPDEV/marine/Matthew.Masarik/dat/sat/AODN/altimeter/JASON2/IMOS_SRS-Surface-Waves_MW_JASON-2 _FV02_090N-356E-DM00.nc does not exist
File /scratch2/NCEPDEV/marine/Matthew.Masarik/dat/sat/AODN/altimeter/JASON2/IMOS_SRS-Surface-Waves_MW_JASON-2 _FV02_090N-357E-DM00.nc does not exist
File /scratch2/NCEPDEV/marine/Matthew.Masarik/dat/sat/AODN/altimeter/JASON2/IMOS_SRS-Surface-Waves_MW_JASON-2 _FV02_090N-358E-DM00.nc does not exist
File /scratch2/NCEPDEV/marine/Matthew.Masarik/dat/sat/AODN/altimeter/JASON2/IMOS_SRS-Surface-Waves_MW_JASON-2 _FV02_090N-359E-DM00.nc does not exist
File /scratch2/NCEPDEV/marine/Matthew.Masarik/dat/sat/AODN/altimeter/JASON2/IMOS_SRS-Surface-Waves_MW_JASON-2 _FV02_090N-360E-DM00.nc does not exist
Done reading and allocating satellite data for JASON-2 
No valid data found for JASON-2 
Job completed.

My concern is the space between JASON-2 and _FV02_090N. However, when I updated the file to only have an input for sdname there was valid data even though I did not change the time period.

JessicaMeixner-NOAA commented 1 year ago

@Ghazal-Mohammadpour I was able to run with my minor updates to your code a few scenarios with different averaging time. I still need to test a different satellite and make some plots to confirm everything is doing what is expected. I'd be interested to see your output for running this file without the modifications to see if I'm doing something wrong on my end while testing, although I did follow your instructions.

Ghazal-Mohammadpour commented 1 year ago

I can send you the path to the results if you need to double check.

On Fri, Oct 13, 2023 at 1:15 PM Jessica Meixner @.***> wrote:

@Ghazal-Mohammadpour https://github.com/Ghazal-Mohammadpour I was able to run with my minor updates to your code a few scenarios with different averaging time. I still need to test a different satellite and make some plots to confirm everything is doing what is expected. I'd be interested to see your output for running this file without the modifications to see if I'm doing something wrong on my end while testing, although I did follow your instructions.

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/WW3-tools/pull/36#issuecomment-1762146388, or unsubscribe https://github.com/notifications/unsubscribe-auth/A5W2POBSLPAM5BPPOAMI6KLX7GOMXAVCNFSM6AAAAAA45A7I7CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONRSGE2DMMZYHA . You are receiving this because you were mentioned.Message ID: @.***>

JessicaMeixner-NOAA commented 1 year ago

I can send you the path to the results if you need to double check.

That would be great, thanks!

Ghazal-Mohammadpour commented 1 year ago

Sure thing. Thank you for your time.

On Fri, Oct 13, 2023 at 1:19 PM Jessica Meixner @.***> wrote:

I can send you the path to the results if you need to double check.

That would be great, thanks!

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/WW3-tools/pull/36#issuecomment-1762150691, or unsubscribe https://github.com/notifications/unsubscribe-auth/A5W2POEX2JK6RDNFHAC2KKTX7GO6JAVCNFSM6AAAAAA45A7I7CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONRSGE2TANRZGE . You are receiving this because you were mentioned.Message ID: @.***>

JessicaMeixner-NOAA commented 1 year ago

@GhazalMohammadpour-NOAA Thanks for sending the paths - I did a quick diff and I see we have some description differences which look like were added just to github but also some code diffs: 

< 
119c95
<                     hsk = fu.variables['SWH_KU'][:]
---
>                     hsk = fu.variables['SWH_C'][:]
127c103
<                     hsk = fu.variables['SWH_KA'][:]
---
>                     hsk = fu.variables['SWH_C'][:]
296d271
<

Is the code in GitHub the correct one?

Ghazal-Mohammadpour commented 1 year ago

Yes, that's it!

On Fri, Oct 13, 2023 at 1:33 PM Jessica Meixner @.***> wrote:

@GhazalMohammadpour-NOAA https://github.com/GhazalMohammadpour-NOAA Thanks for sending the paths - I did a quick diff and I see we have some description differences which look like were added just to github but also some code diffs:

< 119c95 < hsk = fu.variables['SWH_KU'][:]

                hsk = fu.variables['SWH_C'][:]

127c103 < hsk = fu.variables['SWH_KA'][:]

                hsk = fu.variables['SWH_C'][:]

296d271 <

Is the code in GitHub the correct one?

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/WW3-tools/pull/36#issuecomment-1762163879, or unsubscribe https://github.com/notifications/unsubscribe-auth/A5W2POFJFZDBTEGF57WEGWLX7GQQ5AVCNFSM6AAAAAA45A7I7CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONRSGE3DGOBXHE . You are receiving this because you were mentioned.Message ID: @.***>

JessicaMeixner-NOAA commented 1 year ago

Okay I ran the other script from your directory and got an output file, so I'm not sure what exactly was wrong when I first ran things. Either way, more info to come on Monday.

JessicaMeixner-NOAA commented 1 year ago

It's my understanding that work in this PR is included in PR #37 and this PR can be closed. I will stop reviewing this PR unless I hear differently.

JessicaMeixner-NOAA commented 1 year ago

@GhazalMohammadpour-NOAA was this going to be closed?