Adding functionality to download obs in format compatible with OCS's autoval code

[gseroka]

@SorooshMani-NOAA

Spoke with Saeed today about linking searvey with OCS's autoval (https://github.com/noaa-ocs-modeling/autoval), using the Pacific SCHISM model as a test case. Autoval uses csdllib (https://github.com/noaa-ocs-modeling/csdllib) to download CO-OPS observed water level data.

The specific piece of code in csdllib that does this is https://github.com/noaa-ocs-modeling/csdllib/blob/master/csdllib/data/coops.py -- the getData function (see below).

A sample URL that it downloads is https://opendap.co-ops.nos.noaa.gov/axis/webservices/waterlevelrawsixmin/plain/response.jsp?stationId=8410140&beginDate=20230302%2018:06&endDate=20230310%2012:00&datum=MSL&unit=0&timeZone=0&Submit=Submit

8410140 1 WL 2023-03-02 18:06 -1.868 0.017 1 0 0 0 8410140 1 WL 2023-03-02 18:12 -1.887 0.007 1 0 0 0 8410140 1 WL 2023-03-02 18:18 -1.901 0.010 0 0 0 0 8410140 1 WL 2023-03-02 18:24 -1.905 0.008 1 0 0 0 8410140 1 WL 2023-03-02 18:30 -1.910 0.010 1 0 0 0 8410140 1 WL 2023-03-02 18:36 -1.914 0.009 1 0 0 0 8410140 1 WL 2023-03-02 18:42 -1.923 0.009 0 0 0 0

Would you be able to add capability to searvey to download CO-OPS (and eventually IOC and USGS) observations into this format?

def getData (stationID,  dateRange, tmpDir=None,
             product='waterlevelrawsixmin', datum='MSL', units='meters', verbose=False):

    Allows for downloading the observations from NOAA's 
    Center for Operational Oceanographic Products and Services (COOPS)
    via OpenDAP server    http://opendap.co-ops.nos.noaa.gov .

    Args:
        stationID (str):              7 character-long CO-OPS station ID.
        dateRange (datetime, datetime): start and end dates of retrieval.

[SorooshMani-NOAA]

@gseroka it should be straightforward to add, since we already download the data from all the stations, I think it can be some kind of a utility function. The biggest thing to consider is whether we'd like it to be a part of searvey or the client code; i.e. I can write a function that uses searvey to achieve what you want and pass it to you to include it as a part of autoval.

@pmav99, what's you're view on this?

[SorooshMani-NOAA]

@gseroka, as we discussed, instead of adding anything to searvey, let me try to create an example code for using searvey from user side to achieve this. I'll post the code on this issue, later we can add it in searvey examples.

[gseroka]

Thanks so much Soroosh!

--

Greg Seroka, Ph.D.

Physical Scientist

Coastal Marine Modeling Branch

NOAA/NOS/OCS/Coast Survey Development Laboratory

1315 East West Hwy, SSMC3 Rm 6839, N/CS13

Silver Spring, MD 20910-3282

Phone: (240) 847-8272

On Fri, Mar 17, 2023 at 10:41 AM Soroosh Mani @.***> wrote:

@gseroka https://github.com/gseroka, as we discussed, instead of adding anything to searvey, let me try to create an example code for using searvey from user side to achieve this. I'll post the code on this issue, later we can add it in searvey examples.

— Reply to this email directly, view it on GitHub https://github.com/oceanmodeling/searvey/issues/68#issuecomment-1473948582, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABVFD335UHJZ7UWH6QEX4FLW4RZ2TANCNFSM6AAAAAAVO3BQOM . You are receiving this because you were mentioned.Message ID: @.***>

[SorooshMani-NOAA]

@gseroka I started looking into what is provided by the coops implementation in searvey vs what csdllib does. It seems the current searvey implementation doesn't have the 6 minute raw data query available.

So from your requirement standpoint, I was wondering:

1. What products are you looking for? Are you only interested in water_level or are there are products such as prediction etc. (see https://api.tidesandcurrents.noaa.gov/api/prod/#interval:~:text=hours%20of%20data-,Data%20Products,-Specify%20the%20type for CO-OPS reference and https://github.com/oceanmodeling/searvey/blob/6d097b80bd7840163a8dcb794e0362b537528927/searvey/coops.py#L143-L168 for the ones currently implemented in searvey)?
2. What intervals are you interested in. Right now only hourly and hi/low is implemented in searvey, but it should be easy to add other intervals as well.
3. In the format from your link to coops opendap: 8410140 1 WL 2023-03-02 18:12 -1.887 0.007 1 0 0 0 What does the second column (1) mean? The rest of the columns map nicely to the return value from COOPS API: <wl t="2023-03-20 00:00" v="-0.104" s="0.052" f="1,0,0,0" q="p"/>
4. Do you need the full functionality of csdllib to be ported to searvey/coops? If it's so we have to do so in stages and slowly add capabilities you need one by one.

When these capabilities are added in searvey then it's easy to write the script to reformat it.

Update

I just realized there's a difference between how interval works for water_level vs prediction products. In any case for the products that do support interval argument, coops only implements the hourly and hi/low, but for the ones that don't need it, e.g. water_level it automatically gets the default interval, e.g. 6-min.

[gseroka]

Thanks so much @SorooshMani-NOAA.

1. Yes, water_level only for now. We may eventually expand autoval / csdllib to validate other variables, e.g. currents, but that is in the future.
2. 6-minute intervals is what is needed now for autoval / csdllib
3. To my knowledge, this will always be 1. Please see the below links: https://opendap.co-ops.nos.noaa.gov/axis/webservices/waterlevelrawsixmin/response.jsp?stationId=8410140&beginDate=20230302&endDate=20230310&datum=MSL&unit=0&timeZone=0&format=text&Submit=Submit https://opendap.co-ops.nos.noaa.gov/axis/webservices/waterlevelrawsixmin/index.jsp https://opendap.co-ops.nos.noaa.gov/axis/webservices/
4. For now, no, we do not need the full functionality of csdllib, just the getData function, and just for water levels now

[SorooshMani-NOAA]

@gseroka thank you for your responses. This makes things easier. Later as more variables are needed we can slowly expose them in searvey too. I'll get back to you with the script. Based on your responses it seems that all requirements are already in searvey.

[SorooshMani-NOAA]

@gseroka please see https://github.com/SorooshMani-NOAA/searvey/blob/example/ocs-autoval/examples/OCS-Autoval.ipynb for the first example of how to format searvey coops data into what you need it to look like.

Please let me know if you need help with setting up searvey environment. This notebook example can simply be written in a function that you can call for autoval

[gseroka]

Thank you @SorooshMani-NOAA! Will test out at some point hopefully soon!

[SorooshMani-NOAA]

Just a snapshot of it:

[SorooshMani-NOAA]

I forgot to mention, the current notebook uses a single station ID as input (just like getData) but using searvey you can specify a region and get multiple stations instead, and then write them separately to separate files. It really depends on how autoval wants to use it. For now I just used the single station data fetch call to follow getData way of getting the measurements.

[SorooshMani-NOAA]

@gseroka have you had a chance to look into this?

[gseroka]

@SorooshMani-NOAA not yet. Lei is working on getting STOFS-3D-Pacific up and running in semi-operational mode, and then will look into hooking up searvey with autoval to do the skill assessment for it. Thanks again for doing this!

[SorooshMani-NOAA]

OK thank you, no problem!

[SorooshMani-NOAA]

Are there any updates on this front? Should I close this ticket?

[gseroka]

Hi Soroosh, yes, please close this ticket. We still haven't had a chance to test yet due to operational priorities. I'm thinking Lei should have a chance in the coming months. Thank you for your work on it!! Once he works on it, we can reopen the issue if needed.

--

Greg Seroka, Ph.D.

Physical Scientist

Coastal Marine Modeling Branch

NOAA/NOS/OCS/Coast Survey Development Laboratory

1315 East West Hwy, SSMC3 Rm 6839, N/CS13

Silver Spring, MD 20910-3282

Phone: (240) 847-8272

On Fri, Aug 25, 2023 at 10:46 AM Soroosh Mani @.***> wrote:

Are there any updates on this front? Should I close this ticket?

— Reply to this email directly, view it on GitHub https://github.com/oceanmodeling/searvey/issues/68#issuecomment-1693483992, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABVFD3ZQZGMJJW2IX6VKK2DXXC3A3ANCNFSM6AAAAAAVO3BQOM . You are receiving this because you were mentioned.Message ID: @.***>