search

oceanmodeling / StormEvents

Python interfaces for observational data surrounding named storm events, born from @jreniel's ADCIRCpy
https://stormevents.readthedocs.io
GNU General Public License v3.0
22 stars 8 forks source link

Format Requirement for PaHM and ADCIRC for `BEST` and `OFCL` #84

Open SorooshMani-NOAA opened 1 year ago

SorooshMani-NOAA commented 1 year ago

See https://github.com/noaa-ocs-modeling/EnsemblePerturbation/issues/99.

There seems to be different contradicting requirements for track inputs for ADCIRC vs PaHM. The ones currently identified are

These changes potentially require a separate file extension for PaHM vs ADCIRC compliant input files (i.e. stormevents outputs) as discussed in https://github.com/noaa-ocs-modeling/EnsemblePerturbation/pull/98#issuecomment-1604801773

SorooshMani-NOAA commented 1 year ago

@pvelissariou1 it seems there's a difference between how ADCIRC processes track files vs what PaHM does. @WPringle and I would like to first confirm this before separating the logic of handling both in stormevents and ensembleperturbation.

My understanding based on our prior discussions is that in PaHM, the advisory type (e.g. BEST vs OFCL vs CARQ, etc.) and forecast hour (e.g. 0, 6, 12, 18, ...) don't matter, and all that the model looks at is date tag column for figuring out time. And other than that PaHM cares about the isotach speed column and the rest of the physical variables, such as velocity , direction, pressures, radii.

Please also look at the issue description above and let me know if I'm missing anything.

SorooshMani-NOAA commented 1 year ago

@WPringle I just had a meeting with Takis and it seems I didn't have the full picture for PaHM. Later he will paste the link to locations in PaHM where the ATCF files are processed, as well as the ADCIRC code that does the same. Let's then review those and decide how to move forward with the stormevent handling of different cases.

pvelissariou1 commented 1 year ago

@SorooshMani-NOAA The atcf files are preprocessed in PaHM in the https://github.com/noaa-ocs-modeling/PaHM/blob/main/src/parwind.F90

as follows:

Here there is no checking for BEST, OCFL fields as the loop goes through the sequence of times only. Both were designed having "best" track files in mind. Might need to explicitly consider the OCFL fields filtering out all hours except hour 0.

In ADCIRC the atcf files are processed in a similar way and in file: https://github.com/adcirc/adcirc/blob/bd62a3c1b167cfb95db5af13325b031de2a1cdfb/src/wind.F

a) Holland Model: In SUBROUTINE GetHollandStormData, lines: 5283-5385 b) GAHM: In SUBROUTINE NWS20GET, lines 6578-6662 (no testing on OCFL fields either).

To use forecast files input data need to be filtered to consider only "hour 0" (possibly also hour 3). Furthermore, in the forecast atcf files the radius of the last closed isobar is not estimated (is set to 0) which makes both ADCIRC and PaHM produce zero wind fields.

SorooshMani-NOAA commented 1 year ago

@pvelissariou1 @WPringle I'll try to summarize my concerns and understanding here, including #81 (i.e. explicit normalization):

stormevents must do the following normalizations in any case:

In terms of our use cases, we have the following items, along with what it means for stormevents output:

  1. Simulate BEST track for hindcast
  2. Simulate forecast track for incoming storms (NHC collaboration)
    • OFCL track data from a single advisory needs to be used.
    • The OFCL track needs to be modified to add missing fields
    • The OFCL date and forecast hour need to be modified to feed it to the model. This means:
      • For a single advisory, the first entry's date stays the same, the rest will need to increment by the forecast hour.
      • The forecast hours need to be increasing (remain as they are).
      • This should work for both SymH and GAHM, because SymH gets the first entry for each date if 0 does not exists for that date, and GAHM only cares about forecast time after making sure dates are unique
  3. Simulate past-forecast track for probabilistic analysis method implementation and testing
    • Requirements are similar to item 2.

Please note that:

I'd like to also note that @WPringle has told me that the processing of OFCL in ADCIRC GAHM only cares about the forecast hours, and not the date tags, and I'm not sure if PaHM is the same; so please let me know if my understanding is incorrect.

Please let me know if any part of this is incorrect or anything is missing so that I update the comment! Also please let me know if any part of this is unclear. I want to make sure we're all on the same page before making any major changes!

pvelissariou1 commented 1 year ago

For OFCL only hour-0 or the first available entry for every date (track start date) is taken, the rest are filtered In both models OFCL filtering is not performed at this point

Radius of the last closed isobar [missing] We should add the capability to estimate the radius; methods to do so exist but we need to understand how thay do it at NHC

Radius for quadrants for GAHM [missing] The radii of quadrant wind intensities need to be estimated when their values are missing (again need to check with NHC on how they estimate these values)

The forecast dates need to be increasing for it to work with GAHM (at least in ADCIRC) PaHM checks the timestamps and re-orders them in ascending order if needed

OFCL track data from a single advisory needs to be used. The OFCL time intervals are not constant, using the OFCI field that contains the interpolated values for every 6 hours may be a more useful approach.

Link to format documentation: ATCF Formats

SorooshMani-NOAA commented 1 year ago

@pvelissariou1 thank you for the information. I just have some follow up questions/points:

In both models OFCL filtering is not performed at this point

Isn't this line doing the filtering in PaHM for SymH (not GAHM)? Lines 1024-1026

          IF (iCnt > 1) THEN
            IF ( (strOut%fcstInc(iCnt) /= 0) .AND. (strOut%fcstInc(iCnt) == strOut%fcstInc(iCnt - 1))) CYCLE
          END IF

Also in response to

The OFCL time intervals are not constant, using the OFCI field ...

Right now for SymH only BEST and OFCL are accepted, right? So if we use OFCI we can only use it with GAHM, isn't it so?

And for

The forecast dates need to be increasing for it to work with GAHM (at least in ADCIRC)

I meant to say we cannot have all 0 forecast hours (even with different date tags) for BEST track in ADCIRC. Is it the same in PaHM? In other words the following doesn't work for ADCIRC GAHM I think:

AL, 09, 2022092600,   , BEST,   0, 168N,  809W,  50,  991, TS,  34, NEQ,   60,   60,    0,   30, 1007,  120,  30,  60,   0,   L,   0,    ,   0,   0,        IAN, D, 12, NEQ,   30,    0,    0,    0, genesis-num, 028, 
AL, 09, 2022092600,   , BEST,   0, 168N,  809W,  50,  991, TS,  50, NEQ,   30,    0,    0,    0, 1007,  120,  30,  60,   0,   L,   0,    ,   0,   0,        IAN, D, 12, NEQ,   30,    0,    0,    0, genesis-num, 028, 
AL, 09, 2022092606,   , BEST,   0, 177N,  817W,  65,  985, HU,  34, NEQ,   70,   70,   30,   70, 1007,  150,  15,  80,   0,   L,   0,    ,   0,   0,        IAN, D, 12, NEQ,   45,   30,   30,   30, genesis-num, 028, 
AL, 09, 2022092606,   , BEST,   0, 177N,  817W,  65,  985, HU,  50, NEQ,   30,   30,    0,   20, 1007,  150,  15,  80,   0,   L,   0,    ,   0,   0,        IAN, D, 12, NEQ,   45,   30,   30,   30, genesis-num, 028, 
AL, 09, 2022092606,   , BEST,   0, 177N,  817W,  65,  985, HU,  64, NEQ,   15,    0,    0,    0, 1007,  150,  15,  80,   0,   L,   0,    ,   0,   0,        IAN, D, 12, NEQ,   45,   30,   30,   30, genesis-num, 028, 
AL, 09, 2022092612,   , BEST,   0, 187N,  824W,  70,  981, HU,  34, NEQ,   90,   80,   40,   90, 1008,  150,  15,  85,   0,   L,   0,    ,   0,   0,        IAN, D, 12, NEQ,   60,   30,   30,   60, genesis-num, 028, 
AL, 09, 2022092612,   , BEST,   0, 187N,  824W,  70,  981, HU,  50, NEQ,   40,   40,    0,   30, 1008,  150,  15,  85,   0,   L,   0,    ,   0,   0,        IAN, D, 12, NEQ,   60,   30,   30,   60, genesis-num, 028, 
AL, 09, 2022092612,   , BEST,   0, 187N,  824W,  70,  981, HU,  64, NEQ,   20,    0,    0,    0, 1008,  150,  15,  85,   0,   L,   0,    ,   0,   0,        IAN, D, 12, NEQ,   60,   30,   30,   60, genesis-num, 028, 
AL, 09, 2022092618,   , BEST,   0, 197N,  830W,  80,  976, HU,  34, NEQ,  100,   90,   60,   90, 1008,  150,  20, 100,   0,   L,   0,    ,   0,   0,        IAN, D, 12, NEQ,  120,   60,   30,   75, genesis-num, 028, 
AL, 09, 2022092618,   , BEST,   0, 197N,  830W,  80,  976, HU,  50, NEQ,   50,   50,   20,   30, 1008,  150,  20, 100,   0,   L,   0,    ,   0,   0,        IAN, D, 12, NEQ,  120,   60,   30,   75, genesis-num, 028, 
AL, 09, 2022092618,   , BEST,   0, 197N,  830W,  80,  976, HU,  64, NEQ,   30,   25,    0,   20, 1008,  150,  20, 100,   0,   L,   0,    ,   0,   0,        IAN, D, 12, NEQ,  120,   60,   30,   75, genesis-num, 028, 
AL, 09, 2022092700,   , BEST,   0, 208N,  833W,  85,  965, HU,  34, NEQ,  100,   90,   60,   90, 1006,  130,  20, 105,   0,   L,   0,    ,   0,   0,        IAN, D, 12, NEQ,  150,  120,   60,   90, genesis-num, 028,

instead it should be like:

AL, 09, 2022092600,   , BEST,   0, 168N,  809W,  50,  991, TS,  34, NEQ,   60,   60,    0,   30, 1007,  120,  30,  60,   0,   L,   0,    ,   0,   0,        IAN, D, 12, NEQ,   30,    0,    0,    0, genesis-num, 028, 
AL, 09, 2022092600,   , BEST,   0, 168N,  809W,  50,  991, TS,  50, NEQ,   30,    0,    0,    0, 1007,  120,  30,  60,   0,   L,   0,    ,   0,   0,        IAN, D, 12, NEQ,   30,    0,    0,    0, genesis-num, 028, 
AL, 09, 2022092606,   , BEST,   6, 177N,  817W,  65,  985, HU,  34, NEQ,   70,   70,   30,   70, 1007,  150,  15,  80,   0,   L,   0,    ,   0,   0,        IAN, D, 12, NEQ,   45,   30,   30,   30, genesis-num, 028, 
AL, 09, 2022092606,   , BEST,   6, 177N,  817W,  65,  985, HU,  50, NEQ,   30,   30,    0,   20, 1007,  150,  15,  80,   0,   L,   0,    ,   0,   0,        IAN, D, 12, NEQ,   45,   30,   30,   30, genesis-num, 028, 
AL, 09, 2022092606,   , BEST,   6, 177N,  817W,  65,  985, HU,  64, NEQ,   15,    0,    0,    0, 1007,  150,  15,  80,   0,   L,   0,    ,   0,   0,        IAN, D, 12, NEQ,   45,   30,   30,   30, genesis-num, 028, 
AL, 09, 2022092612,   , BEST,   12, 187N,  824W,  70,  981, HU,  34, NEQ,   90,   80,   40,   90, 1008,  150,  15,  85,   0,   L,   0,    ,   0,   0,        IAN, D, 12, NEQ,   60,   30,   30,   60, genesis-num, 028, 
AL, 09, 2022092612,   , BEST,   12, 187N,  824W,  70,  981, HU,  50, NEQ,   40,   40,    0,   30, 1008,  150,  15,  85,   0,   L,   0,    ,   0,   0,        IAN, D, 12, NEQ,   60,   30,   30,   60, genesis-num, 028, 
AL, 09, 2022092612,   , BEST,   12, 187N,  824W,  70,  981, HU,  64, NEQ,   20,    0,    0,    0, 1008,  150,  15,  85,   0,   L,   0,    ,   0,   0,       
...
pvelissariou1 commented 1 year ago

@Soroosh Mani - NOAA Affiliate @.***>

Isn't this line doing the filtering in PaHM for SymH (not GAHM)?

Lines 1024-1026 IF (iCnt > 1) THEN IF ( (strOut%fcstInc(iCnt) /= 0) .AND. (strOut%fcstInc(iCnt) == strOut%fcstInc(iCnt - 1))) CYCLE END IF

It does it partially, it does not filter out the remaining OCFL fields though. I need to modify PaHM

and ADCIRC to allow for the filtering of these fields.

Also in response to

The OFCL time intervals are not constant, using the OFCI field ... Right now for SymH only BEST and OFCL are accepted, right? So if we use OFCI we can only use it with GAHM, isn't it so?

Correct. You may want to rename the OCFI fields to OFCL from within StormEvents. I will modify PaHM and possibly ADCIRC

to accept these fields as well.

Just have in mind that the OCFL fields are irregularly spaced in time (not every six hours). In PaHM I do a temporal interpolation to bring the data to the time intervals requested by the user. In ADCIRC these fields are temporarily interpolated (linearly) at the end at each model's time step. It will be a good idea to know how NHC does this interpolation.

And for

The forecast dates need to be increasing for it to work with GAHM (at least in ADCIRC) I meant to say we cannot have all 0 forecast hours (even with different date tags) for BEST track in ADCIRC. Is it the same in PaHM? In other words the following doesn't work for ADCIRC GAHM I think:

For BEST fields the 6th column should always have 0 values (the timestamp 2022092600 is exactly the time for the data to be used). So the second block:

instead it should be like:

AL, 09, 2022092600, , BEST, 0, 168N, 809W, 50, 991, TS, 34, NEQ, 60, 60, 0, 30, 1007, 120, 30, 60, 0, L, 0, , 0, 0, IAN, D, 12, NEQ, 30, 0, 0, 0, genesis-num, 028, AL, 09, 2022092600, , BEST, 0, 168N, 809W, 50, 991, TS, 50, NEQ, 30, 0, 0, 0, 1007, 120, 30, 60, 0, L, 0, , 0, 0, IAN, D, 12, NEQ, 30, 0, 0, 0, genesis-num, 028, AL, 09, 2022092606, , BEST, 6, 177N, 817W, 65, 985, HU, 34, NEQ, 70, 70, 30, 70, 1007, 150, 15, 80, 0, L, 0, , 0, 0, IAN, D, 12, NEQ, 45, 30, 30, 30, genesis-num, 028, AL, 09, 2022092606, , BEST, 6, 177N, 817W, 65, 985, HU, 50, NEQ, 30, 30, 0, 20, 1007, 150, 15, 80, 0, L, 0, , 0, 0, IAN, D, 12, NEQ, 45, 30, 30, 30, genesis-num, 028, AL, 09, 2022092606, , BEST, 6, 177N, 817W, 65, 985, HU, 64, NEQ, 15, 0, 0, 0, 1007, 150, 15, 80, 0, L, 0, , 0, 0, IAN, D, 12, NEQ, 45, 30, 30, 30, genesis-num, 028, AL, 09, 2022092612, , BEST, 12, 187N, 824W, 70, 981, HU, 34, NEQ, 90, 80, 40, 90, 1008, 150, 15, 85, 0, L, 0, , 0, 0, IAN, D, 12, NEQ, 60, 30, 30, 60, genesis-num, 028, AL, 09, 2022092612, , BEST, 12, 187N, 824W, 70, 981, HU, 50, NEQ, 40, 40, 0, 30, 1008, 150, 15, 85, 0, L, 0, , 0, 0, IAN, D, 12, NEQ, 60, 30, 30, 60, genesis-num, 028, AL, 09, 2022092612, , BEST, 12, 187N, 824W, 70, 981, HU, 64, NEQ, 20, 0, 0, 0, 1008, 150, 15, 85, 0, L, 0, , 0, 0,

won't work either in ADCIRC or PaHM.

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Tue, Jun 27, 2023 at 1:16 PM Soroosh Mani @.***> wrote:

@pvelissariou1 https://github.com/pvelissariou1 thank you for the information. I just have some follow up questions/points:

In both models OFCL filtering is not performed at this point

Isn't this line doing the filtering in PaHM for SymH (not GAHM)? Lines 1024-1026

      IF (iCnt > 1) THEN
        IF ( (strOut%fcstInc(iCnt) /= 0) .AND. (strOut%fcstInc(iCnt) == strOut%fcstInc(iCnt - 1))) CYCLE
      END IF

Also in response to

The OFCL time intervals are not constant, using the OFCI field ...

Right now for SymH only BEST and OFCL are accepted, right? So if we use OFCI we can only use it with GAHM, isn't it so?

And for

The forecast dates need to be increasing for it to work with GAHM (at least in ADCIRC)

I meant to say we cannot have all 0 forecast hours (even with different date tags) for BEST track in ADCIRC. Is it the same in PaHM? In other words the following doesn't work for ADCIRC GAHM I think:

AL, 09, 2022092600, , BEST, 0, 168N, 809W, 50, 991, TS, 34, NEQ, 60, 60, 0, 30, 1007, 120, 30, 60, 0, L, 0, , 0, 0, IAN, D, 12, NEQ, 30, 0, 0, 0, genesis-num, 028, AL, 09, 2022092600, , BEST, 0, 168N, 809W, 50, 991, TS, 50, NEQ, 30, 0, 0, 0, 1007, 120, 30, 60, 0, L, 0, , 0, 0, IAN, D, 12, NEQ, 30, 0, 0, 0, genesis-num, 028, AL, 09, 2022092606, , BEST, 0, 177N, 817W, 65, 985, HU, 34, NEQ, 70, 70, 30, 70, 1007, 150, 15, 80, 0, L, 0, , 0, 0, IAN, D, 12, NEQ, 45, 30, 30, 30, genesis-num, 028, AL, 09, 2022092606, , BEST, 0, 177N, 817W, 65, 985, HU, 50, NEQ, 30, 30, 0, 20, 1007, 150, 15, 80, 0, L, 0, , 0, 0, IAN, D, 12, NEQ, 45, 30, 30, 30, genesis-num, 028, AL, 09, 2022092606, , BEST, 0, 177N, 817W, 65, 985, HU, 64, NEQ, 15, 0, 0, 0, 1007, 150, 15, 80, 0, L, 0, , 0, 0, IAN, D, 12, NEQ, 45, 30, 30, 30, genesis-num, 028, AL, 09, 2022092612, , BEST, 0, 187N, 824W, 70, 981, HU, 34, NEQ, 90, 80, 40, 90, 1008, 150, 15, 85, 0, L, 0, , 0, 0, IAN, D, 12, NEQ, 60, 30, 30, 60, genesis-num, 028, AL, 09, 2022092612, , BEST, 0, 187N, 824W, 70, 981, HU, 50, NEQ, 40, 40, 0, 30, 1008, 150, 15, 85, 0, L, 0, , 0, 0, IAN, D, 12, NEQ, 60, 30, 30, 60, genesis-num, 028, AL, 09, 2022092612, , BEST, 0, 187N, 824W, 70, 981, HU, 64, NEQ, 20, 0, 0, 0, 1008, 150, 15, 85, 0, L, 0, , 0, 0, IAN, D, 12, NEQ, 60, 30, 30, 60, genesis-num, 028, AL, 09, 2022092618, , BEST, 0, 197N, 830W, 80, 976, HU, 34, NEQ, 100, 90, 60, 90, 1008, 150, 20, 100, 0, L, 0, , 0, 0, IAN, D, 12, NEQ, 120, 60, 30, 75, genesis-num, 028, AL, 09, 2022092618, , BEST, 0, 197N, 830W, 80, 976, HU, 50, NEQ, 50, 50, 20, 30, 1008, 150, 20, 100, 0, L, 0, , 0, 0, IAN, D, 12, NEQ, 120, 60, 30, 75, genesis-num, 028, AL, 09, 2022092618, , BEST, 0, 197N, 830W, 80, 976, HU, 64, NEQ, 30, 25, 0, 20, 1008, 150, 20, 100, 0, L, 0, , 0, 0, IAN, D, 12, NEQ, 120, 60, 30, 75, genesis-num, 028, AL, 09, 2022092700, , BEST, 0, 208N, 833W, 85, 965, HU, 34, NEQ, 100, 90, 60, 90, 1006, 130, 20, 105, 0, L, 0, , 0, 0, IAN, D, 12, NEQ, 150, 120, 60, 90, genesis-num, 028,

instead it should be like:

AL, 09, 2022092600, , BEST, 0, 168N, 809W, 50, 991, TS, 34, NEQ, 60, 60, 0, 30, 1007, 120, 30, 60, 0, L, 0, , 0, 0, IAN, D, 12, NEQ, 30, 0, 0, 0, genesis-num, 028, AL, 09, 2022092600, , BEST, 0, 168N, 809W, 50, 991, TS, 50, NEQ, 30, 0, 0, 0, 1007, 120, 30, 60, 0, L, 0, , 0, 0, IAN, D, 12, NEQ, 30, 0, 0, 0, genesis-num, 028, AL, 09, 2022092606, , BEST, 6, 177N, 817W, 65, 985, HU, 34, NEQ, 70, 70, 30, 70, 1007, 150, 15, 80, 0, L, 0, , 0, 0, IAN, D, 12, NEQ, 45, 30, 30, 30, genesis-num, 028, AL, 09, 2022092606, , BEST, 6, 177N, 817W, 65, 985, HU, 50, NEQ, 30, 30, 0, 20, 1007, 150, 15, 80, 0, L, 0, , 0, 0, IAN, D, 12, NEQ, 45, 30, 30, 30, genesis-num, 028, AL, 09, 2022092606, , BEST, 6, 177N, 817W, 65, 985, HU, 64, NEQ, 15, 0, 0, 0, 1007, 150, 15, 80, 0, L, 0, , 0, 0, IAN, D, 12, NEQ, 45, 30, 30, 30, genesis-num, 028, AL, 09, 2022092612, , BEST, 12, 187N, 824W, 70, 981, HU, 34, NEQ, 90, 80, 40, 90, 1008, 150, 15, 85, 0, L, 0, , 0, 0, IAN, D, 12, NEQ, 60, 30, 30, 60, genesis-num, 028, AL, 09, 2022092612, , BEST, 12, 187N, 824W, 70, 981, HU, 50, NEQ, 40, 40, 0, 30, 1008, 150, 15, 85, 0, L, 0, , 0, 0, IAN, D, 12, NEQ, 60, 30, 30, 60, genesis-num, 028, AL, 09, 2022092612, , BEST, 12, 187N, 824W, 70, 981, HU, 64, NEQ, 20, 0, 0, 0, 1008, 150, 15, 85, 0, L, 0, , 0, 0, ...

— Reply to this email directly, view it on GitHub https://github.com/oceanmodeling/StormEvents/issues/84#issuecomment-1610004548, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP3ZGPHFFFVYFLS4SKTXNMPP3ANCNFSM6AAAAAAZUFQY2I . You are receiving this because you were mentioned.Message ID: @.***>

SorooshMani-NOAA commented 1 year ago

@WPringle please see above. It seems that we don't need the forecast hours for GAHM on BEST track.

WPringle commented 1 year ago

@SorooshMani-NOAA For everything except ADCIRC GAHM (original NHC format):

For ADCIRC GAHM:

SorooshMani-NOAA commented 1 year ago

@WPringle, @pvelissariou1 as we discussed in the meeting, we'd like to focus on theses three columns datetime tag, track type, and forecast hour, i.e. 3rd, 5th and 6th (e.g. 20180901, OFCL, 24) from the NHC track file (ATCF). The main concern here is what the expected track input is for PaHM and ADCIRC for either of the GAHM or Holland models. There are also missing forecast fields that we'll address later.

Note that we're not talking about column spacing or value types or orders, we already assume it's the same for all cases. The question is about the values within each column in relation to other columns.

So far, this is my understanding of where things stand based on all that was discussed:

This means that the output expected from storm events should be different from original NHC file, which means (with focus on the 3 aforementioned columns):

These output requirements mean:

@WPringle @pvelissariou1, please confirm that these statements are valid, then I'll go ahead and make the changes.

WPringle commented 1 year ago

@SorooshMani-NOAA No, for OFCL we do not require the datetime tags to be increasing. They can stay as is. Only change is for best-track GAHM for ADCIRC where forecast hours needs to be present.

SorooshMani-NOAA commented 1 year ago

I think I misunderstood:

For everything except ADCIRC GAHM (original NHC format):

  • Actual time should be validation time + forecast time

What do you mean by "actual time" then?

Also does it matter if for non-GAHM model the forecast hours are not all 0s?

WPringle commented 1 year ago

"actual time" means the time that the values are supposed to be representative of, sorry perhaps in that I mean validation time.

The datetime tag represents the time that the forecast is made, or rather the analysis time.

WPringle commented 1 year ago

In ADCIRC the symmetric model actually reads the datetimes and forecast hours correctly. It is only the GAHM model which for some reason they ignored the datetimes and only read forecast hours assuming that 0 is the start time of the simulation

SorooshMani-NOAA commented 1 year ago

@WPringle I just remembered @pvelissariou1 said (if I remember correctly) that for OFCL, PaHM only picks up hour-0 of each advisory, hence the need to fake best track for the simulation. Based on what you're saying that's different in ADCIRC, right? So it's not all the same between PaHM and ADCIRC?

pvelissariou1 commented 1 year ago

@Soroosh Mani - NOAA Affiliate @.> , @William Pringle @.> Soroosh as William mentioned in GAHM the forecast hours column is used to check the forecast cycle, the subroutine NWS20GET in ADCIRC reads a modified version of fort.22 to read all required variables (not all of them found in the original track file) to run GAHM.

Having said that, in PAHM I read the best track file and internally I adjust the forecast hours column as shown in column 6 ORIGINAL florence_BEST.dat AL, 06, 2018083006, , BEST, 0, ... AL, 06, 2018083012, , BEST, 0, ... AL, 06, 2018083018, , BEST, 0, ... AL, 06, 2018083100, , BEST, 0, ... AL, 06, 2018083106, , BEST, 0, ... AL, 06, 2018083112, , BEST, 0, ... AL, 06, 2018083118, , BEST, 0, ... AL, 06, 2018090100, , BEST, 0, ... AL, 06, 2018090106, , BEST, 0, ... AL, 06, 2018090112, , BEST, 0, ... AL, 06, 2018090118, , BEST, 0, ... AL, 06, 2018090200, , BEST, 0, ...

MODIFIED florence_BEST.dat_adj AL, 06, 2018083006, 0, BEST, 6, ... AL, 06, 2018083012, 0, BEST, 12, ... AL, 06, 2018083018, 0, BEST, 18, ... AL, 06, 2018083100, 0, BEST, 24, ... AL, 06, 2018083106, 0, BEST, 30, ... AL, 06, 2018083112, 0, BEST, 36, ... AL, 06, 2018083118, 0, BEST, 42, ... AL, 06, 2018090100, 0, BEST, 48, ... AL, 06, 2018090106, 0, BEST, 54, ... AL, 06, 2018090112, 0, BEST, 60, ... AL, 06, 2018090118, 0, BEST, 66, ... AL, 06, 2018090200, 0, BEST, 72, ... This format works in both Holland and GAHM models. I have tested this with ADCIRC. Also, in both GAHM and Holland the timestamp column is read and the field is split to year, month, day and hour values.

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Tue, Jul 11, 2023 at 1:37 PM William Pringle @.***> wrote:

In ADCIRC the symmetric model actually reads the datetimes and forecast hours correctly. It is only the GAHM model which for some reason they ignored the datetimes and only read forecast hours assuming that 0 is the start time of the simulation

— Reply to this email directly, view it on GitHub https://github.com/oceanmodeling/StormEvents/issues/84#issuecomment-1631315921, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP6TNCEY5CF372DYKELXPWMO3ANCNFSM6AAAAAAZUFQY2I . You are receiving this because you were mentioned.Message ID: @.***>

WPringle commented 1 year ago

@pvelissariou1 @SorooshMani-NOAA That's true that modified format would work for both Holland and GAHM. So for BEST we can just use that format as the fort.22, for OFCL I would advise against updating the datetime column.

pvelissariou1 commented 1 year ago

@Soroosh Mani - NOAA Affiliate @.***> In PaHM I read the timestamp and the forecast hour (0 for best) and I calculate the proper year, month, day, hour values for that data row. Please refer to my previous reply on how I do things in PaHM. For GAHM the best track file needs to be preprocessed before can be digested by ADCIRC.

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Tue, Jul 11, 2023 at 2:17 PM Soroosh Mani @.***> wrote:

@WPringle https://github.com/WPringle I just remembered @pvelissariou1 https://github.com/pvelissariou1 said (if I remember correctly) that for OFCL, PaHM only picks up hour-0 of each advisory, hence the need to fake best track for the simulation. Based on what you're saying that's different in ADCIRC, right? So it's not all the same between PaHM and ADCIRC?

— Reply to this email directly, view it on GitHub https://github.com/oceanmodeling/StormEvents/issues/84#issuecomment-1631380709, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TPYFT3FE7RYFVYVCHKLXPWRDRANCNFSM6AAAAAAZUFQY2I . You are receiving this because you were mentioned.Message ID: @.***>

pvelissariou1 commented 1 year ago

@Soroosh Mani - NOAA Affiliate @.> , @William Pringle @.> My understanding is that ADCIRC (PaHM as well) uses the timestamp + forecast hours fields to determine the exact time location of the data within the simulation time array (it interpolates the data at the time location of the simulation). So, as long as timestamp+forecast hours are set properly the "time location" can be calculated accurately. My personal preference would be to update both the timestamp and the forecast hour columns and keep track of the forecast analysis time.

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Tue, Jul 11, 2023 at 1:19 PM Soroosh Mani @.***> wrote:

@WPringle https://github.com/WPringle, @pvelissariou1 https://github.com/pvelissariou1 as we discussed in the meeting, we'd like to focus on theses three columns datetime tag, track type, and forecast hour, i.e. 3rd, 5th and 6th (e.g. 20180901, OFCL, 24) from the NHC track file (ATCF). The main concern here is what the expected track input is for PaHM and ADCIRC for either of the GAHM or Holland models. There are also missing forecast fields that we'll address later.

Note that we're not talking about column spacing or value types or orders, we already assume it's the same for all cases. The question is about the values within each column in relation to other columns.

So far, this is my understanding of where things stand based on all that was discussed:

  • ADCIRC and PaHM can work with the same input track files
  • In either ADCIRC or PaHM OFCL vs BEST track types do not matter
  • Both ADCIRC and PaHM's symmetric Holland implementation requires datetime tags to be increasing (for every given isotach)
  • Both ADCIRC and PaHM's GAHM implementation requires forecast hours to be increasing (for every given isotach)

This means that the output expected from storm events should be different from original NHC file, which means (with focus on the 3 aforementioned columns):

  • stormevents output file should always have increasing datetag (to work with symmetric)
  • stormevents output file should always have increasing forecast hour (to work with GAHM)
  • stormevents output file should contain 1 track, at least this should be an option
  • stormevents output file has equal time intervals between track entries (interpolation) -- is this optional or necessary for PaHM to work @pvelissariou1 https://github.com/pvelissariou1?
  • The same stormevents output file should work for ADCIRC and PaHM, for either GAHM or symmetric
  • When stormevents reads track files, there should be a way to distinguish original vs modified track (e.g. by extension)
  • Based on all above, it shouldn't matter what the track type is, so it could be either OFCL or BEST without any change to any of the other columns (among the 3) as long as all other items above are true

These output requirements mean:

  • If input is BEST track, the output file is different in that its forecast hours are not 0 in the output
  • If input is OFCL track, the output file is different because the datetime tags are increasing as opposed to constant in original single track

@WPringle https://github.com/WPringle @pvelissariou1 https://github.com/pvelissariou1, please confirm that these statements are valid, then I'll go ahead and make the changes.

— Reply to this email directly, view it on GitHub https://github.com/oceanmodeling/StormEvents/issues/84#issuecomment-1631281796, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP2TO23XRVD4VWHQCNLXPWKI5ANCNFSM6AAAAAAZUFQY2I . You are receiving this because you were mentioned.Message ID: @.***>

WPringle commented 1 year ago

@SorooshMani-NOAA @pvelissariou1 need to be careful to tag the correct Soroosh 🤣

So, let's just stick to the standard NHC format which PaHM can handle no problems (the .dat). We only require the preprocessing for ADCIRC GAHM best-track as described by Takis, which is either done internally by PaHM or can be done by stormevents to pass to ADCIRC when output with fort.22.

pvelissariou1 commented 1 year ago

@William Pringle @.> @Soroosh Mani - NOAA Affiliate @.> Yes I agree William, Soroosh if I remember correctly, pyADCIRC does this sort of thing (that is, preprocessing the best track file).

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Tue, Jul 11, 2023 at 3:59 PM William Pringle @.***> wrote:

@SorooshMani-NOAA https://github.com/SorooshMani-NOAA @pvelissariou1 https://github.com/pvelissariou1 need to be careful to tag the correct Soroosh 🤣

So, let's just stick to the standard NHC format which PaHM can handle no problems (the .dat). We only require the preprocessing for ADCIRC GAHM best-track as described by Takis, which is either done internally by PaHM or can be done by stormevents to pass to ADCIRC when output with fort.22.

— Reply to this email directly, view it on GitHub https://github.com/oceanmodeling/StormEvents/issues/84#issuecomment-1631501270, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TPZCHYXLPNJL3UY2TEDXPW5C3ANCNFSM6AAAAAAZUFQY2I . You are receiving this because you were mentioned.Message ID: @.***>

SorooshMani-NOAA commented 1 year ago

@pvelissariou1 I think GitHub has issues with in-email reply, let's stick with on-website replies to avoid issues! In any case, let me think about it a bit. I'm more and more convinced that maybe we should remove all implicit conversions and just have methods/functions like normalize_for_adcirc or normalize_for_pahm that user needs to explicitly call before writing to output.

Since that has implications on how ensembleperturbation works, I need to make sure any change is either reflected downstream too.

WPringle commented 1 year ago

@SorooshMani-NOAA I like that idea Soroosh, but what I got from Takis is that I don't think we need a normalize_for_pahm.

I would have the normalize_for_adcirc as an optional kwarg for the write function,

pvelissariou1 commented 1 year ago

William is right, whatever works for ADCIRC it will work for PaHM as well.

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Tue, Jul 11, 2023 at 4:42 PM William Pringle @.***> wrote:

@SorooshMani-NOAA https://github.com/SorooshMani-NOAA I like that idea Soroosh, but what I got from Takis is that I don't think we need a normalize_for_pahm.

— Reply to this email directly, view it on GitHub https://github.com/oceanmodeling/StormEvents/issues/84#issuecomment-1631548769, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP6XMOXSESMM4E3IPD3XPXCCXANCNFSM6AAAAAAZUFQY2I . You are receiving this because you were mentioned.Message ID: @.***>

SorooshMani-NOAA commented 1 year ago

Just to be concrete, so these should work for both GAHM and Holland for both PaHM and ADCIRC without any issues:

WPringle commented 1 year ago

@SorooshMani-NOAA Looks good. I suppose we should actually test this all in PaHM and ADCIRC.

SorooshMani-NOAA commented 1 year ago

See https://github.com/noaa-ocs-modeling/PaHM/issues/27#issuecomment-1664135037 for formatting related issues for OFCL tracks.

SorooshMani-NOAA commented 1 year ago

Issue related to missing fields: https://github.com/noaa-ocs-modeling/PaHM/issues/29

In SCHISM, the GAHM model has been modified slightly to use the radius of the last closed isobar (RRP) to reduce the amount of calculations in the domain (similar to the Holland model) by eliminating the nodal points outside RRP (I disagree with this approach but this is for future discussion with the SCHISM developers).

In our case, in the OFCL track files all the RRPs are set to zero, hence the problem with the OFCL files. My suggestion for a temporary workaround is to replace RRP by the max(radius1, radius2, radius3, radius4) of the 34 isotach.

SorooshMani-NOAA commented 1 year ago

Also see #86