noaa-ocs-modeling / CoupledModelDriver

coupled model configuration generation
https://CoupledModelDriver.readthedocs.io
Creative Commons Zero v1.0 Universal
4 stars 3 forks source link

ADCIRC-only run fails on hotstart #95

Closed ghost closed 3 years ago

ghost commented 3 years ago

when running ADCIRC-only with tidal spinup, besttrack, and ww3data forcing on @WPringle's subsetted mesh (in the directory /scratch2/COASTAL/coastal/save/shared/working/zach/adcirc/run_20210503_hsofs_120m_subsetflo_besttrack), the spinup phase runs fine but the hotstart fails with the following error:

ADCIRC_MESH_PREP.err.log
forrtl: severe (59): list-directed I/O syntax error, unit 15, file /scratch2/COASTAL/coastal/save/shared/working/zach/adcirc/run_20210503_hsofs_120m_subsetflo_besttrack/runs/unperturbed/fort.15
Image              PC                Routine            Line        Source
adcprep            0000000000697ECE  Unknown               Unknown  Unknown
adcprep            00000000006CEC60  Unknown               Unknown  Unknown
adcprep            00000000006CCFC2  Unknown               Unknown  Unknown
adcprep            000000000040E871  presizes_mp_sizeu        1068  presizes.F
adcprep            000000000042D06F  prepinput_                436  adcprep.F
adcprep            000000000042BB2B  MAIN__                    239  adcprep.F
adcprep            000000000040B65E  Unknown               Unknown  Unknown
libc-2.17.so       00002AB82E48D555  __libc_start_main     Unknown  Unknown
adcprep            000000000040B569  Unknown               Unknown  Unknown
srun: error: h20c02: task 0: Exited with exit code 59
srun: launch/slurm: _step_signal: Terminating StepId=18545941.1
ghost commented 3 years ago

here are the contents of the hotstart fort.15:

created on 2021-05-03 18:39                                     ! RUNDES                              - 32 CHARACTER ALPHANUMERIC RUN DESCRIPTION
NOMAD mesh v1e MSL                                              ! RUNID                               - 24 CHARACTER ALPANUMERIC RUN IDENTIFICATION
1                                                               ! NFOVER                              - NONFATAL ERROR OVERRIDE OPTION
1                                                               ! NABOUT                              - ABREVIATED OUTPUT OPTION PARAMETER
100                                                             ! NSCREEN                             - UNIT 6 OUTPUT OPTION PARAMETER
567                                                             ! IHOT                                - HOT START PARAMETER
2                                                               ! ICS                                 - COORDINATE SYSTEM SELECTION PARAMETER
511113                                                          ! IM                                  - MODEL SELECTION PARAMETER
1                                                               ! NOLIBF                              - BOTTOM FRICTION TERM SELECTION PARAM; before NWP==1, '2' was used
2                                                               ! NOLIFA                              - FINITE AMPLITUDE TERM SELECTION PARAMETER
1                                                               ! NOLICA                              - SPATIAL DERIVATIVE CONVECTIVE SELECTION PARAMETER
1                                                               ! NOLICAT                             - TIME DERIVATIVE CONVECTIVE TERM SELECTION PARAMETER
5                                                               ! NWP                                 - VARIABLE BOTTOM FRICTION AND LATERAL VISCOSITY OPTION PARAMETER; default 0
mannings_n_at_sea_floor
primitive_weighting_in_continuity_equation
surface_canopy_coefficient
surface_directional_effective_roughness_length
surface_submergence_state
1                                                               ! NCOR                                - VARIABLE CORIOLIS IN SPACE OPTION PARAMETER
1                                                               ! NTIP                                - TIDAL POTENTIAL OPTION PARAMETER
520                                                             ! NWS                                 - WIND STRESS AND BAROMETRIC PRESSURE OPTION PARAMETER
8                                                               ! NRAMP                               - RAMP FUNCTION OPTION
9.81                                                            ! G                                   - ACCELERATION DUE TO GRAVITY - DETERMINES UNITS
-3                                                              ! TAU0                                - WEIGHTING FACTOR IN GWCE; original, 0.005
7.000000                                                        ! DTDP                                - TIME STEP (IN SECONDS)
0                                                               ! STATIM                              - STARTING TIME (IN DAYS)
0                                                               ! REFTIM                              - REFERENCE TIME (IN DAYS)
2012 10 22 06 18 0.9 1 2                                        ! WTIMINC RSTIMINC                    - meteorological data time increment, wave forcing increment
16.4583                                                         ! RNDAY                               - TOTAL LENGTH OF SIMULATION (IN DAYS)
12.250 0.000 0.000 0.000 12.250 12.250 1.000 0.000 12.250       ! DRAMP                               - DURATION OF RAMP FUNCTION (IN DAYS)
0.5 0.5 0                                                       ! A00 B00 C00                         - TIME WEIGHTING FACTORS FOR THE GWCE EQUATION
0.01 0 0 0.01                                                   ! H0 NODEDRYMIN NODEWETRMP VELMIN
-77.0775 32.4507                                                ! SLAM0 SFEA0                         - CENTER OF CPP PROJECTION (NOT USED IF ICS=1, NTIP=0, NCOR=0)
0.0025                                                          ! FFACTOR
-0.2                                                            ! smagorinsky coefficient             - LATERAL EDDY VISCOSITY COEFFICIENT; IGNORED IF NWP =1
0                                                               ! CORI                                - CORIOLIS PARAMETER - IGNORED IF NCOR = 1
8                                                               ! NTIF                                - NUMBER OF TIDAL POTENTIAL CONSTITUENTS BEING FORCED starting 2008082300
110                                                             ! ANGINN                              - INNER ANGLE THRESHOLD
0 0 0 0                                                         ! NOUTE TOUTSE TOUTFE NSPOOLE         - ELEV STATION OUTPUT INFO (UNIT 61)
0                                                               ! NSTAE                               - TOTAL NUMBER OF ELEVATION RECORDING STATIONS
0 0 0 0                                                         ! NOUTV TOUTSV TOUTFV NSPOOLV         - VELOCITY STATION OUTPUT INFO (UNIT 62)
0                                                               ! NSTAV                               - TOTAL NUMBER OF VELOCITY RECORDING STATIONS
0 0 0 0                                                         ! NOUTM TOUTSM TOUTFM NSPOOLM         - METEOROLOGICAL STATION OUTPUT INFO (UNITS 71/72)
0                                                               ! NSTAM                               - TOTAL NUMBER OF METEOROLOGICAL RECORDING STATIONS
-5 0.000000 16.458333 514                                       ! NOUTGE TOUTSGE TOUTFGE NSPOOLGE     - GLOBAL ELEVATION OUTPUT INFO (UNIT 63)
-5 0.000000 16.458333 514                                       ! NOUTGV TOUTSGV TOUTFGV NSPOOLGV     - GLOBAL VELOCITY OUTPUT INFO (UNIT 64)
0 0.000000 0.000000 0                                           ! NOUTGM TOUTSGM TOUTFGM NSPOOLGM     - GLOBAL METEOROLOGICAL OUTPUT INFO
0                                                               ! NFREQ
0 0 0 0                                                         ! THAS THAF NHAINC FMV                - HARMONIC ANALYSIS PARAMETERS
0 0 0 0                                                         ! NHASE NHASV NHAGE NHAGV             - CONTROL HARMONIC ANALYSIS AND OUTPUT TO UNITS 51,52,53,54
0 0                                                             ! NHSTAR NHSINC                       - HOT START FILE GENERATION PARAMETERS
1 0 1E-08 25                                                    ! ITITER ISLDIA CONVCR ITMAX          - ALGEBRAIC SOLUTION PARAMETERS
                                                                ! NCPROJ                              - PROJECT TITLE
                                                                ! NCINST                              - PROJECT INSTITUTION
                                                                ! NCSOUR                              - PROJECT SOURCE
                                                                ! NCHIST                              - PROJECT HISTORY
                                                                ! NCREF                               - PROJECT REFERENCES
                                                                ! NCCOM                               - PROJECT COMMENTS
                                                                ! NCHOST                              - PROJECT HOST
                                                                ! NCONV                               - CONVENTIONS
                                                                ! NCCONT                              - CONTACT INFORMATION
2012-10-10 00:00                                                ! NCDATE                              - forcing start date
&SWANOutputControl SWAN_OutputHS=False, SWAN_OutputDIR=False, SWAN_OutputTM01=False, SWAN_OutputTPS=False, SWAN_OutputWIND=False, SWAN_OutputTM02=False, SWAN_OutputTMM10=False \

and the difference between the coldstart and hotstart fort.15s:

diff spinup/fort.15 runs/unperturbed/fort.15
1c1
< created on 2021-05-03 18:38                                     ! RUNDES                              - 32 CHARACTER ALPHANUMERIC RUN DESCRIPTION
---
> created on 2021-05-03 18:39                                     ! RUNDES                              - 32 CHARACTER ALPHANUMERIC RUN DESCRIPTION
6c6
< 0                                                               ! IHOT                                - HOT START PARAMETER
---
> 567                                                             ! IHOT                                - HOT START PARAMETER
21,22c21,22
< 0                                                               ! NWS                                 - WIND STRESS AND BAROMETRIC PRESSURE OPTION PARAMETER
< 1                                                               ! NRAMP                               - RAMP FUNCTION OPTION
---
> 520                                                             ! NWS                                 - WIND STRESS AND BAROMETRIC PRESSURE OPTION PARAMETER
> 8                                                               ! NRAMP                               - RAMP FUNCTION OPTION
28,29c28,30
< 12.25                                                           ! RNDAY                               - TOTAL LENGTH OF SIMULATION (IN DAYS)
< 12.25                                                           ! DRAMP                               - DURATION OF RAMP FUNCTION (IN DAYS)
---
> 2012 10 22 06 18 0.9 1 2                                        ! WTIMINC RSTIMINC                    - meteorological data time increment, wave forcing increment
> 16.4583                                                         ! RNDAY                               - TOTAL LENGTH OF SIMULATION (IN DAYS)
> 12.250 0.000 0.000 0.000 12.250 12.250 1.000 0.000 12.250       ! DRAMP                               - DURATION OF RAMP FUNCTION (IN DAYS)
4934,4935c4935,4939
< -5 0.000000 12.250000 514                                       ! NOUTGE TOUTSGE TOUTFGE NSPOOLGE     - GLOBAL ELEVATION OUTPUT INFO (UNIT 63)
< -5 0.000000 0.000000 514                                        ! NOUTGV TOUTSGV TOUTFGV NSPOOLGV     - GLOBAL VELOCITY OUTPUT INFO (UNIT 64)
---
> 0 0 0 0                                                         ! NOUTM TOUTSM TOUTFM NSPOOLM         - METEOROLOGICAL STATION OUTPUT INFO (UNITS 71/72)
> 0                                                               ! NSTAM                               - TOTAL NUMBER OF METEOROLOGICAL RECORDING STATIONS
> -5 0.000000 16.458333 514                                       ! NOUTGE TOUTSGE TOUTFGE NSPOOLGE     - GLOBAL ELEVATION OUTPUT INFO (UNIT 63)
> -5 0.000000 16.458333 514                                       ! NOUTGV TOUTSGV TOUTFGV NSPOOLGV     - GLOBAL VELOCITY OUTPUT INFO (UNIT 64)
> 0 0.000000 0.000000 0                                           ! NOUTGM TOUTSGM TOUTFGM NSPOOLGM     - GLOBAL METEOROLOGICAL OUTPUT INFO
4939c4943
< 5 151200                                                        ! NHSTAR NHSINC                       - HOT START FILE GENERATION PARAMETERS
---
> 0 0                                                             ! NHSTAR NHSINC                       - HOT START FILE GENERATION PARAMETERS
4950a4955
> &SWANOutputControl SWAN_OutputHS=False, SWAN_OutputDIR=False, SWAN_OutputTM01=False, SWAN_OutputTPS=False, SWAN_OutputWIND=False, SWAN_OutputTM02=False, SWAN_OutputTMM10=False \
ghost commented 3 years ago

and finally, the contents of configure_adcirc.json:

{
  "adcirc_executable_path": "/scratch2/COASTAL/coastal/save/shared/repositories/ADC-WW3-NWM-NEMS/ADCIRC/work/padcirc",
  "adcprep_executable_path": "/scratch2/COASTAL/coastal/save/shared/repositories/ADC-WW3-NWM-NEMS/ADCIRC/work/adcprep",
  "modeled_start_time": "2012-10-22 06:00:00",
  "modeled_end_time": "2012-10-26 11:00:00",
  "modeled_timestep": 7.0,
  "fort_13_path": "/scratch2/COASTAL/coastal/save/shared/models/meshes/hsofs/120m/Subsetted_Florence2018_Test/fort.13",
  "fort_14_path": "/scratch2/COASTAL/coastal/save/shared/models/meshes/hsofs/120m/Subsetted_Florence2018_Test/fort.14",
  "tidal_spinup_duration": 1058400.0,
  "tidal_spinup_timestep": 7.0,
  "source_filename": "/scratch2/COASTAL/coastal/save/shared/repositories/ADC-WW3-NWM-NEMS/modulefiles/envmodules_intel.hera",
  "use_original_mesh": false,
  "output_surface": true,
  "surface_output_interval": 3600.0,
  "output_stations": false,
  "stations_file_path": null,
  "stations_output_interval": 360.0,
  "output_spinup": true,
  "output_elevations": true,
  "output_velocities": true,
  "output_concentrations": false,
  "output_meteorological_factors": false,
  "processors": 600,
  "nems_parameters": {},
  "attributes": {
    "predictor_corrector": null,
    "RUNDES": null,
    "_IHOT": null,
    "RUNID": null,
    "NFOVER": null,
    "WarnElev": null,
    "iWarnElevDump": null,
    "WarnElevDumpLimit": null,
    "ErrorElev": null,
    "NABOUT": null,
    "NSCREEN": null,
    "IDEN": null,
    "NOLIBF": null,
    "NOLICA": null,
    "NOLICAT": null,
    "NCOR": null,
    "NTIP": null,
    "G": null,
    "TAU0": null,
    "DTDP": null,
    "STATIM": null,
    "REFTIM": null,
    "DRAMP": null,
    "DRAMPExtFlux": null,
    "FluxSettlingTime": null,
    "DRAMPIntFlux": null,
    "DRAMPElev": null,
    "DRAMPTip": null,
    "DRAMPMete": null,
    "DRAMPWRad": null,
    "DUnRampMete": null,
    "H0": null,
    "NODEDRYMIN": null,
    "NODEWETRMP": null,
    "VELMIN": null,
    "SLAM0": null,
    "SFEA0": null,
    "FFACTOR": null,
    "CF": null,
    "HBREAK": null,
    "FTHETA": null,
    "FGAMMA": null,
    "ESLM": null,
    "NOUTGE": null,
    "TOUTSGE": null,
    "TOUTFGE": null,
    "NSPOOLGE": null,
    "NOUTGV": null,
    "TOUTSGV": null,
    "TOUTFGV": null,
    "NSPOOLGV": null,
    "NOUTGM": null,
    "TOUTSGM": null,
    "TOUTFGM": null,
    "NSPOOLGM": null,
    "NOUTGC": null,
    "TOUTSGC": null,
    "TOUTFGC": null,
    "NSPOOLGC": null,
    "CORI": null,
    "ANGINN": null,
    "THAS": null,
    "THAF": null,
    "NHAINC": null,
    "FMV": null,
    "NHSTAR": null,
    "NHSINC": null,
    "ITITER": null,
    "ISLDIA": null,
    "CONVCR": null,
    "ITMAX": null,
    "NCPROJ": null,
    "NCINST": null,
    "NCSOUR": null,
    "NCHIST": null,
    "NCREF": null,
    "NCCOM": null,
    "NCHOST": null,
    "NCCONV": null,
    "NCCOUT": null,
    "vertical_mode": null,
    "lateral_stress_in_gwce": null,
    "lateral_stress_in_gwce_is_symmetrical": null,
    "adcvection_in_gwce": null,
    "lateral_stress_in_momentum": null,
    "lateral_stress_in_momentum_is_symmetrical": null,
    "lateral_stress_in_momentum_method": null,
    "adcvection_in_momentum": null,
    "area_integration_in_momentum": null,
    "baroclinicity": null,
    "gwce_solution_scheme": "semi-implicit",
    "passive_scalar_transport": null,
    "stress_based_3D": null,
    "smagorinsky": true,
    "smagorinsky_coefficient": null,
    "horizontal_mixing_coefficient": null,
    "CFL": null
  }
}
WPringle commented 3 years ago

So your coldstart was run without SWAN and wind forcing I guess?

If I look at this table for NWS: https://wiki.adcirc.org/wiki/NWS#Extended_NWS_with_Ice_.2B_Waves I don't see any option for NWS equal to "520". Maybe supposed to be 320 (for GAHM + SWAN)?

ghost commented 3 years ago

If I look at this table for NWS: https://wiki.adcirc.org/wiki/NWS#Extended_NWS_with_Ice_.2B_Waves I don't see any option for NWS equal to "520". Maybe supposed to be 320 (for GAHM + SWAN)?

Good catch, I forgot that 520 is reserved for NEMS, and I'm not using NEMS in this specific run. Thanks!

ghost commented 3 years ago

currently running Sandy with gwce_solution_scheme="semi-implicit" in

/scratch2/COASTAL/coastal/save/shared/working/zach/adcirc/run_20210507_hsofs_120m_subsetflo_besttrack_sandy

and Florence with gwce_solution_scheme="explicit" in

/scratch2/COASTAL/coastal/save/shared/working/zach/adcirc/run_20210507_hsofs_120m_subsetflo_besttrack_florence
ghost commented 3 years ago
/scratch2/COASTAL/coastal/save/shared/working/zach/adcirc/run_20210507_hsofs_120m_subsetflo_besttrack_sandy/spinup/ADCIRC_SPINUP.err.log
...
...
...
MPI_Abort(159): MPI_Abort(comm=0x18447c30, errorcode=32766) failed
MPI_Abort(97).: Invalid communicator
...
...
...
libc-2.17.so       00002ABC95B6C555  __libc_start_main     Unknown  Unknown
padcirc            000000000040BDA9  Unknown               Unknown  Unknown
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image              PC                Routine            Line        Source
padcirc            00000000006CA8DD  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B6CA37A7630  Unknown               Unknown  Unknown
libmpi.so.12       00002B6CA2795340  MPI_Abort             Unknown  Unknown
libmpifort.so.12.  00002B6CA23A9C5D  mpi_abort             Unknown  Unknown
padcirc            000000000044581B  msg_abort_               1524  messenger.F
padcirc            00000000004AF54E  sizes_mp_make_dir         332  sizes.F
padcirc            00000000004B91C7  adcirc_mod_mp_adc         180  adcirc.F
padcirc            000000000061A8D2  MAIN__                     18  driver.F
padcirc            000000000040BE9E  Unknown               Unknown  Unknown
libc-2.17.so       00002B6CA3CD8555  __libc_start_main     Unknown  Unknown
padcirc            000000000040BDA9  Unknown               Unknown  Unknown
slurmstepd: error: *** STEP 18641261.0 ON h35m02 CANCELLED AT 2021-05-07T19:27:22 ***
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
slurmstepd: error: _file_write_content: unable to write 6 bytes to cgroup /sys/fs/cgroup/freezer/slurm/uid_20644/job_18641261/step_0/freezer.state: No such device
srun: error: h35m20: tasks 440-444,446-479: Killed
srun: launch/slurm: _step_signal: Terminating StepId=18641261.0
srun: error: h35m05: tasks 80-86,88-119: Killed
srun: error: h35m19: tasks 400-417,419-439: Killed
srun: error: h35m07: tasks 160-180,182-199: Killed
srun: error: h35m06: tasks 120-151,153,155-159: Killed
srun: error: h35m02: tasks 0,2-39: Killed
srun: error: h35m22: tasks 520-529,531-551,553-559: Killed
srun: error: h35m15: tasks 280-294,296-300,302,304-319: Killed
srun: error: h35m04: tasks 40-53,55-57,59-70,72-79: Killed
srun: error: h35m18: tasks 360-364,366-380,382-399: Killed
srun: error: h35m10: tasks 200-215,217-239: Killed
srun: error: h35m24: tasks 560-565,567,569-578,580-599: Killed
srun: error: h35m20: task 445: Exited with exit code 174
srun: error: h35m05: task 87: Exited with exit code 174
srun: error: h35m19: task 418: Exited with exit code 174
srun: error: h35m07: task 181: Exited with exit code 174
srun: error: h35m06: tasks 152,154: Exited with exit code 174
srun: error: h35m02: task 1: Exited with exit code 174
srun: error: h35m22: tasks 530,552: Exited with exit code 174
srun: error: h35m15: tasks 295,301,303: Exited with exit code 174
srun: error: h35m04: tasks 54,58,71: Exited with exit code 174
srun: error: h35m18: tasks 365,381: Exited with exit code 174
srun: error: h35m10: task 216: Exited with exit code 174
srun: error: h35m24: tasks 566,568,579: Exited with exit code 174
srun: error: h35m17: tasks 320-338,340-359: Killed
srun: error: h35m17: task 339: Exited with exit code 174
srun: error: h35m14: tasks 240-279: Killed
srun: error: h35m21: tasks 480-519: Killed
WPringle commented 3 years ago

I think this looks like it is related to the prepped PEs not equaling the requested processors for padcirc

ghost commented 3 years ago

alright, I seem to have found the problem. I was submitting the jobs like this

sbatch adcirc.job --dependency=afterok:$setup_jobid:$spinup_jobid

however, the dependency was not actually being applied, so the hotstart would start immediately without waiting for the spinup OR adcprep to complete. This aligns with the errors that we were seeing in hotstart.

After some testing, I figured out that it works if I write it like this instead:

sbatch --dependency=afterok:$setup_jobid:$spinup_jobid adcirc.job

so I've fixed the script generation and am testing the run now.

ghost commented 3 years ago

it looks like the hotstart run completed,

/scratch2/COASTAL/coastal/save/shared/working/zach/adcirc/run_20210510_hsofs_120m_subsetflo_besttrack_florence/runs/unperturbed

image

but with a few elevation warnings:

/scratch2/COASTAL/coastal/save/shared/working/zach/adcirc/run_20210510_hsofs_120m_subsetflo_besttrack_florence/runs/unperturbed/ADCIRC_HOTSTART.err.log
 TIME STEP =  188082     ITERATIONS =   17     TIME =  0.13165740E+07
  ELMAX =  2.0901E+001 AT NODE    46732  SPEEDMAX =  5.9555E+000 AT NODE    47917  ON MYPROC =    4   ** WARNING: Elevation.gt.WarnElev **
 TIME STEP =  188084     ITERATIONS =   17     TIME =  0.13165880E+07
  ELMAX =  2.2019E+001 AT NODE    46732  SPEEDMAX =  5.2908E+000 AT NODE    47917  ON MYPROC =    4   ** WARNING: Elevation.gt.WarnElev **
 TIME STEP =  188885     ITERATIONS =   17     TIME =  0.13221950E+07
  ELMAX =  2.2710E+001 AT NODE   208899  SPEEDMAX =  4.5079E+000 AT NODE  -220561  ON MYPROC =  248   ** WARNING: Elevation.gt.WarnElev **
 TIME STEP =  188886     ITERATIONS =   17     TIME =  0.13222020E+07
  ELMAX =  2.1132E+001 AT NODE   208899  SPEEDMAX =  7.0585E+000 AT NODE   207072  ON MYPROC =  248   ** WARNING: Elevation.gt.WarnElev **
 TIME STEP =  188887     ITERATIONS =   17     TIME =  0.13222090E+07
  ELMAX =  2.4033E+001 AT NODE   208899  SPEEDMAX =  7.0544E+000 AT NODE   207072  ON MYPROC =  248   ** WARNING: Elevation.gt.WarnElev **
 TIME STEP =  188889     ITERATIONS =   17     TIME =  0.13222230E+07
  ELMAX =  2.1556E+001 AT NODE   207072  SPEEDMAX =  6.8517E+000 AT NODE   207072  ON MYPROC =  248   ** WARNING: Elevation.gt.WarnElev **
 TIME STEP =  189691     ITERATIONS =   17     TIME =  0.13278370E+07
  ELMAX =  2.1822E+001 AT NODE   135311  SPEEDMAX =  1.4178E+000 AT NODE   148766  ON MYPROC =  259   ** WARNING: Elevation.gt.WarnElev **
 TIME STEP =  189692     ITERATIONS =   17     TIME =  0.13278440E+07
  ELMAX =  2.0661E+001 AT NODE   135311  SPEEDMAX =  5.1043E+000 AT NODE   133657  ON MYPROC =  259   ** WARNING: Elevation.gt.WarnElev **
 TIME STEP =  189693     ITERATIONS =   17     TIME =  0.13278510E+07
  ELMAX =  2.5124E+001 AT NODE   135311  SPEEDMAX =  9.9263E-001 AT NODE   152227  ON MYPROC =  259   ** WARNING: Elevation.gt.WarnElev **
 TIME STEP =  189694     ITERATIONS =   17     TIME =  0.13278580E+07
  ELMAX =  2.2977E+001 AT NODE   135311  SPEEDMAX =  5.6609E+000 AT NODE   133657  ON MYPROC =  259   ** WARNING: Elevation.gt.WarnElev **
 TIME STEP =  189695     ITERATIONS =   17     TIME =  0.13278650E+07
  ELMAX =  2.8320E+001 AT NODE   135311  SPEEDMAX =  1.9715E+000 AT NODE   136955  ON MYPROC =  259   ** WARNING: Elevation.gt.WarnElev **
 TIME STEP =  189696     ITERATIONS =   17     TIME =  0.13278720E+07
  ELMAX =  2.5513E+001 AT NODE   135311  SPEEDMAX =  6.2510E+000 AT NODE   133657  ON MYPROC =  259   ** WARNING: Elevation.gt.WarnElev **
 TIME STEP =  189697     ITERATIONS =   17     TIME =  0.13278790E+07
  ELMAX =  3.2227E+001 AT NODE   135311  SPEEDMAX =  1.0183E+000 AT NODE   135310  ON MYPROC =  259   ** WARNING: Elevation.gt.WarnElev **
 TIME STEP =  189698     ITERATIONS =   17     TIME =  0.13278860E+07
  ELMAX =  2.8683E+001 AT NODE   135311  SPEEDMAX =  6.9802E+000 AT NODE   133657  ON MYPROC =  259   ** WARNING: Elevation.gt.WarnElev **
 TIME STEP =  189699     ITERATIONS =   17     TIME =  0.13278930E+07
  ELMAX =  3.6274E+001 AT NODE   135311  SPEEDMAX =  5.0500E+000 AT NODE   136956  ON MYPROC =  259   ** WARNING: Elevation.gt.WarnElev **
 TIME STEP =  189700     ITERATIONS =   17     TIME =  0.13279000E+07
  ELMAX =  2.6005E+001 AT NODE   135311  SPEEDMAX =  5.0487E+000 AT NODE   136956  ON MYPROC =  259   ** WARNING: Elevation.gt.WarnElev **
 TIME STEP =  189701     ITERATIONS =   17     TIME =  0.13279070E+07
  ELMAX =  2.2789E+001 AT NODE   135311  SPEEDMAX =  5.6019E+000 AT NODE   133657  ON MYPROC =  259   ** WARNING: Elevation.gt.WarnElev **
 TIME STEP =  189702     ITERATIONS =   17     TIME =  0.13279140E+07
  ELMAX =  2.3334E+001 AT NODE   135311  SPEEDMAX =  5.5986E+000 AT NODE   133657  ON MYPROC =  259   ** WARNING: Elevation.gt.WarnElev **
ghost commented 3 years ago

here's a run from today, with station data. when running with 15 day spinup, the phase seems to be correct but with shifted amplitude

spinup

/scratch2/COASTAL/coastal/save/shared/working/zach/adcirc/run_20210512_hsofs_120m_subsetflo_besttrack_florence/spinup

should the spinup have a starting amplitude? 8775870 2695540 8577330 8735391

hotstart

/scratch2/COASTAL/coastal/save/shared/working/zach/adcirc/run_20210512_hsofs_120m_subsetflo_besttrack_florence/runs/unperturbed

8447386 8654467 8771013 8652587

jreniel commented 3 years ago

No, that's exactly what the spinup is, we gradually increase the amplitude in order to avoid shocking the system.

ghost commented 3 years ago

alright, I'll try for a longer spinup to see if that makes it reach the correct zeta

jreniel commented 3 years ago

alright, I'll try for a longer spinup to see if that makes it reach the correct zeta

That won't be necessary, the bias difference you see will not be resolved by a longer spinup. We need to enable pseudo-pressure loading to fix this.

jreniel commented 3 years ago

alright, I'll try for a longer spinup to see if that makes it reach the correct zeta

@zacharyburnettNOAA Please use NAVD88 as input for the plot_fort61 command, these are referenced to MLLW, and that accounts for the largest amount of difference you see.

WPringle commented 3 years ago

@zacharyburnettNOAA I would use MSL for the vertical datum of the plots as that should be closest to ADCIRC one. Alternatively for comparison purposes can just offset ADCIRC by adding the mean of the observation over the time period to it.

jreniel commented 3 years ago

@zacharyburnettNOAA I would use MSL for the vertical datum of the plots as that should be closest to ADCIRC one. Alternatively for comparison purposes can just offset ADCIRC by adding the mean of the observation over the time period to it.

To be more precise, which one to chose depends on the CRS of the mesh, but MSL is not a CRS. Meshes should be interpolated to the geoid surface. NAVD88 is the closest approximation for the geoid in North America. However some meshes are interpolated @ MSL, such as the HSOFS one, so it's fair to use MSL in such cases. I did not go into this level of detail on my previous posts, because I was thinking we would gradually reach this point of discussion.

WPringle commented 3 years ago

That's true the bathymetry should ideally be on a geoid. However, the ADCIRC model we are simulating is a 2D barotropic model and therefore it cannot correctly compute the sea surface height relative to the geoid. Therefore, as an approximation in my opinion for 2D barotropic models it is just better to make the mesh on the "MSL" to account for variations in the sea surface height the model cannot capture. So the HSOFS mesh took the correct route in my opinion, and they provided a correction file to correct back to NAVD88 when needed. Alternatively, if the domain is small enough we provide input boundary condition that accounts for the offset from the geoid to the MSL.

I suppose that another alternative is to apply the psuedo-pressure forcing as you mentioned to account for the discrepancy between the geoid and the MSL (basically this is like accounting for baroclinic pressure gradients missing from 2D barotropic model, see https://doi.org/10.1029/2018JC014682).

In any case, for 2D barotropic model we almost always run into problem of aligning the model with the observations because the model cannot properly sustain any real vertical datum. Therefore, often we just add the mean from observations to the model.

jreniel commented 3 years ago

@WPringle I agree with you in general. I think we are on the same page, no issues. @zacharyburnettNOAA Bottom line is we need to plot using MSL for HSOFS.

jreniel commented 3 years ago

A relevant reference: https://www.sciencedirect.com/topics/earth-and-planetary-sciences/sea-surface-topography

jreniel commented 3 years ago

@zacharyburnettNOAA From https://user-images.githubusercontent.com/52422935/117993696-f584cd00-b30d-11eb-9221-f49bb3a4610f.png, it looks like there's some storm surge going on during the spinup phase. Are you certain that the dates are correct? All storm surge signals should be happening on the hotstart phase. I would also suggest to find stations in the hotstart phase that are showing storm surge signals in order to showcase your results.

EDIT: The signal in Dog River Brigde is far-away from landfall, so that's an independent signal. During further conversation with @zacharyburnettNOAA, the plots will be updated at some point to have a better view of storm surge signals of the modeled event.

ghost commented 3 years ago

here are the hotstart zetas of stations within 150km of the track: 8656483 8658120 8658163 8661070 8665530

jreniel commented 3 years ago

here are the hotstart zetas of stations within 150km of the track: @zacharyburnettNOAA Excellent work!