One eigenvalue for ensemble of perturbed tracks on Hercules

[FariborzDaneshvar-NOAA]

The K-L expansion is returning only one eigenvalue for nodes from 30 ensembles. This is the case for all 6 leadtimes of three storms (Sandy, Michael, and Harvey) that I ran recently on Hercules! so there should be something with the workflow on Hercules. @SorooshMani-NOAA is there any part of the workflow on Hercules different from what we had on PW? what's your take on this?

[FariborzDaneshvar-NOAA]

Here some example results for Sandy	Sandy, 12hr leadtime	`Sandy, 72hr leadtime

[SorooshMani-NOAA]

Just for the sake of documentation: As we discussed this is due to an issue with setup (there's no track!). It stems from not having internet access. I'm looking into modifying the prep code to avoid the internet requirement.

[FariborzDaneshvar-NOAA]

As @SorooshMani-NOAA noted, the workflow was not using atmospheric forcing. Thanks to Soroosh for fixing this bug, I re-ran the workflow with updated prep.sif image. But still getting one eigenvalue!

Here are comparison of old and new tracks for some scenarios:	old tracks	new tracks
12-hr	12-hr
24-hr	24-hr

[FariborzDaneshvar-NOAA]

@SorooshMani-NOAA Here are pressure fields for some timesteps of the original track of Sandy with 12hr leadtime, suggesting that there is an issue with the hurricane model!

October 24, 2012, at 19:00

October 23, 2012, at 18:00

October 22, 2012, at 23:00

Path to the output on Hercules: /work2/noaa/nos-surge/shared/nhc_hurricanes/sandy_2012_OFCL_12hr_korobov_30/setup/ensemble.dir/runs/original/outputs/out2d_1.nc

[SorooshMani-NOAA]

@FariborzDaneshvar-NOAA I noticed I get a bunch of these warnings:

 No nodal points found inside the radius of the last closed isobar (rrp = 0.000 m)

in a test run for Florence. Do you see the same/similar thing in your Sandy runs? It should be in outputs/mirror.out. I'm not sure if that means we have to modify the track or there's something in the PaHM settings we need to fix.

Update I checked your mirror.out I see the same warning, but not for all timesteps. But still I see something that looks strange:

 TIME STEP=         6623;  TIME=        993450.000000
  before GetGAHMFields
  Working on time frame:    993600.000000000
  Requested output time:    993600.000000000
  , skipping generating data for this time;  0.000000000000000E+000
    21600.0000000000        43200.0000000000        64800.0000000000
    86400.0000000000        108000.000000000        129600.000000000
    151200.000000000        172800.000000000        194400.000000000
    216000.000000000        237600.000000000        259200.000000000
    280800.000000000        302400.000000000        324000.000000000
    345600.000000000        367200.000000000        388800.000000000
    410400.000000000        432000.000000000        453600.000000000
    475200.000000000        496800.000000000        518400.000000000
    540000.000000000        561600.000000000        583200.000000000
    604800.000000000        626400.000000000        637200.000000000
    669600.000000000        712800.000000000        756000.000000000
    799200.000000000        885600.000000000        972000.000000000
  after GetGAHMFields

Is it skipping generating wind?! Is the issue the timestep settings? We need to ask @pvelissariou1

[pvelissariou1]

@FariborzDaneshvar-NOAA I think we have discussed this issue before. I sent the workaround for missing RRP fields by commenting out a couple of if blocks in the PaHM code contained in SCHISM. I think Soroosh implemented it. I'll push the updated modifications to SCHISM. If the hurricane is way outside the domain at the specific time you will get these warnings as wind fields are not calculated in these cases.

[SorooshMani-NOAA]

@pvelissariou1 I see, I just remembered you had it in a dev branch. I didn't realized it's still not in the main SCHISM code? When do you think you'll be able to merge the code?

[SorooshMani-NOAA]

In any case, I recompiled using the change I got from Takis into my schism fork. I'm now retesting the binaries on Hercules to make sure we get the wind correctly.

[FariborzDaneshvar-NOAA]

@SorooshMani-NOAA @pvelissariou1 Thanks for your efforts. The issue is been resolved! PCA is working as expected! Here are example results for Sandy with 48hr leadtime:

I will close this ticket then.

[pvelissariou1]

@FariborzDaneshvar-NOAA , @SorooshMani-NOAA I have updated PaHM in SCHISM and pushed the changes in the cmmb branch: git clone https://github.com/schism-dev/schism.git -b cmmb This branch is already synchronized with master and it is up to date. Could you please try it and see if it compiles and runs as expected? In my side I saw no issues. After confirmation of successful runs, I will create a PR.

[SorooshMani-NOAA]

@pvelissariou1 thank you. I will recompile binaries using the cmmb branch and run to see what happens. If you don't have a separate ticket I'll close this again after testing your branch.

[SorooshMani-NOAA]

@pvelissariou1 I have issues compiling the cmmb branch. I'll follow up with you on chat and then summarize findings here

[SorooshMani-NOAA]

The cmmb branch can be compiled using BLD_STANDALONE=ON option when running cmake.