search

dtcenter / MET

Model Evaluation Tools
https://dtcenter.org/community-code/model-evaluation-tools-met
Apache License 2.0
77 stars 24 forks source link

Refine tc_gen logic to compare forecast genesis events to all BEST track points. #1448

Closed JohnHalleyGotway closed 3 years ago

JohnHalleyGotway commented 4 years ago

Describe the Enhancement

While testing tc_gen in the met-9.1_beta3 development release, Dan Halperin identified a short-coming in its logic. Currently tc_gen identifies genesis events in the forecast, BEST, and operational tracks. Each event is stored as a single location and time. The current matching logic logic compares only those points.

This task is to collaborate with Dan to refine this approach. Replace the current point-to-point comparison with a point-to-track comparison. Compare the forecast genesis event points to the entire set of BEST tracks. This will enable us to...

(1) Check for genesis forecasts from initializations AFTER the actual genesis event in the BEST track. These were previously counted as FALSE ALARMS but should instead be ignored. (2) Implement NHC's operational logic for verifying Tropical Weather Outlooks.

Time Estimate

Estimate the amount of work required here. Issues should represent approximately 1 to 3 days of work.

Sub-Issues

Consider breaking the enhancement down into sub-issues.

Relevant Deadlines

List relevant project deadlines here or state NONE.

Funding Source

7790901 - TC-Gen key

Define the Metadata

Assignee

Labels

Projects and Milestone

Define Related Issue(s)

Consider the impact to the other METplus components.

Enhancement Checklist

See the METplus Workflow for details.

JohnHalleyGotway commented 3 years ago

On 20201204, the team had a telecon: Kathryn, Dan A, Dan H, John HG, and Tara

In met-9.1, TC-Gen defines genesis event points from the ADECK and BDECK data sources. Then it compares those points to populate the contingency table. This task changes that logic. Instead of comparing point to point, we'll now compare forecast genesis event points to all BEST track points.

A PowerPoint from Dan A illustrating the requested refinements is attached. This task is to enhance TC-Gen to support 2 different methodologies for populating contingency tables. Add a configuration file option to control what logic is applied... one or both of them.

tc_gen_algorithm_details.pptx

Need to decide how to distinguish between them in the output. FCST_VAR = OBS_VAR = GENESIS Recommend choosing a different name for the second set of logic.

Kathryn: Make the "discarding" of genesis events based on initialization time be a configurable option with the default being "discard".

Also need to correct the use of the operational CARQ model in the tc-gen logic. The logic should be this...

In this issue, we're refining how to "pair" the track data first, and then "classify" those pairs second.

JohnHalleyGotway commented 3 years ago

Email chain from 12/17/20...

John HG's Question:

The new logic is...

Dan H's answer:

My first thought is to match using the unfiltered BEST and CARQ data, then filter out the events based on the config file. If we filter the BEST and CARQ data before matching, then we may end up with unwarranted false alarms.

For example, let's say there's a GFS forecast genesis event that matches to BEST storm AL032020. If the config file options result in filtering out AL032020 and the BEST and CARQ filtering occurs first, then the GFS forecast would end up incorrectly being classified as a false alarm because no BEST or CARQ match was found.

If the BEST and CARQ filtering occurs after the matching, then the forecast would be discarded.

I think we should filter the BEST data before calculating the misses.

JohnHalleyGotway commented 3 years ago

We no longer need operational genesis definition criteria in the TC-Gen config file. So DELETE...

oper_genesis = {
   technique   = "CARQ";
   category    = [ "TD", "TS" ];
   vmax_thresh = NA;
   mslp_thresh = NA;
}

But we do still need to know the name of the operational technique. So ADD...

oper_technique = "CARQ";
JohnHalleyGotway commented 3 years ago

Let's say we've found a matching BEST track. But when we search that BEST track, no points actually meet the genesis criteria because the user has setup the TC-Gen config file in a stringent way. Is that just a false alarm... a genesis forecast for which there is no qualifying BEST track genesis event? That would be the simplest way of handling it in the current logic.

halperin-erau commented 3 years ago

Yes, that should be a false alarm. There are practical scenarios when that would happen. For example, let's say that the config file has

category = [ "TD", "TS"];

and there is a GFS genesis forecast that matches a BEST entry for AL012019. However, AL012019 was a subtropical storm (category "SS") and therefore not a BEST genesis event as defined by the config file. So, that GFS genesis forecast would be a false alarm.

Dan

From: johnhg notifications@github.com Sent: Friday, December 18, 2020 5:19 PM To: dtcenter/MET MET@noreply.github.com Cc: Halperin, Daniel J. HALPEA17@erau.edu; Assign assign@noreply.github.com Subject: [EXTERNAL] Re: [dtcenter/MET] Refine tc_gen logic to compare forecast genesis events to all BEST track points. (#1448)

Let's say we've found a matching BEST track. But when we search that BEST track, no points actually meet the genesis criteria because the user has setup the TC-Gen config file in a stringent way. Is that just a false alarm... a genesis forecast for which there is no qualifying BEST track genesis event?

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHubhttps://github.com/dtcenter/MET/issues/1448#issuecomment-748346932, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AI3GTCOKAKJKPW67QTLSRSTSVPIOHANCNFSM4PYBRBWA.

halperin-erau commented 3 years ago

You're correct that we will no longer need that option to match a forecast to BEST or CARQ since we're looking for an exact match between model forecast genesis (valid) time and BEST/CARQ time.

However, we will need a similar option (or options?) to define the temporal window for what constitutes a hit vs false alarm. For verification technique 1, we would be checking whether the model forecast genesis (valid) time and the BEST genesis time are within the user-defined X hours of each other. For verification technique 2, we would be checking whether the BEST genesis time occurs within the user-defined X hours after the model initialization time.

Dan

From: johnhg notifications@github.com Sent: Friday, December 18, 2020 5:39 PM To: dtcenter/MET MET@noreply.github.com Cc: Halperin, Daniel J. HALPEA17@erau.edu; Assign assign@noreply.github.com Subject: [EXTERNAL] Re: [dtcenter/MET] Refine tc_gen logic to compare forecast genesis events to all BEST track points. (#1448)

Great, thanks Dan! Next question. We currently have the following TC-Gen config entry:

// // Genesis matching time window, in hours relative to the forecast genesis time // genesis_window = { beg = -24; end = 24; }

Am I correct in thinking that with the updated logic, this no longer applies?

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHubhttps://github.com/dtcenter/MET/issues/1448#issuecomment-748353384, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AI3GTCMD4I4KWPVLTBZMV2DSVPKZFANCNFSM4PYBRBWA.

JohnHalleyGotway commented 3 years ago

I have a question about Otto from 2016. It crossed from the AL to EP basin. It was AL162016 and then EP222016. The tc_gen tool currently counts the genesis event twice when it process two best track data files (bal162016.dat and bep222016.dat) but prints a warning message about it.

Screen Shot 2020-12-28 at 10 40 59 AM

GenesisInfo: IsSet = true, IsBest = true, IsOper = false, IsAnly = false, StormId = "EP222016", Basin = "EP", Cyclone = "22", StormName = "OTTO", TechniqueNumber = -9999, Technique = "BEST", Initials = "(nul)", GenesisTime = "20161120_180000", InitTime = "NA", LeadTime = "000000", Lat = 11.10000, Lon = -79.70000, DLand = 90.77083, NPoints = 38, MinValidTime = "20161117_180000", MaxValidTime = "20161126_120000", MinWarmCoreTime = "NA", MaxWarmCoreTime = "NA"

GenesisInfo: IsSet = true, IsBest = true, IsOper = false, IsAnly = false, StormId = "AL162016", Basin = "AL", Cyclone = "16", StormName = "OTTO", TechniqueNumber = -9999, Technique = "BEST", Initials = "(nul)", GenesisTime = "20161120_180000", InitTime = "NA", LeadTime = "000000", Lat = 11.10000, Lon = -79.70000, DLand = 90.77083, NPoints = 38, MinValidTime = "20161117_180000", MaxValidTime = "20161126_120000", MinWarmCoreTime = "NA", MaxWarmCoreTime = "NA"

I assume that I should update tc_gen to... (1) When a "duplicate" genesis is encountered, do NOT add it a second time. (2) Change the log message from a Warning to Debug(3).

Please let me know if my assumptions are correct.

halperin-erau commented 3 years ago

Hi John,

Your assumptions sound reasonable to me.

Thanks, Dan

From: johnhg notifications@github.com Sent: Monday, December 28, 2020 12:46 PM To: dtcenter/MET MET@noreply.github.com Cc: Halperin, Daniel J. HALPEA17@erau.edu; Assign assign@noreply.github.com Subject: [EXTERNAL] Re: [dtcenter/MET] Refine tc_gen logic to compare forecast genesis events to all BEST track points. (#1448)

I have a question about Otto from 2016. It crossed from the AL to EP basin. It was AL162016 and then EP222016. The tc_gen tool currently counts the genesis event twice when it process two best track data files (bal162016.dat and bep222016.dat) but prints a warning message about it.

[Screen Shot 2020-12-28 at 10 40 59 AM]https://user-images.githubusercontent.com/21087144/103233086-5ed4e800-48f9-11eb-9bb6-e5e3319d3027.png

GenesisInfo: IsSet = true, IsBest = true, IsOper = false, IsAnly = false, StormId = "EP222016", Basin = "EP", Cyclone = "22", StormName = "OTTO", TechniqueNumber = -9999, Technique = "BEST", Initials = "(nul)", GenesisTime = "20161120_180000", InitTime = "NA", LeadTime = "000000", Lat = 11.10000, Lon = -79.70000, DLand = 90.77083, NPoints = 38, MinValidTime = "20161117_180000", MaxValidTime = "20161126_120000", MinWarmCoreTime = "NA", MaxWarmCoreTime = "NA"

GenesisInfo: IsSet = true, IsBest = true, IsOper = false, IsAnly = false, StormId = "AL162016", Basin = "AL", Cyclone = "16", StormName = "OTTO", TechniqueNumber = -9999, Technique = "BEST", Initials = "(nul)", GenesisTime = "20161120_180000", InitTime = "NA", LeadTime = "000000", Lat = 11.10000, Lon = -79.70000, DLand = 90.77083, NPoints = 38, MinValidTime = "20161117_180000", MaxValidTime = "20161126_120000", MinWarmCoreTime = "NA", MaxWarmCoreTime = "NA"

I assume that I should update tc_gen to... (1) When a "duplicate" genesis is encountered, do NOT add it a second time. (2) Change the log message from a Warning to Debug(3).

Please let me know if my assumptions are correct.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHubhttps://github.com/dtcenter/MET/issues/1448#issuecomment-751804727, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AI3GTCJYPWJYYDFUZVFOSILSXC755ANCNFSM4PYBRBWA.

JohnHalleyGotway commented 3 years ago

Great, thanks. Next question. The updated version of tc_gen reports this case as a HIT:

DEBUG 4: SUI1 20160927_000000 initialization, 30 lead, 20160928_060000 genesis forecast at (13.40000, -57.30000) and BEST track genesis for AL142016 at (13.40000, -59.80000) is a technique 1 HIT.

But it also counts the following as a MISSES:

DEBUG 4: SUI1 20160927_000000 initialization, 36 lead, 20160928_120000 BEST track genesis for AL142016 at (13.4, -59.8) is a technique 1 and 2 MISS.
DEBUG 4: SUI1 20160927_060000 initialization, 30 lead, 20160928_120000 BEST track genesis for AL142016 at (13.4, -59.8) is a technique 1 and 2 MISS.

Note that there's a 6-hour offset, so we don't have any precise matches. It seems like one of those MISSES should be a hit instead, but which one? Unless I hear otherwise, I'll go with the first as being the HIT... so we check the initialization time for a hit, even if the forecast hour doesn't exactly match.

halperin-erau commented 3 years ago

I think we want to count the MISS events based on the model initialization time. For example, if we are verifying forecasts at lead times out to 120 hours and the forecast model is run 4 times per day, then we would have a maximum of 20 misses possible (5 days * 4 initializations/day) for each BEST genesis event.

Based on the example you provided, because there was a HIT for AL142016 in the 20160927_000000 model initialization, I think that the following case should NOT be considered a MISS:

DEBUG 4: SUI1 20160927_000000 initialization, 36 lead, 20160928_120000 BEST track genesis for AL142016 at (13.4, -59.8) is a technique 1 and 2 MISS.

From: johnhg notifications@github.com Sent: Monday, December 28, 2020 5:16 PM To: dtcenter/MET MET@noreply.github.com Cc: Halperin, Daniel J. HALPEA17@erau.edu; Assign assign@noreply.github.com Subject: [EXTERNAL] Re: [dtcenter/MET] Refine tc_gen logic to compare forecast genesis events to all BEST track points. (#1448)

Great, thanks. Next question. The updated version of tc_gen reports this case as a HIT:

DEBUG 4: SUI1 20160927_000000 initialization, 30 lead, 20160928_060000 genesis forecast at (13.40000, -57.30000) and BEST track genesis for AL142016 at (13.40000, -59.80000) is a technique 1 HIT.

But it also counts the following as a MISSES:

DEBUG 4: SUI1 20160927_000000 initialization, 36 lead, 20160928_120000 BEST track genesis for AL142016 at (13.4, -59.8) is a technique 1 and 2 MISS. DEBUG 4: SUI1 20160927_060000 initialization, 30 lead, 20160928_120000 BEST track genesis for AL142016 at (13.4, -59.8) is a technique 1 and 2 MISS.

Note that there's a 6-hour offset, so we don't have any precise matches. It seems like one of those MISSES should be a hit instead, but which one?

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHubhttps://github.com/dtcenter/MET/issues/1448#issuecomment-751880942, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AI3GTCJXY3CGTROWRMNRHR3SXD7UDANCNFSM4PYBRBWA.

JohnHalleyGotway commented 3 years ago

Well I have all the code changes I think are needed for this updated logic, but I'm getting pretty different results! Running the old and new logic on 2016 AL storms using default config file settings, here's the results: In met-9.1...

DEBUG 3: For SUI1 model, contingency table hits = 10, false alarms = 493, and misses = 286.
# Here are the 10 HITS:
DEBUG 4: SUI1 20160525_120000 initialization, 51 lead, 20160527_150000 genesis at (28.4, -73.6) is a HIT for BEST 20160527_180000 genesis at (28.3, -74.4).
DEBUG 4: SUI1 20160813_120000 initialization, 36 lead, 20160815_000000 genesis at (8.5, -23.5) is a HIT for CARQ 20160815_120000 genesis at (9.2, -23.7).
DEBUG 4: SUI1 20160818_000000 initialization, 66 lead, 20160820_180000 genesis at (11.6, -16.3) is a HIT for CARQ 20160820_180000 genesis at (12.4, -15.8).
DEBUG 4: SUI1 20160912_120000 initialization, 33 lead, 20160913_210000 genesis at (12.2, -21.1) is a HIT for CARQ 20160913_060000 genesis at (13, -20).
DEBUG 4: SUI1 20160912_120000 initialization, 117 lead, 20160917_090000 genesis at (14.4, -20) is a HIT for CARQ 20160918_000000 genesis at (12.5, -20.9).
DEBUG 4: SUI1 20160917_000000 initialization, 33 lead, 20160918_090000 genesis at (11.9, -23.6) is a HIT for CARQ 20160918_120000 genesis at (12.7, -22.9).
DEBUG 4: SUI1 20160922_120000 initialization, 93 lead, 20160926_090000 genesis at (10.6, -39.6) is a HIT for CARQ 20160925_180000 genesis at (8.1, -39.4).
DEBUG 4: SUI1 20160927_000000 initialization, 30 lead, 20160928_060000 genesis at (13.4, -57.3) is a HIT for BEST 20160928_120000 genesis at (13.4, -59.8).
DEBUG 4: SUI1 20161002_000000 initialization, 36 lead, 20161003_120000 genesis at (22.9, -59.9) is a HIT for BEST 20161004_060000 genesis at (23.2, -59.8).
DEBUG 4: SUI1 20161111_120000 initialization, 93 lead, 20161115_090000 genesis at (10.9, -77.9) is a HIT for CARQ 20161115_120000 genesis at (11.6, -77.6).

Updated version...

DEBUG 3: For filter 1 (AL_BASIN) SUI1 model, technique 1 contingency table hits = 2, false alarms = 501, and misses = 269.
# Here are the 2 HITS:
DEBUG 4: SUI1 20160927_000000 initialization, 36 lead, 20160928_120000 BEST track AL142016 genesis at (13.40000, -59.80000) and 20160928_060000 forecast genesis at (13.40000, -57.30000) is a technique 1 HIT with a genesis time offset of 6 hours and location offset of 270.72126 km.
DEBUG 4: SUI1 20161002_000000 initialization, 54 lead, 20161004_060000 BEST track AL152016 genesis at (23.20000, -59.80000) and 20161003_120000 forecast genesis at (22.90000, -59.90000) is a technique 1 HIT with a genesis time offset of 18 hours and location offset of 34.93146 km.

So 10 hits went down to 2. Of the 10 HITS, 4 had genesis times of 00, 06, 12, or 18 while 6 had genesis times of 03, 09, 15, or 21. Those non-6 hourly times have 0 chance of matching since the BEST and CARQ track points are only 6-hourly!

So what should we do? Should we only analyze 6-hourly forecast track points instead of including the 3-hourly ones? So any forecast genesis at 03, 09, 15, or 21 would most likely be shifted 3 hours forward.

halperin-erau commented 3 years ago

There's always some scenario that I fail to consider! I think we have two options here:

  1. Only consider the forecast track points with forecast valid times at 00, 06, 12, and 18 UTC.
  2. Interpolate the BEST and CARQ positions to generate 3-hourly tracks.

I think both are reasonable options (with pros and cons to each), so I suggest we choose whichever is the most straightforward to implement.

I'll look at these results in greater detail to confirm, but I suspect that the 2 cases with forecast genesis at 00, 06, 12, or 18 UTC that were HITS in v9.1 but FALSE ALARMS now occurred because we were using a CARQ entry as the obs genesis time and location before the system actually developed as a TD or TS in the BEST track. With the refined logic, the obs genesis time and location always come from the BEST track.

From: johnhg notifications@github.com Sent: Monday, December 28, 2020 9:19 PM To: dtcenter/MET MET@noreply.github.com Cc: Halperin, Daniel J. HALPEA17@erau.edu; Assign assign@noreply.github.com Subject: [EXTERNAL] Re: [dtcenter/MET] Refine tc_gen logic to compare forecast genesis events to all BEST track points. (#1448)

Well I have all the code changes I think are needed for this updated logic, but I'm getting pretty different results! Running the old and new logic on 2016 AL storms using default config file settings, here's the results: In met-9.1...

DEBUG 3: For SUI1 model, contingency table hits = 10, false alarms = 493, and misses = 286.

Here are the 10 HITS:

DEBUG 4: SUI1 20160525_120000 initialization, 51 lead, 20160527_150000 genesis at (28.4, -73.6) is a HIT for BEST 20160527_180000 genesis at (28.3, -74.4). DEBUG 4: SUI1 20160813_120000 initialization, 36 lead, 20160815_000000 genesis at (8.5, -23.5) is a HIT for CARQ 20160815_120000 genesis at (9.2, -23.7). DEBUG 4: SUI1 20160818_000000 initialization, 66 lead, 20160820_180000 genesis at (11.6, -16.3) is a HIT for CARQ 20160820_180000 genesis at (12.4, -15.8). DEBUG 4: SUI1 20160912_120000 initialization, 33 lead, 20160913_210000 genesis at (12.2, -21.1) is a HIT for CARQ 20160913_060000 genesis at (13, -20). DEBUG 4: SUI1 20160912_120000 initialization, 117 lead, 20160917_090000 genesis at (14.4, -20) is a HIT for CARQ 20160918_000000 genesis at (12.5, -20.9). DEBUG 4: SUI1 20160917_000000 initialization, 33 lead, 20160918_090000 genesis at (11.9, -23.6) is a HIT for CARQ 20160918_120000 genesis at (12.7, -22.9). DEBUG 4: SUI1 20160922_120000 initialization, 93 lead, 20160926_090000 genesis at (10.6, -39.6) is a HIT for CARQ 20160925_180000 genesis at (8.1, -39.4). DEBUG 4: SUI1 20160927_000000 initialization, 30 lead, 20160928_060000 genesis at (13.4, -57.3) is a HIT for BEST 20160928_120000 genesis at (13.4, -59.8). DEBUG 4: SUI1 20161002_000000 initialization, 36 lead, 20161003_120000 genesis at (22.9, -59.9) is a HIT for BEST 20161004_060000 genesis at (23.2, -59.8). DEBUG 4: SUI1 20161111_120000 initialization, 93 lead, 20161115_090000 genesis at (10.9, -77.9) is a HIT for CARQ 20161115_120000 genesis at (11.6, -77.6).

Updated version...

DEBUG 3: For filter 1 (AL_BASIN) SUI1 model, technique 1 contingency table hits = 2, false alarms = 501, and misses = 269.

Here are the 2 HITS:

DEBUG 4: SUI1 20160927_000000 initialization, 36 lead, 20160928_120000 BEST track AL142016 genesis at (13.40000, -59.80000) and 20160928_060000 forecast genesis at (13.40000, -57.30000) is a technique 1 HIT with a genesis time offset of 6 hours and location offset of 270.72126 km. DEBUG 4: SUI1 20161002_000000 initialization, 54 lead, 20161004_060000 BEST track AL152016 genesis at (23.20000, -59.80000) and 20161003_120000 forecast genesis at (22.90000, -59.90000) is a technique 1 HIT with a genesis time offset of 18 hours and location offset of 34.93146 km.

So 10 hits went down to 2. Of the 10 HITS, 4 had genesis times of 00, 06, 12, or 18 while 6 had genesis times of 03, 09, 15, or 21. Those non-6 hourly times have 0 chance of matching since the BEST and CARQ track points are only 6-hourly!

So what should we do? Should we only analyze 6-hourly forecast track points instead of including the 3-hourly ones? So any forecast genesis at 03, 09, 15, or 21 would most likely be shifted 3 hours later in time.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHubhttps://github.com/dtcenter/MET/issues/1448#issuecomment-751922372, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AI3GTCOCZU2C2AQUJ5ZGPELSXE4EBANCNFSM4PYBRBWA.

JohnHalleyGotway commented 3 years ago

Definitely easier to only include track points for valid hours 00, 06, 12, and 18 rather than interpolating. So I'll go that route for now. OTTO in 2016 is a pesky little storm. It started in the Atlantic as AL162016 but eventually moved to the East Pac as EP222016. Since the genesis occurred in the Atlantic, I'd like to make tc_gen smart enough to figure that out and discard the EP222016 track and corresponding genesis event entirely. When a storm moves from one basin to another, it seems like we really only care about the track/genesis event for the original basin. Will work on that today.

JohnHalleyGotway commented 3 years ago

Making progress, but still trying to reconcile why 10 previous hits are reduced to 3. For example, in the previous version, we had:

DEBUG 4: SUI1 20160813_120000 initialization, 36 lead, 20160815_000000 genesis at (8.5, -23.5) is a HIT for CARQ 20160815_120000 genesis at (9.2, -23.7).

But the new logic looks for a BEST track match for this time (20160815_000000), but none is present. So it looks for a 0-hour CARQ track point and does find this one:

AL, 06, 2016081500, 01, CARQ,   0,  97N,  204W,  20, 1009, DB,  34, NEQ,    0,    0,    0,    0, 1010,  120,  60,   0,   0,   L,   0,   X, 270,  11,     INVEST, S,

But those locations are 366 km apart, which is >300, so they do NOT MATCH. Increasing the radius to 500km, and then they do match.

JohnHalleyGotway commented 3 years ago

John, consider the following changes:

halperin-erau commented 3 years ago

Hi John,

I looked at the two cases that were HITS in v9.1 and FALSE ALARMS in the new version with forecast valid times at 00, 06, 12, or 18 UTC and I think the refined logic is correct.

Case 1:

Old version:

DEBUG 4: SUI1 20160813_120000 initialization, 36 lead,

20160815_000000 genesis at (8.5, -23.5) is a HIT for CARQ

20160815_120000 genesis at (9.2, -23.7).

New version: FALSE ALARM

As you pointed out, the new logic has no matching BEST entry, but it does match the following CARQ entry if the spatial tolerance is 500 km:

AL, 06, 2016081500, 01, CARQ, 0, 97N, 204W, 20, 1009, DB, 34,

NEQ, 0, 0, 0, 0, 1010, 120, 60, 0, 0, L, 0, X, 270,

11, INVEST, S,

The corresponding BEST genesis information for AL062016 is:

AL, 06, 2016081618, , BEST, 0, 120N, 322W, 25, 1009, TD, 0, , 0, 0, 0, 0, 1012, 160, 80, 35, 0, L, 0, , 0, 0, INVEST, S, 0, , 0, 0, 0, 0, genesis-num, 013,

I'm not sure what the spatial and temporal tolerances for verification are in your namelist, but the temporal error between the forecast genesis valid time and the BEST genesis time is 42 hours, and the spatial error is > 500 km. So, while the new logic was able to find a forecast/storm match, it may still be a FALSE ALARM based on the large spatial and temporal errors between the forecast and observed genesis.

Case 2:

Old version:

DEBUG 4: SUI1 20160818_000000 initialization, 66 lead,

20160820_180000 genesis at (11.6, -16.3) is a HIT for CARQ

20160820_180000 genesis at (12.4, -15.8).

New version: FALSE ALARM

I could not find a BEST or CARQ entry at 20160820_180000 that was within 500 km of (12.4, -15.8), so this forecast would be a FALSE ALARM. I'm not sure how this forecast found a match in v9.1.

So, I think these two cases are doing what we want. I'm happy to test additional cases.

Thanks, Dan

From: johnhg notifications@github.com Sent: Wednesday, December 30, 2020 12:24 PM To: dtcenter/MET MET@noreply.github.com Cc: Halperin, Daniel J. HALPEA17@erau.edu; Assign assign@noreply.github.com Subject: [EXTERNAL] Re: [dtcenter/MET] Refine tc_gen logic to compare forecast genesis events to all BEST track points. (#1448)

John, consider the following changes:

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHubhttps://github.com/dtcenter/MET/issues/1448#issuecomment-752697388, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AI3GTCKI3QK75SL35IH2QX3SXNO3DANCNFSM4PYBRBWA.

halperin-erau commented 3 years ago

Also, regarding Otto: That type of storm is a rare occurrence. Typically, if a storm moves from the AL to EP (or vice versa), they dissipate at least briefly over land and then re-develop, so they receive a new ID and new name in the new basin.

For Otto, I agree with your plan to completely discard the track and genesis for EP222016 since Otto developed in the Atlantic. That is also how NHC treated this storm. They did not issue any genesis forecasts for Otto in the Eastern Pacific. They just continued writing advisories when it switched basins.

Thanks, Dan

From: Halperin, Daniel J. HALPEA17@erau.edu Sent: Wednesday, December 30, 2020 1:17 PM To: dtcenter/MET MET@noreply.github.com; dtcenter/MET reply@reply.github.com Cc: Assign assign@noreply.github.com Subject: Re: [EXTERNAL] Re: [dtcenter/MET] Refine tc_gen logic to compare forecast genesis events to all BEST track points. (#1448)

Hi John,

I looked at the two cases that were HITS in v9.1 and FALSE ALARMS in the new version with forecast valid times at 00, 06, 12, or 18 UTC and I think the refined logic is correct.

Case 1:

Old version:

DEBUG 4: SUI1 20160813_120000 initialization, 36 lead,

20160815_000000 genesis at (8.5, -23.5) is a HIT for CARQ

20160815_120000 genesis at (9.2, -23.7).

New version: FALSE ALARM

As you pointed out, the new logic has no matching BEST entry, but it does match the following CARQ entry if the spatial tolerance is 500 km:

AL, 06, 2016081500, 01, CARQ, 0, 97N, 204W, 20, 1009, DB, 34,

NEQ, 0, 0, 0, 0, 1010, 120, 60, 0, 0, L, 0, X, 270,

11, INVEST, S,

The corresponding BEST genesis information for AL062016 is:

AL, 06, 2016081618, , BEST, 0, 120N, 322W, 25, 1009, TD, 0, , 0, 0, 0, 0, 1012, 160, 80, 35, 0, L, 0, , 0, 0, INVEST, S, 0, , 0, 0, 0, 0, genesis-num, 013,

I'm not sure what the spatial and temporal tolerances for verification are in your namelist, but the temporal error between the forecast genesis valid time and the BEST genesis time is 42 hours, and the spatial error is > 500 km. So, while the new logic was able to find a forecast/storm match, it may still be a FALSE ALARM based on the large spatial and temporal errors between the forecast and observed genesis.

Case 2:

Old version:

DEBUG 4: SUI1 20160818_000000 initialization, 66 lead,

20160820_180000 genesis at (11.6, -16.3) is a HIT for CARQ

20160820_180000 genesis at (12.4, -15.8).

New version: FALSE ALARM

I could not find a BEST or CARQ entry at 20160820_180000 that was within 500 km of (12.4, -15.8), so this forecast would be a FALSE ALARM. I'm not sure how this forecast found a match in v9.1.

So, I think these two cases are doing what we want. I'm happy to test additional cases.

Thanks, Dan

From: johnhg notifications@github.com Sent: Wednesday, December 30, 2020 12:24 PM To: dtcenter/MET MET@noreply.github.com Cc: Halperin, Daniel J. HALPEA17@erau.edu; Assign assign@noreply.github.com Subject: [EXTERNAL] Re: [dtcenter/MET] Refine tc_gen logic to compare forecast genesis events to all BEST track points. (#1448)

John, consider the following changes:

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHubhttps://github.com/dtcenter/MET/issues/1448#issuecomment-752697388, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AI3GTCKI3QK75SL35IH2QX3SXNO3DANCNFSM4PYBRBWA.

JohnHalleyGotway commented 3 years ago

See comments from 1/6/21 meeting here: https://github.com/dtcenter/MET/issues/1430#issuecomment-755755898