HMB minimum latitude?

[egthomas]

There seems to be a hard-coded lower latitude limit to the Heppner-Maynard Boundary (HMB) of 40 degrees when determining the location from gridded velocity vectors with map_addhmb. Normally this is not an issue, however during the recent May 2024 storm I've noticed the HMB becomes "saturated" at 40 degrees magnetic latitude for several hours. Should this limit be pushed farther equatorward to better handle such storms?

Note that even though there is limited observational coverage equatorward of 40 degrees magnetic latitude, the non-circular HMB is compressed by ~10 degrees on the dayside such that higher-latitude observations can still help constrain the solution under extreme conditions (eg, from the mid-latitude radars).

[mtwalach]

Hi Evan and all,

Yes, I lowered the limit to 40 (it used to be ~65 or something similar before then). I think 40 degrees is (or at least was) very close to the observational limit of SD, hence the choice. We can push it lower, but is there enough data to push it? If so, I'd be in favour of it.

As you can see from Fig 2f in my 2019 paper (https://agupubs.onlinelibrary.wiley.com/doi/10.1029/2019JA026816), it's not the first time this has saturated during a storm, so I think there is scientific value in it.

[pasha-ponomarenko]

Indeed: https://github.com/SuperDARN/rst/blob/734542037d86b24475d8da571ed668659a858e27/codebase/superdarn/src.lib/tk/hmb.1.0/src/hmb.c#L201 And the asymmetry of the HMB indeed allows its MLT midnight value to go below the 40-degree limit. I saw it with regard to the May 2024 superstorm. On a constructive side, I am looking at the ways of more physics-justified determination of the HMB extent in place of the currently implemented quick fix based on the most equatorward ionospheric scatter location. Application of the spectral width threshold was a first step in this direction. BTW, I noticed that for a given map the value of CPCP is not overly sensitive to the extent of the HMB.

[mtwalach]

Yes, I think you are right there Pasha because the CPCP is primarily determined by where the minima/maxima of the potentials are placed and therefore mainly determined of what is poleward. I think the CPCP is more sensitive to the number of backscatter echoes and adding in e.g. midlatitude radars (and hence likely lowering the HMB) can increase but also decrease the CPCP (see https://agupubs.onlinelibrary.wiley.com/doi/full/10.1029/2021JA029559).

[pasha-ponomarenko]

@mtwalach, we have already dealt with this problem by introducing the spectral width limit of w_l > 100 m/s. This is based on the fact that the subauroral echoes characterising the co-rotational region exhibit significantly lower SW values as compared to those from auroral irregularities. This limit effectively prevents contamination of the high-latitude velocity maps by the sub-auroral echoes. From this perspective, a hard-coded restriction to the maximum extent of the HMB might become redundant.

[pasha-ponomarenko]

It is also important to make sure that the HMB does not shrink to much due to the lack of echoes from lower latitudes (e.g., caused by unfavourable propagation conditions or a radar or two going down).

[agrocott]

Hi Pasha, all,

I have also given this some thought in the past. As I understand it, the boundary placement is based on the latitude at which the velocity goes above 100 m/s (or whatever the user sets), going from low-to-high latitudes. So, ideally, we have some scatter below 100 m/s at the lower latitudes and then the boundary is placed at the lowest latitude where the velocity goes above 100 m/s. For this method to work, scatter <100 m/s is required, otherwise, the boundary is just placed at the lowest latitude that scatter of any velocity is observed (subject to the n_points condition too). So, if the lowest velocity scatter is 1000 m/s, the HMB is placed there. I think this is what Pasha is saying…?

One factor contributing to this issue that I considered is that the boundary placement is independent of the fit. In such a scenario as above, those equator-most 1000 m/s vectors will be reduced to ~zero by the fit, presumably ramping up the chi^2 in the process. I did some experimentation a little while back where I considered this (I think not the global chi^2 but just for the velocities on the HMB) and tried lowering the boundary below the equatorward extent of the scatter until the ‘local’ chi^2 for those velocities was minimised. This seemed to work quite well in some cases, but not always. As such, I have only used this technique on a case by case basis, and am not sure it would be suitable to implement in RST. But if anyone thinks it’s worth pursuing I could always pick it back up…

Cheers, Ade

From: pasha-ponomarenko @.> Date: Wednesday, 17 July 2024 at 17:52 To: SuperDARN/rst @.> Cc: Subscribed @.***> Subject: [External] Re: [SuperDARN/rst] HMB minimum latitude? (Issue #617)

This email originated outside the University. Check before clicking links or attachments.

It is also important to make sure that the HMB does not shrink to much due to the lack of echoes from lower latitudes (e.g., caused by unfavourable propagation conditions or a radar going down).

— Reply to this email directly, view it on GitHubhttps://github.com/SuperDARN/rst/issues/617#issuecomment-2233759316, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AHEPWYUKNIE3PFE5J3F3KZDZM2OKPAVCNFSM6AAAAABLA2DDCOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMZTG42TSMZRGY. You are receiving this because you are subscribed to this thread.Message ID: @.***>

[agrocott]

Hi Pasha, all,

I have what I suppose is ultimately a physics question… is it right that we want to exclude sub-auroral echoes, and that these should be classed as ‘contamination’? I have seen this issue come up a few times in different contexts recently, with one opinion that all (non-corotational) flow is part of the ‘convection’ pattern (not just polar/auroral flow) and so should all be included, and some opinions that only ‘Dungey cycle’ (i.e. polar/auroral) flow is convection and anything else should be excluded. Certainly, I have seen papers in which e.g. SAPS have been included in the fits. But also others where there are slower flows clearly below the HMB (presumably just due to the somewhat arbitrary choice of the minimum velocity used to specify the boundary) thus excluded from the fits.

On a related note, Pasha, you explicitly mention sub-auroral echoes characterising the co-rotational flow. But the radars co-rotate with the planet, so they should not measure any component of co-rotational flow. i.e. at low latitudes, if corotation is the only motion, then the radars should measure zero flow.

At higher latitudes, on the other hand, things get less clear. I’ve thought about this often and never come up with a satisfactory answer on how to properly deal with corotation: Closed field lines that extend some distance downtail, for example, cannot corotate and so they must be held roughly stationary in the sun-earth frame. As such, an eastward corotating radar will ‘see’ a westward component of the flow associated with its own motion…

Cheers, Ade

From: pasha-ponomarenko @.> Date: Wednesday, 17 July 2024 at 16:34 To: SuperDARN/rst @.> Cc: Subscribed @.***> Subject: [External] Re: [SuperDARN/rst] HMB minimum latitude? (Issue #617)

This email originated outside the University. Check before clicking links or attachments.

@mtwalachhttps://github.com/mtwalach, we have already dealt with this problem by introducing the spectral width limit of w_l > 100 m/s. This is based on the fact that the subauroral echoes characterising the co-rotational region exhibit significantly lower SW values as compared to those from auroral irregularities. This limit effectively prevents contamination from sub-auroral echoes. From this perspective, a hard-coded restriction to the maximum extent of the HMB might be redundant.

— Reply to this email directly, view it on GitHubhttps://github.com/SuperDARN/rst/issues/617#issuecomment-2233610496, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AHEPWYTPWQSKTDIEM7SVNDDZM2FHFAVCNFSM6AAAAABLA2DDCOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMZTGYYTANBZGY. You are receiving this because you are subscribed to this thread.Message ID: @.***>

[mtwalach]

I don't think Chi-squared is a good statistic to use, mainly because 1) we know it increases with the number of backscatter points, but not evenly and 2) the velocity measurements are not normally distributed (and usually not even a Gaussian distribution).

Also on the SAPs: There are pros and cons for inclusion/exclusion but in principle, IF they are truly sub-auroral then an algorithm which looks for a threshold should be able to exclude them if we want. All we'd have to do is reverse the HMB-finding algorithm to move from the pole towards the equator, as opposed to moving towards the pole (but I could see this generating other issues if there isn't enough scatter due to say auroral precipitation etc, so perhaps this wouldn't always work - I haven't tried it).

[pasha-ponomarenko]

"On a related note, Pasha, you explicitly mention sub-auroral echoes characterising the co-rotational flow. But the radars co-rotate with the planet, so they should not measure any component of co-rotational flow. i.e. at low latitudes, if corotation is the only motion, then the radars should measure zero flow." For sure, I have just meant internal processes, whatever they are. ;-)

[pasha-ponomarenko]

@agrocott : "I think this is what Pasha is saying…?" Pretty much so, and my concern always was that the determination of the HMB extent should not depend on the presence of ionospheric scatter at all, the latter being subject to ionospheric conditions, radar frequency, continuity of radar operation and coverage, and other factors unrelated to the magnetospheric processes. Your point about zeroing a 1000-m/s observed velocity in the vicinity of the HMB is a good example of why this should not be done.

[agrocott]

“I don't think Chi-squared is a good statistic to use “

I agree, that’s why I didn’t use chi^2 ;-)

By ‘local” chi^2 I just mean that I minimised the differences between the velocities on the boundary and their fitted counterparts in a similar way to the global minimisation.

From: Maria-Theresia Walach @.> Date: Thursday, 18 July 2024 at 11:42 To: SuperDARN/rst @.> Cc: Grocott, Adrian @.>, Comment @.> Subject: [External] Re: [SuperDARN/rst] HMB minimum latitude? (Issue #617)

This email originated outside the University. Check before clicking links or attachments.

I don't think Chi-squared is a good statistic to use, mainly because 1) we know it increases with the number of backscatter points, but not evenly and 2) the velocity measurements are not normally distributed (and usually not even a Gaussian distribution).

Also on the SAPs: There are pros and cons for inclusion/exclusion but in principle, IF they are truly sub-auroral then an algorithm which looks for a threshold should be able to exclude them if we want. All we'd have to do is reverse the HMB-finding algorithm to move from the pole towards the equator, as opposed to moving towards the pole (but I could see this generating other issues if there isn't enough scatter due to say auroral precipitation etc, so perhaps this wouldn't always work - I haven't tried it).

— Reply to this email directly, view it on GitHubhttps://github.com/SuperDARN/rst/issues/617#issuecomment-2236188871, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AHEPWYSZNKE3DOK2N7EZR4LZM6LYDAVCNFSM6AAAAABLA2DDCOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMZWGE4DQOBXGE. You are receiving this because you commented.Message ID: @.***>

[egthomas]

Thank you all for your comments - this has been a great discussion! I just wanted to add a link to the earlier pull request @mtwalach mentioned (https://github.com/SuperDARN/rst/pull/216), where the lowest possible HMB latitude was shifted from 50 to 40 degrees MLAT (thanks for the reminder!). You can see that in the process of reviewing that pull request we also identified a "fudge factor" where the HMB is pushed a further 1 degree lower than determined from the gridded velocity vectors; I'm not sure we ever properly documented that behavior?

And if we're all piling onto the HMB code, I'd like to mention another aspect that has bothered me: the statistical pattern introduced by map_addmodel can be either stretched or compressed according to the data-derived HMB location. Now I haven't performed any quantitative testing or examined any alternatives, but I can imagine scenarios where the background model could become quite distorted, or introduce model vectors where statistically none should exist.

[agrocott]

Just picking this up as I’ve been on leave…

I have also wondered about the issue Evan mentions about the background model. In fact, I had mainly wondered if the pattern was stretched (as Evan notes) or if it was just dealt with in some other (more suitable) way, but had never actually looked into it. So thanks, Evan, for confirming!

In that case, I agree that the stretching /compressing could well be problematic in some circumstances. Presumably, if the pattern is just expanding and contracting per the ECPC model then this isn’t so bad (assuming a Dungey cycle pattern looks largely similar whatever size it is – although this is perhaps still not strictly true). But if the HMB drops because some sub-auroral feature starts to be captured (with the ‘Dungey’ part of the pattern not changing) then a stretched model pattern is surely going to be misaligned with the corresponding geophysical regions.

From a ‘user’ perspective, I guess this just comes under the ‘know what you’re doing’ caveat. Decent data coverage should override the model. Failing that, appropriate choice of model is probably relevant with TS18 presumably including some sub-auroral flow and (e.g.) RG96 not doing so.

From: Evan Thomas @.> Date: Friday, 19 July 2024 at 17:07 To: SuperDARN/rst @.> Cc: Grocott, Adrian @.>, Mention @.> Subject: [External] Re: [SuperDARN/rst] HMB minimum latitude? (Issue #617)

This email originated outside the University. Check before clicking links or attachments.

Thank you all for your comments - this has been a great discussion! I just wanted to add a link to the earlier pull request @mtwalachhttps://github.com/mtwalach mentioned (#216https://github.com/SuperDARN/rst/pull/216), where the lowest possible HMB latitude was shifted from 50 to 40 degrees MLAT (thanks for the reminder!). You can see that in the process of reviewing that pull request we also identified a "fudge factor" where the HMB is pushed a further 1 degree lower than determined from the gridded velocity vectors; I'm not sure we ever properly documented that behavior?

And if we're all piling onto the HMB code, I'd like to mention another aspect that has bothered me: the statistical pattern introduced by map_addmodel can be either stretched or compressed according to the data-derived HMB location. Now I haven't performed any quantitative testing or examined any alternatives, but I can imagine scenarios where the background model could become quite distorted, or introduce model vectors where statistically none should exist.

— Reply to this email directly, view it on GitHubhttps://github.com/SuperDARN/rst/issues/617#issuecomment-2239522741, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AHEPWYUP5KLSLXARF5VRKADZNE2TPAVCNFSM6AAAAABLA2DDCOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMZZGUZDENZUGE. You are receiving this because you were mentioned.Message ID: @.***>

[mtwalach]

RG96 is unable to pick up anything below 65 degrees in latitude. Ruohoniemi & Greenwald (1996) write:

However, the convection generally extends equatorward of the 65 degree limit of the observations and it is not possible to unambiguously establish a zero potential reference. The potential is uncertain by an additive constant. In this analysis we have adopted the following convention: the potential is set so that its average on the nightside (18-06 MLT) at the low-latitude limit (65 degree) of the observations is zero. While not ideal, this convention is objective and usually identifies the zero potential condition with a contour that exits the nightside polar cap close to the 0 MLT meridian. So, in effect, the HMB was forced but "objectively" so.

This suggests that they were not able to pick up expansions below 65 degrees either but should the convection have moved above 65 degrees, then yes, RG96 would have picked up sub-auroral flows. However, I think TS18 handpicked the HMB to avoid picking up sub-auroral flows and other ambiguities. Maybe @egthomas has more to comment on this.

@egthomas, re I'm not sure we ever properly documented that behavior?, no I don't think it's been properly documented. I mention it in a sentence or two in one of my papers, and perhaps you do too, but I think that's it. Unless anyone else knows otherwise?

[billetd]

We had some discussion about this yesterday in the USask slack. Although seems like this thread has gone fairly passed @egthomas's original point about a hardcoded limit, I think most HMB issues (i.e. finding, and trusting a lower limit, excluding sub-auroral scatter, etc) stem from basing the HMB purely on radar scatter. Ideally, there would be a completely separate, always available dataset of perfect HMBs that we could pull from to use instead of scatter, but unfortunately that perfect world doesn't exist. Something like AMPERE might come close (e.g. https://agupubs.onlinelibrary.wiley.com/doi/full/10.1029/2019JA027218), but it's less than ideal to rely on for everything. I know Pasha is currently looking at this, so maybe he can comment further. I do know that trying to fit high-latitude convection flows to sub-auroral, high-velocity echoes (like SAPS) with map potential does cause issues, and often requires manual intervention with the HMB setting.

I believe Mike Ruohoniemi @ VT was also going to be chairing a HMB/convection task force to look more closely at this issue, but I don't think it's in full swing currently. Others in this thread might be interested in contributing when that happens.