Redefine "Overlap" bit as "Overage"

[esilvia]

From a private conversation I had with Karl back in 2014:

Overlap is any area that is covered by 2 or more swaths. Nice, but not the phenomenon or condition we want and need to identify. What we are concerned with is more properly called OVERAGE. Harder to define without a picture, but, imagine a project intended for single coverage at 20% sidelap, with swaths 1km wide. The intent is to use the center 800m of each swath … what I refer to as “the tenderloin”, and to edge-match all of the project’s tenderloins together for a nice seamless, gapless single coverage.

In this scenario (which was and remains a VERY common practice for years), for each swath, the 100m outboard of the tenderloin (on each side) is “excess” data. Still perfectly good data, just not needed to accomplish the single coverage. That 100m on each side of each swath is the OverAGE for that swath. And those are the points that are to be flagged with the (regrettably mis-named) “Overlap” Flag. (On the other hand, if you tagged all of the points in the true “OverLAP”, only 60% of the swath would remain, and without those points, you’d have a 200m gap between swaths).

The genesis of all this was that data producers would classify all of the overage points as Class 12, and then do their DEMs based on the tenderloin-based single coverage. Which made nice DEMs, because you eliminated the stripes of double-density (higher variability) data that were plainly visible in the resulting rasters. The PROBLEM with that is users who were interested in something OTHER than the DEM could not do anything with those points without “damaging” the dataset classification (losing the differentiation of the “overage” points). It’s all a remnant of the days when DEMs were King and nobody regarded the point cloud as anything but a means to that end. The practices became entrenched, but then we realized that Lidar was not just for DEMs anymore. Now we are struggling to break old bad habits.

In the v1.0 Spec (using LAS v1.2/1.3), we required that “overage” point be flagged using some means other than Class 12, that allowed both “normal” classification AND differentiation of the overage condition. Makes the data useful beyond just the DEM. How this was done was defined, because data producers had their own systems and processes and there could not be a one-size-fits-all solution. NOW, however, using LAS v1.4, there IS a one-size-fits-all solution: The “Overlap Flag”.

And from a followup conversation:

For “Overlap/age”, your definition is keyed on modeling, and gives the impression that these points should not be used for analysis, in a general or broad-ranging sense. I think the real key is the spatial location, relative to the swath and its overlapping swaths. I would say: Overlap: bit flag to mark points that are unnecessary to achieve a consistent “depth of coverage” (meaning single, double, whatever). No implications are made regarding point quality. To be certain, that description needs a great deal of refinement before “publication”, but it properly limits the term to spatial location/relationships. I would not want to make any implication as to whether anybody should or should not use the points however they choose. That’s their business. Additionally, SOME people may interpret your definition as extending to not “modeling” these points for classification purposes.

[HKHeidemann]

I concur with Evon. Both OverLAP and OverAGE are valid descriptors, but of different things.

OverLAP is a static positional condition of one swath (or lift, or project, or ...) relative to another swath (or other piece of data). Once data is geometrically finalized, the OverLAP is also finalized. Whether or not a point is "OverLAP" is immutable: It either is, or is not, in that area.

OverAGE is a spatial (x-y) subset of a swath, which as Evon VERY nicely put it, "[is] unnecessary to achieve a consistent “depth of coverage” (meaning single, double, whatever)." The condition of OverAGE IS mutable; any user may choose to redefine the exact boundaries of the "tenderloins", while remaining complete and non-overlapped between them, and thus, which points are OverAGE.

Evon's point - suggesting suitability of these points for any application is beside the point and should be avoided - is also spot-on.

[esilvia]

Funny that you like the wording, since most of that was copied directly from some of your posts. :)

I think my primary concern with altering the label/wording in the specification is that some may have implemented use of the Overlap flag based on the current definition and therefore this change would be construed as breaking binary compatibility. Does anyone know of any implementations that implement the Overlap bit as OVERLAP rather than USGS's desired meaning of OVERAGE?

[HKHeidemann]

I believe that "overage" was the original intent of both the earlier classification as well as the current flag -- that the "problem" lies merely with the term being used. I will be interested in hearing if anybody has been using it for actual overlap. I don't think I've ever seen one here at EROS.

[esilvia]

The consensus from the 5/18 as I understood it was that renaming this field would only add confusion to a feature that is little understood and used even less, especially since the scant software that does support Overlap already calls it that. Therefore, the best solution seems to be to add an example and leave the name alone (see screenshot).

Does that help address the concerns from the last LWG meeting without causing any damage to existing implementations?

[jdnimetz]

@esilvia I believe the descriptions you've listed in preceding post are good as-is. Regarding overlap: I agree that usage of overlap bit flag is not very well understood, as evident by the variability in its usage in multiple airborne lidar datasets available in the public domain.

Of particular concern is how to assess point cloud density and distribution when this flag is used. Should overlap-flagged points be counted for acquisition-designed (i.e. "nominal") first-return density? Should these points be ignored in this assessment? What about terrain (ground) density? Should these points be counted or ignored if they are not used to create bare earth DEMs derived from the point cloud?

Use of the overlap bit flag is sometimes confused or conflated with use of the withheld flag. I've seen airborne lidar datasets where the same point is flagged as both withheld and overlap because it meets the data producer's selection criteria. The differentiation can be muddy; if a point should be ignored for the purpose of deriving a model, doesn't this indicate a problem with the point's geometry? If so, isn't the withheld flag alone the right choice for differentiation?

Curious to hear others' thoughts on the use of overlap and withheld flags, either together or independently.

[esilvia]

@jdnimetz I think it's up to the user as to whether or not Overlap points are included in density computations. Generally, I'd say they should be included when assessing whether a dataset was acquired to meet a certain specification, but excluded when producing DEM rasters.

It's correct to say that Overlap and Withheld represent two different situations. Withheld points are either invalid returns or geometrically unreliable. Overlap points are potentially valid but "extra" in the sense that they are not required to meet the needs of the application and were identified as the least important points in that overlapping region. Points that fall within the "extra" region for that swath and are also geometrically unreliable may be both Overlap and Withheld, such as "stacked" points at the very edge of an oscillating mirror scanner.

[esilvia]

I believe this discussion is closed. Feel free to reopen if we need to revisit.