search

wmo-im / wmds

WIGOS Metadata Standard: Semantic standard and code tables
16 stars 22 forks source link

5-02-01 Lidar in Measurement/Observing Method (atmosphere) #266

Closed ejwelton closed 3 years ago

ejwelton commented 3 years ago

Branch

https://github.com/wmo-im/wmds/blob/issue266/tables_en/5-02-01.csv

Summary and Purpose

There are 5 lidar entries in the Measurement/Observing Method (atmosphere) code list, all have similar names “Light Detection and Ranging (lidar)” and none are defined. This makes it impossible for users to select the appropriate method for lidar. There are also two Differential absorption lidar (DIAL) entries (320,335) that have no definitions and no way to distinguish one from the other.

Stakeholder(s)

WG ACV

Proposal

Add unambiguous descriptions to the Lidar and DIAL methods that include information on the observed variables related to the methods.

notation path name definition
142 \Atmosphere\Wind\Remote-sensing, passive\Light detection and ranging (Lidar) profiler Light detection and ranging (LIDAR) profiler Light detection and ranging (Lidar) profiler for wind observations
143 \Atmosphere\Humidity\Remote-sensing, active\Light detection and ranging (Lidar) Light detection and ranging (Lidar) Light detection and ranging (Lidar) for humidity measurements
243 \Atmosphere\Aerosol\Remote-sensing, active\Optical properties\Light detection and ranging (Lidar) profiler Light detection and ranging (Lidar) profiler Light detection and ranging (Lidar) profiler for aerosol observations
269 \Atmosphere\Temperature\Remote-sensing, active\Light detection and ranging (Lidar) profiler Light detection and ranging (Lidar) profiler Light detection and ranging (Lidar) profiler for temperature measurements
320 \Atmosphere\Trace gas\Remote-sensing, active\Laser-based methods\Differential absorption Lidar (DIAL) Differential absorption Lidar (DIAL) Differential absorption Lidar (DIAL) for trace gas observations
324 \Atmosphere\Visual range\Remote-sensing, active\Laser-based methods\Light detection and ranging (Lidar) Light detection and ranging (Lidar) Light detection and ranging (Lidar) for observations of visibility and optical range
335 \Atmosphere\Ozone\Remote-sensing, active\Laser-based methods\Differential absorption Lidar (DIAL) Differential absorption Lidar (DIAL) Differential absorption Lidar (DIAL) for ozone measurements

Reason

The descriptons make it possible for the users to distiguish between methods of the same name.

Original comment:

Branch https://github.com/wmo-im/wmds/blob/issue266/tables_en/5-02-01.csv (edited)

Summary and Purpose There are 5 lidar entries in the Measurement/Observing Method (atmosphere) code list, all have similar names “Light Detection and Ranging (lidar)” and none are defined. This makes it impossible for users to select the appropriate method for lidar. There are also two Differential absorption lidar (DIAL) entries (320,335) that have no definitions and no way to distinguish one from the other.

Stakeholder(s) WG ACV

Proposal Redefine the lidar entries as follows: Notation: 143 Backscatter Lidar Definition: Elastic backscatter lidar Notation: 324 Raman Lidar Definition: Raman lidar instrument Notation: 142 HSRL Definition: High Spectral Resolution Lidar Notation: 243 Ozone Lidar Definition: Ozone lidar Notation: 269 Polarized Lidar Definition: Polarized lidar instrument capable of determining depolarization ratio Determine why there are two DIAL entries in the code list, and provide definitions. If they are the same, deprecate one.

Reason This proposal will allow lidar providers to correctly state the type of lidar instrument used to provide the observed variable, this also informs the nature of the variable(s) associated with the lidar type. For instance, aerosol extinction from backscatter lidar is less accurate than from Raman or HSRL.

fstuerzl commented 3 years ago

I checked the entries in table 5-02-01.csv (Observin method (atmosphere). There, the entries you propose to redefine are distinguished only in the paths, which is impractical since the paths are not part of the tables in the codes registry and they are only visible to users in the OSCAR application (method tree). The paths in the table below show that while the observing methods are the same they differ in the target (observed variable).

notation path name definition
142 \Atmosphere\Wind\Remote-sensing, passive\Light detection and ranging (LIDAR) profiler Light detection and ranging (LIDAR) profiler
143 \Atmosphere\Humidity\Remote-sensing, active\Light detection and ranging (Lidar) Light detection and ranging (Lidar)
243 \Atmosphere\Aerosol\Remote-sensing, active\Optical properties\Light detection and ranging (Lidar) profiler Light detection and ranging (Lidar) profiler
269 \Atmosphere\Temperature\Remote-sensing, active\Light detection and ranging (Lidar) profiler Light detection and ranging (Lidar) profiler
320 \Atmosphere\Trace gas\Remote-sensing, active\Laser-based methods\Differential absorption Lidar (DIAL) Differential absorption Lidar (DIAL)
324 \Atmosphere\Visual range\Remote-sensing, active\Laser-based methods\Light detection and ranging (Lidar) Light detection and ranging (Lidar)
335 \Atmosphere\Ozone\Remote-sensing, active\Laser-based methods\Differential absorption Lidar (DIAL) Differential absorption Lidar (DIAL)

If I understand it correctly it is possible to just replace the names and add definitions, because the entries 142, 143, 243, 269 and 324 all describe the general concept of lidar observing methods, whithout distinguishing between the types of lidar. For the different lidar types we should create new entries, I think.

notation path name definition
Backscatter Lidar Elastic backscatter lidar
Raman Lidar Raman lidar instrument
HSRL High Spectral Resolution Lidar
Ozone Lidar Ozone lidar
Polarized Lidar Polarized lidar instrument capable of determining depolarization ratio
Raman Lidar High Spectral Resolution Lidar

Therefore a revision of the path structure is also necessary.

fstuerzl commented 3 years ago

Element with notation 142 contains an error: Path should be \Atmosphere\Wind\Remote-sensing, active\Light detection and ranging (LIDAR) profiler. This has been changed in OSCAR/Surface.

https://github.com/wmo-im/wmds/compare/a031428161fee0677870d264e62d6e5bf69fdc05...2a720056704e4466184b98794148b3e3c4609c87

ejwelton commented 3 years ago

I would recommend a change in path structure coupled with moving lidars to the /Atmosphere path level. Is it possible to do this, and not have them distributed in sub groups by observed variable? Lidar are inherently multi-disciplinary instruments, and it is too difficult to group them by variable. I do not think it is a good idea to have multiple similar sounding instruments mixed between all these different sub-groups (and a new path structure). It is already confusing to users who have to select an observingMethod to submit to OSCAR. Adding new entries without addressing the existing ones will just add to the confusion. For instance, every lidar is capable of providing aerosol and cloud profiling (even DIAL, ozone, etc). Thus all of these other lidar "methods" would have to be duplicated into the \Atmosphere\Aerosol\Remote-sensing, active\ path. Moving them to the \Atmosphere level and cleaning up the entires would be a better fix.

joergklausen commented 3 years ago

We are exploring an approach of using tags rather than paths for the variables, and if this works, the methods could be treated in a similar manner. Here, tags would be [remote-sensing, active] [DIAL] [gas] [particle phase] [cloud] [profiling] ... as to be determined by experts. As a side, thinking of lidar as an instrument rather than a method is not the intention, although the acronym is used for both. The intention is to attach the method of observation to an observed variable, in order to distinguish different observations of the same observed variable. Obviously, method and instrument are related, as each instrument uses a particular method.

amilan17 commented 3 years ago

For reference, if helpful, the Task Team on Table Driven Code Formats has been recently evolving the BUFR/GRIB codes for lidar. 

adopted in FT2021-1

in progress

ejwelton commented 3 years ago

@amilan17 I looked over the BUFR links and they do not align well with actual lidar requirements for WIGOS. BUFR4#65 is more about establishing what looks like a lidar file format (with variables included). That is outside the scope of WIGOS metadata, where we already have variables established that trace back to RRR. For instance it would be great to include the lidar ratio and depolarization ratio in WIGOS, but they have not been included in the final recommendations from RRR. BUFR4#79 is about adding "ground based lidar" in BUFR. We already have lidar instruments (based on capability) included in the discussion here, and much more expansive than what I saw in BUFR. We need a way to identify the type of lidar, not just that one is present at a location. @fstuerzl, is there something left to do to move this issue forward? I'm not sure if it has stalled out or not.

fstuerzl commented 3 years ago

@ejwelton, since we have not applied the tag approach to the method table yet, we would need to integrate the variables you proposed into the existing structure of the method hierarchy and add descriptions.

By adding the following descriptions (please make corrections if necessary) we at least avoid having multiple entries with the same name, which are only distinguished by the paths.

notation path name definition
142 \Atmosphere\Wind\Remote-sensing, passive\Light detection and ranging (LIDAR) profiler Light detection and ranging (LIDAR) profiler Light detection and ranging (Lidar) profiler for wind observations
143 \Atmosphere\Humidity\Remote-sensing, active\Light detection and ranging (Lidar) Light detection and ranging (Lidar) Light detection and ranging (Lidar) for humidity measurements
243 \Atmosphere\Aerosol\Remote-sensing, active\Optical properties\Light detection and ranging (Lidar) profiler Light detection and ranging (Lidar) profiler Light detection and ranging (Lidar) profiler for aerosol observations
269 \Atmosphere\Temperature\Remote-sensing, active\Light detection and ranging (Lidar) profiler Light detection and ranging (Lidar) profiler Light detection and ranging (Lidar) profiler for temperature measurements
320 \Atmosphere\Trace gas\Remote-sensing, active\Laser-based methods\Differential absorption Lidar (DIAL) Differential absorption Lidar (DIAL) Differential absorption Lidar (DIAL) for trace gas observations
324 \Atmosphere\Visual range\Remote-sensing, active\Laser-based methods\Light detection and ranging (Lidar) Light detection and ranging (Lidar) Light detection and ranging (Lidar) for observations of visibility and optical range
335 \Atmosphere\Ozone\Remote-sensing, active\Laser-based methods\Differential absorption Lidar (DIAL) Differential absorption Lidar (DIAL) Differential absorption Lidar (DIAL) for ozone measurements

For the methods you proposed, I need more information on the paths. Do we need to add them multiple times? Can any of the existing variables be replaced?

See also: https://github.com/wmo-im/wmds/wiki/Guidance-for-Removal-of-Term

ejwelton commented 3 years ago

The proposal above would provide the current definitions for the existing entries, so yes that would avoid the current confusion. However, the existing entries are not adequate to describe a method or instrument used. If an aerosol variable exists in an observation and its method says "Light detection and ranging (Lidar) profiler for aerosol observations" then such information is useless (it just says a lidar of some type provided aerosol data). If the intent here for observing method was not just to document the instrument type, but also processing methodology (ie, observingMethod = instrument + retrieval technique) then it might be possible to expand the different paths that currently exist. For instance, there are many combinations of Aerosol Lidar type and retrieval technique that could go in \Atmosphere\Aerosol\Remote-sensing, active\Optical properties\Light detection and ranging (Lidar) profiler/: single wavelength backscatter lidar, Fernald retrieval single wavelength polarized backscatter lidar, Fernald retrieval single wavelength backscatter lidar, Fernald retrieval, Sunphotometer constrained single wavelength polarized backscatter lidar, Fernald retrieval, Sunphotometer constrained single wavelength raman lidar, raman retrieval single wavelength polarized raman lidar, raman retrieval single wavelength HSRL lidar, HSRL retrieval single wavelength polarized HSRL lidar, HSRL retrieval ... then dual wavelength versions of above ... then multi-wavelength versions of above. Each of those methods provides contextual information on the quality of the aerosol data provided. But this would be a huge expansion of the lidar methods, requiring a lot of new codes. This why I was trying to keep it simple and restrict the lidar methods to instrument type only, and utilize a new wavelength feature to fully describe a data variable. For instance, aerosol backscatter could be linked to two methods (backscatter lidar, and polarized lidar) and wavelength to provide context. I'm not sure how to proceed here since the paths and internal structure of OSCAR are beyond me.

joergklausen commented 3 years ago

I agree with @ejwelton 's arguments. There is still a lot to be improved. However, keep in mind the original objective of this issue, which was to be able to distinguish between all the lidar methods currently in the code list, with different codes (corresponding to the different paths), but similar/identical names. The WDMR information model requires a method to be assigned, and we should assist users - such as @ejwelton @gaochen-larc - in making a choice. We also have run out of time for this FT, and the only solution I see for the time being that will not have negative consequences and at least provide a small improvement is to add a notion of the observed variable, as suggested by @fstuerzl . @fstuerzl Please proceed as suggested, so that I can push this issue to 'validated' by EOB today. Also, please open a new issue for FT-2022-1. As a starting point, we can use Judd's previous comment, although I would follow a general concept of observedMethod = measurement principle (x retrieval)(x wavelength(s))(x more specifics), where terms in parentheses are optional. For the lidar case, I would see 4 measurement principles, namely backscatter lidar, polarized lidar, raman lidar, differential absorption lidar.

fstuerzl commented 3 years ago

Branch updated, see: https://github.com/wmo-im/wmds/compare/issue266 Final proposal with smaller changes to the method table included in the comment on top of this issue.

ejwelton commented 3 years ago

@joergklausen I understand, given the time constraints, and the smaller change is fine with me. Further modification (if any) can be addressed later.