search

OceanDataTools / openrvdas

An open source data acquisition system designed for use on research vessels and other scientific installations
http://openrvdas.org
Other
43 stars 21 forks source link

Store/retrieve device calibration data #169

Open davidpablocohn opened 5 years ago

davidpablocohn commented 5 years ago

We need to be able to store/retrieve/use device calibration data:

Transforms should have access to these metadata so that they can perform the adjustment (and pass the metadata of how the adjustment was made along with the new, adjusted values).

There should be mechanisms for associating those data with data streams, such as outputting as header in a log file or including in JSON or DASRecord messages.

webbpinner commented 5 years ago

For what it's worth I do not think this should be handled within a data acquisition system and instead should be handled by a dedicated metadata tool. I see where it would be a slick feature for sensors that are handled by OpenRVDAS, however not all data systems on a research vessel are run through OpenRVDAS.

My personal preference would be to develop a stand-alone sensor metadata tracking system that could be used to manage metadata for all sensor systems on a vessel. I've actually been thinking recently that this metadata tool might be the next project OpenOceanTools takes on.

davidpablocohn commented 5 years ago

It sounds reasonable to me to have an external system manage metadata. But I think OpenRVDAS components need to be able to retrieve/parse/work with it. Would be interesting to consider whether simply having a "MetadataReader" component that could retrieve things like calibration values (to be used, e.g., in a DerivedDataTransform) would be sufficient.

On Mon, Oct 21, 2019 at 11:36 AM Webb Pinner notifications@github.com wrote:

For what it's worth I do not think this should be handled within a data acquisition system and instead should be handled by a dedicated metadata tool. I see where it would be a slick feature for sensors that are handled by OpenRVDAS, however not all data systems on a research vessel are run through OpenRVDAS.

My personal preference would be to develop a stand-alone sensor metadata tracking system that could be used to manage metadata for all sensor systems on a vessel. I've actually been thinking recently that this metadata tool might be the next project OpenOceanTools takes on.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/OceanDataTools/openrvdas/issues/169?email_source=notifications&email_token=AFO7V3SKJOCYPNV3GHNQVWTQPYAE5A5CNFSM4JC7BXBKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEB3Q3TI#issuecomment-544673229, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFO7V3SHYAYBYQACLC533VLQPYAE5ANCNFSM4JC7BXBA .

webbpinner commented 5 years ago

Been thinking about this a bit more. I still believe a DAS should just deal with reading and writing data. Transforms for things like prepending timestamps and true-wind calculations are a necessary complication but I believe anything beyond that should be handled by an external mechanism.

All that said, I am now in favor of having a transform that can apply a user-defined lambda function to a data stream. This would allow an operator to perform static corrections (i.e. adding the distance from a depth sensor to the water line for proper depth-from-surface) and unit conversions (knots <--> kph, C <--> F, etc). This type of transform could also be used to apply a calibration curve if the operator so desired. I can't take credit for this idea. An older version of the DAS used on WHOI ships included this functionality. I think SCS can do this as well.

My apprehension about getting into automated corrections based on calibration data comes from several bad experiences and was recently reinforced by a incident where a piece of mid-water equipment made contact with the seafloor because the wrong correction equation was being unknowingly applied in real-time to a depth sensor.

With regards to metadata tracking... I hope you can talk with Chris Romsos about work he's done in this area that I just uncovered yesterday.

davidpablocohn commented 5 years ago

Yup - this kind of conversion is what the DerivedDataTransform is all about. Of course, OpenRVDAS should (always) be storing the raw values that come out of the sensors; I just want to make sure it has access to the various calibration factors that let it turn those raw values into numbers that are meaningful to the scientists and crew on board.

On Tue, Oct 22, 2019 at 8:38 AM Webb Pinner notifications@github.com wrote:

Been thinking about this a bit more. I still believe a DAS should just deal with reading and writing data. Transforms for things like prepending timestamps and true-wind calculations are a necessary complication but I believe anything beyond that should be handled by an external mechanism.

All that said, I am now in favor of having a transform that can apply a user-defined lambda function to a data stream. This would allow an operator to perform static corrections (i.e. adding the distance from a depth sensor to the water line for proper depth-from-surface) and unit conversions (knots <--> kph, C <--> F, etc). This type of transform could also be used to apply a calibration curve if the operator so desired. I can't take credit for this idea. An older version of the DAS used on WHOI ships included this functionality. I think SCS can do this as well.

My apprehension about getting into automated corrections based on calibration data comes from several bad experiences and was recently reinforced by a incident where a piece of mid-water equipment made contact with the seafloor because the wrong correction equation was being unknowingly applied in real-time to a depth sensor.

With regards to metadata tracking... I hope you can talk with Chris Romsos about work he's done in this area that I just uncovered yesterday.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/OceanDataTools/openrvdas/issues/169?email_source=notifications&email_token=AFO7V3QVEDQLHVNF3O5EBILQP4UBJA5CNFSM4JC7BXBKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEB6MX2I#issuecomment-545049577, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFO7V3WDT2CHLSUGFPMNFZDQP4UBJANCNFSM4JC7BXBA .

LMG-ET commented 3 years ago

Yes.... I see Webb's point, but I consider it apples and oranges. Both tasty fruit, but different. Let's take a common example: A transmissometer. You've got calibration factors you copy from the vendor's calibration sheet. And WAY YES! It would be cool to have a tool where I could upload that and it would get indexed with make/model/serial-number/date, when I installed it, etc. Especially if it had a API I could query. But to get the actual beam transissance (is that the right word?) I need those factors, plus ideally in-situ dark and air measurements (which are data points sent to the DAS via the sea-water wall data stream so that the values used are in the raw data somewhere). The meta-data system wouldn't necessarily know about those since they'll change every time we clean the instrument and sample the air/dark values, and let's face it, tech's get busy and don't always have time to futz with metadata.

So yes... a function using (basically) the derived data transform is great. I think it might be hard to figure out how to enter that into the existing config structure though. I'd almost suggest a separate config file and a special-case thing like the true-winds transform that reads it. That way you could describe the metadata, the equation(s), the required data from the DAS, and the data you're providing TO the das. Since such a transform might apply to more than one logger stream, you maybe can't just it in the chain for one instrument.

I've actually been working on something like that, and can't wait for Pablo to show up so I can have him tell me I'm doing it wrong and show me the OpenRVDAS way. :-)

davidpablocohn commented 3 years ago

Sounds good - looking forward to seeing what you've got in mind.

On Tue, Jul 20, 2021, 4:26 PM LMG-ET @.***> wrote:

Yes.... I see Webb's point, but I consider it apples and oranges. Both tasty fruit, but different. Let's take a common example: A transmissometer. You've got calibration factors you copy from the vendor's calibration sheet. And WAY YES! It would be cool to have a tool where I could upload that and it would get indexed with make/model/serial-number/date, when I installed it, etc. Especially if it had a API I could query. But to get the actual beam transissance (is that the right word?) I need those factors, plus ideally in-situ dark and air measurements (which are data points sent to the DAS via the sea-water wall data stream so that the values used are in the raw data somewhere). The meta-data system wouldn't necessarily know about those since they'll change every time we clean the instrument and sample the air/dark values, and let's face it, tech's get busy and don't always have time to futz with metadata.

So yes... a function using (basically) the derived data transform is great. I think it might be hard to figure out how to enter that into the existing config structure though. I'd almost suggest a separate config file and a special-case thing like the true-winds transform that reads it. That way you could describe the metadata, the equation(s), the required data from the DAS, and the data you're providing TO the das. Since such a transform might apply to more than one logger stream, you maybe can't just it in the chain for one instrument.

I've actually been working on something like that, and can't wait for Pablo to show up so I can have him tell me I'm doing it wrong and show me the OpenRVDAS way. :-)

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/OceanDataTools/openrvdas/issues/169#issuecomment-883676009, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFO7V3XH2EYU25B4XXFUROLTYXLZDANCNFSM4JC7BXBA .