OceanDataTools / openrvdas

An open source data acquisition system designed for use on research vessels and other scientific installations
http://openrvdas.org
Other
39 stars 20 forks source link

Easier data retrieval from CDS #367

Open LMG-ET opened 5 months ago

LMG-ET commented 5 months ago

Currently if an external script (perhaps running on another machine) wants to grab a quick value from the CDS (i.e., What's the current air temperature) it involves a conversation: subscribe, ready, etc on a websocket. While this is fine for data you expect to be monitoring for a while (like, say, a display), it's less useful for a shell script.

What might be a useful enhancement is the ability to query the most recent data value for a variable, like (e.g., https://openvdas.mydomain/CDS/variable_name) and get a string back (data_value)

Possible approach: Since nginx can serve http and websockets from the same location block, on connection the CDS could check if it's receiving a normal HTTP request or a websocket connection, either returning a quick reply to a GET or proceeding as it does now for websocket traffic.

Alternative approach: Since small and modular is often better than a monolithic swiss army knife for embeds like ROV's, that may not be the way to go. A separate process (which you could optionally turn on with supervisord) that just gets proxied via nginx would work, so you'd do something like https://openrvdas.mydomain/latest_data/variable_name, you'd define a location block "latest_data/" in the nginx config, then nginx would proxy the request to a separate process, That process could query the cached data server in the normal way. Still the same conversation, but the complexity gets hidden from the end-user.

Alternate Alternate approach: An nginx module to handle this functionality. It seems like it might be both serious overkill and not in keeping with the python feng-shui of OpenRVDAS, but I don't think we'd find an approach with less latency.

What I'm specifically thinking about for this is getting data into elog. USAP's current common configuration for elog automatically fills in things like latitude, longitude, depth, sea-surface temperature, etc when you create a new log entry. Using a python script to talk to the CDS on the elog host would require installing the script, the correct modules (the websocket library, for example, is not part of the python standard library), so much better to have integral support for this kind of data query.

davidpablocohn commented 5 months ago

Yesterday (or was it the day before?), while we were talking about reviving a RESTful API for the control UI, we discussed whether it also made sense to have something RESTful for the CDS. I don't think it would be all that hard.

But I'm going to snooze this in my queue until I get back from a bit of west Tasmania exploration next week.

On Fri, Mar 29, 2024 at 3:52 AM LMG-ET @.***> wrote:

Currently if an external script (perhaps running on another machine) wants to grab a quick value from the CDS (i.e., What's the current air temperature) it involves a conversation: subscribe, ready, etc on a websocket. While this is fine for data you expect to be monitoring for a while (like, say, a display), it's less useful for a shell script.

What might be a useful enhancement is the ability to query the most recent data value for a variable, like (e.g., https://openvdas.mydomain/CDS/variable_name) and get a string back (data_value)

Possible approach: Since nginx can serve http and websockets from the same location block, on connection the CDS could check if it's receiving a normal HTTP request or a websocket connection, either returning a quick reply to a GET or proceeding as it does now for websocket traffic.

Alternative approach: Since small and modular is often better than a monolithic swiss army knife for embeds like ROV's, that may not be the way to go. A separate process (which you could optionally turn on with supervisord) that just gets proxied via nginx would work, so you'd do something like https://openrvdas.mydomain/latest_data/variable_name, you'd define a location block "latest_data/" in the nginx config, then nginx would proxy the request to a separate process, That process could query the cached data server in the normal way. Still the same conversation, but the complexity gets hidden from the end-user.

Alternate Alternate approach: An nginx module to handle this functionality. It seems like it might be both serious overkill and not in keeping with the python feng-shui of OpenRVDAS, but I don't think we'd find an approach with less latency.

What I'm specifically thinking about for this is getting data into elog. USAP's current common configuration for elog automatically fills in things like latitude, longitude, depth, sea-surface temperature, etc when you create a new log entry. Using a python script to talk to the CDS on the elog host would require installing the script, the correct modules (the websocket library, for example, is not part of the python standard library), so much better to have integral support for this kind of data query.

— Reply to this email directly, view it on GitHub https://github.com/OceanDataTools/openrvdas/issues/367, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFO7V3WSZTWJQRVQGCWPU4DY2RDEVAVCNFSM6AAAAABFNEK7QKVHI2DSMVQWIX3LMV43ASLTON2WKOZSGIYTGNRZGI2DCOA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

LMG-ET commented 5 months ago

I’ve already done a crude implementation of option 2 here (just for testing), but if it didn’t add much to the size (both in-memory and on disk), I’d vote for option 1.

Option 3 would be fun, but it’s just silly.

--Kevin

From: David Pablo Cohn @.> Sent: Thursday, March 28, 2024 8:59 PM To: OceanDataTools/openrvdas @.> Cc: LMG-ET @.>; Author @.> Subject: Re: [OceanDataTools/openrvdas] Easier data retrieval from CDS (Issue #367)

WARNING: This email originated from outside of USAP’s email system. Do not click links or open attachments without verifying with the sender and confirming the link or document is safe.

Yesterday (or was it the day before?), while we were talking about reviving a RESTful API for the control UI, we discussed whether it also made sense to have something RESTful for the CDS. I don't think it would be all that hard.

But I'm going to snooze this in my queue until I get back from a bit of west Tasmania exploration next week.

On Fri, Mar 29, 2024 at 3:52 AM LMG-ET @. <mailto:@.> > wrote:

Currently if an external script (perhaps running on another machine) wants to grab a quick value from the CDS (i.e., What's the current air temperature) it involves a conversation: subscribe, ready, etc on a websocket. While this is fine for data you expect to be monitoring for a while (like, say, a display), it's less useful for a shell script.

What might be a useful enhancement is the ability to query the most recent data value for a variable, like (e.g., https://openvdas.mydomain/CDS/variable_name https://protect2.fireeye.com/v1/url?k=31323334-50bba2bf-3132e9a4-4544474f5631-52a2d0d7ff8e83f5&q=1&e=e4f927bb-6f6d-4e2a-80f6-10b4684fc002&u=https%3A%2F%2Fopenvdas.mydomain%2FCDS%2Fvariable_name ) and get a string back (data_value)

Possible approach: Since nginx can serve http and websockets from the same location block, on connection the CDS could check if it's receiving a normal HTTP request or a websocket connection, either returning a quick reply to a GET or proceeding as it does now for websocket traffic.

Alternative approach: Since small and modular is often better than a monolithic swiss army knife for embeds like ROV's, that may not be the way to go. A separate process (which you could optionally turn on with supervisord) that just gets proxied via nginx would work, so you'd do something like https://openrvdas.mydomain/latest_data/variable_name https://protect2.fireeye.com/v1/url?k=31323334-50bba2bf-3132e9a4-4544474f5631-1b723602bc9a929a&q=1&e=e4f927bb-6f6d-4e2a-80f6-10b4684fc002&u=https%3A%2F%2Fopenrvdas.mydomain%2Flatest_data%2Fvariable_name , you'd define a location block "latest_data/" in the nginx config, then nginx would proxy the request to a separate process, That process could query the cached data server in the normal way. Still the same conversation, but the complexity gets hidden from the end-user.

Alternate Alternate approach: An nginx module to handle this functionality. It seems like it might be both serious overkill and not in keeping with the python feng-shui of OpenRVDAS, but I don't think we'd find an approach with less latency.

What I'm specifically thinking about for this is getting data into elog. USAP's current common configuration for elog automatically fills in things like latitude, longitude, depth, sea-surface temperature, etc when you create a new log entry. Using a python script to talk to the CDS on the elog host would require installing the script, the correct modules (the websocket library, for example, is not part of the python standard library), so much better to have integral support for this kind of data query.

— Reply to this email directly, view it on GitHub <https://github.com/OceanDataTools/openrvdas/issues/367 https://protect2.fireeye.com/v1/url?k=31323334-50bba2bf-3132e9a4-4544474f5631-46fc8830fe30812c&q=1&e=e4f927bb-6f6d-4e2a-80f6-10b4684fc002&u=https%3A%2F%2Fgithub.com%2FOceanDataTools%2Fopenrvdas%2Fissues%2F367 >, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFO7V3WSZTWJQRVQGCWPU4DY2RDEVAVCNFSM6AAAAABFNEK7QKVHI2DSMVQWIX3LMV43ASLTON2WKOZSGIYTGNRZGI2DCOA https://protect2.fireeye.com/v1/url?k=31323334-50bba2bf-3132e9a4-4544474f5631-6aa2dfadb0563346&q=1&e=e4f927bb-6f6d-4e2a-80f6-10b4684fc002&u=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAFO7V3WSZTWJQRVQGCWPU4DY2RDEVAVCNFSM6AAAAABFNEK7QKVHI2DSMVQWIX3LMV43ASLTON2WKOZSGIYTGNRZGI2DCOA > . You are receiving this because you are subscribed to this thread.Message ID: @. <mailto:@.> >

— Reply to this email directly, view it on GitHub https://protect2.fireeye.com/v1/url?k=31323334-50bba2bf-3132e9a4-4544474f5631-1f0f30825d03c35c&q=1&e=e4f927bb-6f6d-4e2a-80f6-10b4684fc002&u=https%3A%2F%2Fgithub.com%2FOceanDataTools%2Fopenrvdas%2Fissues%2F367%23issuecomment-2026102113 , or unsubscribe https://protect2.fireeye.com/v1/url?k=31323334-50bba2bf-3132e9a4-4544474f5631-ce505f12d9089c76&q=1&e=e4f927bb-6f6d-4e2a-80f6-10b4684fc002&u=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAB74QHSICGW4ISLMZZFHT73Y2R77NAVCNFSM6AAAAABFNEK7QKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRWGEYDEMJRGM . You are receiving this because you authored the thread. https://github.com/notifications/beacon/AB74QHX7RHRUXJZV5OXLFSLY2R77NA5CNFSM6AAAAABFNEK7QKWGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTTYYPOWC.gif Message ID: @. @.> >