search

rburghol / om_agman

0 stars 0 forks source link

HOBO JSON Feed Development #278

Open rburghol opened 3 years ago

rburghol commented 3 years ago

Tasks:

Use

Call Each Sensor Update (from file list)

#!/bin/bash
$single=$1
while IFS= read -r line; do
  #echo "Text read from file: $line"
  IFS="$IFS," read fid hobo_logger hobo_userid days_behind <<< "$line"
  if [ $single -eq 1 ]; then
    ./catch_up_hobo 1 $fid $hobo_logger $hobo_userid 1
  else
    catch_up_hobo $days_behind $fid $hobo_logger $hobo_userid 0
  fi
  drush scr modules/dh_weather/src/summarise_last24.php dh_feature $fid
done < '/tmp/hobo_devices.txt'

Generate list of devices

COPY (
select a.hydroid, b.propcode as hobo_logger, c.propcode as hobo_userid,
  min(ceil((extract(epoch from now()) - tsw.tstime)/86400)) as days_behind
from dh_properties as b 
left outer join dh_feature as a 
on (
  a.hydroid = b.featureid) 
left outer join dh_properties as c 
on (
  c.featureid = b.featureid 
  and c.propname = 'hobo_userid' 
  and c.entity_type = 'dh_feature'
) 
left outer join dh_timeseries_weather as tsw 
on (
  tsw.featureid = a.hydroid
  and tsw.entity_type = 'dh_feature'
  and tsw.varid in (select hydroid from dh_variabledefinition where varkey = 'weather_obs')
)
where b.entity_type = 'dh_feature' 
and b.propname = 'hobo_logger'
group by a.hydroid, b.propcode, c.propcode
) to '/tmp/hobo_devices.txt' with CSV
;

Data Model

This needs to have an updated model to handle the new station config (single sensor instead of multiple as before). But for now it is keeping the old method.

R Oauth token access

Code 1: Download data from HOBO.

get_url = paste(site,"ws/data/file", "json", "user", userid, sep="/")
hobo_rest <- httr::GET(
  get_url,
  encode = "application/json",
  query = inputs,
  config = list(token = access_token),
  httr::add_headers(Authorization=paste("bearer", access_token)),
  httr::verbose()
);
hobo_rest
# gives a list of freaky formatting, but the data is there...
raw_data = httr::content(hobo_rest)$observation_list

# put into crosstab

PHP Testing:

R Code Dev:

HOBO Station REST Documentation:

Drupal REST client

rburghol commented 3 years ago

Installing Guzzle:

# intall and ebable curl php extension
sudo apt-get install php5-curl
sudo apachectl restart

# See: https://guzzle3.readthedocs.io/getting-started/installation.html
# install composer  is this needed or is only the php one?
sudo apt-get install composer
# composer php install. This is needed, is the previous?
cd /var/www/d.live
curl -sS https://getcomposer.org/installer | php

# install guzzle - see also: https://guzzle3.readthedocs.io/getting-started/installation.html
composer require guzzlehttp/guzzle

Then, we can test Guzzle with this:

<?php

require 'vendor/autoload.php';

use GuzzleHttp\Client;

$client = new Client([
    // Base URI is used with relative requests
    'base_uri' => 'http://httpbin.org',
    // You can set any number of default request options.
    'timeout'  => 2.0,
]);

dpm($client,'client');

?>
rburghol commented 3 years ago

NOTE: UPdated 9/15/2021, changed httr command from "GET" to "POST" @hdaniel7 the following doesn't work, but it gets a small foundation based on the hydrotools rest code. The contents of "/opt/model/config.private" is that json string with username and password. Hope it is helpful:

library("rjson")
site = "https://webservice.hobolink.com"

client_info <- rjson::fromJSON(file="/opt/model/config.private")
username <- as.character(client_info$onset_client_id)
userpass <- as.character(client_info$onset_client_secret)

inputs <- list(
  client_id = username,
  client_secret = userpass
)
hobo_rest <- httr::POST(
  paste0(site, "/","ws/auth/token"), 
  query = inputs, 
  encode = "json"
);

httr::content(hobo_rest)
rburghol commented 3 years ago

Yo @hdaniel7 !! I got it, this works to get the token (what we do with the toke after is anyone's guess!):

hobo_rest <- httr::POST(
  paste0(site, "/","ws/auth/token"),
  encode = "form",
  httr::content_type("application/x-www-form-urlencoded"),
  query = inputs

);

httr::content(hobo_rest)
hdaniel7 commented 3 years ago

Great to know! I haven't had a chance to dig further into it but I'll try to figure how to utilize the token in the coming days

On Wed, Sep 15, 2021, 2:19 PM rburghol @.***> wrote:

Yo @hdaniel7 https://github.com/hdaniel7 !! I got it, this works to get the token (what we do with the toke after is anyone's guess!):

hobo_rest <- httr::POST( paste0(site, "/","ws/auth/token"), encode = "form", httr::content_type("application/x-www-form-urlencoded"), query = inputs

);

httr::content(hobo_rest)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/rburghol/om_agman/issues/278#issuecomment-920266837, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIXJA62OAGOYW7U4YULWXBLUCDPTFANCNFSM44DPQOWQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

hdaniel7 commented 3 years ago

interestingly enough, the chunk of code you posted doesn't seem to work for me -- running it gives me the error "Error in curl::curl_fetch_memory(url, handle = handle) : Protocol "" not supported or disabled in libcurl". Digging into it a bit, it seems to take issue with the query parameter and the lack of a body parameter -- to make sure I understand what exactly the parameters were doing, I changed it into

hobo_rest <- httr::POST( url = paste0(site, "/","ws/auth/token"), encode = "form", config = httr::content_type("application/x-www-form-urlencoded"), query = inputs

);

httr::content(hobo_rest)

and am having the same issue. Interestingly enough, if I add a body parameter and comment out the query parameter (i.e.

hobo_rest <- httr::POST( url = paste0(site, "/","ws/auth/token"), encode = "form", config = httr::content_type("application/x-www-form-urlencoded"),

query = inputs,

body = 'hi'

);

it seems to at least communicate with the site, even if I get an error 405 (method not allowed) returned instead of anything useful. Any ideas why it's working for you and not for me? Looking at some of the old hydro_tools code, it became readily apparent that you've used code with very similar syntax before that worked perfectly... so I'm a bit confused as to why I'm struggling so much to get this to function. Perhaps you're using some package besides curl as the libcurl binding?

Looking ahead, I realized that I'll probably need the serial number of the HOBO device in order to test any code after I'm successfully able to get a token -- could you send that my way or put it in the authentification document on the drive or something when you have a chance?

rburghol commented 3 years ago

@hdaniel7 - the loggerid you can use is 20622331 as for the connection issue, 2 things:

From my read of that error message, it is saying that something with the library "curl" that your machine is using does not allow and empty protocol. Looking at my code, it appears that I added an additional variable in the inputs list, which may be the reason that the protocol fails. An updated set of code that is working for me is below:

library("rjson")
site = "https://webservice.hobolink.com"

client_info <- rjson::fromJSON(file="/opt/model/config.private")
username <- as.character(client_info$onset_client_id)
userpass <- as.character(client_info$onset_client_secret)

inputs <- list(
  client_id = username,
  client_secret = userpass,
  grant_type = 'client_credentials'
)
hobo_rest <- httr::POST(
  paste0(site, "/","ws/auth/token"),
  encode = "form",
  httr::content_type("application/x-www-form-urlencoded"),
  query = inputs

);

hobo_response <- httr::content(hobo_rest)
access_token <- hobo_response$access_token

If that doesn't work, it suggests to me that maybe there is some version information that has changed, or perhaps that I have some setting on my machine that is separate from yours. In other words, it does not seem, to me, that the error message is coming from the server at hobolink. Especially since I have successfully authenticated with the same code, and gotten the authentication token back from them. But anyhow, pop that up there.

hdaniel7 commented 3 years ago

Backup copy of the V3 HOBOlink web developer's guide -- just in case they move it around or get rid of it again: 25113-A HOBOlink Web Services V3 Developer's Guide.pdf