balena-labs-projects / connector

Auto-configured data connector block based on Telegraf
15 stars 8 forks source link

Facilitate input of strings and boolean values via JSON #10

Closed chrisys closed 3 years ago

chrisys commented 4 years ago

This applies at the least to HTTP pull and push, unsure about other inputs.

If we add something like json_string_fields = ['string_*', 'boolean_*'] to the telegraf configuration it means fields in the received JSON with these prefixes will be allowed to pass to the database, without this, strings and boolean values are dropped indiscriminately.

chrisys commented 4 years ago

An expansion upon this, I'm using HTTP pull with an API endpoint that returns a mix of field types; in this situation the suggested prefix of string_* doesn't solve for the use case as of course as an end user I can't control the names of the fields provided by the 3rd party API.

I'm not sure if there is a way we can tell telegraf to bring in all the fields it encounters rather than just numeric ones, if not then perhaps we should allow the user of this block to be able to specify string fields to retrieve as part of the plugin configuration.

Example output:

      "WindDirection": "W",
      "WindGust": 1.03,
      "SustainedWindSpeed": 0.28

In this case the WindDirection field is discarded and not stored, because it is a string.

chrisys commented 4 years ago

@phil-d-wilson I was testing this but couldn't get it working. It initially breaks if the env var is not set and needs a check to see if it exists before .split, something like:

string_fields = os.environ.get('EXTERNAL_HTTP_PULL_STRINGS_FIELDS')
    if(string_fields is not None):

But even after that, and after adding the env var, I'm getting stuff like the below, and telegraf.conf doesn't exist. Screenshot 2020-08-23 at 22 31 44

chrisys commented 4 years ago

OK nm I got it, PR incoming :D