source http map: Support nested keys

[jgroom33]

Currently, keys must exist at the top level of the http response

https://github.com/chengguangnan/vine

The above gem would allow nested object access such as:

source:
  default: http
  http:
    url: https://url/api
    scheme: https
    delimiter: !ruby/regexp /:/
    user: username
    pass: password
    map:
      name: properties.hostname
      model: properties.os
      username: properties.credentials.username
      password: properties.credentials.password

[ytti]

I'm bit confused. Can you explain how things are now, now they would be ideally and how would the change be useful.

[jgroom33]

current behavior

http source response data

{"hostname":"1.1.1.1",
   "os":"cisco",
    "username":"cisco",
     "password":"pays"
}

configuration

source:
  default: http
  http:
    url: https://url/api
    scheme: https
    delimiter: !ruby/regexp /:/
    user: username
    pass: password
    map:
      name: hostname
      model: os
      username: username
      password: password

desired behavior

http source response data

{ 
   "properties":{
      "hostname":"1.1.1.1",
      "os":"cisco",
      "credentials":{
          "username":"cisco",
          "password":"pays"
      }
   }
}

configuration

source:
  default: http
  http:
    url: https://url/api
    scheme: https
    delimiter: !ruby/regexp /:/
    user: username
    pass: password
    map:
      name: properties.hostname
      model: properties.os
      username: properties.credentials.username
      password: properties.credentials.password

[jgroom33]

If the feature seems useful, there are implementation options: vine gem hashie gem native ruby way??

[ytti]

We could do it, adding it here: https://github.com/ytti/oxidized/blob/master/lib/oxidized/source/http.rb#L40-L52

Is the rationale that you have this HTTP API, which does return this hierarchical data, and you cannot use HTTP source now because the HTTP source expects flat data?

in SQL we also expect flat data, but because we support arbitrary SQL statements, users can just join tables etc, to get single flat results. So in a sense SQL is flexible, HTTP is not, so what you say seems important, provided we have practical use-case.

[slarimore02]

I would also like to use nested keys. I am a new user to oxidized and I wanted to import devices for config backup from the Netbox IPAM solution. I've included an example response body from the netbox api for devices. I would like to utilize the platform.slug object for mapping the operating system.

[ { "id": 2, "name": "nxs01sxsccc", "display_name": "nxs01sxsccc", "device_type": { "id": 13, "manufacturer": { "id": 2, "name": "Cisco", "slug": "cisco" }, "model": "Nexus 5672", "slug": "nexus-5672" }, "device_role": { "id": 13, "name": "Management", "slug": "management" }, "tenant": null, "platform": { "id": 2, "name": "Cisco NX-OS", "slug": "cisco-nx-os" }, "serial": "serial number", "asset_tag": null, "rack": { "id": 18, "name": "rack5", "facility_id": null, "display_name": "rack5" }, "position": 17, "face": 1, "parent_device": null, "status": true, "primary_ip": { "id": 169, "family": 4, "address": "192.168.1.10/32" }, "primary_ip4": { "id": 169, "family": 4, "address": "192.168.1.10/32" }, "primary_ip6": null, "comments": "", "custom_fields": {} } ]

[ytti]

@jgroom33 @slarimore02 this makes perfect sense, even more so now that there is actual real-life example of an API we'd like to support, but currently cannot.

This is something I definitely want to support and I'm ready to write the code myself.

Is there any chance to expose the API to some of my machines, so that I can verify if my change works correctly or not? Other option is that we just go back-and-forth until we make it work.

[slarimore02]

Great! Here's the link to the docker installation guide for netbox if you want to deploy them on your machine or I'd be happy to test with my installation if that would be easier. http://netbox.readthedocs.io/en/latest/installation/docker/

Once the container is up you can navigate to /api/docs/ to get to the Swagger API documentation. http://netbox.readthedocs.io/en/latest/api-integration/

Let me know if you have any questions or if I can help in any way!

[ytti]

Yeah I'm not planning to install anything, on account of lazy. But it would be grand if I'd have some URL I can use to get the data, but I guess I can just fake it and use the JSON you already provided.

[slarimore02]

Unfortunately I don't have a public URL but I'd be more than willing to run any commands and test any code on my instance. Let me know if you need any output to help write the code.

[ytti]

Please test this: https://github.com/ytti/oxidized/commit/ddbfedc97c5add661286d0f12c4ac98275dbeb82

[slarimore02]

No dice. I get a unknown user/password. The API isn't authenticated so I left both out of the http config of the config file. I've attached both the crash and config file.

config.txt crash.txt

[ytti]

undefined method `dig' for ["detail", "Invalid username/password."]:Array [NoMethodError]

It looks like you've not received object from the HTTP. We expect object, you received Array, and clearly the Array didn't contain your nodes.

Looks like your API is complaining about your auth.

[slarimore02]

Here's the curl output for a API call for one devices from the same machine that oxidized was installed on. It's not authenticating API calls so I'm not sure why the crash file is complaining about auth. Would it be possible to support Arrays like this or are objects only going to be supported?

root@spoc-tools01:/etc/oxidized# curl -ik https://netbox.spoc.linux/api/dcim/devices/?name=nxs01sxsccc HTTP/1.1 200 OK Date: Tue, 14 Feb 2017 04:00:15 GMT Server: gunicorn/19.4.5 Vary: Accept,Cookie X-Frame-Options: SAMEORIGIN Content-Type: application/json Allow: GET, HEAD, OPTIONS Access-Control-Allow-Origin: * Transfer-Encoding: chunked

[{"id":2,"name":"nxs01sxsccc","display_name":"nxs01sxsccc","device_type":{"id":13,"manufacturer":{"id":2,"name":"Cisco","slug":"cisco"},"model":"Nexus 5672","slug":"nexus-5672"},"device_role":{"id":13,"name":"Management","slug":"management"},"tenant":null,"platform":{"id":2,"name":"Cisco NX-OS","slug":"cisco-nx-os"},"serial":"FOC153","asset_tag":null,"rack":{"id":18,"name":"AE.R02.R05","facility_id":null,"display_name":"R05"},"position":17,"face":1,"parent_device":null,"status":true,"primary_ip":{"id":169,"family":4,"address":"192.168.1.1/32"},"primary_ip4":{"id":169,"family":4,"address":"192.168.1.10/32"},"primary_ip6":null,"comments":"","custom_fields":{}}]

[matejv]

I bet you ran into the same issue as me. See #720

Try to add user: false to your config:

source:
  http:
    user: false

[ytti]

Array of objects is fine, but you didn't receive one. @matejv is probably right, I fixed that too.

[slarimore02]

After changing user=false it was able to pull the device OS from a nested key successfully. I'll download the latest changes and test again without the user=false.

[slarimore02]

Tested this morning with the latest code and without the user=false. Ran into no startup issues and everything is running well. Thanks for your help!

[jgroom33]

Looks good to me. Nice work!