influxdata / telegraf

Agent for collecting, processing, aggregating, and writing metrics, logs, and other arbitrary data.
https://influxdata.com/telegraf
MIT License
14.6k stars 5.57k forks source link

HTTP plugin code 400 on connection to Citrix ADC #11478

Closed SpiderD555 closed 2 years ago

SpiderD555 commented 2 years ago

Relevant telegraf.conf

[[inputs.http]]
  urls = [
        "https://netscaler/nitro/v1/stat/lbvserver"
        ]
  method = "GET"

  cookie_auth_url = "https://netscaler/nitro/v1/config/login"
  cookie_auth_method = "POST"
  cookie_auth_headers = { Content-Type = "application/json"}
  cookie_auth_body = '{"login":{"username": "username", "password":"password"}}'
  ## cookie_auth_renewal not set or set to "0" will auth once and never renew the cookie
  cookie_auth_renewal = "1h"
  data_format = "json_v2"

  insecure_skip_verify = true

Logs from Telegraf

E! [telegraf] Error running agent: could not initialize input inputs.http: cookie auth renewal received status code: 400 (Bad Request)

System info

Telegraf 1.22.4 (also tried 1.24.0~e945364b), Redhat 8.6

Docker

No response

Steps to reproduce

  1. Configure HTTP plugin as shown in example
  2. Configure Citrix ADC with basic credentials
  3. Start Telegraf
  4. Telegraf stops with error

Expected behavior

I expect Telegraf HTTP plugin will connect to Citrix ADC and initiate the session, so I can draw stats using API calls

Moreover Telegraf stops on that error code 400, which in my humble opinion may be a bad design choice since it may crash otherwise a perfectly fine running Telegraf instance. Telegraf should just ignore the error and maybe retry later instead of stopping entirely. Imagine a situation where we have 1 HTTP endpoint and 99 SNMP devices. When we poll a device with HTTP plugin and it replies with non-200 code (because of some internal error), then whole Telegraf instance crashes and stops gathering statistics for those 99 perfectly working devices.

Actual behavior

E! [telegraf] Error running agent: could not initialize input inputs.http: cookie auth renewal received status code: 400 (Bad Request)

Additional info

When I use curl then it all works:

Establishing new session curl -ik https://netscaler/nitro/v1/config/login -X POST -H 'Content-Type: application/json' -d '{ "login" :{"username": "username", "password": "password"}}' HTTP/1.1 201 Created Date: Fri, 08 Jul 2022 06:56:16 GMT Server: Apache X-Frame-Options: SAMEORIGIN Set-Cookie: SESSID=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; Max-Age=0; path=/ Expires: Thu, 19 Nov 1981 08:52:00 GMT Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Pragma: no-cache Set-Cookie: sessionid=%23%23465611085B10AEECBC084499AA564FABA40BACAE435A343F2E40A96A12191918377A434FE2959F11C56076F0E08DDA840EFF0A9968DA9BAA0CA70555A6D168D5155CEBBAB02F9101FBED406B01164D0E617790FE785EF572F90C11FBDFE84FA7437466B824ECA2BC9652FB39FFCAF3DEF23B3AD421A53C32A8D52F75B874; path=/nitro/v1 Feature-Policy: camera 'none'; microphone 'none'; geolocation 'none' Referrer-Policy: no-referrer X-XSS-Protection: 1; mode=block X-Content-Type-Options: nosniff Content-Length: 328 Content-Type: application/json; charset=utf-8

{ "errorcode": 0, "message": "Done", "severity": "NONE", "sessionid": "##465611085B10AEECBC084499AA564FABA40BACAE435A343F2E40A96A12191918377A434FE2959F11C56076F0E08DDA840EFF0A9968DA9BAA0CA70555A6D168D5155CEBBAB02F9101FBED406B01164D0E617790FE785EF572F90C11FBDFE84FA7437466B824ECA2BC9652FB39FFCAF3DEF23B3AD421A53C32A8D52F75B874" }

Getting example statistics curl -ik https://netscaler/nitro/v1/stat/lbvserver -H 'Content-Type: application/json' --cookie "sessionid=%23%23465611085B10AEECBC084499AA564FABA40BACAE435A343F2E40A96A12191918377A434FE2959F11C56076F0E08DDA840EFF0A9968DA9BAA0CA70555A6D168D5155CEBBAB02F9101FBED406B01164D0E617790FE785EF572F90C11FBDFE84FA7437466B824ECA2BC9652FB39FFCAF3DEF23B3AD421A53C32A8D52F75B874; path=/nitro/v1" HTTP/1.1 200 OK Date: Fri, 08 Jul 2022 06:57:05 GMT Server: Apache X-Frame-Options: SAMEORIGIN Expires: Thu, 19 Nov 1981 08:52:00 GMT Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Pragma: no-cache Vary: Accept-Encoding Feature-Policy: camera 'none'; microphone 'none'; geolocation 'none' Referrer-Policy: no-referrer X-XSS-Protection: 1; mode=block X-Content-Type-Options: nosniff Content-Length: 1640 Content-Type: application/json; charset=utf-8

{ "errorcode": 0, "message": "Done", "severity": "NONE", "lbvserver": [ { "name": "lb-test-ssl", "avgcltttlb": "0", "cltresponsetimeapdex": 1.000000, "vsvrsurgecount": "0", "establishedconn": "0", "inactsvcs": "0", "vslbhealth": "0", "primaryipaddress": "169.254.1.1", "primaryport": 443, "type": "SSL", "state": "DOWN", "actsvcs": "0", "cpuusagepm": "0", "tothits": "0", "hitsrate": 0, "totalrequests": "0", "requestsrate": 0, "totalresponses": "0", "responsesrate": 0, "totalrequestbytes": "0", "requestbytesrate": 0, "totalresponsebytes": "0", "responsebytesrate": 0, "totalh2requests": "0", "h2requestsrate": 0, "totalh2responses": "0", "h2responsesrate": 0, "totalpktsrecvd": "0", "pktsrecvdrate": 0, "totalpktssent": "0", "pktssentrate": 0, "curclntconnections": "0", "cursrvrconnections": "0", "curpersistencesessions": "0", "curbackuppersistencesessions": "0", "surgecount": "0", "svcsurgecount": "0", "sothreshold": "0", "totspillovers": "0", "labelledconn": "0", "pushlabel": "0", "deferredreq": "0", "deferredreqrate": 0, "invalidrequestresponse": "0", "invalidrequestresponsedropped": "0", "totvserverdownbackuphits": "0", "curmptcpsessions": "0", "cursubflowconn": "0", "totalconnreassemblyqueue75": "0", "totalconnreassemblyqueueflush": "0", "totalsvrbusyerr": "0", "svrbusyerrrate": 0, "reqretrycount": "0", "reqretrycountexceeded": "0", "httpmaxhdrszpkts": "0", "httpmaxhdrfldlenpkts": "0", "tcpmaxooopkts": "0", "totcltttlbtransactions": "0", "cltttlbtransactionsrate": 0, "toleratingttlbtransactions": "0", "toleratingttlbtransactionsrate": 0, "frustratingttlbtransactions": "0", "frustratingttlbtransactionsrate": 0 } ] }

At first I thought that maybe Telegraf just won't accept code 201 as per https://github.com/influxdata/telegraf/issues/11134 so I tested new build https://github.com/influxdata/telegraf/pull/11472 but it still fails

powersj commented 2 years ago

Hi,

curl -ik https://netscaler/nitro/v1/config/login -X POST -H 'Content-Type: application/json' -d '{ "login" :{"username": "username", "password": "password"}}' HTTP/1.1 201 Created

It does look like the system you are connecting to returns a 201, which will eventually require the fix in #11472. However, Telegraf is not getting that far and instead getting a 400 per your telegraf log. The Citrix ADC docs say the reason is in the body, so let's try to find out why.

I have put up #11482, which should have some artifacts attached shortly. Can you give that a try and see what the body says?

SpiderD555 commented 2 years ago

Hi,

Same story with latest build (with more debug information this time):

Jul 11 08:59:54 localhost telegraf[2894337]: 2022-07-11T06:59:54Z E! [telegraf] Error running agent: could not initialize input inputs.http: cookie auth renewal received status code: 400 (Bad Request) - <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
Jul 11 08:59:54 localhost telegraf[2894337]: <html><head>
Jul 11 08:59:54 localhost telegraf[2894337]: <title>400 Bad Request</title>
Jul 11 08:59:54 localhost telegraf[2894337]: </head><body>
Jul 11 08:59:54 localhost telegraf[2894337]: <h1>Bad Request</h1>
Jul 11 08:59:54 localhost telegraf[2894337]: <p>Your browser sent a request that this server could not understand.<br />
Jul 11 08:59:54 localhost telegraf[2894337]: </p>
Jul 11 08:59:54 localhost telegraf[2894337]: </body></html>
Jul 11 08:59:54 localhost telegraf[2894337]: {"errorcode":-1,"message":"Invalid payload format. Default format is json","severity":"ERROR"}

Because of "Invalid payload format. Default format is json" I think we may need to see exactly what Telegraf is sending to Citrix ADC, to get to the root cause of the problem.

powersj commented 2 years ago

Thanks for trying that out!

We saw something similar in #11083 and learned that some devices are expected the content length and did not like a request formed by Go's NopCloser. Given this is for a request to generate a cookie, the request will not be huge, and there is no need to use the NopCloser.

I have pushed an update to the PR that will change the body to use a io.Reader over an io.ReadCloser and also print out some debug information before doing the request.

Can you give that update a shot once new artifacts are attached?

SpiderD555 commented 2 years ago

I am not sure which artifact you are referring to. I tried this one: https://github.com/influxdata/telegraf/pull/11482

Anyway the result is following: Telegraf is not stopping, but I also don't see any data from the Citrix device in my InfluxDB instance (however I do get data from few test SNMP probes). The only log I see is only this:

https://netscaler/nitro/v1/config/login
POST
map[Content-Type:[application/json]]

So far I didn't see any other connection attempts to Citrix in logs, so I assume that it authenticates successfully, but it is not sending any proper API calls to draw data.

Now the situation is getting a bit more interesting when I set cookie_auth_renewal = "5m"

I am getting inital session authentication, then Telegraf tries to renew the session with old session cookie (which I suppose it shouldn't as per the log):


Jul 12 12:03:27 localhost telegraf[2974996]: 2022-07-12T10:03:27Z D! [agent] Initializing plugins
Jul 12 12:03:27 localhost telegraf[2974996]: https://netscaler/nitro/v1/config/login
Jul 12 12:03:27 localhost telegraf[2974996]: POST
Jul 12 12:03:27 localhost telegraf[2974996]: map[Content-Type:[application/json]]

Jul 12 12:08:27 localhost telegraf[2974996]: https://netscaler/nitro/v1/config/login
Jul 12 12:08:27 localhost telegraf[2974996]: POST
Jul 12 12:08:27 localhost telegraf[2974996]: map[Content-Type:[application/json] Cookie:[sessionid=%23%238543669E93EC9727F77E001DCF46FF973974DCBD31F174C64A6DC
F2302E28B28E86C6B5B9867FD642CA6C2856D138218E78151433B843E4DF51D4BE87C5662B9B79495237AE13ECD4B6D8CF12553552D37AAD71F59BF3312878E8FB88E650928DADA7322F5DFE4BACC88713E0F5A29ADA1B52DD7D8386
E61F5FD2FE3A41D]]
Jul 12 12:08:27 localhost telegraf[2974996]: 2022-07-12T10:08:27Z E! [inputs.http] renewal failed for "https://netscaler/nitro/v1/c
onfig/login": cookie auth renewal received status code: 401 (Unauthorized) - { "errorcode": 354, "message": "Invalid username or password", "severity": "ERROR" }

Jul 12 12:13:27 localhost telegraf[2974996]: https://netscaler/nitro/v1/config/login
Jul 12 12:13:27 localhost telegraf[2974996]: POST
Jul 12 12:13:27 localhost telegraf[2974996]: map[Content-Type:[application/json] Cookie:[sessionid=%23%238543669E93EC9727F77E001DCF46FF973974DCBD31F174C64A6DC
F2302E28B28E86C6B5B9867FD642CA6C2856D138218E78151433B843E4DF51D4BE87C5662B9B79495237AE13ECD4B6D8CF12553552D37AAD71F59BF3312878E8FB88E650928DADA7322F5DFE4BACC88713E0F5A29ADA1B52DD7D8386
E61F5FD2FE3A41D]]
Jul 12 12:13:27 localhost telegraf[2974996]: 2022-07-12T10:13:27Z E! [inputs.http] renewal failed for "https://netscaler/nitro/v1/c
onfig/login": cookie auth renewal received status code: 401 (Unauthorized) - { "errorcode": 354, "message": "Invalid username or password", "severity": "ERROR" }
powersj commented 2 years ago

So far I didn't see any other connection attempts to Citrix in logs, so I assume that it authenticates successfully, but it is not sending any proper API calls to draw data.

It sounds like progress and indeed sounds like the Cirtix server was possibly expecting the content length similar to #11083.

I am getting inital session authentication, then Telegraf tries to renew the session with old session cookie (which I suppose it shouldn't as per the log):

Can you demonstrate a renewal with curl 1) with the cookie in the headers and 2) without the cookie in the headers, please? The 401 makes it pretty clear that the user credentials are wrong on renewal, so it seems unable to get a new cookie which is why the old cookie is used.

SpiderD555 commented 2 years ago

Let me demonstrate 2)

curl -ik https://netscaler/nitro/v1/config/login -X POST -H 'Content-Type: application/json' -d '{ "login" :{"username": "username", "password": "password"}}'
HTTP/1.1 201 Created
Date: Wed, 13 Jul 2022 06:07:51 GMT
Server: Apache
X-Frame-Options: SAMEORIGIN
Set-Cookie: SESSID=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; Max-Age=0; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Set-Cookie: sessionid=%23%238F90C1B1BEE9369035623B2BFD6448F32C7F2EC3FDCFF3D81C5847DB174ADEA8EA277F4D6AC7695E37C829C5E0BA0675E81BD15DCC4C226CE3A4EAD38F64B40E4CD5C2623775D0A4070EEDA0434DB0657BAD7AED4933D57AC75B6790EF946F0F1FBE1A1D0C35F7E406D1C5FADD5242585B6163DC8E039EE19166F27F68A3; path=/nitro/v1
Feature-Policy: camera 'none'; microphone 'none'; geolocation 'none'
Referrer-Policy: no-referrer
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Content-Length: 328
Content-Type: application/json; charset=utf-8

{ "errorcode": 0, "message": "Done", "severity": "NONE", "sessionid": "##8F90C1B1BEE9369035623B2BFD6448F32C7F2EC3FDCFF3D81C5847DB174ADEA8EA277F4D6AC7695E37C829C5E0BA0675E81BD15DCC4C226CE3A4EAD38F64B40E4CD5C2623775D0A4070EEDA0434DB0657BAD7AED4933D57AC75B6790EF946F0F1FBE1A1D0C35F7E406D1C5FADD5242585B6163DC8E039EE19166F27F68A3" }

=============================================

curl -ik https://netscaler/nitro/v1/stat/lbvserver -H 'Content-Type: application/json' --cookie "sessionid=%23%238F90C1B1BEE9369035623B2BFD6448F32C7F2EC3FDCFF3D81C5847DB174ADEA8EA277F4D6AC7695E37C829C5E0BA0675E81BD15DCC4C226CE3A4EAD38F64B40E4CD5C2623775D0A4070EEDA0434DB0657BAD7AED4933D57AC75B6790EF946F0F1FBE1A1D0C35F7E406D1C5FADD5242585B6163DC8E039EE19166F27F68A3; path=/nitro/v1"
HTTP/1.1 200 OK
Date: Wed, 13 Jul 2022 06:08:55 GMT
Server: Apache
X-Frame-Options: SAMEORIGIN
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Vary: Accept-Encoding
Feature-Policy: camera 'none'; microphone 'none'; geolocation 'none'
Referrer-Policy: no-referrer
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Content-Length: 1640
Content-Type: application/json; charset=utf-8

{ "errorcode": 0, "message": "Done", "severity": "NONE", "lbvserver": [ { "name": "lb-test-ssl", "avgcltttlb": "0", "cltresponsetimeapdex": 1.000000, "vsvrsurgecount": "0", "establishedconn": "0", "inactsvcs": "0", "vslbhealth": "0", "primaryipaddress": "169.254.1.1", "primaryport": 443, "type": "SSL", "state": "DOWN", "actsvcs": "0", "cpuusagepm": "0", "tothits": "0", "hitsrate": 0, "totalrequests": "0", "requestsrate": 0, "totalresponses": "0", "responsesrate": 0, "totalrequestbytes": "0", "requestbytesrate": 0, "totalresponsebytes": "0", "responsebytesrate": 0, "totalh2requests": "0", "h2requestsrate": 0, "totalh2responses": "0", "h2responsesrate": 0, "totalpktsrecvd": "0", "pktsrecvdrate": 0, "totalpktssent": "0", "pktssentrate": 0, "curclntconnections": "0", "cursrvrconnections": "0", "curpersistencesessions": "0", "curbackuppersistencesessions": "0", "surgecount": "0", "svcsurgecount": "0", "sothreshold": "0", "totspillovers": "0", "labelledconn": "0", "pushlabel": "0", "deferredreq": "0", "deferredreqrate": 0, "invalidrequestresponse": "0", "invalidrequestresponsedropped": "0", "totvserverdownbackuphits": "0", "curmptcpsessions": "0", "cursubflowconn": "0", "totalconnreassemblyqueue75": "0", "totalconnreassemblyqueueflush": "0", "totalsvrbusyerr": "0", "svrbusyerrrate": 0, "reqretrycount": "0", "reqretrycountexceeded": "0", "httpmaxhdrszpkts": "0", "httpmaxhdrfldlenpkts": "0", "tcpmaxooopkts": "0", "totcltttlbtransactions": "0", "cltttlbtransactionsrate": 0, "toleratingttlbtransactions": "0", "toleratingttlbtransactionsrate": 0, "frustratingttlbtransactions": "0", "frustratingttlbtransactionsrate": 0 } ] }

=============================================

curl -ik https://netscaler/nitro/v1/config/login -X POST -H 'Content-Type: application/json' -d '{ "login" :{"username": "username", "password": "password"}}'
HTTP/1.1 201 Created
Date: Wed, 13 Jul 2022 06:09:37 GMT
Server: Apache
X-Frame-Options: SAMEORIGIN
Set-Cookie: SESSID=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; Max-Age=0; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Set-Cookie: sessionid=%23%2368BD0AF9A3A9134600B0C207E7EBED684541F64FF85884273CF676CB9D411DF693B4F1420EB77BE061F1D655C2486E025A05E26D14A513D1FBD6641D218F42EA9B8FB05619A2AD2715DD0A6AC95CCA11D44BD7B1A082095F84629C295542431A4DA9BC902A5762E60849037EEDD2EE4C9C0D05B8D600B9040822D6BEA804; path=/nitro/v1
Feature-Policy: camera 'none'; microphone 'none'; geolocation 'none'
Referrer-Policy: no-referrer
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Content-Length: 328
Content-Type: application/json; charset=utf-8

{ "errorcode": 0, "message": "Done", "severity": "NONE", "sessionid": "##68BD0AF9A3A9134600B0C207E7EBED684541F64FF85884273CF676CB9D411DF693B4F1420EB77BE061F1D655C2486E025A05E26D14A513D1FBD6641D218F42EA9B8FB05619A2AD2715DD0A6AC95CCA11D44BD7B1A082095F84629C295542431A4DA9BC902A5762E60849037EEDD2EE4C9C0D05B8D600B9040822D6BEA804" }

=============================================

curl -ik https://netscaler/nitro/v1/stat/lbvserver -H 'Content-Type: application/json' --cookie "sessionid=%23%2368BD0AF9A3A9134600B0C207E7EBED684541F64FF85884273CF676CB9D411DF693B4F1420EB77BE061F1D655C2486E025A05E26D14A513D1FBD6641D218F42EA9B8FB05619A2AD2715DD0A6AC95CCA11D44BD7B1A082095F84629C295542431A4DA9BC902A5762E60849037EEDD2EE4C9C0D05B8D600B9040822D6BEA804; path=/nitro/v1"
HTTP/1.1 200 OK
Date: Wed, 13 Jul 2022 06:10:41 GMT
Server: Apache
X-Frame-Options: SAMEORIGIN
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Vary: Accept-Encoding
Feature-Policy: camera 'none'; microphone 'none'; geolocation 'none'
Referrer-Policy: no-referrer
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Content-Length: 1640
Content-Type: application/json; charset=utf-8

{ "errorcode": 0, "message": "Done", "severity": "NONE", "lbvserver": [ { "name": "lb-test-ssl", "avgcltttlb": "0", "cltresponsetimeapdex": 1.000000, "vsvrsurgecount": "0", "establishedconn": "0", "inactsvcs": "0", "vslbhealth": "0", "primaryipaddress": "169.254.1.1", "primaryport": 443, "type": "SSL", "state": "DOWN", "actsvcs": "0", "cpuusagepm": "0", "tothits": "0", "hitsrate": 0, "totalrequests": "0", "requestsrate": 0, "totalresponses": "0", "responsesrate": 0, "totalrequestbytes": "0", "requestbytesrate": 0, "totalresponsebytes": "0", "responsebytesrate": 0, "totalh2requests": "0", "h2requestsrate": 0, "totalh2responses": "0", "h2responsesrate": 0, "totalpktsrecvd": "0", "pktsrecvdrate": 0, "totalpktssent": "0", "pktssentrate": 0, "curclntconnections": "0", "cursrvrconnections": "0", "curpersistencesessions": "0", "curbackuppersistencesessions": "0", "surgecount": "0", "svcsurgecount": "0", "sothreshold": "0", "totspillovers": "0", "labelledconn": "0", "pushlabel": "0", "deferredreq": "0", "deferredreqrate": 0, "invalidrequestresponse": "0", "invalidrequestresponsedropped": "0", "totvserverdownbackuphits": "0", "curmptcpsessions": "0", "cursubflowconn": "0", "totalconnreassemblyqueue75": "0", "totalconnreassemblyqueueflush": "0", "totalsvrbusyerr": "0", "svrbusyerrrate": 0, "reqretrycount": "0", "reqretrycountexceeded": "0", "httpmaxhdrszpkts": "0", "httpmaxhdrfldlenpkts": "0", "tcpmaxooopkts": "0", "totcltttlbtransactions": "0", "cltttlbtransactionsrate": 0, "toleratingttlbtransactions": "0", "toleratingttlbtransactionsrate": 0, "frustratingttlbtransactions": "0", "frustratingttlbtransactionsrate": 0 } ] }

I think there is no such process to go with 1) method. Citrix ADC API documentation says you can logout/terminate a session using a cookie, but to initiate new session it is required to send login and password without previous session cookie. Old session will just time-out on Citrix ADC at some point of time. In the example above Curl does use a cookie to initiate a session SESSID=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; Max-Age=0; path=/ but according to my knowledge setting cookie expiration date in the past is like saying it is expired/not for use. Therefore each new session should be initiated from scratch in my humble opinion.

powersj commented 2 years ago

Thanks for the response. I've pushed another update just now that will omit the cookie from the headers during an auth. Once new artifacts build, can you give that one more shot?

Thanks again!

SpiderD555 commented 2 years ago

Similar story with the latest artifact, although there is no mention about cookie in the reply from Citrix ADC this time.

Jul 14 08:42:13 localhost telegraf[3105480]: 2022-07-14T06:42:13Z I! Loaded inputs: http snmp (15x)
Jul 14 08:47:15 localhost telegraf[3105480]: 2022-07-14T06:47:15Z E! [inputs.http] renewal failed for "https://netscaler/nitro/v1/config/login": cookie auth renewal received status code: 401 (Unauthorized) [{ "errorcode": 354, "message": "Invalid username or password", "severity": "ERROR" }]
Jul 14 08:52:15 localhost telegraf[3105480]: 2022-07-14T06:52:15Z E! [inputs.http] renewal failed for "https://netscaler/nitro/v1/config/login": cookie auth renewal received status code: 401 (Unauthorized) [{ "errorcode": 354, "message": "Invalid username or password", "severity": "ERROR" }]
Jul 14 08:57:15 localhost telegraf[3105480]: 2022-07-14T06:57:15Z E! [inputs.http] renewal failed for "https://netscaler/nitro/v1/config/login": cookie auth renewal received status code: 401 (Unauthorized) [{ "errorcode": 354, "message": "Invalid username or password", "severity": "ERROR" }]
Jul 14 09:02:15 localhost telegraf[3105480]: 2022-07-14T07:02:15Z E! [inputs.http] renewal failed for "https://netscaler/nitro/v1/config/login": cookie auth renewal received status code: 401 (Unauthorized) [{ "errorcode": 354, "message": "Invalid username or password", "severity": "ERROR" }]
Jul 14 09:07:15 localhost telegraf[3105480]: 2022-07-14T07:07:15Z E! [inputs.http] renewal failed for "https://netscaler/nitro/v1/config/login": cookie auth renewal received status code: 401 (Unauthorized) [{ "errorcode": 354, "message": "Invalid username or password", "severity": "ERROR" }]
powersj commented 2 years ago

Hmm I re-added the debug info to print out the headers. Maybe it is getting set again? Can you try one more time? I am not sure what else this might be.

Your CURL examples in your previous comment omitted the example of logging in with the cookie in the header. I wanted to see if you could reproduce the same failure on the CLI.

SpiderD555 commented 2 years ago

Below is the flow you request. First is login, then regular API call, third request is a second authentication attempt.

curl -ik https://netscaler/nitro/v1/config/login -X POST -H 'Content-Type: application/json' -d '{ "login" :{"username": "username", "password": "password"}}'

HTTP/1.1 201 Created
Date: Thu, 14 Jul 2022 15:22:27 GMT
Server: Apache
X-Frame-Options: SAMEORIGIN
Set-Cookie: SESSID=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; Max-Age=0; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Set-Cookie: sessionid=%23%238DDDC7D90D1BB7A5973D89EDB1DEB52A545D7D4EBB19684B16C5C2375342D9681DD03752A058F240120DC1DC6183256E12992765BB50515DBD32E874F6A587A5413AB425317064910B4678CCC088F16944A3FDA1F9553F89DCA33431FDBF49856692929B04140A8F3E0386988F56E3472471D061D1F78321F43FEC68EDB7; path=/nitro/v1
Feature-Policy: camera 'none'; microphone 'none'; geolocation 'none'
Referrer-Policy: no-referrer
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Content-Length: 328
Content-Type: application/json; charset=utf-8

{ "errorcode": 0, "message": "Done", "severity": "NONE", "sessionid": "##8DDDC7D90D1BB7A5973D89EDB1DEB52A545D7D4EBB19684B16C5C2375342D9681DD03752A058F240120DC1DC6183256E12992765BB50515DBD32E874F6A587A5413AB425317064910B4678CCC088F16944A3FDA1F9553F89DCA33431FDBF49856692929B04140A8F3E0386988F56E3472471D061D1F78321F43FEC68EDB7" }

=============================================

curl -ik https://netscaler/nitro/v1/stat/lbvserver -H 'Content-Type: application/json' --cookie "sessionid=%23%238DDDC7D90D1BB7A5973D89EDB1DEB52A545D7D4EBB19684B16C5C2375342D9681DD03752A058F240120DC1DC6183256E12992765BB50515DBD32E874F6A587A5413AB425317064910B4678CCC088F16944A3FDA1F9553F89DCA33431FDBF49856692929B04140A8F3E0386988F56E3472471D061D1F78321F43FEC68EDB7; path=/nitro/v1"
HTTP/1.1 200 OK
Date: Wed, 13 Jul 2022 06:08:55 GMT
Server: Apache
X-Frame-Options: SAMEORIGIN
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Vary: Accept-Encoding
Feature-Policy: camera 'none'; microphone 'none'; geolocation 'none'
Referrer-Policy: no-referrer
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Content-Length: 1640
Content-Type: application/json; charset=utf-8

{ "errorcode": 0, "message": "Done", "severity": "NONE", "lbvserver": [ { "name": "lb-test-ssl", "avgcltttlb": "0", "cltresponsetimeapdex": 1.000000, "vsvrsurgecount": "0", "establishedconn": "0", "inactsvcs": "0", "vslbhealth": "0", "primaryipaddress": "169.254.1.1", "primaryport": 443, "type": "SSL", "state": "DOWN", "actsvcs": "0", "cpuusagepm": "0", "tothits": "0", "hitsrate": 0, "totalrequests": "0", "requestsrate": 0, "totalresponses": "0", "responsesrate": 0, "totalrequestbytes": "0", "requestbytesrate": 0, "totalresponsebytes": "0", "responsebytesrate": 0, "totalh2requests": "0", "h2requestsrate": 0, "totalh2responses": "0", "h2responsesrate": 0, "totalpktsrecvd": "0", "pktsrecvdrate": 0, "totalpktssent": "0", "pktssentrate": 0, "curclntconnections": "0", "cursrvrconnections": "0", "curpersistencesessions": "0", "curbackuppersistencesessions": "0", "surgecount": "0", "svcsurgecount": "0", "sothreshold": "0", "totspillovers": "0", "labelledconn": "0", "pushlabel": "0", "deferredreq": "0", "deferredreqrate": 0, "invalidrequestresponse": "0", "invalidrequestresponsedropped": "0", "totvserverdownbackuphits": "0", "curmptcpsessions": "0", "cursubflowconn": "0", "totalconnreassemblyqueue75": "0", "totalconnreassemblyqueueflush": "0", "totalsvrbusyerr": "0", "svrbusyerrrate": 0, "reqretrycount": "0", "reqretrycountexceeded": "0", "httpmaxhdrszpkts": "0", "httpmaxhdrfldlenpkts": "0", "tcpmaxooopkts": "0", "totcltttlbtransactions": "0", "cltttlbtransactionsrate": 0, "toleratingttlbtransactions": "0", "toleratingttlbtransactionsrate": 0, "frustratingttlbtransactions": "0", "frustratingttlbtransactionsrate": 0 } ] }

=============================================

curl -ik https://netscaler/nitro/v1/config/login -X POST -H 'Content-Type: application/json' -d '{ "login" :{"username": "username", "password": "password"}}' --cookie "sessionid=%23%238DDDC7D90D1BB7A5973D89EDB1DEB52A545D7D4EBB19684B16C5C2375342D9681DD03752A058F240120DC1DC6183256E12992765BB50515DBD32E874F6A587A5413AB425317064910B4678CCC088F16944A3FDA1F9553F89DCA33431FDBF49856692929B04140A8F3E0386988F56E3472471D061D1F78321F43FEC68EDB7; path=/nitro/v1"
HTTP/1.1 401 Unauthorized
Date: Thu, 14 Jul 2022 15:24:07 GMT
Server: Apache
X-Frame-Options: SAMEORIGIN
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Feature-Policy: camera 'none'; microphone 'none'; geolocation 'none'
Referrer-Policy: no-referrer
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Content-Length: 84
Content-Type: application/json; charset=utf-8

{ "errorcode": 354, "message": "Invalid username or password", "severity": "ERROR" }
powersj commented 2 years ago

thank you! That does show that setting the cookie is causing a 401.

I just pushed another change which will recreate the cookie jar at every reauth. This should ensure the cookie is never used during the auth process.

SpiderD555 commented 2 years ago

It looks better now. Session renewal is working. I see that because of those messages:

Jul 15 11:16:15 localhost telegraf[3180145]: map[Content-Type:[application/json]]
Jul 15 11:21:15 localhost telegraf[3180145]: map[Content-Type:[application/json]]
Jul 15 11:26:15 localhost telegraf[3180145]: map[Content-Type:[application/json]]

However I still don't see any real statistic data from Citrix ADC.

powersj commented 2 years ago

It looks better now. Session renewal is working.

Awesome! Thanks for hanging in there. I am going to get this branch merged since it already fixes a couple different issues.

However I still don't see any real statistic data from Citrix ADC.

Based on your previous messages you are getting JSON back that looks like:

{
  "errorcode": 0,
  "message": "Done",
  "severity": "NONE",
  "lbvserver": [
    {
      "name": "lb-test-ssl",
      "avgcltttlb": "0",
      "cltresponsetimeapdex": 1,
      "vsvrsurgecount": "0",
      "establishedconn": "0",
      "inactsvcs": "0",
      "vslbhealth": "0",
      "primaryipaddress": "169.254.1.1",
      "primaryport": 443,
      "type": "SSL",
      "state": "DOWN",
      "actsvcs": "0",
      "cpuusagepm": "0",
      "tothits": "0",
      "hitsrate": 0,
      "totalrequests": "0",
      "requestsrate": 0,
      "totalresponses": "0",
      "responsesrate": 0,
      "totalrequestbytes": "0",
      "requestbytesrate": 0,
      "totalresponsebytes": "0",
      "responsebytesrate": 0,
      "totalh2requests": "0",
      "h2requestsrate": 0,
      "totalh2responses": "0",
      "h2responsesrate": 0,
      "totalpktsrecvd": "0",
      "pktsrecvdrate": 0,
      "totalpktssent": "0",
      "pktssentrate": 0,
      "curclntconnections": "0",
      "cursrvrconnections": "0",
      "curpersistencesessions": "0",
      "curbackuppersistencesessions": "0",
      "surgecount": "0",
      "svcsurgecount": "0",
      "sothreshold": "0",
      "totspillovers": "0",
      "labelledconn": "0",
      "pushlabel": "0",
      "deferredreq": "0",
      "deferredreqrate": 0,
      "invalidrequestresponse": "0",
      "invalidrequestresponsedropped": "0",
      "totvserverdownbackuphits": "0",
      "curmptcpsessions": "0",
      "cursubflowconn": "0",
      "totalconnreassemblyqueue75": "0",
      "totalconnreassemblyqueueflush": "0",
      "totalsvrbusyerr": "0",
      "svrbusyerrrate": 0,
      "reqretrycount": "0",
      "reqretrycountexceeded": "0",
      "httpmaxhdrszpkts": "0",
      "httpmaxhdrfldlenpkts": "0",
      "tcpmaxooopkts": "0",
      "totcltttlbtransactions": "0",
      "cltttlbtransactionsrate": 0,
      "toleratingttlbtransactions": "0",
      "toleratingttlbtransactionsrate": 0,
      "frustratingttlbtransactions": "0",
      "frustratingttlbtransactionsrate": 0
    }
  ]
}

As you are using the json_v2 parser, you are going to need to specify some settings to correctly parse some of the data provided to you:

  data_format = "json_v2"
  [[inputs.http.json_v2]]
    [[inputs.http.json_v2.object]]
      path = "lbvserver"

Which should produce something similar to this below. Note I used the file input to produce this example:

file name="lb-test-ssl",avgcltttlb="0",cltresponsetimeapdex=1,vsvrsurgecount="0",establishedconn="0",inactsvcs="0",vslbhealth="0",primaryipaddress="169.254.1.1",primaryport=443,type="SSL",state="DOWN",actsvcs="0",cpuusagepm="0",tothits="0",hitsrate=0,totalrequests="0",requestsrate=0,totalresponses="0",responsesrate=0,totalrequestbytes="0",requestbytesrate=0,totalresponsebytes="0",responsebytesrate=0,totalh2requests="0",h2requestsrate=0,totalh2responses="0",h2responsesrate=0,totalpktsrecvd="0",pktsrecvdrate=0,totalpktssent="0",pktssentrate=0,curclntconnections="0",cursrvrconnections="0",curpersistencesessions="0",curbackuppersistencesessions="0",surgecount="0",svcsurgecount="0",sothreshold="0",totspillovers="0",labelledconn="0",pushlabel="0",deferredreq="0",deferredreqrate=0,invalidrequestresponse="0",invalidrequestresponsedropped="0",totvserverdownbackuphits="0",curmptcpsessions="0",cursubflowconn="0",totalconnreassemblyqueue75="0",totalconnreassemblyqueueflush="0",totalsvrbusyerr="0",svrbusyerrrate=0,reqretrycount="0",reqretrycountexceeded="0",httpmaxhdrszpkts="0",httpmaxhdrfldlenpkts="0",tcpmaxooopkts="0",totcltttlbtransactions="0",cltttlbtransactionsrate=0,toleratingttlbtransactions="0",toleratingttlbtransactionsrate=0,frustratingttlbtransactions="0",frustratingttlbtransactionsrate=0 1657891626000000000

From there you can continue to customize these settings further.

SpiderD555 commented 2 years ago

I can confirm it is working. I finally get data in InfluxDB from Citrix ADC. Thank you for your help