access_token missing on second call to Mulesoft

[stevanbt]

Hi, I've started using your project to pull logs and stats from Mulesoft and push into ES. I have an issue when I use them... the first call that the script makes to Mulesoft works, it gets an access_token and uses that to get the org id, etc. The information is then pushed into ES and I can see graphs.

However, the issue I'm seeing is on a subsequent call to Mulesoft it appears that the access_token is blank, when I enable the ruby debug statement at the end of the login.conf script the first time it displays now_ms, access_token, organization_id, environment_id and environment.name. The second time it displays the following:-

{
              "now_ms" => 1591650558000,
    "environment.name" => "Production",
     "organization_id" => "%{[body][user][organization][id]}"
}

So organization_id isn't populated because the access_token is missing is what I'm assuming. I've tried running the script in debug mode and I've tried changing the blacklist and whitelist statements without success.

Any suggestions as to what I can try?

Thanks, Steve.

[michaelhyatt]

Hi Steve

I tried it on my end with Postman, the first call was successful and the second one returned 403 Forbidden with payload saying invalid csrf token. When I am clearing the cookies it starts to work again. I suggest adding cookies => "false" to the authentication HTTP call in login.conf, as per this: https://www.elastic.co/guide/en/logstash/current/plugins-inputs-http_poller.html#plugins-inputs-http_poller-cookies

[stevanbt]

Hi, That worked, I had briefly tried it, but must have backed it out when I was troubleshooting.

Thanks for your help with this... and the prompt response.

Thanks, Steve.

[pankajmangal2007]

@stevanbt @michaelhyatt Could you please help me to find out where exactly this 'cookies=>"false"' need to add, if possible could you please share the snippet ?

I tried to add it in header section as well as outside the header but second call getting failed with error 'invalid csrf token'

sample login.conf

input { http_poller { urls => { login => {

Supports all options supported by ruby's Manticore HTTP client

    method => post
    url => "https://anypoint.mulesoft.com/accounts/login" 
    headers => {
      "Content-Type" => "application/x-www-form-urlencoded"
       "cookies" => "false"
    }
    body => "username=${CH_USER}&password=${CH_PASSWORD}"
 }
}
request_timeout => 120
# Supports "cron", "every", "at" and "in" schedules by rufus scheduler
# schedule => { cron => "* * * * * UTC"}
schedule => { "every" => "30s"}
codec => "json"
# A hash of request metadata info (timing, response headers, etc.) will be sent here
# metadata_target => "http_poller_metadata"

} }

output { stdout { codec => rubydebug }

Response:

2020-11-27T08:35:39,684][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600} { "@timestamp" => 2020-11-27T03:06:10.522Z, "access_token" => "117868ad-ecc7-45be-a197-aadfa047d65a", "redirectUrl" => "/home/", "token_type" => "bearer", "@version" => "1" } [2020-11-27T08:36:38,579][ERROR][logstash.codecs.json ] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Unrecognized token 'invalid': was expecting ('true', 'false' or 'null') at [Source: (String)"invalid csrf token"; line: 1, column: 8]>, :data=>"invalid csrf token"} { "@timestamp" => 2020-11-27T03:06:38.585Z, "tags" => [ [0] "_jsonparsefailure" ], "@version" => "1", "message" => "invalid csrf token"

[michaelhyatt]

It is a http_poller level configuration option:

https://www.elastic.co/guide/en/logstash/current/plugins-inputs-http_poller.html#plugins-inputs-http_poller-options

On Fri, 27 Nov 2020 at 14:12, Pankaj Mangal notifications@github.com wrote:

@stevanbt https://github.com/stevanbt @michaelhyatt https://github.com/michaelhyatt Could you please help me to find out where exactly this 'cookies=>"false"' need to add, if possible could you please share the snippet ?

I tried to add it in header section as well as outside the header but second call getting failed with error 'invalid csrf token'

sample login.conf

input {

1. Kick off with the login poll

http_poller { urls => { login => {

Supports all options supported by ruby's Manticore HTTP client

method => post url => "https://anypoint.mulesoft.com/accounts/login" headers => { "Content-Type" => "application/x-www-form-urlencoded" "cookies" => "false" } body => "username=${CH_USER}&password=${CH_PASSWORD}" } } request_timeout => 120

Supports "cron", "every", "at" and "in" schedules by rufus scheduler

schedule => { cron => " * UTC"}

schedule => { "every" => "30s"} codec => "json"

A hash of request metadata info (timing, response headers, etc.) will be

sent here

metadata_target => "http_poller_metadata"

} }

output { stdout { codec => rubydebug }

Response:

2020-11-27T08:35:39,684][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600} { "@timestamp https://github.com/timestamp" => 2020-11-27T03:06:10.522Z, "access_token" => "117868ad-ecc7-45be-a197-aadfa047d65a", "redirectUrl" => "/home/", "token_type" => "bearer", "@Version https://github.com/Version" => "1" } [2020-11-27T08:36:38,579][ERROR][logstash.codecs.json ] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Unrecognized token 'invalid': was expecting ('true', 'false' or 'null') at [Source: (String)"invalid csrf token"; line: 1, column: 8]>, :data=>"invalid csrf token"} { "@timestamp https://github.com/timestamp" => 2020-11-27T03:06:38.585Z, "tags" => [ [0] "_jsonparsefailure" ], "@Version https://github.com/Version" => "1", "message" => "invalid csrf token"

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/michaelhyatt/elastic-cloudhub-observability/issues/2#issuecomment-734622343, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADXR5DJXRUWRI75S5MX2D5TSR4KKJANCNFSM4NYZA2OQ .

[pankajmangal2007]

Thanks @michaelhyatt. It's working for me, now just one more question how to keep the Logstash running, as soon as I close the Ubuntu terminal Logstash also gets stopped (Elasticserach and Kibana don't stop)

[michaelhyatt]

I'd say use something like nohup to run logstash, or just configure it as a daemon/service.