search

demisto / demisto-py

Demisto Client for Python
Apache License 2.0
70 stars 41 forks source link

BASE_URL Confusing #38

Closed gachmoon8 closed 4 years ago

gachmoon8 commented 4 years ago

Hi everyone.

I know this is not a bug, but I do not know where can i get information about this. I have xsoar access information, and website url is https://demisto.example.com/ for example.

What will be BASE_URL in my case ? I want to work with Cortex XSOAR server using REST API.

Thanks in advance.

glicht commented 4 years ago

You base url is simply the url you access Demisto at. If you are using env variables you should do:

export DEMISTO_BASE_URL=https://demisto.example.com
gachmoon8 commented 4 years ago

@glicht Thank you so much for reply.

I generated API_KEY from demisto website. Is it correct way ?

Basically, after try to make http call it returns:

HTTP response body: {"id":"forbidden","status":403,"title":"Forbidden","detail":"Issue with CSRF code","error":"http: named cookie not present","encrypted":false,"multires":null}

glicht commented 4 years ago

Please post full details.

gachmoon8 commented 4 years ago

Demisto = 5.5.0 Python = 3.7 demisto-py = 2.0.12

from __future__ import print_function
import os
import demisto_client.demisto_api
from demisto_client.demisto_api.rest import ApiException
from pprint import pprint

# RestAPIIntegration
api_key = '<MY_API_KEY>'  # generated from web UI
base_url = 'https://demisto.mydomain.net/'

# create an instance of the API class
api_instance = demisto_client.configure(base_url=base_url, api_key=api_key, debug=True)
filter = demisto_client.demisto_api.SearchIncidentsData()

# Create incident filter object
inc_filter = demisto_client.demisto_api.IncidentFilter()
inc_filter.name = ['test']

filter.filter = inc_filter

try:
    # Search incidents by filter
    api_response = api_instance.search_incidents(filter=filter)
    pprint(api_response)
except ApiException as e:
    print("Exception when calling DefaultApi->search_incidents: %s\n" % e)

I got a response like:

Exception when calling DefaultApi->search_incidents: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=10886400000000000; includeSubDomains', 'Date': 'Wed, 03 Jun 2020 23:53:03 GMT', 'Content-Length': '159'})
HTTP response body: {"id":"forbidden","status":403,"title":"Forbidden","detail":"Issue with CSRF code","error":"http: named cookie not present","encrypted":false,"multires":null}
glicht commented 4 years ago

The error you are getting means the API key is invalid. I would try generating the key again.

Also you can try using username/password authentication by using the env variables:

DEMISTO_USERNAME DEMISTO_PASSWORD

Make sure to unset DEMISTO_API_KEY if using username/password.

gachmoon8 commented 4 years ago

@glicht

I tried to re-generate new API key, but response is still same. I changed strategy as you told me (I used username/password), and i was authenticated only once. But, every next try was unsuccessful.

api_instance = demisto_client.configure(base_url=base_url, username=username, password=password, verify_ssl=False, debug=True)  # script is stucked on this line

Logs looks like this

2020-06-04 12:22:47,971 DEBUG Starting new HTTPS connection (1): demisto.mydomain.net:443
2020-06-04 12:22:47,971 DEBUG Starting new HTTPS connection (1): demisto.mydomain.net:443
2020-06-04 12:22:47,971 DEBUG Starting new HTTPS connection (1): demisto.mydomain.net:443
header: Accept-Ranges header: Content-Length header: Content-Type header: Last-Modified header: Set-Cookie header: Set-Cookie header: Strict-Transport-Security header: X-Content-Type-Options header: X-Frame-Options header: X-Xss-Protection header: Date send: b'POST /login HTTP/1.1\r\nHost: demisto.mydomain.net\r\nAccept-Encoding: identity\r\nContent-Length: 53\r\nAccept: application/json\r\nContent-Type: application/json\r\nX-XSRF-TOKEN: w3l1eQW9aaILRKCzu3Fdmz9p65iyHhjGFQ+BAqSu1Nqo98bVGu1UDFmzE/Iaixndnn1PzYPDg2TaU+1hMxJ5Zeko7r9UvhZyccxlh4Nv2S70hrqEv0CWvM3Hqj4ievVo01G7b8nsy22iHaQdDjkuDFuLJ2vpjfUhGtVTiV5aCYY=\r\nUser-Agent: demisto-py/dev\r\nCookie: XSRF-TOKEN=w3l1eQW9aaILRKCzu3Fdmz9p65iyHhjGFQ+BAqSu1Nqo98bVGu1UDFmzE/Iaixndnn1PzYPDg2TaU+1hMxJ5Zeko7r9UvhZyccxlh4Nv2S70hrqEv0CWvM3Hqj4ievVo01G7b8nsy22iHaQdDjkuDFuLJ2vpjfUhGtVTiV5aCYY=; Path=/; Expires=Fri, 04 Jun 2021 08:22:47 GMT; Max-Age=31536000; Secure; SameSite=Lax, inc-term=eyJzaW5ndWxhciI6ImluY2lkZW50Iiwic2luZ3VsYXJDYXBpdGFsIjoiSW5jaWRlbnQiLCJwbHVyYWwiOiJpbmNpZGVudHMiLCJwbHVyYWxDYXBpdGFsIjoiSW5jaWRlbnRzIn0=; Secure; SameSite=Lax\r\n\r\n'
send: b'{"user": "<myuser>", "password": "<mypassword>"}'

Also, why I can not use API_KEY ? It will be so helpful to use PyCharm plugin.

Thanks.

glicht commented 4 years ago

You can test your api key via curl. You can do the following request to see if your api key is valid:

curl -i -k -H 'Authorization: YOUR_API_KEY'  https://your-demisto.server.url/about

You will get a response similar to this:

HTTP/1.1 200 OK
Strict-Transport-Security: max-age=10886400000000000; includeSubDomains
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-Xss-Protection: 1; mode=block
Date: Thu, 04 Jun 2020 09:27:53 GMT
Content-Length: 189
Content-Type: text/plain; charset=utf-8

{"demistoVersion":"5.5.0","buildSHA":"223c7594a95c182db7f5a3a1d47894623e34631c","webClientHA":"35f79617453f551f010d14b7a4464cf6e6dee1a0","buildNum":"67560","engineProtocolVersion":"1.0.5"}

Regarding user/pass auth. Not sure what you mean:

But, every next try was unsuccessful.

Please provide your full script and full debug output.

gachmoon8 commented 4 years ago

Now, I will show you everything.

1) Check if API_KEY is valid

Settings -> Integrations -> API Keys -> Get your key
Request:

curl -i -k -H 'Authorization: <generated_api_key>' https://demisto.mydomain.net/about

Response:

HTTP/1.1 401 Unauthorized
Content-Type: application/json
Set-Cookie: XSRF-TOKEN=Lk+Yidk17zHKmyI3HL3O65BqeouRueTsGJDyRTnVpCGmZabMID3/FB9quNhT1fRP0ZS61Rmbf7XImf7sx9o3U8REl/Vgk+/DK8TGI9cB/PHCbHifpYPWxcn/3o/BC8+gSzc6wRE/QpIk7b/SoyU2PG6qQJH8l3CbOMLA3OiubPQ=; Path=/; Expires=Fri, 04 Jun 2021 10:39:09 GMT; Max-Age=31536000; Secure; SameSite=Lax
Strict-Transport-Security: max-age=10886400000000000; includeSubDomains
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-Xss-Protection: 1; mode=block
Date: Thu, 04 Jun 2020 10:39:09 GMT
Content-Length: 179

{"id":"unauthorized","status":401,"title":"Unauthorized","detail":"The request requires authorization","error":"http: named cookie not present","encrypted":false,"multires":null}

2) demisto_client implementation using API_KEY

from __future__ import print_function
import os
import demisto_client.demisto_api
from demisto_client.demisto_api.rest import ApiException
from pprint import pprint

api_key = None  # set to your 'YOUR_API_KEY' or set environment variable: DEMISTO_API_KEY
host = None  # set to your 'http://DEMISTO_HOST' or set environment variable: DEMISTO_BASE_URL

'''
Just to check that variables are set correctly
print(os.getenv('DEMISTO_BASE_URL'))
print(os.getenv('DEMISTO_API_KEY'))
'''

# create an instance of the API class
api_instance = demisto_client.configure(base_url=host, api_key=api_key, debug=True)
filter = demisto_client.demisto_api.SearchIncidentsData()

# Create incident filter object
inc_filter = demisto_client.demisto_api.IncidentFilter()
inc_filter.name = ['test']

filter.filter = inc_filter

try:
    # Search incidents by filter
    api_response = api_instance.search_incidents(filter=filter)
    pprint(api_response)
except ApiException as e:
    print("Exception when calling DefaultApi->search_incidents: %s\n" % e)
Log:
2020-06-04 14:49:38,992 DEBUG Starting new HTTPS connection (1): demisto.mydomain.net:443
send: b'POST /incidents/search HTTP/1.1\r\nHost: demisto.mydomain.net\r\nAccept-Encoding: identity\r\nContent-Length: 873\r\nAccept: application/json\r\nContent-Type: application/json\r\nUser-Agent: demisto-py/dev\r\nAuthorization: 2807346F840FC8FDBF461B1F7496DFAE\r\n\r\n'
send: b'{"filter": {"Cache": null, "andOp": null, "category": null, "details": null, "files": null, "firstIncidentInPage": null, "fromActivatedDate": null, "fromClosedDate": null, "fromDate": null, "fromDateLicense": null, "fromDueDate": null, "fromReminder": null, "id": null, "includeTmp": null, "investigation": null, "lastIncidentInPage": null, "level": null, "name": ["test"], "nextPage": null, "notCategory": null, "notInvestigation": null, "notStatus": null, "page": null, "parent": null, "period": null, "query": null, "reason": null, "searchAfter": null, "searchBefore": null, "sequentialPagesSearch": null, "size": null, "sort": null, "status": null, "systems": null, "timeFrame": null, "toActivatedDate": null, "toClosedDate": null, "toDate": null, "toDueDate": null, "toReminder": null, "totalOnly": null, "type": null, "urls": null, "users": null}, "userFilter": null}'
reply: 'HTTP/1.1 403 Forbidden\r\n'
2020-06-04 14:49:40,101 DEBUG https://demisto.mydomain.net:443 "POST /incidents/search HTTP/1.1" 403 159
2020-06-04 14:49:40,101 DEBUG response body: {"id":"forbidden","status":403,"title":"Forbidden","detail":"Issue with CSRF code","error":"http: named cookie not present","encrypted":false,"multires":null}

header: Content-Type header: Strict-Transport-Security header: Date header: Content-Length Exception when calling DefaultApi->search_incidents: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=10886400000000000; includeSubDomains', 'Date': 'Thu, 04 Jun 2020 10:49:39 GMT', 'Content-Length': '159'})
HTTP response body: {"id":"forbidden","status":403,"title":"Forbidden","detail":"Issue with CSRF code","error":"http: named cookie not present","encrypted":false,"multires":null}

3) demisto_client implementation using username/password

# {api_key is unset}
from __future__ import print_function
import os
import demisto_client.demisto_api
from demisto_client.demisto_api.rest import ApiException
from pprint import pprint

username = None
password = None
host = None  # set to your 'http://DEMISTO_HOST' or set environment variable: DEMISTO_BASE_URL

'''
Just to check that variables are set correctly
print(os.getenv('DEMISTO_USERNAME'))
print(os.getenv('DEMISTO_PASSWORD'))
'''

api_instance = demisto_client.configure(base_url=host, username=username, password=password, debug=True)
filter = demisto_client.demisto_api.SearchIncidentsData()

# Create incident filter object
inc_filter = demisto_client.demisto_api.IncidentFilter()
inc_filter.name = ['test']

filter.filter = inc_filter

try:
    # Search incidents by filter
    api_response = api_instance.search_incidents(filter=filter)
    pprint(api_response)
except ApiException as e:
    print("Exception when calling DefaultApi->search_incidents: %s\n" % e)
Log:
2020-06-04 14:56:12,870 DEBUG Starting new HTTPS connection (1): demisto.mydomain.net:443
2020-06-04 14:56:12,870 DEBUG Starting new HTTPS connection (1): demisto.mydomain.net:443
send: b'GET / HTTP/1.1\r\nHost: demisto.mydomain.net\r\nAccept-Encoding: identity\r\nAccept: application/json\r\nContent-Type: application/json\r\nUser-Agent: demisto-py/dev\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
2020-06-04 14:56:13,731 DEBUG https://demisto.mydomain.net:443 "GET / HTTP/1.1" 200 1268
2020-06-04 14:56:13,731 DEBUG https://demisto.mydomain.net:443 "GET / HTTP/1.1" 200 1268
2020-06-04 14:56:13,732 DEBUG response body: <!DOCTYPE html>
<html lang="en">
  <head>
    <title>Cortex XSOAR</title>
    <meta charset="utf-8">
    <meta name="description" content="Cortex XSOAR">
    <meta name="keywords" content="security,incident response,attack playback,forensics">

    <!-- social media tags -->
    <meta name="twitter:card" content="summary">
    <meta name="twitter:site" content="@Cortex XSOAR">
    <meta name="twitter:title" content="Cortex XSOAR">
    <meta name="twitter:description" content="Cortex XSOAR">
    <meta name="twitter:creator" content="@Cortex XSOAR">
    <meta name="twitter:image:src" content="">
    <meta property="og:title" content="Cortex XSOAR" />
    <meta property="og:type" content="website" />
    <meta property="og:url" content="" />
    <meta property="og:image" content="" />
    <meta property="og:description" content="Cortex XSOAR">

    <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1">
    <link rel="shortcut icon" href="/favicon.ico?v=1585601864566">
    <link href="/assets/light-bundle-1585601864566.css?v=1585601864566" rel="stylesheet">
  </head>
  <body>
    <div id="app"></div>
    <script type="text/javascript" src="/assets/light-bundle-1585601864566.js?v=1585601864566"></script>
  </body>
</html>

2020-06-04 14:56:13,732 DEBUG response body: <!DOCTYPE html>
<html lang="en">
  <head>
    <title>Cortex XSOAR</title>
    <meta charset="utf-8">
    <meta name="description" content="Cortex XSOAR">
    <meta name="keywords" content="security,incident response,attack playback,forensics">

    <!-- social media tags -->
    <meta name="twitter:card" content="summary">
    <meta name="twitter:site" content="@Cortex XSOAR">
    <meta name="twitter:title" content="Cortex XSOAR">
    <meta name="twitter:description" content="Cortex XSOAR">
    <meta name="twitter:creator" content="@Cortex XSOAR">
    <meta name="twitter:image:src" content="">
    <meta property="og:title" content="Cortex XSOAR" />
    <meta property="og:type" content="website" />
    <meta property="og:url" content="" />
    <meta property="og:image" content="" />
    <meta property="og:description" content="Cortex XSOAR">

    <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1">
    <link rel="shortcut icon" href="/favicon.ico?v=1585601864566">
    <link href="/assets/light-bundle-1585601864566.css?v=1585601864566" rel="stylesheet">
  </head>
  <body>
    <div id="app"></div>
    <script type="text/javascript" src="/assets/light-bundle-1585601864566.js?v=1585601864566"></script>
  </body>
</html>

2020-06-04 14:56:13,733 DEBUG Starting new HTTPS connection (1): demisto.mydomain.net:443
2020-06-04 14:56:13,733 DEBUG Starting new HTTPS connection (1): demisto.mydomain.net:443
2020-06-04 14:56:13,733 DEBUG Starting new HTTPS connection (1): demisto.mydomain.net:443
header: Accept-Ranges header: Content-Length header: Content-Type header: Last-Modified header: Set-Cookie header: Set-Cookie header: Strict-Transport-Security header: X-Content-Type-Options header: X-Frame-Options header: X-Xss-Protection header: Date send: b'POST /login HTTP/1.1\r\nHost: demisto.mydomain.net\r\nAccept-Encoding: identity\r\nContent-Length: 53\r\nAccept: application/json\r\nContent-Type: application/json\r\nX-XSRF-TOKEN: oUfSwyLkDiYug5WrU8I1kwVUQF+nQ1wiykhwNXQeDdEzcovg6S/YG/53aj/k1jCQzBwvydSkd6oGo6leBR4WsqrqF9de8UHRydIyGuuBtxkfy7s+puUOACTt2xPouipeWErPqG2wGiftvPJs9j+LpZhPKpIAWH3kaFL6ab1fTbU=\r\nUser-Agent: demisto-py/dev\r\nCookie: XSRF-TOKEN=oUfSwyLkDiYug5WrU8I1kwVUQF+nQ1wiykhwNXQeDdEzcovg6S/YG/53aj/k1jCQzBwvydSkd6oGo6leBR4WsqrqF9de8UHRydIyGuuBtxkfy7s+puUOACTt2xPouipeWErPqG2wGiftvPJs9j+LpZhPKpIAWH3kaFL6ab1fTbU=; Path=/; Expires=Fri, 04 Jun 2021 10:56:13 GMT; Max-Age=31536000; Secure; SameSite=Lax, inc-term=eyJzaW5ndWxhciI6ImluY2lkZW50Iiwic2luZ3VsYXJDYXBpdGFsIjoiSW5jaWRlbnQiLCJwbHVyYWwiOiJpbmNpZGVudHMiLCJwbHVyYWxDYXBpdGFsIjoiSW5jaWRlbnRzIn0=; Secure; SameSite=Lax\r\n\r\n'
send: b'{"user": <myuser:str>, "password": <mypassword:str>}' <<< Stucked infinitely here !

Maybe this will be more helpful ? I provided all possible scenario i did.

glicht commented 4 years ago

Hi @gachmoon8 ,

Unfortunately, it looks like you are experiencing problems beyond the scope of this project.

From what I see the API_KEY is invalid. It doesn't work via curl. You need to get this resolved as a start. Best option is to take this up with Cortex XSOAR support.

Regarding user/pass I am not sure why it is getting stuck. This probably also requires investigating the sever. I suggest also taking this up with support when checking the API key issue.

gachmoon8 commented 4 years ago

@glicht

I understand. Thank's for your time.

forrest-greene-sp commented 1 year ago

@gachmoon8 did you ever find a resolution? I have the exact same problem. I can connect to the api via curl from the demisto server, but if i try python requests, or if I use curl from another box I get the exact same error. My api is un usable :/

forrest-greene-sp commented 1 year ago

just for fyi. the issue was i was using "Authorization": f"Bearer {api_key}" instead of "Authorization": f"{api_key}". So in short They do not support bearer auth, only basic auth. while the docs are technically true, calling this out would have saved some headache. Luckliy this all seems to be going away on v8.