search

PaloAltoNetworks / minemeld

Main MineMeld documentation repo
379 stars 64 forks source link

Read Timeout for all o365 API Miners #49

Open mnooseman opened 4 years ago

mnooseman commented 4 years ago

For the last few days every one of my O365 miners have been erroring out with "HTTPSConnectionPool(host="endpoints.office365.com", port="443"): Read Timed out (read timeout=20)"

Not sure why this is happening but I can access the API from elsewhere within my network. Wondering if it's perhaps not cycling GUIDs on the API Calls?

Engine log is as below: 2020-01-04T09:54:38 (31903)basepoller._actor_loop INFO: wl_o365-api_any - command: 1578131678131 poll 2020-01-04T09:54:38 (31903)basepoller._polling_loop INFO: Polling wl_o365-api_any 2020-01-04T09:54:58 (31903)basepoller._poll ERROR: Exception in polling loop for wl_o365-api_any: HTTPSConnectionPool(host='endpoints.office.com', port=443): Read timed out. (read timeout=20) Traceback (most recent call last): File "/opt/minemeld/engine/core/minemeld/ft/basepoller.py", line 732, in _poll performed = self._polling_loop() File "/opt/minemeld/engine/core/minemeld/ft/basepoller.py", line 579, in _polling_loop iterator = self._build_iterator(now) File "/opt/minemeld/engine/core/minemeld/ft/o365.py", line 373, in _build_iterator latest_version = self._check_version() File "/opt/minemeld/engine/core/minemeld/ft/o365.py", line 288, in _check_version **rkwargs File "/opt/minemeld/engine/current/local/lib/python2.7/site-packages/requests/api.py", line 75, in get return request('get', url, params=params, **kwargs) File "/opt/minemeld/engine/current/local/lib/python2.7/site-packages/requests/api.py", line 60, in request return session.request(method=method, url=url, **kwargs) File "/opt/minemeld/engine/current/local/lib/python2.7/site-packages/requests/sessions.py", line 524, in request resp = self.send(prep, **send_kwargs) File "/opt/minemeld/engine/current/local/lib/python2.7/site-packages/requests/sessions.py", line 637, in send r = adapter.send(request, **kwargs) File "/opt/minemeld/engine/current/local/lib/python2.7/site-packages/requests/adapters.py", line 529, in send raise ReadTimeout(e, request=request) ReadTimeout: HTTPSConnectionPool(host='endpoints.office.com', port=443): Read timed out. (read timeout=20)

jtschichold commented 4 years ago

@mnooseman GUID should not be cycled between calls as it is used to identify the client. I see the same issues where requests from curl or browser to endpoints.office.com sometimes are successful and sometimes hang. It seems to be a problem with MSFT API endpoint.

mnooseman commented 4 years ago

Just odd that every attempt in the last 4 days would fail, when I can make the same call from other locations on the same network and have it work.

Thanks for the info

jtschichold commented 4 years ago

@mnooseman I totally agree this is odd but I am seeing the same issue without using MineMeld.

Could you try using curl from the MineMeld instance and from one of the working hosts. Example:

curl "https://endpoints.office.com/version?clientrequestid=3fb1f126-e221-490f-857e-c4332d38979a"

It should timeout on MineMeld instance. You can change the Client ID, result should be the same.

baxter888 commented 3 years ago

@mnooseman I totally agree this is odd but I am seeing the same issue without using MineMeld.

Could you try using curl from the MineMeld instance and from one of the working hosts. Example:

curl "https://endpoints.office.com/version?clientrequestid=3fb1f126-e221-490f-857e-c4332d38979a"

It should timeout on MineMeld instance. You can change the Client ID, result should be the same.

In the prototype [https://github.com/PaloAltoNetworks/minemeld-node-prototypes/blob/master/prototypes/o365-api.yml] is a harcoded GUID appended to the url. But MS wrote that every instance which is pulling the list, should have a own GUID. I think, they block requests if to much requests with the same GUID but diffrent IPs are access. Therefore it should be able to change the url if you create your own local prototype of minemeld.ft.o365.O365API Class. I'am not sure if it can overwriten in the new local prototype, if not its a feature request, or you add url as variable [required].