search

davidhamann / python-fmrest

Python wrapper around the FileMaker Data API
MIT License
100 stars 27 forks source link

Timeout when pulling a large set of records #2

Closed HawkinsJM closed 6 years ago

HawkinsJM commented 6 years ago

When I try to pull the 2700x200 records from this layout I get a timeout error. foundset = fms.get_records(range_=3000) foundset

returns

`--------------------------------------------------------------------------- RequestException Traceback (most recent call last)

in () ----> 1 foundset = fms.get_records(range_=3000) 2 foundset ~/Documents/kai_ming/data_analysis/venv_km3/lib/python3.6/site-packages/fmrest/server.py in get_records(self, offset, range_, portals) 265 params['offset'] = offset 266 params['range'] = range_ --> 267 response = self._call_filemaker('GET', path, params=params) 268 269 return Foundset(self._process_foundset_response(response)) ~/Documents/kai_ming/data_analysis/venv_km3/lib/python3.6/site-packages/fmrest/server.py in _call_filemaker(self, method, path, data, params) 412 data=data, 413 verify=self.verify_ssl, --> 414 params=params 415 ) 416 ~/Documents/kai_ming/data_analysis/venv_km3/lib/python3.6/site-packages/fmrest/utils.py in request(*args, **kwargs) 26 return requests.request(*args, timeout=TIMEOUT, **kwargs) 27 except Exception as ex: ---> 28 raise RequestException(ex, args, kwargs) from None 29 30 def build_portal_params(portals, names_as_string=False): RequestException: Request error: HTTPSConnectionPool(host='fake.address', port=443): Read timed out. (read timeout=10)` Also, is there a better way to pull all records there than setting range to very large value you are sure is larger than the number of records?
davidhamann commented 6 years ago

The timeout cannot be configured at the moment. It should be, so I'll put this on the feature list (Update: See here: https://github.com/davidhamann/python-fmrest/issues/3).

To solve your problem in the meantime you could monkey patch the constant value like so:

import fmrest
fmrest.utils.TIMEOUT = 20
# now do your call

TIMEOUT is in seconds.

Let me know if this helps.

It seems that your FMS needs to do a lot of work / a lot of time to respond. If you are already using FMS17 you could use the new response layout feature to limit the amount of data being returned and/or query only the portal you need and ignore others. On 16 you could try to reduce the number of fields/portals on the find layout.

As for getting the right number of records: you could try to fetch them in chunks, check if you get the full amount, and then work with the offset parameter to fetch the next chunk. Unfortunately, it is not possible to easily get the number of matching records before you make the actual call (in theory you could call a FM script first to get the number, and then use that returned number to set the limit on your next request, but not sure how practical that would be 🙃). The default limit set by FMS is always 100 if you're not overwriting it.

davidhamann commented 6 years ago

Closed for now. Use fmrest_timeout environment variable to increase timeout.

If you have further problems, please re-open.