Implement a means of getting more than 1000 data points

[lcmcninch]

This PR resolves #108, at least partially. The end result is that the data method can use the limit query parameter and link header to return any number of data points (or all data points) of a feed. Current functionality returns all data up to 1000 points and doesn't provide the option to either decrease or increase that limit.

Scope of change:

• The method _get of the Client class has a new keyword argument "params" which is passed directly to requests.get. The default value is None, which is also the default for requests.get so this should be completely backward compatible.
• client.py now imports from re and urllib.parse. I'm not an expert on the various versions of python but I don't think this will cause any compatibility issues.
• An attribute lastresponse is added to the Client class to store the response of the last request. It is updated with each get_, _post and _delete call. This isn't ideal but I thought it would be the least intrusive way of allowing access to properties of the response other than the json (such as the 'link' header). I'm open to other ideas.
• A _get_nextlink method is added which parses the 'link' header from the __lastresponse. This method is based on this example which is referenced in the API documentation here. However, as discussed in this forum post, there is a bug in the API. If and when this bug is fixed, the _get_nextlink method would break and need to be updated.
• Finally, a keyword argument _maxresults is added to the data method of Client. If max_results is less than the API page limit of 1000, it simply applies the lower limit. If it is larger than 1000, it will use the pagination link in the get response to retrieve more data until max_responses (or the end of the data) is reached. If max_responses is None, all of the data for the feed is retrieved. This should be backwards compatible as the current implementation always returns all data (up to 1000). If max_results isn't given, it defaults to 1000 and previous behavior should be retained.

Limitations:

• Ideally we would also implement the _starttime, _endtime and include parameters of the API, but I wanted to get this pull request in and discuss before taking it any further. There is also the issue with _starttime mentioned in this post which would need to be addressed.
• I did not add the new functionality to the examples. Because this could move a lot of data, I'm not sure how to best implement it in the examples. Let me know if you have thoughts on how you'd want an example implemented.

[lcmcninch]

I made the requested change based on option two from my response. I query the feed count, and use that as max_results, if it is given as None. I'm not 100% sure that that query will always work, but it was the way I was able to find to get the number of records in a feed. If there's a cleaner way, please let me know.

[lcmcninch]

Apologies, I inadvertently deleted this branch, which closed the PR. I restored the branch and I am re-opening.

[brentru]

@lcmcninch Hi - could you resolve the conflicting files and I'll look at merging this in.

[lcmcninch]

@lcmcninch Hi - could you resolve the conflicting files and I'll look at merging this in.

@brentru No problem, conflict resolved. Thanks for looking at this!

[jwcooper]

I think we should drop the default from 1000 records to something much lower. In fact, I think we may change the server to default to a lower limit as well. I'd like to think 100 records is a better default limit.

[lcmcninch]

@jwcooper I'm happy to make the change, do you want me to reduce it to 100?

[lcmcninch]

I lowered the default limit to 100.

[unlimitedbacon]

Any update on getting this merged?