datacommonsorg / api-python

Python client library to access Data Commons
https://www.datacommons.org
Apache License 2.0
70 stars 39 forks source link

add chunked query to dc python api #162

Open clincoln8 opened 4 years ago

clincoln8 commented 4 years ago

adding conduct_chunked_query and its helper function recursive_query to be able to conduct large queries without overloading Data Commons server.

I had tried to add in the tests in chunked_query_test.py to tests/query_test.py, but was having a lot of trouble doing so. Should the tests be put in once conduct_chunked_query can be reached using just the api (as in dc.conduct_chunked_query() )?

Additionally, the test test_large_query() in chunked_query_test.py occasionally fails when running over command line, but I have run the same test many, many times using a colab notebook, and it never fails there. Any insight to why that may be happening would be appreciated :) I don't think it has to do with the code in the two functions since it works fine in colab. Maybe either the testing environment or the server, since it works fine in colab? I am not sure.

Thank you for the help!