Batch downloads

[daikman]

I've added an "offset" argument to get_resource() to enable batch downloading. I also created get_resource_batched() to wrap around get_resource() to easily download a resource in batches.

I haven't written tests for the new function yet, but get_resource() still passes its tests despite the changes made to it.

[csillasch]

The new batched wrapper function around get-resource looks great! Couple of initial thoughts:

• we need to keep the code in the get-resource function that handles rows > 99999. This is because of a CKAN quirk, if the row number is above this but doesn't match the total number of records, the query fails. Also a speed technique, as the query becomes very slow over that number therefore it is more efficient to use the dump API endpoint and leave it to users to filter the data after that themselves.
• in the batched function, could we have a default number of rows, so it would still run if users don't set their own number of batches or rows? Just to make it a bit easier to use when you people don't necessarily know the size of the file and just want it downloaded quick and easy.

[Moohan]

Agree with Csilla's comments.

I would suggest calling the offset parameter skip, skip_n, skip_rows or something like that as I think that is more intuitive. The offset name makes most sense only in the context of the batch function.

I wonder if you could run some tests for speed/number of timeouts to determine what is a good default size. This could then also be used 'baked-in' to the get_resource function so for example when requesting > 10, 000 rows it reverts to the batch function with n_rows = 2, 000.

[csillasch]

Closing this as using dump endpoint over rows >99999 achieves sufficient efficiency (can revisit in future if needed).