search

CityofSantaMonica / mds-provider

Python tools for working with MDS Provider data
https://github.com/openmobilityfoundation/mobility-data-specification
MIT License
18 stars 20 forks source link

remove incorrect break condition in paging #58

Closed yoimnik closed 5 years ago

yoimnik commented 5 years ago

we should not be breaking the paging loop when we have a page that has no data.

it's possible for a MDS provider to paginate their data at small time windows. for example, and this can be discovered by looking at the next_url, bird's api paginates for every 2 minutes.

it's possible that if i'm requesting the past hour of data, then the time window of 8-10 minutes is empty (ie. no status_changes occured in that time window), but the rest of the time windows have data.

the current code will then only ingest time windows 0-2 minutes, 2-4 minutes, 4-6 minutes, 6-8 minutes, and not ingest any of 10-12 minutes, 12-14 minutes, ..., 58-60 minutes because the while loop will break prematurely.

yoimnik commented 5 years ago

mds spec for pagination:

https://github.com/CityOfLosAngeles/mobility-data-specification/tree/0.2.x/provider#pagination

yoimnik commented 5 years ago

@thekaveman requesting review

thekaveman commented 5 years ago

Thanks @niketanpatel, this seems like a good change.

Can you please rebase on dev and resolve conflicts?

yoimnik commented 5 years ago

rebased

edit: oops, this rebase was a mistake, fixing

yoimnik commented 5 years ago

rebased

thekaveman commented 5 years ago

@niketanpatel you rebased, but in the process of adding your change you deleted what you rebased on top of. Please add the rate_limit code back, with a line of spacing between the if block and your new block.

yoimnik commented 5 years ago

@thekaveman whoops! added that back. thanks for the catch.

thekaveman commented 5 years ago

Thanks, this looks good. Merging.