estately / rets

A pure-ruby library for fetching data from RETS servers
https://github.com/estately/rets
127 stars 94 forks source link

Support count 1 (records and count) #132

Closed hfaulds closed 8 years ago

hfaulds commented 9 years ago

Rets accepts 3 values for count, if unset or 0 it returns the records for the query without the count. For 1 it returns the records and the count and for 2 it returns just the count.

It's been buggging me that we don't support count=1 and return both records and count. The reason I want this is that some RETS servers set a limit on the number of records returned (<MAXROWS> but doesn't return said limit all the time). When using COUNT=1 the count returned is the count of all records ignoring limts, offsets or maxrows (partly rets spec 7.4.1).

Here's some pagination logic I whipped up with this (I tested some similar code which worked):

def paginated_fetch(query, old_rows=[])
  response = fetch(query.merge(count: 1))

  batch_rows = response.rows
  total_rows = old_rows + batch_rows

  if total_rows.size < response.count && batch_rows.size != 0
    # After the first query set the limit, offset and no_records_not_an_error
    # - If an mls has a limit of 1000 and has 1001 records
    # - We fetch 1000 and see we should have fetched 1001
    # - They delete a property before our second batch
    # - We need to not error if offset=1000 yields no results response code
    next_query = query.merge( limit: batch_rows.size, offset: total_rows.size + 1, no_records_not_an_error: true )
    fetch_with_custom_query(next_query, total_rows)
  else
    total_rows.uniq
  end
end

I haven't checked to see if the count returned with COUNT=1 matches the number of records, when the number of records < MAXROWS and limit/offset are unset (i.e. in database speak is the count done in a transaction with the select). But the above code seems like it would handle that reasonably anyway This is a breaking change for any-one using the gem but trivial for people to fix in their code.

dougcole commented 8 years ago

Seems like this has been abondoned, is that right @hfaulds?