pulibrary / pdc_discovery

Princeton Data Commons discovery portal for Research Data
10 stars 0 forks source link

Add last_page metadata to PPPL data feed #425

Closed bess closed 6 months ago

bess commented 1 year ago

Chun requests that the pppl data feed be split into a meta section and a data section to match the way solr provides paginated results.

He wants to be able to write something like this:

next_page = 1
while True:
    query = PDC_QUERY | {"page": next_page}
    r = requests.get(PDC_URI, params=urlencode(query, safe="+"))
    j = r.json()
    all_items.extend(j["data"])
    next_page = j["meta"]["pages"]["next_page"]
    if j["meta"]["pages"]["last_page?"]:
        break
astrochun commented 1 year ago

I realized that there are a lot of other features, so this is not too critical. I can paginate until an empty response is provided.

astrochun commented 6 months ago

We have a working solution in place so this isn't critical. I'll be closing.