bulk iNat import

[JoeCohen]

Import many Observations. For instance, users should have the option to import all their iNat Obss. Possibly date/location limited imports

Tasks

• [x] Allow multiple listed IDs
  • [x] Change form iNat observation number to numbers
  • [x] Mention comma separated list
  • [x] Change work flow: get one page of results, Iterate over the results, repeat until out of pages
  • [x] Refactor params to always include: ids; sort by id, asc; allow optional additional params
  • [x] #2250
  • [x] Revise controller test stub
  • [x] Revise controller params
  • [x] Fix the stubs in the model tests. The stubs must return entire search results (which include the obs), rather than just the obs. Reason: controller has changed so that it does a search, which it will go per page; each page per result (obs)
  • [x] Revert one mock
  • [x] Fix the model stub to do a search, return a single result
  • [x] Import multiple IDs if user included multiple ids on form
• [x] iNat Username
  • [x] Add it to form
  • [x] make it mandatory
  • [x] Add it to InatImport and to search string
• [ ] Add option to import all a user's Obss
  • [x] Modify form
• [ ] Give user feedback on # of qualifying iNat Obss (those that meet checked criteria && not already imported && fungal)

[JoeCohen]

iNat limits # of ids to 30 when using https://api.inaturalist.org/v1/observations/10000,10001,... so must provide id param and comma-separated list: https://api.inaturalist.org/v1/observations?id=10000%2C10001%2C10002&order=desc&order_by=created_at Return is like this: { "total_results": 37, "page": 1, "per_page": 30, "results": [ How do I get subsequent pages?
I could use the page param, but iNat says:

The large size of the observations index prevents us from supporting the page parameter when retrieving records from large result sets. If you need to retrieve large numbers of records, use the per_page and id_above or id_below parameters instead.

So I need to do something like tis pseudo_code

import_observations do
  last_import ||= 0
  import with per_page=200&id_above=#{last_import}
    import_one_obs
    ...
  last_import = results.last[:id]
  next if  total_results > page * per_page
end

[JoeCohen]

This is Erroring in MO RestClient::UnprocessableEntity in Observations::InatImportsController#authenticate @inat = RestClient.get("#{API_BASE}#{operation}", headers)

>> "#{API_BASE}#{operation}"
=> "https://api.inaturalist.org/v1/observations?id=&id_above=0&only_id=false&per_page=200&order=asc&order_by=id&user_login=dev189&iconic_taxa=Fungi,Protozoa"
>> headers
=> {"Authorization"=>"Bearer {\"access_token\":\"_VLnaihHahFntdN2nne_U2Clv7eJyJkpo8i-WutIdeg\",\"token_type\":\"Bearer\",\"scope\":\"write login\",\"created_at\":1722123661}"}

I've tried it in the console with the same result. Also tried it without a header. Tried it with and empty header { }. This is working on the iNat API test page: https://api.inaturalist.org/v1/observations?user_login=devin189&iconic_taxa=Fungi&id_above=0&per_page=200&order=asc&order_by=id&only_id=true

[nimmolo]

Are you sure headers is encoded right? I spot two different hash notations, the => and the :

[JoeCohen]

thanks! that could be it. will check tomorrow

Sent from Gmail Mobile

El El sáb, jul 27, 2024 a la(s) 17:47, andrew nimmo < @.***> escribió:

Are you sure headers is encoded right? I spot two different hash notations, the => and the :

— Reply to this email directly, view it on GitHub https://github.com/MushroomObserver/mushroom-observer/issues/2224#issuecomment-2254297371, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAALDFAWNNIBVD6YK4X32A3ZOQ5T7AVCNFSM6AAAAABKRCV6CSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENJUGI4TOMZXGE . You are receiving this because you were assigned.Message ID: @.***>