search

paleobot / pbot-dev

Codebase and initial design documents for pbot client
MIT License
2 stars 2 forks source link

Notes from session with Matt on bulk upload (break out into better issues later) #255

Closed aazaff closed 4 months ago

aazaff commented 7 months ago

  1. Better error messaging for invalid API-driven data uploads. Currently only get an http failure message with no additional information. (This may actually partially be an R issue).

  2. What options are there for bulk roll-back (e.g., what is someone bulk uploads a bunch of junk by accident?). Does the user have mechanisms to do this through the API? If not, do the administrators have a mechanism?

  3. Bulk upload seems to be on the slow side. Maybe about an hour for 500 transactions. Not sure if anything can or should be done about this, but making note of it.

ecurrano commented 5 months ago

I met with Matt yesterday and he has made tremendous progress on the bulk uploader and can now bulk create specimen nodes, description nodes, and the character state descriptions. He is working on the OTU node creation as well. We discussed publishing an R package that would provide the full documentation and hopefully make this as idiot-proof as possible. We do need to still consider Andrews point 2 above.

aazaff commented 5 months ago

I am not sure what can practically be done about 2. I think it has to be punted completely to the next grant (I'm pretty sure it is explicitly called out there). For the short term. I would say that only real way forward is just for Matt to really test the hell out of it on the test site and only publish it on production when he is confident - i.e., just be really careful and don't make any mistakes.

Obviously that's not a very robust strategy, but it is what we practically can do.

ecurrano commented 5 months ago

Agreed. I told Matt that I will be the "idiot" tester to help make sure we have everything as well-documented as possible.