search

ArctosDB / documentation-wiki

Arctos Documentation and How-To Guides
https://handbook.arctosdb.org
GNU General Public License v3.0
13 stars 13 forks source link

[CONTACT] What prevents others from using the bulkloader? #173

Open alexkrohn opened 5 years ago

alexkrohn commented 5 years ago

I had a question about what precisely prevents others from using the bulkloader when someone has specimens in there. I would like to amend our protocols so that I have time to check the loaded specimens before approving them to go "live," while also ensuring that others have access to the bulkloader in a timely manner.

I currently have multiple students who are either bulkloading legacy specimens, or entering legacy specimens one-by-one. When I look at the bulkloader (Enter Data -> Bulkloader -> Browse and Edit), I see specimens in three possible stages: 1) bulkoaded, 2) waiting approval or 3) with some error. If I check the bulkloader status (Enter Data -> Bulkloader -> Bulkloader Status), specimens with any of those errors may be there -- either in the failed section or in the top section.

If I (or the student) do not remove the specimens, which of these errors prevents other users from using the bulkloader? I assume it's just specimens that have been bulkloaded, but not the waiting for approval specimens, right?

Also, and maybe tangentially, is there a reason that only one person can bulkload specimens at a time? Is it just to prevent two people from simultaneously loading thousands of specimens at once and potentially crashing the servers?

dustymc commented 5 years ago

The bulkloader STAGE - the tool to upload a CSV file to the bulkloader - is single-user. That just makes it slightly easier to handle errors and makes things go a bit faster for users. I think it could be made multi-user if someone had a compelling reason to; we could make the process asynchronous and send email when it's done or similar, but I think that would make it much more difficult for an individual users to know what's where.

The "real" bulkloader - where specimens entered with the data entry screen (and other 'local' apps), or those that have made it through the bulkloader stage, are located - is multi-user. Leaving records there for as long as you need to is fine. (We may limit that at some point, but it'll be on the order of a year if we do.)

The functional distinction in LOADED is binary - NULL records will be checked and either loaded or returned with an error, records with any value in LOADED will be ignored by the loader. "Bulkloaded" is just intended to help users understand where things came from.

campmlc commented 3 years ago

Once the records are visible in Browse and Edit, you can take your time to review them and mark to load. Only one collection can load to the bulkloader at a time, but that bottleneck just puts records in a temporary staging table, like an anteroom, where they can be deleted and reloaded or pushed to Browse and Edit. We do need different terms for these two stages to clarify.

On Fri, Jan 22, 2021, 9:07 AM alexkrohn notifications@github.com wrote:

  • [EXTERNAL]*

I had a question about what precisely prevents others from using the bulkloader when someone has specimens in there. I would like to amend our protocols so that I have time to check the loaded specimens before approving them to go "live," while also ensuring that others have access to the bulkloader in a timely manner.

I currently have multiple students who are either bulkloading legacy specimens, or entering legacy specimens one-by-one. When I look at the bulkloader (Enter Data -> Bulkloader -> Browse and Edit), I see specimens in three possible stages: 1) bulkoaded, 2) waiting approval or 3) with some error. If I check the bulkloader status (Enter Data -> Bulkloader -> Bulkloader Status), specimens with any of those errors may be there -- either in the failed section or in the top section.

If I (or the student) do not remove the specimens, which of these errors prevents other users from using the bulkloader? I assume it's just specimens that have been bulkloaded, but not the waiting for approval specimens, right?

Also, and maybe tangentially, is there a reason that only one person can bulkload specimens at a time? Is it just to prevent two people from simultaneously loading thousands of specimens at once and potentially crashing the servers?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ArctosDB/documentation-wiki/issues/173, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADQ7JBBTA6X2IPCTIZFXPB3S3GPFLANCNFSM4WOWIAVA .