Bulkloader Column Names - Reprise

[Jegelewicz]

If you want background see https://github.com/ArctosDB/arctos/issues/6103 and https://github.com/ArctosDB/arctos/issues/5193#issuecomment-1512200579

In an effort to make the bulkloader better while it is being rebuilt, the Code Table Admins went through the currently proposed column headers and revised them for consistency and logical grouping. You can review our efforts here: Bulkloader Rebuild

The first column includes the headers from the current bulkloader, the middle (hidden) column was Dusty's original proposal and the last column is the Code Table Admins' proposal. This is expected to be our bulkloader for about a five year time frame, so please review this and record any objections or provide your support for the revision in this issue.

Note there will be many more columns in the actual bulkloader as some sections will be duplicated. For more information see

Current table is always https://arctos.database.museum/tblbrowse.cfm?tbl=bulkloader

---

START DLM bits

temp_maybe_new_bulk_cols(2).csv

temp_maybe_new_bulkloader(11).csv

Column Count:1137

lastedit: https://github.com/ArctosDB/arctos/issues/6255

lastedit: associated_species

END DLM bits

---

Summary: https://docs.google.com/spreadsheets/d/1ssNP_jAiOok7TYIPq8b-OKcLCw9NngtCElPwbS6ajFo/edit#gid=0

1136

Barring a block by #6170 (which you should also review!), we plan to adopt the revised bulkloader column headers at the AWG Issues Meeting on May 4, 2023

@ewommack could you please send an email to the AWG with the text of this issue? Thanks!

[ewommack]

@Jegelewicz - as a reply to the last bulkloader email?

@lin-fred - just an FYI

[Jegelewicz]

as a reply to the last bulkloader email?

I do not know what was in that email. I don't think it matters if it is a reply or a new email though.

[ewommack]

Okey dokey

[ewommack]

And sent. Text below: Arctos members,

Arctos needs your help.

In an effort to make the bulkloader better while it is being rebuilt, the Code Table Admins went through the currently proposed column headers and revised them for consistency and logical grouping. You can review our efforts here: Bulkloader Rebuild

The first column includes the headers from the current bulkloader, the middle column is Dusty's current proposal and the last column is the Code Table Admins' revision. This is expected to be our bulkloader for about a five year time frame, so please review this and record any objections or provide your support for the revision in this issue.

And for additional background please see #6103 and #5193 (comment)

Barring a block by #6170 (which you should also review!), we plan to adopt the revised bulkloader column headers at the AWG Issues Meeting on May 4, 2023

--

[dustymc]

I think this probably should not be closed, and I think it should be communicated to the whole community rather than just the working group??

[Jegelewicz]

I think it should be communicated to the whole community

The only way I know to do this is with a banner - we did not discuss that - should we add one? @ArctosDB/arctos-working-group-officers

[dustymc]

I certainly don't know what nor how, but if something emerges please add it to https://github.com/ArctosDB/internal/issues/258#issuecomment-1515440177 - the mechanics of (2) are still pretty foggy from here, this is something that will/may require changes from near everybody, and this is something that will not be easily un-done.

[campmlc]

I agree with a banner notification.

[Jegelewicz]

New Banner

Notice: A rebuild of the bulkloader is planned soon. For more details see this Github Issue. We plan to vote to adopt the new bulkloader at the AWG Issues Meeting on May 4, 2023. Please post any objection, request, or agreement in the Issue before May 2, 2023.

Any edits before I post?

[campmlc]

Maybe "A planned rebuild of the bulkloader is scheduled for final approval and implementation by the end of the month." for first sentence? At least say this is happening right now - and people need to weigh in.

On Mon, Apr 24, 2023 at 1:41 PM Teresa Mayfield-Meyer < @.***> wrote:

• [EXTERNAL]*

New Banner

Notice: A rebuild of the bulkloader is planned soon. For more details see this Github Issue https://github.com/ArctosDB/arctos/issues/6171#issuecomment-1520559021. We plan to vote to adopt the new bulkloader at the AWG Issues Meeting on May 4, 2023. Please post any objection, request, or agreement in the Issue before May 2, 2023.

Any edits before I post?

— Reply to this email directly, view it on GitHub https://github.com/ArctosDB/arctos/issues/6171#issuecomment-1520726237, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADQ7JBAPLVA2IHMKLYBFSU3XC3JM5ANCNFSM6AAAAAAXGEYCCI . You are receiving this because you are on a team that was mentioned.Message ID: @.***>

[dustymc]

Your link is going to a comment.

I think the proposed column names should be more explicit.

"My" proposal (its not??) can be removed.

It should be explained (somehow) that the linked sheet isn't complete, eg there are some ridiculous number of various things in the current proposal, resulting in over a thousand columns.

The implications of changing all the names should probably be summarized somewhere.

What precisely does "scheduled for final approval and implementation" mean? (For both me and someone who might want a change in a couple weeks, please.)

"We plan to vote to adopt " - who's "we"? (Maybe that's known from other stuff?)

[Jegelewicz]

Notice: A planned rebuild of the bulkloader is scheduled for final approval at the AWG Issues Meeting on May 4, 2023. For more details see https://github.com/ArctosDB/arctos/issues/6171. Please post any objection, request, or agreement in the Issue before May 2, 2023.

[Jegelewicz]

I think the proposed column names should be more explicit.

Ideas accepted

"My" proposal (its not??) can be removed.

I want this there for the purpose of evolution - if everyone else thinks it can go away, then I can delete it - I have hidden it now.

It should be explained (somehow) that the linked sheet isn't complete, eg there are some ridiculous number of various things in the current proposal, resulting in over a thousand columns.

That is in #5193 and I placed a summary in the first comment for those who won't review the background.

The implications of changing all the names should probably be summarized somewhere.

And those are?

What precisely does "scheduled for final approval and implementation" mean? (For both me and someone who might want a change in a couple weeks, please.)

Removed implementation - only final approval is on the table.

"We plan to vote to adopt " - who's "we"? (Maybe that's known from other stuff?)

Changed - shouldn't be a problem now?

[dustymc]

That is in https://github.com/ArctosDB/arctos/issues/5193

That should be closed; this is no longer additive and has become something very different.

And those are?

• Anything that uses or references "old" column names will no longer function.
• I (and the dev environment) will be unavailable for other tasks for ~a month
• Beyond that core - you tell me.
  • Do we need to update component loaders?
  • Do we need to update any other part of the UI?
  • Does the pre-bulkloader need to survive this?
  • Can eg https://github.com/ArctosDB/arctos/issues/6183 (and various other dependencies) go anywhere radical, or are those things being finalized at the same time?
  • ???

Changed

In part, I was wondering if the meeting was open.

[Jegelewicz]

Thanks for documenting the implications

In part, I was wondering if the meeting was open.

All of our AWG meetings are "open".

[ewommack]

@lin-fred - just including you to double check I have the listserv info correct.

Just a quick note for this comment:

I think this probably should not be closed, and I think it should be communicated to the whole community rather than just the working group??

The email listserv goes to everyone now, so everyone who gets communication us through email should have gotten the email.

[lin-fred]

Yes the AWG working group email is not just the AWG but everyone who wanted to sign up. If something like this needs to go out to more people, then maybe a newsletter write up, which then can be linked in a tweet/in the banner. I am unsure how else to reach more Arctos users.

[Jegelewicz]

Look at Geome model for building a template? We can't because of Bootstrap. But if we can get close it would be great. Adding definitions would be helpful.

[campmlc]

if Bootstrap were possible, we could solve a lot of difficult interface issues. I agree with trying to emulate Geome model.

[mkoo]

FYI, I was using that as a model for functionality (not so much style but it's nice too) So having definitions for fields and allowing flexible template building are key!

[dustymc]

I think perhaps these last few comments refer to https://github.com/ArctosDB/arctos/issues/6183??

Bootstrap doesn't make anything "possible" - it's just a tool, it makes some things easier, refuses to address others at all. I think maybe we've built up some misplaced mysticism or something??

Arctos field definitions are https://arctos.database.museum/tblbrowse.cfm?tbl=bulkloader (and almost none of that is going to survive this).

[Jegelewicz]

Arctos field definitions are https://arctos.database.museum/tblbrowse.cfm?tbl=bulkloader (and almost none of that is going to survive this).

Assuming there will still be this thing? Working on this in Sheet 3 of https://docs.google.com/spreadsheets/d/1qstLM0xpW8gkkEnRxUpZWOZGtJkv2NTu8zIgKxv-nYc/edit?usp=sharing

[Jegelewicz]

@dustymc In today's AWG - it was made clear that two things are missing from the columns listed in the file above

associated species identification confidence - is this becoming identification rank? or are we also missing this and rank from the new list?

[dustymc]

associated species

I think that got lost in https://github.com/ArctosDB/arctos/issues/4707, I'll add it back in.

identification confidence

See https://github.com/ArctosDB/arctos/issues/4829#issuecomment-1474064536 via https://github.com/ArctosDB/arctos/issues/5331 - the whole package should be considered.

becoming identification rank?

Is this a reference to https://github.com/ArctosDB/arctos/issues/3540?

[Jegelewicz]

becoming identification rank?

Is this a reference to https://github.com/ArctosDB/arctos/issues/3540?

Yes - how will that get added in data entry/bulkloading? IDENTIFICATION_ORDER?

[dustymc]

identification_1_order | character varying(4) | | |

[Jegelewicz]

OK I think that covers it

associated_species being added back identification confidence becoming an attribute of identification

Cool - do I need to fire up issues to create ctidentification_attributes? Then we will also need ctidentification_att_att so that we can link the controlled vocabularies for the new nature of id attribute to ctnature_of_id and the new identification confidence to ctidentification_confidence. So also issues for those terms....

[dustymc]

o also issues

Yea, probably with the way things have been going. That's the code I'd written before one of the times this got killed, but I suppose I can just toss that.

If you're feeling brave can we PLEASE lose the "WE HAVE NOTHING TO SAY!!" option while we're in there?