search

ArctosDB / arctos

Arctos is a museum collections management system
https://arctos.database.museum
60 stars 13 forks source link

Panama serology upload #6073

Closed jldunnum closed 1 year ago

jldunnum commented 1 year ago

Panamanian pathogen serology data for upload into new "examined_detected" attributes. Also contains a column for microhabitat data, can this be loaded as well? This is critical for modelling pathogen prevalence with habitat. Panama serology upload to Dusty.xlsx

dustymc commented 1 year ago

CSV please - anyone but me, Excel hates me!

I can append the habitat in, if there's one event, but - it's sorta not? "Domestic"? Is there a better place for that??

Can the publication be changed to a more findable format, eg https://arctos.database.museum/publication/1 is nice (and I'm 100% sure I can recover it if this gets more formalized)?

I can't do this until someone checks some boxes at https://github.com/ArctosDB/arctos/issues/6042 (or we find a better way, or SOMETHING) - help?? (And then I'll hit the other issues, but they're not preventing me from doing stuff NOW.)

Jegelewicz commented 1 year ago

I didn't do anything magic - it should be OK

Panama.serology.upload.to.Dusty.csv

Jegelewicz commented 1 year ago

I can append the habitat in, if there's one event, but - it's sorta not? "Domestic"? Is there a better place for that??

I think Domestic - means "where people live"?

I would imagine that definitions for these terms will end up being useful to someone in the future and I feel like they are great candidates for a collecting event attribute?

jldunnum commented 1 year ago

CSV attached

Not sure how to describe it better. Domestic is the direct translation of their word in that field "Domestico". Basically the animal was trapped in or around the human habited area (house, yard, sheds, barns, etc).

This is the publication where the method comes from but the paper itself doesn't cite all these specimens. Hjelle, B, S Jenison, N Torrez-Martinez, B Herring, S Quan, A Polito, S Pichuantes, T Yamada, C Morris, F Elgh, H W Lee, H Artsob, R Dinello. 1997. Journal of Clinical Microbiology 35(3):600-608. https://doi.org/10.1128/jcm.35.3.600-608.1997 Going to https://github.com/ArctosDB/arctos/issues/6042 to review now. [https://opengraph.githubassets.com/3e46198e0f1689b7b19a0b3a7537915a07d6a945c57a70d63da2584bb42fa1ad/ArctosDB/arctos/issues/6042]https://github.com/ArctosDB/arctos/issues/6042 Code Table Request - New Code Table = examined_detected · Issue #6042 · ArctosDB/arctoshttps://github.com/ArctosDB/arctos/issues/6042 Instructions This is a template to facilitate communication with the Arctos Code Table Committee. Submit a separate request for each relevant value. This form is appropriate for exploring how data ... github.com

Jonathan L. Dunnum Ph.D. (he, him, his) Senior Collection Manager Division of Mammals, Museum of Southwestern Biology University of New Mexico Albuquerque, NM 87131 (505) 277-9262 Fax (505) 277-1351

Chair, Systematic Collections Committee, American Society of Mammalogists Latin American Fellowship Committee, ASM

MSB Mammals website: http://www.msb.unm.edu/mammals/index.html Facebook: http://www.facebook.com/MSBDivisionofMammals

Shipping Address: Museum of Southwestern Biology Division of Mammals University of New Mexico CERIA Bldg 83, Room 204 Albuquerque, NM 87131

From: dustymc @.> Sent: Tuesday, March 28, 2023 3:03 PM To: ArctosDB/arctos @.> Cc: Jonathan Dunnum @.>; Author @.> Subject: Re: [ArctosDB/arctos] [CONTACT] (Issue #6073)

[EXTERNAL]

CSV please - anyone but me, Excel hates me!

I can append the habitat in, if there's one event, but - it's sorta not? "Domestic"? Is there a better place for that??

Can the publication be changed to a more findable format, eg https://arctos.database.museum/publication/1 is nice (and I'm 100% sure I can recover it if this gets more formalized)?

I can't do this until someone checks some boxes at #6042https://github.com/ArctosDB/arctos/issues/6042 (or we find a better way, or SOMETHING) - help?? (And then I'll hit the other issues, but they're not preventing me from doing stuff NOW.)

— Reply to this email directly, view it on GitHubhttps://github.com/ArctosDB/arctos/issues/6073#issuecomment-1487588129, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AED2PA3LRPU3WDDJMUYUSVTW6NGZRANCNFSM6AAAAAAWLCTK7E. You are receiving this because you authored the thread.Message ID: @.***>

campmlc commented 1 year ago

The term I've seen in use for that kind of zoonotic surveillance is "peridomestic", which according at least one source includes the household area. But this article distinguishes between the two, so perhaps we need both: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4296493/

On Tue, Mar 28, 2023 at 3:33 PM Jonathan Dunnum @.***> wrote:

  • [EXTERNAL]*

CSV attached

Not sure how to describe it better. Domestic is the direct translation of their word in that field "Domestico". Basically the animal was trapped in or around the human habited area (house, yard, sheds, barns, etc).

This is the publication where the method comes from but the paper itself doesn't cite all these specimens. Hjelle, B, S Jenison, N Torrez-Martinez, B Herring, S Quan, A Polito, S Pichuantes, T Yamada, C Morris, F Elgh, H W Lee, H Artsob, R Dinello. 1997. Journal of Clinical Microbiology 35(3):600-608. https://doi.org/10.1128/jcm.35.3.600-608.1997 Going to https://github.com/ArctosDB/arctos/issues/6042 to review now. [ https://opengraph.githubassets.com/3e46198e0f1689b7b19a0b3a7537915a07d6a945c57a70d63da2584bb42fa1ad/ArctosDB/arctos/issues/6042 ]https://github.com/ArctosDB/arctos/issues/6042 Code Table Request - New Code Table = examined_detected · Issue #6042 · ArctosDB/arctoshttps://github.com/ArctosDB/arctos/issues/6042 Instructions This is a template to facilitate communication with the Arctos Code Table Committee. Submit a separate request for each relevant value. This form is appropriate for exploring how data ... github.com

Jonathan L. Dunnum Ph.D. (he, him, his) Senior Collection Manager Division of Mammals, Museum of Southwestern Biology University of New Mexico Albuquerque, NM 87131 (505) 277-9262 Fax (505) 277-1351

Chair, Systematic Collections Committee, American Society of Mammalogists Latin American Fellowship Committee, ASM

MSB Mammals website: http://www.msb.unm.edu/mammals/index.html Facebook: http://www.facebook.com/MSBDivisionofMammals

Shipping Address: Museum of Southwestern Biology Division of Mammals University of New Mexico CERIA Bldg 83, Room 204 Albuquerque, NM 87131

From: dustymc @.> Sent: Tuesday, March 28, 2023 3:03 PM To: ArctosDB/arctos @.> Cc: Jonathan Dunnum @.>; Author @.> Subject: Re: [ArctosDB/arctos] [CONTACT] (Issue #6073)

[EXTERNAL]

CSV please - anyone but me, Excel hates me!

I can append the habitat in, if there's one event, but - it's sorta not? "Domestic"? Is there a better place for that??

Can the publication be changed to a more findable format, eg https://arctos.database.museum/publication/1 is nice (and I'm 100% sure I can recover it if this gets more formalized)?

I can't do this until someone checks some boxes at #6042< https://github.com/ArctosDB/arctos/issues/6042> (or we find a better way, or SOMETHING) - help?? (And then I'll hit the other issues, but they're not preventing me from doing stuff NOW.)

— Reply to this email directly, view it on GitHub< https://github.com/ArctosDB/arctos/issues/6073#issuecomment-1487588129>, or unsubscribe< https://github.com/notifications/unsubscribe-auth/AED2PA3LRPU3WDDJMUYUSVTW6NGZRANCNFSM6AAAAAAWLCTK7E

. You are receiving this because you authored the thread.Message ID: @.***>

— Reply to this email directly, view it on GitHub https://github.com/ArctosDB/arctos/issues/6073#issuecomment-1487622476, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADQ7JBC6T5FIOTHCTXFIG2TW6NKJHANCNFSM6AAAAAAWLCTK7E . You are receiving this because you are subscribed to this thread.Message ID: @.***>

dustymc commented 1 year ago

Domestico

AHA! - so - don't translate? Or peridomestic seems to get the idea across. And can you attach the raw to an accession or project or whatever's handy?

jldunnum commented 1 year ago

Just emailed our Panamanian collaborators to see what their meaning is.

Jonathan L. Dunnum Ph.D. (he, him, his) Senior Collection Manager Division of Mammals, Museum of Southwestern Biology University of New Mexico Albuquerque, NM 87131 (505) 277-9262 Fax (505) 277-1351

Chair, Systematic Collections Committee, American Society of Mammalogists Latin American Fellowship Committee, ASM

MSB Mammals website: http://www.msb.unm.edu/mammals/index.html Facebook: http://www.facebook.com/MSBDivisionofMammals

Shipping Address: Museum of Southwestern Biology Division of Mammals University of New Mexico CERIA Bldg 83, Room 204 Albuquerque, NM 87131

From: dustymc @.> Sent: Tuesday, March 28, 2023 4:25 PM To: ArctosDB/arctos @.> Cc: Jonathan Dunnum @.>; Author @.> Subject: Re: [ArctosDB/arctos] Panama serology upload (Issue #6073)

[EXTERNAL]

Domestico

AHA! - so - don't translate? Or peridomestic seems to get the idea across. And can you attach the raw to an accession or project or whatever's handy?

— Reply to this email directly, view it on GitHubhttps://github.com/ArctosDB/arctos/issues/6073#issuecomment-1487681608, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AED2PA5PRIJP6P4L2HT3RYDW6NQNLANCNFSM6AAAAAAWLCTK7E. You are receiving this because you authored the thread.Message ID: @.***>

dustymc commented 1 year ago

great candidates for a collecting event attribute?

Yea, probably, but not now - that will have been entered 67 ways and etc. I'll check that but...

CSV attached

@jldunnum that doesn't work via email.

didn't do anything

You should have, its got invalid headers (which I can easily fix), and a ton of NULL columns (which I can't so easily fix).

I'll fix that and play with test, but I need valid CSV before this can go to prod.

dustymc commented 1 year ago

Bah - I still need valid csv @jldunnum

jldunnum commented 1 year ago

Sorry, will fix that now

Jonathan L. Dunnum Ph.D. (he, him, his) Senior Collection Manager Division of Mammals, Museum of Southwestern Biology University of New Mexico Albuquerque, NM 87131 (505) 277-9262 Fax (505) 277-1351

Chair, Systematic Collections Committee, American Society of Mammalogists Latin American Fellowship Committee, ASM

MSB Mammals website: http://www.msb.unm.edu/mammals/index.html Facebook: http://www.facebook.com/MSBDivisionofMammals

Shipping Address: Museum of Southwestern Biology Division of Mammals University of New Mexico CERIA Bldg 83, Room 204 Albuquerque, NM 87131

From: dustymc @.> Sent: Tuesday, April 4, 2023 9:56 AM To: ArctosDB/arctos @.> Cc: Jonathan Dunnum @.>; Mention @.> Subject: Re: [ArctosDB/arctos] Panama serology upload (Issue #6073)

[EXTERNAL]

Bah - I still need valid csv @jldunnumhttps://github.com/jldunnum

— Reply to this email directly, view it on GitHubhttps://github.com/ArctosDB/arctos/issues/6073#issuecomment-1496224738, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AED2PA3L2U2CE5PK4O45JVDW7RAEXANCNFSM6AAAAAAWLCTK7E. You are receiving this because you were mentioned.Message ID: @.***>

dustymc commented 1 year ago

And I think maybe the data need some more review, although it's possible I borked something trying to deexcel:

arctosprod@arctos>> select guid from temp_ph group by guid having count(*) > 1;
      guid       
-----------------
 MSB:Mamm:342307
 MSB:Mamm:342315
 MSB:Mamm:342326
 MSB:Mamm:342312
 MSB:Mamm:342329
 MSB:Mamm:342325
 MSB:Mamm:342313
 MSB:Mamm:342318
 MSB:Mamm:342317
 MSB:Mamm:264590
 MSB:Mamm:342303
 MSB:Mamm:342311
 MSB:Mamm:264591
 MSB:Mamm:342316
 MSB:Mamm:342560
 MSB:Mamm:342308
 MSB:Mamm:342327
 MSB:Mamm:342324
 MSB:Mamm:342309
 MSB:Mamm:342306
 MSB:Mamm:342314
 MSB:Mamm:342322
 MSB:Mamm:342330
 MSB:Mamm:342331
 MSB:Mamm:342310
 MSB:Mamm:342304
 MSB:Mamm:342328
 MSB:Mamm:342320
 MSB:Mamm:342559
 MSB:Mamm:342321
 MSB:Mamm:342305
 MSB:Mamm:342319
(32 rows)

eg https://arctos.database.museum/guid/MSB:Mamm:342307 claims to be wandering around in multiple places presumably after having its whole organism pickled....

I created https://arctos.database.museum/publication/10011855, I will plan to turn

(following method in Hjelle et al. 1997_J Clin Microbiol. 35(3):600-8. doi: 10.1128/jcm.35.3.600-608.1997).

into 

(following method in https://arctos.database.museum/publication/10011855).

if there are no strong objections (because I believe that's more recoverable if we end up with some new method structure)

jldunnum commented 1 year ago

What exactly is the problem with these records? Is it the attributed dates I assigned 10 years after the collection date?

Jonathan L. Dunnum Ph.D. (he, him, his) Senior Collection Manager Division of Mammals, Museum of Southwestern Biology University of New Mexico Albuquerque, NM 87131 (505) 277-9262 Fax (505) 277-1351

Chair, Systematic Collections Committee, American Society of Mammalogists Latin American Fellowship Committee, ASM

MSB Mammals website: http://www.msb.unm.edu/mammals/index.html Facebook: http://www.facebook.com/MSBDivisionofMammals

Shipping Address: Museum of Southwestern Biology Division of Mammals University of New Mexico CERIA Bldg 83, Room 204 Albuquerque, NM 87131

From: dustymc @.> Sent: Tuesday, April 4, 2023 10:07 AM To: ArctosDB/arctos @.> Cc: Jonathan Dunnum @.>; Mention @.> Subject: Re: [ArctosDB/arctos] Panama serology upload (Issue #6073)

[EXTERNAL]

And I think maybe the data need some more review, although it's possible I borked something trying to deexcel:

@.**>> select guid from temp_ph group by guid having count() > 1; guid

MSB:Mamm:342307 MSB:Mamm:342315 MSB:Mamm:342326 MSB:Mamm:342312 MSB:Mamm:342329 MSB:Mamm:342325 MSB:Mamm:342313 MSB:Mamm:342318 MSB:Mamm:342317 MSB:Mamm:264590 MSB:Mamm:342303 MSB:Mamm:342311 MSB:Mamm:264591 MSB:Mamm:342316 MSB:Mamm:342560 MSB:Mamm:342308 MSB:Mamm:342327 MSB:Mamm:342324 MSB:Mamm:342309 MSB:Mamm:342306 MSB:Mamm:342314 MSB:Mamm:342322 MSB:Mamm:342330 MSB:Mamm:342331 MSB:Mamm:342310 MSB:Mamm:342304 MSB:Mamm:342328 MSB:Mamm:342320 MSB:Mamm:342559 MSB:Mamm:342321 MSB:Mamm:342305 MSB:Mamm:342319 (32 rows)

eg https://arctos.database.museum/guid/MSB:Mamm:342307 claims to be wandering around in multiple places presumably after having its whole organism pickled....

I created https://arctos.database.museum/publication/10011855, I will plan to turn

(following method in Hjelle et al. 1997_J Clin Microbiol. 35(3):600-8. doi: 10.1128/jcm.35.3.600-608.1997).

into

(following method in https://arctos.database.museum/publication/10011855).

if there are no string objections (because I believe that's more recoverable if we end up with some new method structure)

— Reply to this email directly, view it on GitHubhttps://github.com/ArctosDB/arctos/issues/6073#issuecomment-1496241301, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AED2PAYCZ3L7QMBYWIYPMC3W7RBNBANCNFSM6AAAAAAWLCTK7E. You are receiving this because you were mentioned.Message ID: @.***>

dustymc commented 1 year ago

There are multiples of them in the file - not necessarily a problem, but I didn't expect it.

jldunnum commented 1 year ago

Hmm, OK let me review

Jonathan L. Dunnum Ph.D. (he, him, his) Senior Collection Manager Division of Mammals, Museum of Southwestern Biology University of New Mexico Albuquerque, NM 87131 (505) 277-9262 Fax (505) 277-1351

Chair, Systematic Collections Committee, American Society of Mammalogists Latin American Fellowship Committee, ASM

MSB Mammals website: http://www.msb.unm.edu/mammals/index.html Facebook: http://www.facebook.com/MSBDivisionofMammals

Shipping Address: Museum of Southwestern Biology Division of Mammals University of New Mexico CERIA Bldg 83, Room 204 Albuquerque, NM 87131

From: dustymc @.> Sent: Tuesday, April 4, 2023 10:30 AM To: ArctosDB/arctos @.> Cc: Jonathan Dunnum @.>; Mention @.> Subject: Re: [ArctosDB/arctos] Panama serology upload (Issue #6073)

[EXTERNAL]

There are multiples of them in the file - not necessarily a problem, but I didn't expect it.

— Reply to this email directly, view it on GitHubhttps://github.com/ArctosDB/arctos/issues/6073#issuecomment-1496271651, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AED2PAZQJ3NA3XJ3DFWBOVLW7REBDANCNFSM6AAAAAAWLCTK7E. You are receiving this because you were mentioned.Message ID: @.***>

jldunnum commented 1 year ago

Definitely an issue of some sort so don't load anything until I work through it. Thansk!

Jonathan L. Dunnum Ph.D. (he, him, his) Senior Collection Manager Division of Mammals, Museum of Southwestern Biology University of New Mexico Albuquerque, NM 87131 (505) 277-9262 Fax (505) 277-1351

Chair, Systematic Collections Committee, American Society of Mammalogists Latin American Fellowship Committee, ASM

MSB Mammals website: http://www.msb.unm.edu/mammals/index.html Facebook: http://www.facebook.com/MSBDivisionofMammals

Shipping Address: Museum of Southwestern Biology Division of Mammals University of New Mexico CERIA Bldg 83, Room 204 Albuquerque, NM 87131

From: Jonathan Dunnum @.> Sent: Tuesday, April 4, 2023 10:31 AM To: ArctosDB/arctos @.>; ArctosDB/arctos @.> Cc: Mention @.> Subject: Re: [ArctosDB/arctos] Panama serology upload (Issue #6073)

Hmm, OK let me review

Jonathan L. Dunnum Ph.D. (he, him, his) Senior Collection Manager Division of Mammals, Museum of Southwestern Biology University of New Mexico Albuquerque, NM 87131 (505) 277-9262 Fax (505) 277-1351

Chair, Systematic Collections Committee, American Society of Mammalogists Latin American Fellowship Committee, ASM

MSB Mammals website: http://www.msb.unm.edu/mammals/index.html Facebook: http://www.facebook.com/MSBDivisionofMammals

Shipping Address: Museum of Southwestern Biology Division of Mammals University of New Mexico CERIA Bldg 83, Room 204 Albuquerque, NM 87131

From: dustymc @.> Sent: Tuesday, April 4, 2023 10:30 AM To: ArctosDB/arctos @.> Cc: Jonathan Dunnum @.>; Mention @.> Subject: Re: [ArctosDB/arctos] Panama serology upload (Issue #6073)

[EXTERNAL]

There are multiples of them in the file - not necessarily a problem, but I didn't expect it.

— Reply to this email directly, view it on GitHubhttps://github.com/ArctosDB/arctos/issues/6073#issuecomment-1496271651, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AED2PAZQJ3NA3XJ3DFWBOVLW7REBDANCNFSM6AAAAAAWLCTK7E. You are receiving this because you were mentioned.Message ID: @.***>

jldunnum commented 1 year ago

Hey Dusty, Teresa and Mariel, Here is another go at the Orthohanta serology load (a smaller subset which should be clean. Panama Orthohanta serology_1st batch.csv

campmlc commented 1 year ago

Hi Jon, just a quick lookover, and looks like you need to change attribute

  1. The attribute types would either be "detected" or "not detected" and the value would be "virus: Orthohantavirus". Also, the format needs to be in the format for the attribute bulkload component loader. Do you want me to tweak the file?

On Mon, May 1, 2023 at 1:42 PM Jonathan Dunnum @.***> wrote:

  • [EXTERNAL]*

Hey Dusty, Teresa and Mariel, Here is another go at the Orthohanta serology load (a smaller subset which should be clean. Panama Orthohanta serology_1st batch.csv https://github.com/ArctosDB/arctos/files/11367631/Panama.Orthohanta.serology_1st.batch.csv

— Reply to this email directly, view it on GitHub https://github.com/ArctosDB/arctos/issues/6073#issuecomment-1530122358, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADQ7JBBVQXNWYHLBI7RTCS3XEAG3HANCNFSM6AAAAAAWLCTK7E . You are receiving this because you commented.Message ID: @.***>

dustymc commented 1 year ago

@campmlc yes please, I think the setup is all happy now so this can go through the attribute loader, you're more likely to notice problem than I am, goferit, let me know if anything causes any kind of problem.

Or let me know, I don't mind pushing buttons, I can set this up for the component loaders too.

I think all of these should have one

and one of either

or

https://arctos.database.museum/info/ctDocumentation.cfm?table=ctexamined_detected#virus__orthohantavirus

and it took me longer than it should have to find those, can we pretty please prioritize getting rid of the old stuff without slamming into the usual halfway-there hardblock? I can open Issue, although that also might go smoother if it starts with someone with some idea of the intent of a bunch of attributes.

campmlc commented 1 year ago

OK - will do. I have the cleaned up csvs for the three attribute bulkloads to send to Jon right now. Once we determine they work, I will attach here for reference.

On Mon, May 1, 2023 at 2:31 PM dustymc @.***> wrote:

  • [EXTERNAL]*

@campmlc https://github.com/campmlc yes please, I think the setup is all happy now so this can go through the attribute loader, you're more likely to notice problem than I am, goferit, let me know if anything causes any kind of problem.

Or let me know, I don't mind pushing buttons, I can set this up for the component loaders too.

I think all of these should have one

- https://arctos.database.museum/info/ctDocumentation.cfm?table=CTATTRIBUTE_TYPE&field=detected#examined_for

and one of either

- https://arctos.database.museum/info/ctDocumentation.cfm?table=CTATTRIBUTE_TYPE&field=detected#detected

or

- https://arctos.database.museum/info/ctDocumentation.cfm?table=ctattribute_type#not_detected

all with values of

https://arctos.database.museum/info/ctDocumentation.cfm?table=ctexamined_detected#virus__orthohantavirus

and it took me longer than it should have to find those, can we pretty please prioritize getting rid of the old stuff without slamming into the usual halfway-there hardblock? I can open Issue, although that also might go smoother if it starts with someone with some idea of the intent of a bunch of attributes.

— Reply to this email directly, view it on GitHub https://github.com/ArctosDB/arctos/issues/6073#issuecomment-1530198454, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADQ7JBAWJCKMSWTSPCY5BNTXEAMTPANCNFSM6AAAAAAWLCTK7E . You are receiving this because you were mentioned.Message ID: @.***>

campmlc commented 1 year ago

This looks to be completed. Closing.