search

PmagPy / PmagPy

Python-based tools for paleomagnetic research
https://pmagpy.github.io/PmagPy-docs/
BSD 3-Clause "New" or "Revised" License
69 stars 42 forks source link

Some column names need to be removed from 3.0 upload files #119

Closed moonshoes87 closed 7 years ago

moonshoes87 commented 7 years ago

Some columns are added into tables where they don't belong. For example, the specimens table might have 'site' and 'location'. Those should be removed prior to upload. This will be a function for new_builder (a method of Contribution).

moonshoes87 commented 7 years ago

I've now implemented functionality that does this: 641847bcd

The way this currently works is that all headers that aren't in the 3.0 data model are removed from all MagIC tables prior to upload. The problem with this approach is that if someone has a slightly incorrect MagIC column, this will be deleted as well. With Rupert's new interface, it's pretty easy to fix a misspelled column name, but a user won't have the option to do that if I chuck that column before the upload.

Some options:

Misspelled columns aren't too likely to be a problem for people who are using PmagPy the whole way through their process, since we are pretty consistent. But if they're mixing and matching with their process, maybe doing some editing by hand, the problem could come up.

@ltauxe, any thoughts about this?

moonshoes87 commented 7 years ago

From meeting: Add a warning, and an are you sure?

njarboe commented 7 years ago

My two cents worth is that when creating a MagIC file for upload the columns that are used internally in the PmagPy programs, but not found in the data model, should be dropped without a warning. Columns added by the user, but not in the data model, should be kept with a warning. The user, as you said, can link an unknown column to the correct one in the data model when uploading. I'm generally against correcting spelling automatically and instead like just pointing out the possible problem.

On Tue, May 16, 2017 at 11:11 AM, Lori J notifications@github.com wrote:

From meeting: Add a warning, and an are you sure?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/PmagPy/PmagPy/issues/119#issuecomment-301868166, or mute the thread https://github.com/notifications/unsubscribe-auth/ABODRe1Korb6rRjx-9zCDiDuos1btB5Rks5r6ebCgaJpZM4LvYeD .

moonshoes87 commented 7 years ago

@njarboe, thanks for your input. I've implemented a solution that does more or less what you suggest, although the problem with blacklisting PmagPy columns is that I suspect I don't have a complete list of what should be dropped. I will keep trying to add those (in ipmag.upload_magic3, to RmKeys and extra_RmKeys) as I find them. Everyone else should feel free to do the same, or let me know what they find.