Missing data after upgrading a 2.5 contribution

[moonshoes87]

I tried upgrading this contribution with the MagIC upgrade tool:

zmab0094214tmp03.txt

And got this result.

Upgraded Contribution v3.0.txt

There appears to be some significant loss of data. For example, the 2.5 contribution has vgp_lat and vgp_lon for each result, but those data are missing in the 3.0 file. Class and lithology are also present in the 2.5 contribution but missing in the upgrade.

It's possible that the original contribution was actually in 2.4 or lower, but the upgrade gave no warnings to imply that data was dropped.

[rminnett]

Thanks for letting me know about this. I just ran the upgrade tool on Windows and MacOS on that file and got this both times:

Upgraded Contribution v3.0 - Rupert.txt

Do you mind upgrading your file one more time to see if your results are reproducible, please?

zmab0094214tmp03.txt

If so, we might need to do a screen share meeting when it's a good time for you to see what the difference is between our computers.

[moonshoes87]

The upgraded file I'm getting now matches yours. Apparently I did something different initially and didn't realize it.

However, in both upgraded files I'm still missing some data from the pmag_results table. Looking at the data model, it appears that pmag_results.vgp_lat should be converted either into sites.vgp_lat or locations.pole_lat, but neither is present in the upgraded contribution. Am I misunderstanding how the conversion should work, or are data being lost?

[rminnett]

We decided that in 3.0 there would be sites VGPs and locations Poles, but no samples or specimens VGPs (https://earthref.org/MagIC/data-models and searching for "vgp" in the top right tab, should clarify this). Those look like specimens VGPs in that file, so they're being dropped by design. If that's not what we intended, we might have to revise our rules.

[moonshoes87]

I see, so they are ignored? Ok, perhaps this should be a discussion at the MagIC meeting...

On Apr 26, 2018, at 2:38 PM, Rupert notifications@github.com wrote:

We decided that in 3.0 there would be sites VGPs and locations Poles, but no samples or specimens VGPs (https://earthref.org/MagIC/data-models https://earthref.org/MagIC/data-models and searching for "vgp" in the top right tab, should clarify this). Those look like specimens VGPs in that file, so they're being dropped by design. If that's not what we intended, we might have to revise our rules.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/earthref/MagIC/issues/342#issuecomment-384797252, or mute the thread https://github.com/notifications/unsubscribe-auth/ABwGIpS0TVqbYMFqVbctOnjLQuFtFRNVks5tsj5ngaJpZM4TkVvV.

[rminnett]

Yes, they're currently getting ignored. I was able to trace the discussion back to some emails on 8/9/16 where Lisa and Ron were talking about other samples and specimens issues. It doesn't look like the VGP issue was specifically resolved in an email because my last email on that thread never got a response, but perhaps we discussed it further in a meeting after that. We could certainly revisit this in the meeting tomorrow, though.

[rminnett]

I searched through the 2.5 and older contributions for ones that had pmag_results VGPs only at samples and specimens and luckily these are the only ones that matched:

https://earthref.org/MagIC/10894 https://earthref.org/MagIC/10507 https://earthref.org/MagIC/8732

I'm downgrading the priority on this issue and handing it over to @njarboe to determine if any of the data from 2.5 and older versions should be reintroduced to the latest versions.