Contributions with bad data

[moonshoes87]

As @njarboe suggested, I'm making an issue to keep track of contributions with bad data. These will be mainly contributions on which make_magic_plots.py has failed due to bad/incomplete data. I will add to this list as I find more problems.

14019 -- no site column in the sites table, and generally very incomplete sites table.

[moonshoes87]

14290 -- no sample columns in samples table, no locations table.

[moonshoes87]

14575 -- not sure if this is technically "bad" data, but the sites table is missing many entries for location. This is one of those places where it would be very helpful if minimal info like location name were propagated throughout the rows where available. Doing so in PmagPy is possible but slow to do every time. It also looks like there are 7 locations but only 1 is actually linked to any sites, which seems a little suspicious.

[moonshoes87]

15417 -- this has only a contribution table.

[rminnett]

Thanks for doing this, Lori!

[njarboe]

16454 - contribution ID missing

[moonshoes87]

Contribution 11821 has values in measurements.specimen that are not present in the specimens table.

It also has at least one missing value in measurements.dec (specimen SLB05.4a, may be others).

[moonshoes87]

Contribution https://earthref.org/MagIC/11846: no 'sample' column in samples table.

[moonshoes87]

Contribution https://earthref.org/MagIC/14127 has no 'site' column in sites table

[moonshoes87]

Contribution https://earthref.org/MagIC/15417 is missing a locations table (or anything besides a contribution table).

[moonshoes87]

https://earthref.org/MagIC/15640 is missing locations table (or anything besides contribution)

[moonshoes87]

https://earthref.org/MagIC/16273 is missing locations table (but has samples --> measurements)

Also has two samples tables, two specimens tables, two measurements tables -- this one definitely needs some attention.

[moonshoes87]

contribution 16302 is missing at least one value in measurements.dir_dec

[moonshoes87]

15736 has no locations table: https://earthref.org/MagIC/15736

[moonshoes87]

16072 has no locations table: https://earthref.org/MagIC/16072

[moonshoes87]

16320 has negative values in sites.vgp_lat with a space between the '-' and the number, i.e.: − 69.1. Should this be fixed for download? It means that Python, at least, does not correctly translate this value into a float. I will put a fix in PmagPy, but seems like this could be annoying for others as well.

Edit: Actually, I had to do some weird manipulations because of the unicode characters in these strings. Not sure what is going on with this sites table, but the vgp_lat field contained values like \u2212 which have to be translated.

[moonshoes87]

16338 has many blank fields in measurements.method_codes.

[moonshoes87]

16410 downloads with an extra header row in the sites table:

>>>>>>>>>>
tab delimited   sites
site    location    lithologies lat lon elevation   dir_tilt_correction dir_dec dir_inc dir_alpha95 dir_r   dir_k   dir_n_samples   vgp_lat vgp_lon

Site    Locality    Lithology   Latitude    Longitude   Elev (m)    Direction Tilt Correction   Dec Inc a95 R   K   N   VGP Lat VGP Lon
DB0708  Haas Paleosol   Greyish brown mudstone  39.41202    -104.34196  1874.31 0.00000 335 42  39.9    2.81    10.62   3   64  137.4
DB0707  Haas Paleosol   Olive brown mudstone    39.41200    -104.34194  1873.14 0.00000 327.5   54.9    23.9    2.93    27.6    3   64  167.2

16416 https://earthref.org/MagIC/16416 has the same problem.

[moonshoes87]

https://earthref.org/MagIC/16418

No contribution id in contribution table.

[moonshoes87]

https://earthref.org/MagIC/16497

Several controlled vocabularies in the sites table have incorrect entries. method_codes, result_quality, and result_type are all filled with 1s.

[moonshoes87]

https://earthref.org/MagIC/16501 has no locations table.

[moonshoes87]

16416 https://earthref.org/MagIC/16416 has the descriptive row included as well as the actual headers.

[moonshoes87]

Something wrong in the naming hierarchy (can't propagate locations down to the measurement level):

13742, 16279, 16426, 15444, 16335, 15349, 15897, 16240, 16619, 16450, 16308, 16501, 16238, 16291, 16497, 16515, 13709, 14359, 16334, 15890, 16452, 16358, 11943, 14868, 12450, 16609, 16263, 16273, 16505, 11881, 15221, 14614, 16416, 11189, 16269, 14575, 16421, 11883, 16353, 11821, 16301, 15435, 16508, 16280, 16305, 12638, 16258, 16237, 16233, 14891, 16410, 15551, 11906, 13538, 16624, 16411, 16529, 16313, 15803, 15040, 14384, 16626, 15461, 15085, 11773, 11929, 11846, 13969, 16618, 16623

Locations table has many blanks in the 'location' column:

11189

Problem with naming hierarchy and missing column (treat_temp):

13727 14809 16457 16015

quick_hyst.py LP-HYS method code present, but required column(s) [treat_temp] missing

15283 15840 16458 16460

No tables found:

16277