search

andrewxhill / MOL

The Map of Life
mol.colorado.edu/
19 stars 4 forks source link

config.yaml has unexpected fields #110

Closed tucotuco closed 12 years ago

tucotuco commented 13 years ago

After a fresh install on Windows, running loader.py shows errors in the config.yaml file provided.

C:\Projects\MOL\workflow\mol-data>loader.py --url=http://localhost:8080/_ah/remote_api --config_file=bulkload.yaml INFO:root:Processing source directories: ['iucn', 'jetz']

ERROR:root: Unexpected fields found in section 'Collections:Required': collectionorigin, collectionpresence, collectionseasonality, collectionstartdate, contributor, coverage, creator, date, description, email, publisher, rights, scientificname, source, subject, title

ERROR:root: Unexpected fields found in section 'Collections:Optional': accessright, enddateaccuracy, format, identifier, language, relation, rights, startdateaccuracy, type

ERROR:root: Fields missing from section 'Collections:DBFMapping:Required': contributor, creator, format, pubdate, publisher, rights, title, type

ERROR:root: Unexpected fields found in section 'Collections:DBFMapping:Optional': contributor, dateend, datestart, infraspecificepithet, measurementcomment, polygonid

ERROR:root:'iucn\config.yaml', directory 'mammals' could not be validated. Please fix the errors reported above and retry.

gaurav commented 13 years ago

Yup, I'll bring the config.yamls into compliance once Rob and I have finalized the fields spec. For now, you can run loader.py -V to turn off the validation, for instance if you just need to test the bulk upload. I'll add that note to the validation errors also.

gaurav commented 12 years ago

Quick update: I've added the note to the validation option as promised. I've been working on bringing the config.yamls into compliance, but that requires getting halfway to the new config.yaml format we discussed a few weeks ago (since the current field spec specifically expects fields to work as per the new format, i.e. with a single set of fields which can be set at either the shapefile level or the layer level). I'm mostly done with the code in @305aedba2d; now I need to actually upgrade the config files. Should be finished soon, hopefully tomorrow!

gaurav commented 12 years ago

Okay, I think this is fixed as of @1770a58318. I've glanced through the uploaded data and it looks okay to me, but I won't merge it into master until Monday, when I'll have had a chance to really make sure the DBF fields are being mapped correctly. But I'm pretty sure most things are working okay!

Because of the constraints of the field listings (which involves merging "Source" fields and "DBF" fields into fields divided only by their Required/Optional status), we're now halfway to @tucotuco's idea about have each field define default values, column names, and so on. I could work on that next week, if that makes sense.

tucotuco commented 12 years ago

+1 It would be nice to have that all wrapped up.

gaurav commented 12 years ago

Okay, mapping and bulkloading are working for me as of @83c19169f71c. I'll close this issue and open another one for the new config.yaml file format.