Add additional warnings about potential data model definition problems

[murphyke]

There are a couple circumstances which cause the data-models-sqlalchemy service to throw exceptions, and it might make sense to flag them in the data models service. Currently it's impossible for users to know what fields or tables are responsible for the exceptions, so either the data models service or DMSA should be changed. People tend to treat the data models service as the validator for the data model definitions, so it makes sense to put the warnings here.

The problematic situations are:

• schema/sometable.csv doesn't exist or does not contain rows for all fields of sometable. I realize that the physical implementation of fields is optional as far as the data models service is concerned, but in the real world, an advisory warning would seem to be practical.
• definitions/sometable.csv doesn't exist even though it exists in tables.csv.

[murphyke]

I have spent way too much time today trying to debug the latest pcornet data model submission. There were various issues, one of which was a typo in a field name in schema/sometable.csv that caused DMSA to die unhelpfully. Hopefully the next time this happens I will have a total freakout and will improve the validation in DMS.

[burrowse]

THANK YOU @murphyke for all your help!!

In adding the PCORnet 4.0 DDL, which started as a copy of what was in place for the PCORnet 3.1 DDL there were issues around formatting, data types etc that were discovered.

If the tool is intended to be used for future work, it would be great to have the following features/warnings:

• Model name warning, if a model name supplied does not exist.
• A validator that checks and reports on fields that do not align between schema and definitions (or ways to report that issues exist)
• Warnings for missing foreign key and/or primary key declarations
• Documentation/Information on what file types and/or encoding is or is not supported by the tool
• Information on how to use/debug with CircleCI
• Documentation around how to represent composite keys
• Invalid data type warnings (documentation on what are the valid data types/expected values for each field)