monfresh
commented
10 years ago Thanks for the feedback, Francis! First, I'd like to make sure I understand the issue. It sounds like there are two issues here:
- Some validation rules are too strict.
- Need ability to flag incorrect entries.
The ability to flag invalid entries was added 18 days ago. Currently, the import script allows all valid entries to go through, and then saves the invalid entries to a separate file. This allows you to have a working version of the API that serves valid data, while also allowing you to correct the invalid data at a later time, albeit via the data files, not the admin interface.
As for the validation rules, that is something we are still iterating on via the OpenReferral spec. If there are specific validations you think are too strict, please mention them here and also open an issue in the OpenReferral repo. One validation that I think you ran into was the address/mail_address requirement for a location because you have many "virtual" locations. One way to get around that is to add a virtual boolean column to the Locations table (set to false by default), then change the validation rule to only apply if the location's virtual field is false, then set the virtual field to true in your dataset for all locations that don't have an address.
Feel free to submit a pull request if you think that's the right approach for virtual locations.
francisli
I thought I'd bring up a suggestion for discussion, as I've started participating in the SF Brigade's project with OpenReferral/Ohana. While the spec and the API platform are being worked on, we've been spending most of the time gathering data and preparing them for import. The process is basically gather, transform, attempt import- see failures, rollback additions, edit data, re-attempt import- see failures, rollback, edit, - wash, rinse, repeat. Managing this process in the source import files themselves is a very technical and rather arduous task (especially when the data set is large).
A more user-focused process would be to allow importing and storing incomplete data through the admin interface, with views for highlighting invalid records for editing/correction at a later time. For example, right now to get data to import, we're often putting empty placeholder strings in various required fields. Now those records are "valid", but with placeholder values that are not always easily distinguishable from real "valid" data. Contrast that to being able to import them with clear flags that they are in an invalid/draft state, so that they can be easily recalled for reference when confirming/editing/correcting the data- say, by calling the organization, which could be days later.
This is more than just a UI/design issue- for this to be possible, the data model will need to support it- i.e. by removing database constraints (i.e. NOT NULL) and adding conditionals to the validations. Note that this is not completely removing the validations- validations can still be run conditionally, and validation state could be stored as a boolean column in each table. Clearly if a record is incomplete/invalid as per spec, it should not be fetched from the API nor exported.