Organisation Reference Collection is Incomplete

[AnotherCodeArtist]

The latest import of RTR (June 2016) data has shown that there are several problems with the import routine. Apparently a good part of organizations is missing in the organization collection including for example "Bundesministerium für Inneres". This is important, since for every imported transfer, the corresponding document needs to be looked up and a reference to it is stored. When no such reference document was found, the entire tranfser was ignored. Since this behavior is erroneous and inacceptable the following changes have been made to the master branch:

• A new "Unknown" organization was added to the CSV document (Rechtsträger) and imported
• If no corresponding entry for a new transfer can be found, it gets linked to this "Unknown" organization
• The error-handling for organizations and transfer upload was improved and in the case of the transfer upload at least the number of errors gets reported in the user interface

So there's the following TODO's:

• fix the missing organizations and find out, why they aren't found (this could also involve refinement of organization names in the old reports from 2012-2013?).
• make sure that errors are reported in the user interface
• the error message should include a list of records that have caused these errors
• do some quality checks (e.g. comparing the number of entries in CSV files with the number of documents in the database

[relative-progressio]

This was really interesting. The list of zip codes did not include a few zip codes (e.g. the one for Bundesministerium für Inneres). To resolve all possible missing references, I edited the list of organisations and the newest report, since the newest report seems to include all previous data. This can be now automated with OpenRefine. I uploaded all the steps so OpenRefine can execute them. Before that, the steps have to be uploaded into the OpenRefine project.

Errors (from MongoDB or during upload) and the entries caused them are now shown at the user interface.

The mentioned quality check is implemented, additionally all information about possible sources of error is shown.

Implemented at: refined-upload-branch

[relative-progressio]

We need to leave the Bundesminierium-entries without changes.