1) Take the list of publishers/files from the registry
2) In each multiprocess take one of the entries from the list and add the metadata information
3) recreate the list of publishers/files from the registry from the returned results of (2)
If something goes wrong we should return the original entry from the registry to rebuild the list, this is done by catching all the exceptions and returning the original data. This is fine.. but we don't want to rely on that mechanism, especially in the case where one of those multiprocesses gets terminated without an exception but doesn't terminate the parent process (if that is even possible).
In https://github.com/ThreeSixtyGiving/datagetter/blob/master/getter/get.py#L297 we currently
1) Take the list of publishers/files from the registry 2) In each multiprocess take one of the entries from the list and add the metadata information 3) recreate the list of publishers/files from the registry from the returned results of (2)
If something goes wrong we should return the original entry from the registry to rebuild the list, this is done by catching all the exceptions and returning the original data. This is fine.. but we don't want to rely on that mechanism, especially in the case where one of those multiprocesses gets terminated without an exception but doesn't terminate the parent process (if that is even possible).