Closed lukecampbell closed 7 years ago
Shouldn't some sort of UUID be enforcing the uniqueness, rather than the title?
I'm not the designer of this software, I don't know what assumptions or why it was done this way. But the issue is that it fails and prevents us from running concurrent harvesters.
I can't think of an easy way to do this without making some architecture changes, i.e. pushing the name key up to Redis prior to the update. There is, however, a unique key on the name column of the "packages" table.
what about catching the exception and if it's not unique, adding an integer to the title?
Imagine two datasets with the title "Fake Data"
The title key becomes fake-data
and when the unique key constraint fails, we would change it to fake-data-1
and try again?
There's actually UUID/hex appending logic in the harvesters which should in theory substantially reduce the number of collisions over sequentially incrementing a number (16^5 random vs 999 sequential). I'm going to opt for that path for the time being, but I'd like to also handle the case where a failure does occur by catching the exception and at least logging it.
When CKAN creates a package entry for a dataset, it creates a package name based on the
title
from the XML document. Duplicate names are kind of ok, in that the harvester will query the database to see if the name already exists, if it doesn't then create it, if it does, append a numeral to the end of the number for the next one.The problem is that if there are concurrent workers, this creates a race condition of multiple workers are processing datasets the same title and it will ultimately lead to a crash of the worker if the right race condition is met:
We need to find a way to ensure titles are unique before the harvesters attempt to create the record.