search

not-nexus / shelf

REST API for AWS S3 meant to be an interface to immutable artifact storage.
MIT License
8 stars 3 forks source link

Parallel requests to update/create metadata/artifacts cause inconsistent results #98

Open kyle-long opened 7 years ago

kyle-long commented 7 years ago 
TransportError(409, u'VersionConflictEngineException[[metadata][3] [metadata][e2eaa4779fe88c8087873098fe6db01e38e86f26a403149e41323b958b0f57f1]: version conflict, current [8], provided [7]]')

This happens with elasticsearch. It's discussed here https://www.elastic.co/guide/en/elasticsearch/guide/current/partial-updates.html.

We could attempt to do partial updates to mitigate this problem. What I could see is somebody not having all the metadata. Instead of getting it, this user just wants to update a few metadata properties, but in parallel.

I could also see one user trying to update all metadata and the same time another user is updating a single property.

The solution for the former could be to change updating metadata properties to use partial updates. I have no solution for the latter. Which version do we use? Does it matter? Probably not unless they are trying to put an immutable property and the other user is putting it as a non immutable property. Even then I am not sure it matters.

I would be open to suggestions.

kyle-long commented 7 years ago

After a discussion with @fidian this problem is worse than I had thought.

The real problem is that parallel updates don't do any locking. Elasticsearch blows up but likely S3 just takes the last request to come in.

Updating title