search

aptly-dev / aptly

aptly - Debian repository management tool
https://www.aptly.info/
MIT License
2.57k stars 374 forks source link

Updating on S3 needs mutual exclusion on API #374

Closed mpuchol-netquest closed 2 months ago

mpuchol-netquest commented 8 years ago

When a repository is being updated to S3, if there is another thread doing the same action, last action can fails.

The following log show previous case:

[GIN] 2016/04/01 - 07:39:26 | 200 |  724.63868ms | 10.50.30.251:58859 |   POST    /api/files/folder1
[GIN] 2016/04/01 - 07:39:26 | 200 | 555.043702ms | 10.50.30.251:58860 |   POST    /api/repos/repository/file/folder1
Signing file 'Release' with gpg, please enter your passphrase when prompted:
Clearsigning file 'Release' with gpg, please enter your passphrase when prompted:
[GIN] 2016/04/01 - 07:39:43 | 200 | 17.203494609s | 10.50.30.251:58861 |   PUT     /api/publish/s3:repo:/precise
[GIN] 2016/04/01 - 07:39:52 | 200 | 1.324651495s | 10.50.30.251:58862 |   POST    /api/files/folder2
[GIN] 2016/04/01 - 07:39:53 | 200 | 950.391809ms | 10.50.30.251:58863 |   POST    /api/repos/repository/file/folder2
Signing file 'Release' with gpg, please enter your passphrase when prompted:
Clearsigning file 'Release' with gpg, please enter your passphrase when prompted:
[GIN] 2016/04/01 - 07:40:10 | 500 | 17.75603111s | 10.50.30.251:58864 |   PUT     /api/publish/s3:repo:/precise
Error #01: unable to update: unable to rename: error copying dists/precise/main/binary-amd64/Release.tmp -> dists/precise/main/binary-amd64/Release in S3: us-east-1:repo/: NoSuchKey: The specified key does not exist.
    status code: 404, request id: 164EAA42579D5CE2 
     Meta: Operation aborted

I guess this error should happen even updating on disk.

Adding a mutual exclusion on PUT could fix this problem.

Aptly version: 0.9.7~dev

smira commented 8 years ago

yep, thanks for the report!

easy fix would be allowing only one publish operation concurrently. but probably we can allow one per underlying storage.

dpedu2 commented 6 years ago

+1 for this

smira commented 6 years ago

I checked the code and I don't think aptly can do any two concurrent publishes in API mode (not mentioning two concurrent publishes to S3).

marcinswigon commented 4 years ago

Hi, I am hitting this issue quite often recently. Looks like for S3 some delay is needed before https://github.com/aptly-dev/aptly/blob/master/s3/public.go#L395 On versioned buckets .tmp files are created but looks like move is performed before storage gets consistency. Further reading: https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#CoreConcepts "Amazon S3 Data Consistency Model"