search

aptly-dev / aptly

aptly - Debian repository management tool
https://www.aptly.info/
MIT License
2.56k stars 374 forks source link

Error #01: Needed resources are used by other tasks. #1276

Closed ishaulov closed 2 weeks ago

ishaulov commented 5 months ago

Hi! Need help with publishing packages!

Packages are assembled on Jenkins, and then they are further uploaded and published to the repository using the aptly api:

curl -f -X POST -F file=@"libson-1.11.6248_amd64.deb" http://repos.local/deb/api/files/uploaded-files?_async=true -u repouser:repopassword 
curl -f -X POST http://repos.local/deb/api/repos/debrepo/file/uploaded-files/libson-1.11.6248_amd64.deb?_async=true -u repouser:repopassword 
curl -f -X PUT http://repos.local/deb/api/publish/:./debrepo?_async=true -H 'Content-Type: application/json' --data '{"Signing":{"Batch":true,"GpgKey":"gpgkey"}}' -u repouser:repopassword 
curl -f -X PUT http://repos.local/deb/api/publish/:./ubuntu-20.04?_async=true -H 'Content-Type: application/json' --data '{"Signing":{"Batch":true,"GpgKey":"gpgkey"}}' -u repouser:repopassword 
curl -f -X PUT http://repos.local/deb/api/publish/:./mint?_async=true -H 'Content-Type: application/json' --data '{"Signing":{"Batch":true,"GpgKey":"gpgkey"}}' -u repouser:repopassword 
curl -f -X PUT http://repos.local/deb/api/publish/:./ubuntu-22.04??_async=true -H 'Content-Type: application/json' --data '{"Signing":{"Batch":true,"GpgKey":"gpgkey"}}' -u repouser:repopassword

The problem is that there may be parallel builds, downloads, and publications at the same time, and I get the following error: aptly — "Error #01: Needed resources are used by other tasks." curl — (22) The requested URL returned error: 409 Conflict


{"level":"info","time":"2024-04-18T09:15:42+05:00","message":"Executing task asynchronously"}

{"level":"info","remote":"172.27.0.4","method":"POST","path":"/api/repos/debrepo/file/uploaded-files/libson-1.11.6248_amd64.deb?_async=true","protocol":"HTTP/1.0","code":"202","latency":"166.954µs","agent":"curl/7.58.0","time":"2024-04-18T09:15:42+05:00"}

{"level":"info","time":"2024-04-18T09:15:42+05:00","message":"Executing task asynchronously"}

{"level":"warn","remote":"172.27.0.4","method":"PUT","path":"/api/publish/:./debrepo?_async=true","protocol":"HTTP/1.0","code":"409","latency":"4.423243ms","agent":"curl/7.58.0","time":"2024-04-18T09:15:42+05:00","message":"Error #01: Needed resources are used by other tasks."}

{"level":"info","time":"2024-04-18T09:15:42+05:00","message":"Executing task asynchronously"}

{"level":"warn","remote":"172.27.0.4","method":"PUT","path":"/api/publish/:./ubuntu-20.04?_async=true","protocol":"HTTP/1.0","code":"409","latency":"4.386ms","agent":"curl/7.58.0","time":"2024-04-18T09:15:42+05:00","message":"Error #01: Needed resources are used by other tasks."}

{"level":"info","time":"2024-04-18T09:15:42+05:00","message":"Executing task asynchronously"}

{"level":"warn","remote":"172.27.0.4","method":"PUT","path":"/api/publish/:./mint?_async=true","protocol":"HTTP/1.0","code":"409","latency":"4.258301ms","agent":"curl/7.58.0","time":"2024-04-18T09:15:42+05:00","message":"Error #01: Needed resources are used by other tasks."}

{"level":"info","time":"2024-04-18T09:15:42+05:00","message":"Executing task asynchronously"}

{"level":"warn","remote":"172.27.0.4","method":"PUT","path":"/api/publish/:./ubuntu-22.04?_async=true","protocol":"HTTP/1.0","code":"409","latency":"4.272133ms","agent":"curl/7.58.0","time":"2024-04-18T09:15:42+05:00","message":"Error #01: Needed resources are used by other tasks."}

Have I tried using this version: https://github.com/aptly-dev/aptly/issues/1125#issuecomment-1331843568 and using ?_async=true, but the error persists.

I tried to build aptly from the master branch, errors persist. What else can I do, or maybe I'm doing something wrong?

neolynx commented 5 months ago

Hi, thanks for reporting !

yes, this might be fixed with #1125, which is currently being rebased on master.

the ?_async=true will make the API calls return a task_id. the caller should wait for the task to complete, before issuing a next api call. this can be done with GET /tasks/:id/wait maybe you could try this ?

ishaulov commented 5 months ago

Thanks for the reply!

Yes, I can. But let me clarify: do I understand correctly that I need to write a shell handler that will process each curl request, take the ID and put it into GET /tasks/:ID/wait ?

neolynx commented 5 months ago

yes, something like this: curl http://repos.local/api/tasks/1/wait

ishaulov commented 4 months ago

Hello, @neolynx!

I apologize for the delay in replying, it took a while to check everything out.

I've added /tasks/:ID/wait in my shell script, but I still get this error when building in parallel on Jenkins:

[GIN] 2024/05/21 - 13:41:39 | 200 |   83.949261ms |      172.19.0.2 | GET      "/api/tasks/9/wait"
2024/05/21 13:41:39 Executing task asynchronously
[GIN] 2024/05/21 - 13:41:39 | 202 |    2.403509ms |      172.19.0.2 | PUT      "/api/publish/:./ubuntu-20.04?_async=true"
[GIN] 2024/05/21 - 13:41:39 | 200 |   87.470772ms |      172.19.0.2 | GET      "/api/tasks/10/wait"
[GIN] 2024/05/21 - 13:41:47 | 200 |  310.719834ms |      172.19.0.2 | POST     "/api/files/uploaded-files?_async=true"
2024/05/21 13:41:47 Executing task asynchronously
[GIN] 2024/05/21 - 13:41:47 | 202 |      115.25µs |      172.19.0.2 | POST     "/api/repos/libs/file/uploaded-files/libs-generator.87.6316_amd64.deb?_async=true"
[GIN] 2024/05/21 - 13:41:47 | 200 |  307.632802ms |      172.19.0.2 | POST     "/api/files/uploaded-files?_async=true"
2024/05/21 13:41:47 Executing task asynchronously
[GIN] 2024/05/21 - 13:41:47 | 409 |     116.276µs |      172.19.0.2 | POST     "/api/repos/libs/file/uploaded-files/libs-generator.87.6317_amd64.deb?_async=true"
Error #01: Needed resources are used by other tasks.

Aptly version 1.5.0+ds1-1+b4 .

This is stdout of shell script with set -x:

+ echo 'Moving deb package to repository...'
Moving deb package to repository...
++ curl -f -X POST 'http://repos.local/deb/api/repos/libs/file/uploaded-files//libs-generator.87.6317_amd64.deb?_async=true' -u repo-api:aGF#EQLr
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0    53    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (22) The requested URL returned error: 409

And do I understand correctly that I can't get ID and STATE on this request?

+ echo 'Uploading deb package: libsdeb///libs-generator.87.6317_amd64.deb'
Uploading deb package: libsdeb///libs-generator.87.6317_amd64.deb
++ curl -s -f -X POST -F file=@libsdeb///libs-generator.87.6317_amd64.deb 'http://repos.local/deb/api/files/uploaded-files?_async=true'
+ upload_response='["uploaded-files//libs-generator.87.6317_amd64.deb"]'
+ '[' 0 -ne 0 ']'
+ echo '["uploaded-files//libs-generator.87.6317_amd64.deb"]'
+ json_pp
[
   "uploaded-files//libs-generator.87.6317_amd64.deb"
]

And is it correct that on GET /tasks/:ID/wait I should get STATE=2? Is there any actual documentation on this function somewhere?

neolynx commented 3 months ago

Hi !

I looked in the problem and I think aptly does not support parallel operation, i.e. different clients sending requests at the same time. For this reason in the molior project there is a queue before all aptly operations, making in work with parallel request. The background API also does not solve the problem, is is merely allowing to do other things while waiting for aptly to complete, the requests still must be sent sequentially.

There are ideas to introduce a mutex to wait until the resources are available, but I am not sure this is safe and requests will be executed in order (imagine creating, deleting and recreating same repo. those should be executed in order as they arrive). So probably aptly should queue requests and process them in order, which whould be a bigger implementation.

In the meanwhile, I there a way to wait for all jenkins tasks to complete, and then do all aptly operations in one go, not using async ?

neolynx commented 3 months ago

I implemented a queue in https://github.com/aptly-dev/aptly/pull/1271, this should allow to publish from different clients instead of returning 409.

Something like this should work for all commands which support async:

task_id=`curl -f -X PUT http://localhost:3142/api/publish/bookworm_12.5-archs-copy_repos_test_1.0-copy2/stable?_async=true -H 'Content-Type: application/json' --data '{"Signing":{"Batch":true,"GpgKey":"reposign@molior", "PassphraseFile": "/var/lib/aptly/gpg-pass"}}'  | jq .ID`

curl http://repos.local/api/tasks/$task_id/wait

Could you give this a try?

Regarding your questions:

The question now is, how to deal with non async requests. Maybe they should go through the queue as well and additionally just wait for the task to finish before returning 200 ?

neolynx commented 3 months ago

The normal non-async requests are not also queued in https://github.com/aptly-dev/aptly/pull/1271, and also file uploads should work concurrently now.

Could you please give it a try?

ishaulov commented 2 weeks ago

@neolynx Hi! Sorry for the wait and thank you so much for fix! Builds have been falling less frequently.

I checked parallel builds with aptly version: 1.5.0+207+g372ce3c4. Error Needed resources are used by other tasks is no longer present

I keep getting an error when doing parallel builds on Jenkins with new error: Error #01: open /var/www/html/devrepo/deb/upload/uploaded-files/lib_fstyle_4.4.3.7368_amd64.deb: no such file or directory

Maybe, I must open new issue?

Jenkins log from the broken job:

13:29:29 + echo '++ curl -f -X POST -F file=@/home/deb//lib_fstyle_4.4.3.7368_amd64.deb '\''http://192.168.100.150/deb/api/files/uploaded-files?_async=true'\'' -u api:pass
13:29:29 ++ json_pp
13:29:29   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
13:29:29                                  Dload  Upload   Total   Spent    Left  Speed
13:29:29 
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
 11  813M    0     0   11 92.6M      0  95.9M  0:00:08 --:--:--  0:00:08 95.9M
 23  813M    0     0   23  192M      0  97.8M  0:00:08  0:00:01  0:00:07 97.8M
 35  813M    0     0   35  290M      0  98.0M  0:00:08  0:00:02  0:00:06 98.0M
 47  813M    0     0   47  389M      0  98.1M  0:00:08  0:00:03  0:00:05 98.1M
 58  813M    0     0   58  478M      0  96.2M  0:00:08  0:00:04  0:00:04 96.2M
 71  813M    0     0   71  578M      0  96.9M  0:00:08  0:00:05  0:00:03 97.1M
 83  813M    0     0   83  679M      0  97.5M  0:00:08  0:00:06  0:00:02 97.4M
 95  813M    0     0   95  780M      0  98.0M  0:00:08  0:00:07  0:00:01 98.0M
100  813M    0     0  100  813M      0  87.5M  0:00:09  0:00:09 --:--:-- 79.6M
100  813M    0     0  100  813M      0  79.2M  0:00:10  0:00:10 --:--:-- 63.2M
13:29:29 curl: (22) The requested URL returned error: 500 Internal Server Error

aptly json log:

    {
      "level": "info",
      "remote": "192.168.100.113",
      "method": "POST",
      "path": "/api/files/uploaded-files?_async=true",
      "protocol": "HTTP/1.0",
      "code": "200",
      "latency": "26.75636ms",
      "agent": "curl/7.58.0",
      "time": "2024-09-14T13:32:04+05:00"
    },
    {
      "level": "debug",
      "time": "2024-09-14T13:32:04+05:00",
      "message": "Executing task asynchronously"
    },
    {
      "level": "info",
      "remote": "192.168.100.113",
      "method": "POST",
      "path": "/api/repos/devrepo/file/uploaded-files/lib_masterstyle_4.4.3.7369_amd64.deb?_async=true",
      "protocol": "HTTP/1.0",
      "code": "202",
      "latency": "232.443µs",
      "agent": "curl/7.58.0",
      "time": "2024-09-14T13:32:04+05:00"
    },
    {
      "level": "debug",
      "time": "2024-09-14T13:32:04+05:00",
      "message": "Executing task asynchronously"
    },
    {
      "level": "info",
      "remote": "192.168.100.113",
      "method": "PUT",
      "path": "/api/publish/:./devrepo?_async=true",
      "protocol": "HTTP/1.0",
      "code": "202",
      "latency": "5.076542ms",
      "agent": "curl/7.58.0",
      "time": "2024-09-14T13:32:04+05:00"
    },
    {
      "level": "debug",
      "time": "2024-09-14T13:32:04+05:00",
      "message": "Executing task asynchronously"
    },
    {
      "level": "info",
      "remote": "192.168.100.113",
      "method": "PUT",
      "path": "/api/publish/:./ubuntu-20.04?_async=true",
      "protocol": "HTTP/1.0",
      "code": "202",
      "latency": "4.331976ms",
      "agent": "curl/7.58.0",
      "time": "2024-09-14T13:32:04+05:00"
    },
    {
      "level": "debug",
      "time": "2024-09-14T13:32:04+05:00",
      "message": "Executing task asynchronously"
    },
    {
      "level": "info",
      "remote": "192.168.100.113",
      "method": "PUT",
      "path": "/api/publish/:./ubuntu-22?_async=true",
      "protocol": "HTTP/1.0",
      "code": "202",
      "latency": "4.326821ms",
      "agent": "curl/7.58.0",
      "time": "2024-09-14T13:32:04+05:00"
    },
    {
      "level": "debug",
      "time": "2024-09-14T13:32:04+05:00",
      "message": "Executing task asynchronously"
    },
    {
      "level": "info",
      "remote": "192.168.100.113",
      "method": "PUT",
      "path": "/api/publish/:./ubuntu-24?_async=true",
      "protocol": "HTTP/1.0",
      "code": "202",
      "latency": "4.303508ms",
      "agent": "curl/7.58.0",
      "time": "2024-09-14T13:32:04+05:00"
    },
    {
      "level": "debug",
      "time": "2024-09-14T13:32:04+05:00",
      "message": "Executing task asynchronously"
    },
    {
      "level": "info",
      "remote": "192.168.100.113",
      "method": "PUT",
      "path": "/api/publish/:./astra-ce-2.12?_async=true",
      "protocol": "HTTP/1.0",
      "code": "202",
      "latency": "4.404118ms",
      "agent": "curl/7.58.0",
      "time": "2024-09-14T13:32:04+05:00"
    },
    {
      "level": "error",
      "remote": "192.168.100.113",
      "method": "POST",
      "path": "/api/files/uploaded-files?_async=true",
      "protocol": "HTTP/1.0",
      "code": "500",
      "latency": "1.978734261s",
      "agent": "curl/7.58.0",
      "time": "2024-09-14T13:32:04+05:00",
      "message": "Error #01: open /var/www/html/devrepo/deb/upload/uploaded-files/lib_fstyle_4.4.3.7368_amd64.deb: no such file or directory"
    },
    {
      "level": "debug",
      "time": "2024-09-14T13:32:04+05:00",
      "message": "Executing task asynchronously"
    },
    {
      "level": "info",
      "remote": "192.168.100.113",
      "method": "POST",
      "path": "/api/repos/devrepo/file/uploaded-files/lib_fstyle_4.4.3.7368_amd64.deb?_async=true",
      "protocol": "HTTP/1.0",
      "code": "202",
      "latency": "229.132µs",
      "agent": "curl/7.58.0",
      "time": "2024-09-14T13:32:04+05:00"
    },
    {
      "level": "debug",
      "time": "2024-09-14T13:32:04+05:00",
      "message": "Executing task asynchronously"
    },
    {
      "level": "info",
      "remote": "192.168.100.113",
      "method": "PUT",
      "path": "/api/publish/:./devrepo?_async=true",
      "protocol": "HTTP/1.0",
      "code": "202",
      "latency": "21.189441ms",
      "agent": "curl/7.58.0",
      "time": "2024-09-14T13:32:04+05:00"
    },
    {
      "level": "debug",
      "time": "2024-09-14T13:32:04+05:00",
      "message": "Executing task asynchronously"
    },
    {
      "level": "info",
      "remote": "192.168.100.113",
      "method": "PUT",
      "path": "/api/publish/:./ubuntu-20.04?_async=true",
      "protocol": "HTTP/1.0",
      "code": "202",
      "latency": "4.111851ms",
      "agent": "curl/7.58.0",
      "time": "2024-09-14T13:32:04+05:00"
    },
    {
      "level": "debug",
      "time": "2024-09-14T13:32:04+05:00",
      "message": "Executing task asynchronously"
    },
    {
      "level": "info",
      "remote": "192.168.100.113",
      "method": "PUT",
      "path": "/api/publish/:./ubuntu-22?_async=true",
      "protocol": "HTTP/1.0",
      "code": "202",
      "latency": "3.787162ms",
      "agent": "curl/7.58.0",
      "time": "2024-09-14T13:32:04+05:00"
    },
    {
      "level": "debug",
      "time": "2024-09-14T13:32:04+05:00",
      "message": "Executing task asynchronously"
    },
    {
      "level": "info",
      "remote": "192.168.100.113",
      "method": "PUT",
      "path": "/api/publish/:./ubuntu-24?_async=true",
      "protocol": "HTTP/1.0",
      "code": "202",
      "latency": "3.901205ms",
      "agent": "curl/7.58.0",
      "time": "2024-09-14T13:32:04+05:00"
    },
    {
      "level": "debug",
      "time": "2024-09-14T13:32:04+05:00",
      "message": "Executing task asynchronously"
    },
    {
      "level": "info",
      "remote": "192.168.100.113",
      "method": "PUT",
      "path": "/api/publish/:./astra-ce-2.12?_async=true",
      "protocol": "HTTP/1.0",
      "code": "202",
      "latency": "3.703295ms",
      "agent": "curl/7.58.0",
      "time": "2024-09-14T13:32:04+05:00"
    }

And I want to clarify: when I query curl curl -f -X POST -F file=@/home/deb/lib_mastertest_amd64.deb 'http://192.168.100.150/deb/api/files/uploaded-files?_async=true -u api:pass

I get json

[
    "uploaded-files/lib_mastertest_amd64.deb"
]

Do I understand correctly that this query does not need to reverse ?_async=true, since I can't get the state?

neolynx commented 2 weeks ago

if you are using parallel uploads / add files, make sure a unique upload directory is used:

folder=`mktemp -u tmp.XXXXXXXXXXXXXXX`
curl -f -X POST -F file=@/home/deb/lib_mastertest_amd64.deb 'http://192.168.100.150/deb/api/files/$folder -u api:pass
...

if the same folder is used from multiple sources, you might get the no such file or directory error message because when adding the uploaded files to a repo, the upload folder is deleted.

the background operation via ?_async=true is only implemented for a few API calls which take a long time, in order to avoid API timeouts. so for uploading a file, the call will block anyway until the file is uploaded.

for the subsequent publishing of the repo, you could use the async operating and poll the corresponsing task until successful (State: 2).

ishaulov commented 2 weeks ago

@neolynx thanks! 😃

The doc says that the directory will be created if it doesn't exist, and a successfully downloaded package will be deleted. So the directory will be deleted too.... Got it. I'll check it out right now.

That is, in parallel builds, when one has already uploaded the package and is publishing, the file and directory are deleted. At the same time a file from another build was added to this directory...

ishaulov commented 2 weeks ago

@neolynx it's worked! Your support and advice have been incredible helpful 😃