search

coollabsio / coolify

An open-source & self-hostable Heroku / Netlify / Vercel alternative.
https://coolify.io
Apache License 2.0
34.04k stars 1.84k forks source link

[Bug]: PullHelperImageJob failed with: Error response from daemon: toomanyrequests #3395

Open davidpp opened 1 month ago

davidpp commented 1 month ago

Description

After upgrading Coolify, I'm receiving repeated error messages in my Discord notif channel. These errors indicate issues with pulling Docker images due to rate limiting. The errors specifically mention the PullHelperImageJob failing due to too many requests.

Minimal Reproduction (if possible, example repository)

Can't tell exactly what is needed to get this error, but there are other mentions in Discord

Exception or Error

PullHelperImageJob failed with: Error response from daemon: toomanyrequests: retry-after: 244.706µs, allowed: 44000/minute PullHelperImageJob failed with: Error response from daemon: toomanyrequests: retry-after: 385.552µs, allowed: 44000/minute

Version

v4.0.0-beta.333

Cloud?

ohbob commented 1 month ago

haven't been upgrading for a while, yesterday did upgrade, now get the same issues as OP.

davidpp commented 1 month ago

Probably related/is #2930

peaklabs-dev commented 1 month ago

What we know so far: -> Rate limiting issue by ghrc.io -> Even though the rate limit is much higher than 1 request and hour: Anonymous users | 100 pulls per 6 hours per IP address

Reproduction:

  1. Configure any S3 storage
  2. Setup automatic backups to trigger every hour and upload to S3
  3. Database backups start failing with: 
    PullHelperImageJob failed with: Error response from daemon: toomanyrequests: retry-after: 244.706µs, allowed: 44000/minute
  4. Trigger a manual backup to fix this issue.

-> Unfortunately, I cannot reproduce the problem on my end.

heyJordanParker commented 1 month ago

Same issue on my end with automatic backups to Digital Ocean.

andrasbacsai commented 1 month ago

This will be fixed in the upcoming version in #3393, releasing soon.

truemiller commented 1 month ago

@andrasbacsai, still experiencing this in v4.0.0-beta.335 unfortunately

via Discord webhook image

peaklabs-dev commented 1 month ago

@truemiller Is it a backup job? If so, could you delete and recreate the backup job and create a new one?

graphiostudio commented 1 month ago

Same here, on 335 and still getting this. I will try what @peaklabs-dev suggested.

EDIT: Found which one, it's a coolify backup from settings, removed failed ones, did a manual backup and it worked, set to daily, I guess we'll see tomorrow if it works or not.

EDIT 2: Got another error but don't know where it originated from, I have to go one by one and find it. I wish Discord messages would be clear on which containers were backed up, and which ones failed.

liamwh commented 1 month ago

I am running v4.0.0-beta.335. Experiencing the same. +1 on better error messages here please. I'd like to know at least which backup job and which images. Links to the page where the failing stuff in the UI could be managed would be ideal.

alynaug commented 1 month ago

Same here. On 335 and getting spammed with 2 error messages every hour sharp, cannot find which ones though. image

graphiostudio commented 1 month ago

Again, on 335. I removed all the backups, created new ones, coolify database backups, and postgres backups... now I'm getting spammed with failed jobs every hour as well. but still don't know which ones are failing, no info on the discord message.

Xiloe commented 1 month ago

Same issue here on 335, I don't have any backup or S3 buckets at all, so I don't know where that could come from, is there some backup turned on by default ?

graphiostudio commented 1 month ago

I checked, and none of my backups are failing, but I keep getting this discord messages every hour. I don't know what else to do. So I just turned off the discord notifications for now.

Xiloe commented 1 month ago
image

changing the update check frequency from "0 " to "0 0 *" actually fixed the issue for me, not sure if it will reappear tomorrow at 12AM but I don't get it every hour anymore.

Edit: it did In fact send me the same error code over telegram at 12AM, so it comes from there, in my case at least.

thefeli73 commented 1 month ago

Same here. i dont have any database backups configured, so i agree with @Xiloe that i seems to come from the update checker (i have auto updates disabled).

liamwh commented 1 month ago

I believe there is some noise happening here.

I tried disabling the following Discord notification categories:

I also tried disabling auto update, but I still received the notifications.

I am now trying with disabling the Container Status Changes and if I still do then I will test again by disabling only the Application Deployments notification category. But my testing leads me to believe this notification is caused by either Container Status Changes or Application Deployments, not backups and not Coolify auto update checking.

andrasbacsai commented 1 month ago

I will add another potential fix in v340 (releasing in a few minutes).

Tbh, not sure why this rate limit occurs. Coolify does not do anything special, just tries to download an image from a Docker Registry. Not a rocket science.

Xiloe commented 1 month ago

Seems like v340 fixed it for me, changed the frequency back to hourly and it doesn't spam me anymore.

Nonolanlan1007 commented 1 month ago

Did not change anything for me in 340

husarcik commented 1 month ago

Issue persists on v4.0.0-beta.341 with daily backups. Manually backups work as expected, even if deployed immediately after failure.

Failed to pull helper image: toomanyrequests: retry-after: 505.387µs, allowed: 44000/minute

Cristy94 commented 1 month ago

I am encountering the same issue on v4.0.0-beta.355. My backups are failing, maybe because I have multiple databases set to daily:

 [07/10/2024 02:00]
DatabaseBackupJob failed with: Failed to pull helper image: Error response from daemon: toomanyrequests: retry-after: 179.767µs, allowed: 44000/minute

[07/10/2024 02:00]
Coolify: Database backup for coolify-db (db:coolify) with frequency of 0 0 * * * was FAILED.

Reason:
Failed to pull helper image: Error response from daemon: toomanyrequests: retry-after: 179.767µs, allowed: 44000/minute

Failed to pull helper image: Error response from daemon: toomanyrequests: retry-after: 179.767µs, allowed: 44000/minute

EDIT: I am not even sure why a database backup would require to pull an image.

pascalchristian commented 1 month ago

same issue on v4.0.0-beta.355

a-w-1806 commented 4 weeks ago

Same here on 355. A couple of weeks ago I start to notice some of my dbs are not backing up at all because my backup schedule settings just disappeared, and now seeing this. Not sure if they are related

graphiostudio commented 4 weeks ago

same here, 355, backups are still disappearing, and I keep getting the same error on discord.