🐛 Source Facebook Marketing: There have been too many calls to this ad-account.

[mihir-kanzariya]

Hey all, I am getting this error after configuring facebook marketing connectors.

Status:  400
  Response:
    {
      "error": {
        "message": "(#80004) There have been too many calls to this ad-account. Wait a bit and try again. For more info, please refer to https://developers.facebook.com/docs/graph-api/overview/rate-limiting.",
        "type": "OAuthException",
        "code": 80004,
        "error_subcode": 2446079,
        "fbtrace_id": "AFm-w0VSqFxoRyuKXJgg48H"
      }
    }

When I check it also says it also says Utilization is too high (118)%, pausing for 47 minutes .

As per @alafanechere, The connector already has a backoff logic that should avoid the error. Maybe there could be some backoff optimizations to be made.

You can find logs in the attachment. logs-1731-0.txt

[alafanechere]

To add more context the Slack discussion can be found here. From Facebook API's documentation, the rate limit is dynamically computed according to the number of ads on the user account. What they are calling the Business Use Case Rate Limits which is documented here. I don't think our connector is handling this dynamic rate limit and it might be the root cause of the error encountered by @mihir-kanzariya .

[sherifnada]

@mihir-kanzariya do you have the advanced access token tier enabled like described in this section?

[alafanechere]

@sherifnada I asked @mihir-kanzariya on Slack to upgrade his app to the advanced access token tier.

[misteryeo]

@alafanechere @mihir-kanzariya following up here to see if the advanced access token being enabled fixes this issue or it still persists?

[misteryeo]

Closing this issue since there's no response from @mihir-kanzariya - feel let us know if we need to re-open this.

[wub]

Hey @misteryeo, we're encountering this issue even with advanced access, so I imagine @alafanechere's hypothesis is correct re: BUC rate limits.

BUC rate limit for ads_insights: Calls within one hour = 60 + 400 * Number of Active ads - 0.001 * User Errors

[plorenzatto]

@wub @misteryeo We are experiencing the same issue even after upgrading our tokens for advanced access. We tried changing the pagination and other settings to no avail.

cc @lahgzz

Airbyte Version: 0.39.21-alpha Source: Facebook Marketing (0.2.63) Destination: Postgres (0.3.20) logs-1604.txt

[misteryeo]

Sorry for the delay here folks - @YowanR can you take a look at this with the team?

[YowanR]

@davydov-d @lazebnyi Adding this to our backlog for prioritization. Cc @YuliiaNahoha

[plorenzatto]

We had some progress by splitting the source in multiple ones each with campaigns from a different region. We suppose that this worked because of the lower number of concurrent requests which may be an issue when fetching a large number of records to fill a large lookup window. If that's the case I would suggest adding a way to control fetching concurrency or at least some indications on how to resolve the issue in the error message.

[tinomerl]

@plorenzatto what did you do for the screencast for the advanced access? Did you show your Airbyte Syncs and the dashboards for the data? And did you use the custom Insights for splitting the source up?

[midavadim]

This is should be fixed by this PR: https://github.com/airbytehq/airbyte/pull/29994

There we removed batch processing for a lot of streams