Can not run action update filter list

[nightmare199x]

Hi, i have error and can not update filter after sync folk. Please help!

Run npm run cloudflare-create

cloudflare-create npm run cloudflare-create:list && npm run cloudflare-create:rule

cloudflare-create:list node cf_list_create.js

Processing allowlist.txt Processing blocklist.txt

Number of processed domains: 687961 Number of duplicate domains: 347126 Number of unnecessary domains: 113312 Number of allowed domains: 768 Number of blocked domains: 226755 Number of lists to be created: 227

Creating 227 lists for 226755 domains... Error occurred while creating lists - TypeError: Cannot use 'in' operator to search for 'errors' in undefined file:///home/runner/work/cloudflare-gateway-pihole-scripts/cloudflare-gateway-pihole-scripts/lib/helpers.js:129 throw new Error(HTTP error! Status: ${response.status} - ${ 'errors' in data ? data.errors[0].message : JSON.stringify(data) }); ^

TypeError: Cannot use 'in' operator to search for 'errors' in undefined at request (file:///home/runner/work/cloudflare-gateway-pihole-scripts/cloudflare-gateway-pihole-scripts/lib/helpers.js:129:79) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async Promise.all (index 172) at async createZeroTrustListsAtOnce (file:///home/runner/work/cloudflare-gateway-pihole-scripts/cloudflare-gateway-pihole-scripts/lib/api.js:77:5) at async file:///home/runner/work/cloudflare-gateway-pihole-scripts/cloudflare-gateway-pihole-scripts/cf_list_create.js:139:5

Node.js v20.14.0 Error: Process completed with exit code 1.

[arunsathiya]

I saw the same issue after https://github.com/mrrfv/cloudflare-gateway-pihole-scripts/commit/a5d3fdb4b4659f85355c43fe654acceb327fcd02, but the overall underlying issue remains this: https://github.com/mrrfv/cloudflare-gateway-pihole-scripts/issues/111.

[galpt]

@arunsathiya What about using the previous 3-second sleep (the example I wrote there https://github.com/mrrfv/cloudflare-gateway-pihole-scripts/issues/111)? I guess it'll work for you.

If you're getting rate limited, the best practice is to wait, not to keep sending more requests until they accept it. If they're detecting more requests coming from you after they placed the rate limit on you, and you're not slowing down, you're just giving them more reasons to keep the rate limit even longer.

They offer DDoS protection as their main product, and one of the core features is to prevent non-human requests from reaching their servers faster than what their servers can actually handle. This usually includes blocking requests until the sender of the requests is slowing everything down.

What I've noticed from the latest PR is that it's not slowing down until you're getting rate limited, which is already too late. You don't want to trigger the rate limit in the first place, so adding sleep() between requests should be the correct logic.

[galpt]

Someone might want to experiment with a logic where, let's say we're sending 3 requests to their API, and then sleep for 3 seconds, and sending 3 more requests again, and sleep again for 3 seconds.

So that way you're not waiting 3 seconds for every request, but for every 3 requests, which is probably faster and won't trigger the rate limit at all.

The above example is what I could think of right now. 3-sec sleep for every request feels slow but it's a guaranteed success, but there should be a better logic.

[erik-de-bont]

The 3 seconds solution (#111) works fine for me, and I think it's a better approach than the current implemented workaround (keep sending requests until it continuous). I prefer stability over speed in this case. It's updating at 03:00 AM automatically so I don't mind if it takes 30 minutes or longer, since I am not waiting for it.

[arunsathiya]

Confirm, the delayed retry approach (https://github.com/mrrfv/cloudflare-gateway-pihole-scripts/issues/111#issuecomment-2151733701, https://github.com/mrrfv/cloudflare-gateway-pihole-scripts/commit/a3ade648a49b5352a22f8e9069fe3b7596370645) works well with fast mode disabled.