List uploading stop working

[LexterS999]

Here is a log:

Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "/home/runner/work/Cloudflare-Gateway-Pihole/Cloudflare-Gateway-Pihole/src/main.py", line 148, in asyncio.run(main()) File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/asyncio/runners.py", line 190, in run return runner.run(main) ^^^^^^^^^^^^^^^^ File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/asyncio/runners.py", line 118, in run return self._loop.run_until_complete(task) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete return future.result() ^^^^^^^^^^^^^^^ File "/home/runner/work/Cloudflare-Gateway-Pihole/Cloudflare-Gateway-Pihole/src/main.py", line 145, in main await cloudflaremanager.run() File "/home/runner/work/Cloudflare-Gateway-Pihole/Cloudflare-Gateway-Pihole/src/main.py", line 98, in run cf_lists = await asyncio.gather(create_list_tasks) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/runner/work/Cloudflare-Gateway-Pihole/Cloudflare-Gateway-Pihole/src/cloudflare.py", line 14, in wrapper return await func(args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/runner/work/Cloudflare-Gateway-Pihole/Cloudflare-Gateway-Pihole/src/cloudflare.py", line 43, in create_list raise Exception("Failed to create Cloudflare list") Exception: Failed to create Cloudflare list Error: Process completed with exit code 1.

And this error happens almost for all lists in domains / host format. But yesterday all works fine.

[misanthroope]

same here , i delete the repository and fork a new one but it's the same!

[nhathoang24]

same here !

[misanthroope]

do you guys know if there's a good easy one on GitHub with auto refresh list? i found one but it's manual list

[n0w33d]

I have this same issue. Not only did my existing block lists ALL get deleted, I now cannot upload new lists. No idea how to fix this.

[luxysiv]

Some time python works fail. So I make retry for that issue

[luxysiv]

Oh, I was check. Problem belonging to Cloudflare server.

[n0w33d]

Do we simply wait and try running the python script at a later time? Is that the solution you are implying?

[luxysiv]

Do we simply wait and try running the python script at a later time? Is that the solution you are implying?

Sometimes Cloudflare Gateway Zero Trust does not respond causing script errors. So I let it run 3 times. I just tested the script that runs on the requests library. It also fails like aiohttp

[n0w33d]

My error is similar to OP, but not the same (OP error shows it fails to create the list). Mine fails to delete lists. See below

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/data/data/com.termux/files/home/Cloudflare-Gateway-Pihole/src/__main__.py", line 148, in <module>
    asyncio.run(main())
  File "/data/data/com.termux/files/usr/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/data/data/com.termux/files/usr/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/data/com.termux/files/usr/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/data/data/com.termux/files/home/Cloudflare-Gateway-Pihole/src/__main__.py", line 145, in main
    await cloudflaremanager.run()
  File "/data/data/com.termux/files/home/Cloudflare-Gateway-Pihole/src/__main__.py", line 89, in run
    await asyncio.gather(*delete_list_tasks)
  File "/data/data/com.termux/files/home/Cloudflare-Gateway-Pihole/src/cloudflare.py", line 14, in wrapper
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/data/com.termux/files/home/Cloudflare-Gateway-Pihole/src/cloudflare.py", line 54, in delete_list
    raise Exception("Failed to delete Cloudflare list")
Exception: Failed to delete Cloudflare list

It likely failed because it already deleted them but then was unable to complete due to Cloudflare Zero Trust issues you mentioned.

How can we all help you confirm the issue is not our side and actually Cloudflares fault?

[luxysiv]

In this case the error is the same as many months ago. Yesterday I tried to access the Cloudflare Workers Pages but kept getting errors

[luxysiv]

[LexterS999]

Seems like Cloudflare changed something on worker page side.

[luxysiv]

I think it's a problem CF API. If the list < 70 , it works fine

[LexterS999]

Thought the same before and even changed CF API, but still the same.

[luxysiv]

I think lists on CF Zero Trust less 65

[luxysiv]

It is very possible that Cloudflare rejects asyncio requests. If next week the situation is still the same, I will publish the code using request and patch the lists. It's better that when we update the lists the blocking ability is still valid

[luxysiv]

Now you can visit https://github.com/luxysiv/Cloudflare-Gateway-Package Choose mode patch - run and total domains ~ 70k

[n0w33d]

Now you can visit https://github.com/luxysiv/Cloudflare-Gateway-Package Choose mode patch - run and total domains ~ 70k

The instructions say: "Set Variables, Please read Cloudflare-Gateway-Pihole", but the link provided does not mention how to do this.

Also, the instructions mention: "So you can create and paste this code to run on Cloudflare Workers.", but I do not know how to do this. What do you recommend I read or watch to understand how to do this?

[luxysiv]

Now you can visit https://github.com/luxysiv/Cloudflare-Gateway-Package Choose mode patch - run and total domains ~ 70k

The instructions say: "Set Variables, Please read Cloudflare-Gateway-Pihole", but the link provided does not mention how to do this.

Also, the instructions mention: "So you can create and paste this code to run on Cloudflare Workers.", but I do not know how to do this. What do you recommend I read or watch to understand how to do this?

The same like CGP

[n0w33d]

The same like CGP

Okay, apologies if I sound stupid. But you have Cloudflare Gateway PiHole, and also Cloudflare Gateway Package. Which one are you effing talking about. Can you please be more detailed?

[luxysiv]

Package is Cloudflare Gateway Pihole add python environment and modules

[n0w33d]

Package is Cloudflare Gateway Pihole add python environment and modules

I apologize again for being dumb, but your replies continue to be cryptic. Do you mean we add the Python files to our existing local git package and run it in Termux?

[luxysiv]

Package is Cloudflare Gateway Pihole add python environment and modules

I apologize again for being dumb, but your replies continue to be cryptic. Do you mean we add the Python files to our existing local git package and run it in Termux? Maybe. I not try do it on Termux. But I think It can

[n0w33d]

If I run python -m Cloudflare-Gateway-Pihole the error results with the console printing No module named Cloudflare-Gateway-Pihole.

If I rename the extension by adding a ".py" and try running again, it says

Traceback (most recent call last):
  File "<frozen runpy>", line 189, in _run_module_as_main
  File "<frozen runpy>", line 112, in _get_module_details
SyntaxError: source code string cannot contain null bytes

[luxysiv]

chmod +x Cloudflare-Gateway-Pihole && ./Cloudflare-Gateway-Pihole leave chmod +x Cloudflare-Gateway-Pihole && ./Cloudflare-Gateway-Pihole run

[luxysiv]

Do not use Cloudflare-Gateway-Pihole, Pls use Cloudflare-Gateway-Pyhole

[n0w33d]

Do not use Cloudflare-Gateway-Pihole, Pls use Cloudflare-Gateway-Pyhole

That is so confusing.

Well I've done a git pull on both repositories you have, copied both files over ([...]Pyhole and [...]Pihole) to one directory above "src" and tried to run both with chmod 777 properties, and the error is similar for both files: bash: ./Cloudflare-Gateway-Pyhole: cannot execute binary file: Exec format error

However running python -m src still works but always fails with the same error OP posted as the first message of this discussion.

[luxysiv]

Ok, I try to sleep 2s before update or creat list. Maybe Cloudflare blocks sending requests to the system too quickly

[luxysiv]

This code no care about Github Action run. It patches lists. And while patching lists, block ads still vallid

[luxysiv]

I think true. Script works well when sleep 2s before send next request

[luxysiv]

Please wait, I will update script soon

[n0w33d]

This code no care about Github Action run. It patches lists. And while patching lists, block ads still vallid

I do not know how to use Clouflare Workers and for you to teach me how to do it is out of your scope.

So the Termux instructions, which for some reason you removed from the main Cloudflare-Gateway-Pihole repo, works for me running in Termux on my phone which has more than enough performance to run your script.

We will wait for you to update the repo and try again and report here to you any changes

[luxysiv]

This code no care about Github Action run. It patches lists. And while patching lists, block ads still vallid

I do not know how to use Clouflare Workers and for you to teach me how to do it is out of your scope.

So the Termux instructions, which for some reason you removed from the main Cloudflare-Gateway-Pihole repo, works for me running in Termux on my phone which has more than enough performance to run your script.

We will wait for you to update the repo and try again and report here to you any changes

Because last year, I tried to install aiohttp on Termux but not success. New update will use requests, it can install on termux fine. So you can run on Termux. But this script will take 4 minutes to run script or more. So Github Action is best. Pls wait, I go to sleep. Good night

[luxysiv]

Script is back in operation. This time the mechanism works to patch lists if they already exist, delete or create the necessary lists. Script will operate at a slow speed. When Script is active the blocking effect is still available. Please delete old lists manually (clean any lists) so that the Script works correctly. Thank you.

[LexterS999]

No. It still doesn't work. I made sync fork and error still happening even with new lists. All firewall rules were cleared.

[LexterS999]

Run python -m src 2024-06-07 08:10:18.452 | INFO | src:info:50 - Downloaded file from https://raw.githubusercontent.com/SystemJargon/filters/main/ads.txt File size: 5364997 2024-06-07 08:10:18.542 | INFO | src:info:50 - Downloaded file from https://raw.githubusercontent.com/SystemJargon/filters/main/telemetry.txt File size: 77051 2024-06-07 08:10:18.961 | INFO | src:info:50 - Downloaded file from https://raw.githubusercontent.com/missdeer/blocklist/master/convert/alldomains.txt File size: 8089224 2024-06-07 08:10:18.968 | INFO | src:info:50 - Number of whitelisted domains: 0 2024-06-07 08:10:20.773 | INFO | src:info:50 - Number of blocked domains: 296212 2024-06-07 08:10:20.883 | INFO | src:info:50 - Number of final domains: 296212 2024-06-07 08:10:24.282 | ERROR | src:error:43 - Failed to get current lists after retries Error: Process completed with exit code 1.

[luxysiv]

It works fine. Maybe you not yet add variables

[LexterS999]

If you mean CF and API tokens, it's added already. I even start from the beginning, but still the same.

[luxysiv]

Please see my Github Action, it works fine

[luxysiv]

@asterriya clean your lists on CF

[LexterS999]

@asterriya clean your lists on CF

Yes. I did that already. It's clear. Also, I forked repository from the scratch. And followed the same instructions. But still the same error.

[luxysiv]

@asterriya Update the script and try running again. If you haven’t succeeded then run again. The code will run successfully. See the log here https://github.com/luxysiv/Cloudflare-Gateway-Pihole/actions/runs/9418540542/job/25946246736

[luxysiv]

@misanthroope @n0w33d @nhathoang24 update code

[LexterS999]

@luxysiv I synced fork, and it goes now for infinite loop. 6 minutes passed, but it's still showing update rules and passed over 200 rules. Which isn't real. I even try to create from the start repo and still the same infinite loop.

[suricloud]

Please alspo update the cf gateway package with these latest update.

[LexterS999]

Please alspo update the cf gateway package with these latest update.

what do you mean?

[suricloud]

Please alspo update the cf gateway package with these latest update.

what do you mean? sorry this is for @luxysiv

[luxysiv]

@luxysiv I synced fork, and it goes now for infinite loop. 6 minutes passed, but it's still showing update rules and passed over 200 rules. Which isn't real. I even try to create from the start repo and still the same infinite loop.

First you can run delete CF lists. This update I was check all today. It works very well

[luxysiv]

Please alspo update the cf gateway package with these latest update.

Done

[suricloud]

Please alspo update the cf gateway package with these latest update.

Done

Thanks :)