search

boredazfcuk / docker-icloudpd

An Alpine Linux container for the iCloud Photos Downloader command line utility
1.58k stars 149 forks source link

Retry loop with failure to find icloud.com IP address #470

Closed phurth closed 2 months ago

phurth commented 6 months ago

I just updated the container to the latest version and since then have been experiencing a retry/fail loop:

2024/01/01 15:21:13 | stdout | 2024-01-01 15:21:13 WARNING  Cannot find icloud.com IP address - retrying
-- | -- | --
2024/01/01 15:20:49 | stdout | 2024-01-01 15:20:49 INFO     Loading configuration from: /config/icloudpd.conf
2024/01/01 15:20:49 | stdout | 2024-01-01 15:20:49 INFO     Python version: 3.11.6
2024/01/01 15:20:49 | stdout | 2024-01-01 15:20:49 INFO     Alpine Linux 3.19.0
2024/01/01 15:20:49 | stdout | 2024-01-01 15:20:49 INFO     ***** For support, please go here: https://github.com/boredazfcuk/docker-icloudpd *****
2024/01/01 15:20:49 | stdout | 2024-01-01 15:20:49 INFO     ***** boredazfcuk/icloudpd container for icloud_photo_downloader v1.0.705 started *****
2024/01/01 15:20:49 | stdout |  
2024/01/01 15:18:42 | stdout | 2024-01-01 15:18:42 ERROR    Cannot find icloud.com IP address. Please check your DNS/Firewall settings. DNS server must be available using TCP port 53 - exiting
2024/01/01 15:11:53 | stdout | 2024-01-01 15:11:53 WARNING  Cannot find icloud.com IP address - retrying
2024/01/01 15:11:29 | stdout | 2024-01-01 15:11:29 INFO     Loading configuration from: /config/icloudpd.conf
2024/01/01 15:11:29 | stdout | 2024-01-01 15:11:29 INFO     Python version: 3.11.6
2024/01/01 15:11:29 | stdout | 2024-01-01 15:11:29 INFO     Alpine Linux 3.19.0
2024/01/01 15:11:29 | stdout | 2024-01-01 15:11:29 INFO     ***** For support, please go here: https://github.com/boredazfcuk/docker-icloudpd *****
2024/01/01 15:11:29 | stdout | 2024-01-01 15:11:29 INFO     ***** boredazfcuk/icloudpd container for icloud_photo_downloader v1.0.705 started *****

Looking at the change log, there was a change made with DNS lookups. I can access icloud.com outside of the container and this was working prior to the container update.

khargy commented 6 months ago

Started have the same issue with v1.0.705. (v1.0.696 works as expected)

boredazfcuk commented 6 months ago

This change in behaviour was due to an issue identified a couple of days ago. The error message should contain details of the new requirements for the container.

khargy commented 6 months ago

it says:

2024-01-01 19:18:18 INFO boredazfcuk/icloudpd container for icloud_photo_downloader v1.0.705 started 2024-01-01 19:18:18 INFO For support, please go here: https://github.com/boredazfcuk/docker-icloudpd 2024-01-01 19:18:18 INFO Alpine Linux 3.19.0 2024-01-01 19:18:18 INFO Python version: 3.11.6 2024-01-01 19:18:18 INFO Loading configuration from: /config/icloudpd.conf 2024-01-01 19:18:18 WARNING Cannot find icloud.com IP address - retrying 2024-01-01 19:20:19 ERROR Cannot find icloud.com IP address. Please check your DNS/Firewall settings. DNS server must be available using TCP port 53 - exiting

It's not clear what needs to be updated/changed

phurth commented 6 months ago

The error in the log indicates that DNS on TCP port 53 is unreachable. My icloudpd container is running on the host network and should have no trouble reaching DNS.

Edit: running nslookup in the container's shell does resolve icloud.com, si I'm not sure what I'm supposed to change.

rogerc66 commented 6 months ago

I'm having the exact issue as well, since the update on 1/1/2024

kfstorm commented 6 months ago

I'm having the same issue, but with icloud.com.cn. It seems that TCP 53 doesn't work

Container logs:

024-01-03 11:55:19 INFO     ***** boredazfcuk/icloudpd container for icloud_photo_downloader v1.0.705 started *****
2024-01-03 11:55:19 INFO     ***** For support, please go here: https://github.com/boredazfcuk/docker-icloudpd *****
2024-01-03 11:55:19 INFO     Alpine Linux 3.19.0
2024-01-03 11:55:19 INFO     Python version: 3.11.6
2024-01-03 11:55:20 INFO     Loading configuration from: /config/icloudpd.conf
2024-01-03 11:55:44 WARNING  Cannot find icloud.com.cn IP address - retrying
2024-01-03 12:02:33 ERROR    Cannot find icloud.com.cn IP address. Please check your DNS/Firewall settings. DNS server must be available using TCP port 53 - exiting

Dig in the existing container shows that TCP 53 doesn't work:

docker exec -it icloudpd-kai sh
/ # dig icloud.com.cn

; <<>> DiG 9.18.19 <<>> icloud.com.cn
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 33522
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; MBZ: 0x0001, udp: 1232
; COOKIE: 0e0d343dc5b07bf9 (echoed)
;; QUESTION SECTION:
;icloud.com.cn.         IN  A

;; ANSWER SECTION:
icloud.com.cn.      1   IN  A   198.18.15.31

;; Query time: 0 msec
;; SERVER: 127.0.0.11#53(127.0.0.11) (UDP)
;; WHEN: Wed Jan 03 12:11:50 CST 2024
;; MSG SIZE  rcvd: 70

/ # dig +tcp icloud.com.cn
;; communications error to 127.0.0.11#53: end of file
;; communications error to 127.0.0.11#53: end of file
;; communications error to 127.0.0.11#53: end of file

; <<>> DiG 9.18.19 <<>> +tcp icloud.com.cn
;; global options: +cmd
;; no servers could be reached

But if I start a new container, TCP 53 works:

docker run --rm -it --entrypoint /bin/sh boredazfcuk/icloudpd
/ # dig icloud.com.cn

; <<>> DiG 9.18.19 <<>> icloud.com.cn
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 20558
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; MBZ: 0x0001, udp: 1232
; COOKIE: c9ad543497bf055c (echoed)
;; QUESTION SECTION:
;icloud.com.cn.         IN  A

;; ANSWER SECTION:
icloud.com.cn.      1   IN  A   198.18.15.31

;; Query time: 0 msec
;; SERVER: 8.8.8.8#53(8.8.8.8) (UDP)
;; WHEN: Wed Jan 03 15:12:46 UTC 2024
;; MSG SIZE  rcvd: 70

/ # dig +tcp icloud.com.cn

; <<>> DiG 9.18.19 <<>> +tcp icloud.com.cn
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 37470
;; flags: qr aa rd ra ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; MBZ: 0x0001, udp: 1232
; COOKIE: f9d3fe6d8842e24b (echoed)
;; QUESTION SECTION:
;icloud.com.cn.         IN  A

;; ANSWER SECTION:
icloud.com.cn.      1   IN  A   198.18.15.31

;; Query time: 0 msec
;; SERVER: 8.8.8.8#53(8.8.8.8) (TCP)
;; WHEN: Wed Jan 03 15:12:51 UTC 2024
;; MSG SIZE  rcvd: 83

The existing container runs in a docker-compose-created bridge network:

        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "9f7c50fd32c113e636a807c91457e9eebe7ebd755e7bc3c1c152d9c1a3e21483",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/var/run/docker/netns/9f7c50fd32c1",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "mediacenter_default": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": [
                        "icloudpd-kai",
                        "icloudpd-kai",
                        "1b5ea5bcdd02"
                    ],
                    "NetworkID": "401bea4f0eb7a2e7124ae7f5a8f61893c400ce1c155d743bf5475ef64917a573",
                    "EndpointID": "2d91b938c7db717a4b6536b7100ea1c8e853d7fb0edc3ac42138cc688ab883f9",
                    "Gateway": "172.20.0.1",
                    "IPAddress": "172.20.0.2",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:ac:14:00:02",
                    "DriverOpts": null
                }
            }
        }

The newly created container runs in the default bridge network:

        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "b716e59f8432c84a690dcd459388779441c31a21118f81e78299384645593ffe",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/var/run/docker/netns/b716e59f8432",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "b3e9c76c319d5a603c56fae22f8d3d9ac19ce2491925e790805687888d12420d",
            "Gateway": "172.17.0.1",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "172.17.0.3",
            "IPPrefixLen": 16,
            "IPv6Gateway": "",
            "MacAddress": "02:42:ac:11:00:03",
            "Networks": {
                "bridge": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "54dfe28a7ba53e526287b248337bc24e5ab5306adc83478084a460c6c9240899",
                    "EndpointID": "b3e9c76c319d5a603c56fae22f8d3d9ac19ce2491925e790805687888d12420d",
                    "Gateway": "172.17.0.1",
                    "IPAddress": "172.17.0.3",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:ac:11:00:03",
                    "DriverOpts": null
                }
            }
        }
khargy commented 6 months ago

I tried making a new docker and it doesn't seem to make a difference

rogerc66 commented 6 months ago

I'm having the following error:

nslookup: parse of /etc/resolv.conf failed

kfstorm commented 6 months ago

I managed to work around this issue by removing the -vc parameter of nslookup in ./usr/local/bin/sync-icloud.sh manually in my container.

wildenrou commented 6 months ago

having the same retry loop failure since 2024. Thanks in advance for your precious help

khargy commented 6 months ago

I managed to work around this issue by removing the -vc parameter of nslookup in ./usr/local/bin/sync-icloud.sh manually in my container.

That worked for me as well

phurth commented 6 months ago

If I understand the change implemented in 1.0.705, -vc was added to fix some other issue. Removing it undoes that fix and at any rate will be overwritten if the container is rebuilt. Is there a better fix than editing the script in the container?

kfstorm commented 6 months ago

If I understand the change implemented in 1.0.705, -vc was added to fix some other issue. Removing it undoes that fix and at any rate will be overwritten if the container is rebuilt. Is there a better fix than editing the script in the container?

I believe the issue -vc tries to fix is https://github.com/boredazfcuk/docker-icloudpd/issues/459.

I'd suggest adding an option to control whether to use -vc or not. For people who don't encounter issue #459, they can disable -vc.

myevit commented 6 months ago

Same issue. If I am running the container as user (kubernetes on TrueNas). if container runs as root - all works as intended

khargy commented 5 months ago

This change in behaviour was due to an issue identified a couple of days ago. The error message should contain details of the new requirements for the container.

Still getting this error in v1.0.716 I assume this isn't an error for all users, but I still haven't figured out a simple fix for it. I'm using Unraid for my docker setup. The only error msg I see is:

2024-01-23 19:29:39 WARNING Cannot find icloud.com IP address - retrying 2024-01-23 19:31:41 ERROR Cannot find icloud.com IP address. Please check your DNS/Firewall settings. DNS server must be available using TCP port 53 - exiting

Which doesn't help me as to how to fix it.

BallanceHZ commented 5 months ago

Still getting this error in v1.0.716

2024-01-27 22:43:29 INFO boredazfcuk/icloudpd container for icloud_photo_downloader v1.0.716 started 2024-01-27 22:43:29 INFO For support, please go here: https://github.com/boredazfcuk/docker-icloudpd 2024-01-27 22:43:29 INFO Alpine Linux 3.19.0 2024-01-27 22:43:29 INFO Python version: 3.11.6 2024-01-27 22:43:29 INFO Loading configuration from: /config/icloudpd.conf 2024-01-27 22:43:29 WARNING Cannot find icloud.com IP address - retrying 2024-01-27 22:45:30 ERROR Cannot find icloud.com IP address. Please check your DNS/Firewall settings. DNS server must be available using TCP port 53 - exiting

Suyknow commented 4 months ago

I managed to work around this issue by removing the -vc parameter of nslookup in ./usr/local/bin/sync-icloud.sh manually in my container.

Nice job!

boredazfcuk commented 4 months ago

I've amended the code slightly, that it will only use the -vc parameter for nslookup if notifications are enabled.

Notifications are sent using curl which requires tcp based name resolution to work. If you guys don't have notifications configured, then it should revert back to previous behaviour.

mikexfreeze commented 4 months ago

I've amended the code slightly, that it will only use the -vc parameter for nslookup if notifications are enabled.

Notifications are sent using curl which requires tcp based name resolution to work. If you guys don't have notifications configured, then it should revert back to previous behaviour.

Not quite understand what is notifications configure. And how to revert back to previous behaviour. I tried to remove -vc parameter and created a new image to run, still don't work.

Suyknow commented 4 months ago

I've amended the code slightly, that it will only use the -vc parameter for nslookup if notifications are enabled.

Notifications are sent using curl which requires tcp based name resolution to work. If you guys don't have notifications configured, then it should revert back to previous behaviour.

Thank you, I've enabled notifications and removed the -vc, and it still works fine (my notifications are sent via Telegram). As mentioned above, with -vc, iCloud.com cannot resolve IP.

(I didn't look carefully at the implementation logic of the entire code, and I'm not CS professional, so I apologize if the following question is too basic.) Additionally, there is another question: notifications should only connect to Telegram or other services, so why does this script for finding icloud_domain also need the -vc parameter? This part should be unrelated to notifications, right?

boredazfcuk commented 4 months ago

Additionally, there is another question: notifications should only connect to Telegram or other services, so why does this script for finding icloud_domain also need the -vc parameter? This part should be unrelated to notifications, right?

When curl performs name resolution, it is using TCP DNS requests. I have confirmed this by blocking TCP:53 on my container host which resulted in the curl requests to the notification servers hanging, because it cannot resolve the hostname. UDP:53 DNS requests were successful. This puts the container in the position that name resolution will work (via UDP:53) but curl requests will fail (TCP:53) and the container hangs when attempting to send the startup notification, or other notifications if it is disabled.

Adding the -vc parameter to the nslookup command when the notification_type variable is set, forces the icloud.com name resolution to be performed over TCP, pre-empting the curl problem before it happens.

Hope this helps.

boredazfcuk commented 2 months ago

I've backed this change out as it seems that it actually works OK for some users. I'm guessing there must be something else affecting whether these UDP lookups are successful or not. Possibly dependent on host platform.

mikexfreeze commented 2 months ago

I've backed this change out as it seems that it actually works OK for some users. I'm guessing there must be something else affecting whether these UDP lookups are successful or not. Possibly dependent on host platform.

For me it block my local Linux machine work but works in cloud VM.

boredazfcuk commented 2 months ago

For me it block my local Linux machine work but works in cloud VM.

What Linux is your local machine running? Debian, Ubuntu, Fedora? And what container platform is it running? Docker, Podman, Kubernetes?

I run Debian with Docker and can't replicate half the errors that are logged here. I'm guessing that certain platforms/configurations prevent TCP DNS requests and break the container. Others run containers with reduced permissions which again, breaks the container.

dotnjet commented 3 weeks ago

I had the same problem but then I realized, that I added more IPs to the GEO Blocking (not USA though) on the firewall. Does anyone know the IP Range which has to be allowed? After removing the extra GEO Blocks it worked again.

boredazfcuk commented 3 weeks ago

I ended up scrapping my GeoIP blocks after it started thinking I was in Denmark. It would bounce between there and my home country every update.

After reading up on how it works, it's only an approximate guess. The problem nowadays is that people like Apple own a lot of IP addresses and as their services are now elastic and defined by code. They can stand up a whole raft of cloud services in the morning in one country, to cope with increased demand, then when that demand has ceased, destroy all those services and stand them up in another country/continent lasted that day. It means IP addresses are no longer regionalised.

dotnjet commented 3 weeks ago

thank you @boredazfcuk What I don't understand is how an inbound block can interfere with the Docker image. From my perspective, it should simply be an outbound call to the icloud.com URL, initiating the traffic flow.

boredazfcuk commented 3 weeks ago

Yeah, that's the way it works if you have a connection tracking rule to allow incoming packets. This needs to be above the rule to drop based on IP match.

If you don't have that connection tracking rule, then the incoming responses to your requests will be dropped.

Edit: I still have the GeoIP rules in my firewall config script... Basically this:

iptables --insert INPUT --match conntrack --ctstate ESTABLISHED,RELATED --jump ACCEPT

Should put a rule at the top of your iptables which tracks your connections. For any connections that are established or related to your outgoing connections, they will be accepted.

dotnjet commented 3 weeks ago

Thanks again @boredazfcuk ! This must be it. It seems that my additional Firewall on my NAS is not smart enough :-(

Would you by chance know which Ports have to be allowed, then I could block the rest for extra security ;-)

Have a great day

boredazfcuk commented 3 weeks ago

It should just be port 443. The app in the container just emulates logging on to iCloud.com with a web browser, then downloading all the files. It's the same as what would happen if you were doing it from a computer using Firefox/Chromium/Safari.