Not able to bootstrap the container on macOS Ventura 13.5.2/13.6.0

[walnutist]

Description: I am trying to deploy the docker-icloudpd on a macOS server for downloading multiple users' photos timely. And the container launch failed as below:

docker run  -it\
        --privileged \
        --network bridge \
        --restart=always \
        --env user=hank \
        --env user_id=501 \
        --env group=staff \
        --env group_id=20 \
        --env force_gid=True \
        --env apple_id="<MY_ID>" \
        --env authentication_type=2FA \
        --env folder_structure={:%Y/%m} \
        --env auto_delete=True \
        --env synchronisation_interval=43200 \
        --env icloud_china=True \
        --env auth_china=True \
        --env TZ="Asia/Shanghai" \
        --volume $(pwd)/config:/config \
        --volume $(pwd)/photos:/home/root/iCloud \
        boredazfcuk/icloudpd

2023-09-22 11:49:45 INFO     ***** boredazfcuk/icloudpd container for icloud_photo_downloader v1.0.628 started *****
2023-09-22 11:49:45 INFO     ***** For support, please go here: https://github.com/boredazfcuk/docker-icloudpd *****
2023-09-22 11:49:45 INFO     Alpine Linux 3.18.2
2023-09-22 11:49:45 INFO     Python version: 3.11.5
2023-09-22 11:49:45 INFO     Loading configuration from: /config/icloudpd.conf
2023-09-22 11:50:13 INFO     Apple ID: <MYID>
2023-09-22 11:50:13 INFO     Authentication Type: MFA
2023-09-22 11:50:13 INFO     Cookie path: /config/<MYID>
2023-09-22 11:50:13 INFO     Cookie expiry notification period: 7
2023-09-22 11:50:13 INFO     Download destination directory: /home/<MYID>/iCloud
2023-09-22 11:50:13 INFO     Download directory does not exist.
2023-09-22 11:50:13 INFO     Creating /home/<MYID>/iCloud and configuring permissions.
2023-09-22 11:50:13 INFO     Folder structure: {:%Y/%m}
2023-09-22 11:50:13 INFO     Synchronisation interval: 43200
2023-09-22 11:50:13 INFO     Synchronisation delay (minutes): 0
2023-09-22 11:50:13 INFO     Set EXIF date/time: false
2023-09-22 11:50:13 INFO     Auto delete: true
2023-09-22 11:50:13 INFO     Delete after download: false
2023-09-22 11:50:13 INFO     Photo size: original
2023-09-22 11:50:13 INFO     Single pass mode: false
2023-09-22 11:50:13 INFO     Skip download check: false
2023-09-22 11:50:13 INFO     Skip live photos: false
2023-09-22 11:50:13 INFO     Number of most recently added photos to download: Download All Photos
2023-09-22 11:50:13 INFO     Downloading photos from album: Download All Photos
2023-09-22 11:50:13 INFO     Stop downloading when prexisiting files count is: Download All Photos
2023-09-22 11:50:13 INFO     Live photo size: original
2023-09-22 11:50:13 INFO     Skip videos: false
2023-09-22 11:50:13 INFO     Convert HEIC to JPEG: false
2023-09-22 11:50:13 INFO     Downloading from: icloud.com.cn
2023-09-22 11:50:13 INFO     Authentication domain: cn
2023-09-22 11:50:13 INFO     Ignore Synology extended attribute directories: Disabled
2023-09-22 11:50:13 ERROR    Group id, 20, already in use by the group: staff - exiting. If you must to add your user to this pre-existing system group, please set the force_gid variable to True

I don't understand why the script says the force_gid is False even it was already explicitly set to True

[walnutist]

If I change the command below, different exception throwed:

docker run -it\
    --privileged \
    --network bridge \
    --restart=always \
    --env apple_id="<MYID>" \
    --env authentication_type=2FA \
    --env folder_structure={:%Y/%m} \
    --env auto_delete=True \
    --env synchronisation_interval=43200 \
    --env icloud_china=True \
    --env auth_china=True \
    --env TZ="Asia/Shanghai" \
    --volume $(pwd)/config:/config \
    --volume $(pwd)/photos:/home/root/iCloud \
    boredazfcuk/icloudpd sync-icloud.sh --init

......
2023-09-22 12:07:15 INFO     Script launch parameters: --init
chown: changing ownership of '/config/.DS_Store': Operation not permitted
chown: changing ownership of '/config/icloudpd.conf': Operation not permitted
chown: changing ownership of '/config/python_keyring': Operation not permitted
chown: changing ownership of '/config': Operation not permitted
2023-09-22 12:07:15 INFO     Directory is writable: /config/python_keyring/
2023-09-22 12:07:15 INFO     Directory is writable: /home/user/.local/share/
Enter iCloud password for <MYID>:
Save password in keyring? [y/N]: y
Traceback (most recent call last):
  File "/opt/icloudpd_v1.7.2_china/bin/icloud", line 33, in <module>
    sys.exit(load_entry_point('pyicloud==1.0.0', 'console_scripts', 'icloud')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/icloudpd_v1.7.2_china/lib/python3.11/site-packages/pyicloud/cmdline.py", line 197, in main
    utils.store_password_in_keyring(username, password)
  File "/opt/icloudpd_v1.7.2_china/lib/python3.11/site-packages/pyicloud/utils.py", line 55, in store_password_in_keyring
    return keyring.set_password(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/icloudpd_v1.7.2_china/lib/python3.11/site-packages/keyring/core.py", line 61, in set_password
    get_keyring().set_password(service_name, username, password)
  File "/opt/icloudpd_v1.7.2_china/lib/python3.11/site-packages/keyrings/alt/file_base.py", line 129, in set_password
    self._write_config_value(service, username, password_base64)
  File "/opt/icloudpd_v1.7.2_china/lib/python3.11/site-packages/keyrings/alt/file_base.py", line 137, in _write_config_value
    self._ensure_file_path()
  File "/opt/icloudpd_v1.7.2_china/lib/python3.11/site-packages/keyrings/alt/file_base.py", line 166, in _ensure_file_path
    with open(self.file_path, 'w'):
         ^^^^^^^^^^^^^^^^^^^^^^^^^
PermissionError: [Errno 13] Permission denied: '/home/user/.local/share/python_keyring/keyring_pass.cfg'
2023-09-22 12:07:28 ERROR    Keyring file does not exist. Please try again.

[boredazfcuk]

All the env variables are written to /config/icloudpd.conf and then loaded from there, it's possible the logic is off for force_gid. Try recreating your container without any env variables then editing that file to force_gid=true

[walnutist]

All the env variables are written to /config/icloudpd.conf and then loaded from there, it's possible the logic is off for force_gid. Try recreating your container without any env variables then editing that file to force_gid=true

Correct. I also figured out. But even with the force_gid properly configured, the script still fail as my second comment. So I did a bit investigation myself. looks like the Permission denied error occurs because the line 750 within ConfigurePassword function:

su "${user}" -c 'icloud --username "${0}"' -- "${apple_id}"

Within my container environment, this command eventually got translated into:

su "user" -c 'icloud --username "${0}"' -- "<MYID>"

The thing is, if I log into the container via terminal as inside root, then the icloud --username <MYID> works flawlessly. But if I try the above command which try to execute via user "", it throws the same permission denied error.

Why and how to fix?

Thanks & regards

[boredazfcuk]

Great work. I should be able to fix this now you've found the root cause. Hopefully I'll be able to take a look at this later today.

[walnutist]

Now I believe the issue is really at line 833. cause the below error occurred right before calling ConfigurePassword:

2023-09-22 07:28:34 DEBUG    Set owner and group on icloudpd temp directory
2023-09-22 07:28:34 DEBUG    Set owner and group on config directory
chown: changing ownership of '/config/.DS_Store': Operation not permitted
chown: changing ownership of '/config/icloudpd.conf': Operation not permitted
chown: changing ownership of '/config/python_keyring': Operation not permitted
chown: changing ownership of '/config': Operation not permitted

Indeed, inside the container, the /config/ folder belongs to root:nobody and is set to 755 (drwxr-xr-x). Also manually try to chown <my_host_user>:<my_host_group> /config/ throws exception "Operation not permitted". This could be the root cause.

both and exists inside the container after container creation.

[walnutist]

In fact, looks like the chown inside the container does NOT work at all, not even as root, as long as under the /config/ directory.

And here is my docker run command and parameters:

docker run -it\
    --privileged \
    --network bridge \
    --restart=always \
    --volume $(pwd)/config:/config \
    --volume $(pwd)/photos:/home/root/iCloud \
    boredazfcuk/icloudpd sync-icloud.sh --init

icloudpd.conf with privacy info removed:

#force_gid=true
albums_with_dates=false
apple_id=<MYID>
auth_china=true
authentication_type=MFA
auto_delete=true
bark_device_key=
bark_server=
convert_heic_to_jpeg=false
debug_logging=true
delete_accompanying=false
delete_after_download=false
delete_notifications=true
dingtalk_token=
directory_permissions=750
discord_id=
discord_token=
download_notifications=true
download_path=
file_permissions=640
folder_structure={:%Y/%m}
gotify_app_token=
group=<MACOS_HOST_GROUP>
group_id=502
icloud_china=true
iyuu_token=
jpeg_path=
jpeg_quality=90
nextcloud_delete=false
nextcloud_password=
nextcloud_upload=false
nextcloud_url=
nextcloud_username=
notification_days=7
notification_type=
photo_album=
photo_size=original
prowl_api_key=
pushover_sound=
pushover_token=
pushover_user=
recent_only=100
set_exif_datetime=false
single_pass=false
skip_album=
skip_check=false
skip_download=false
skip_live_photos=false
synchronisation_delay=0
synchronisation_interval=43200
synology_ignore_path=false
telegram_chat_id=
telegram_polling=true
telegram_server=
telegram_silent_file_notifications=
telegram_token=
trigger_nextlcoudcli_synchronisation=
until_found=
user=<MACOS_HOST_USER>
user_id=501
webhook_https=false
webhook_id=
webhook_path=/api/webhook/
webhook_port=8123
webhook_server=
wecom_id=
wecom_proxy=
wecom_secret=

[walnutist]

I just realized the underlying podman might be the real cirminal here... See https://github.com/containers/podman/issues/20077 for further information. In that case, I would then switch to the old plain docker on macOS and give a try. Keep tuned and no need to further invest your time

[walnutist]

Switched to orbstack, now the initialization run through flawlessly. Confirmed this was caused by podman underline filesystem, and seems same issue exists for colima too. Closing the issue