Fails to save credentials to keychain when using login command with token option

[Joel-Bell-okcupid]

Description

Hi,

When attempting to execute the package-registry login command using the token option, the credentials fail to save to the keychain resulting in a -25308 error.

Since Xcode only supports registry credentials saved in Keychain, using netrc to authorize does not work either (as mentioned in this issue discussion).

This makes it impossible to do things like xcodebuild -resolvePackageDependencies in a CI pipeline. Please note suggested solutions like this one and this one assume access to the Keychain UI which is not available in a CI pipeline/context (e.g., Creating a build using Xcode Cloud).

Thank you in advance for your help!

Expected behavior

The credentials should be persisted in the keychain without error.

Actual behavior

When executing the command mentioned, the following error occurs:

Failed to save credentials for \‘https://my_domain.d.codeartifact.the_region.amazonaws.com/[\](https://my_domain.d.codeartifact.the_region.amazonaws.com/)’ to keychain: status -25308”

Steps to reproduce

CI Pipeline example 1 - Amazon EC2 running macOS 13.5.2, Xcode 15.0:

(Assumes AWS CLI is installed and configured)

1. Get authorization token

   export CODEARTIFACT_AUTH_TOKEN=`aws codeartifact get-authorization-token --domain my_domain --domain-owner my_domain_owner_id --region my_region --query authorizationToken --output text --profile my_profile`

2. Set registry

   swift package-registry set --global https://my_domain.d.codeartifact.my_region.amazonaws.com/swift/my_repo/

3. Log in to registry

   swift package-registry login https://my_domain.d.codeartifact.my_region.amazonaws.com/swift/my_repo/login --token ${CODEARTIFACT_AUTH_TOKEN}

4. Login is successful (but credentials are not saved to keychain)

CI Pipeline example 2 - Xcode Cloud running macOS 13.5.2, Xcode 15.1:

(occurs post clone of repo as part of ci_post_clone.sh script)

1. Install AWS CLI and configure
2. Get authorization token

   export CODEARTIFACT_AUTH_TOKEN=`aws codeartifact get-authorization-token --domain my_domain --domain-owner my_domain_owner_id --region my_region --query authorizationToken --output text --profile my_profile`

3. Set registry

   swift package-registry set --global https://my_domain.d.codeartifact.my_region.amazonaws.com/swift/my_repo/

4. Log in to registry

   swift package-registry login https://my_domain.d.codeartifact.my_region.amazonaws.com/swift/my_repo/login --token ${CODEARTIFACT_AUTH_TOKEN}

5. Login is successful (but credentials are not saved to keychain)

Swift Package Manager version/commit hash

Swift Package Manager - Swift 5.9.0

Swift & OS version (output of swift --version ; uname -a)

CI Pipeline example 1 - Amazon EC2 running macOS 13.5.2, Xcode 15.0:

swift-driver version: 1.87.1 Apple Swift version 5.9 (swiftlang-5.9.0.128.108 clang-1500.0.40.1)
Target: arm64-apple-macosx13.0
Darwin github-runner-macos-02 22.6.0 Darwin Kernel Version 22.6.0: Wed Jul  5 22:22:52 PDT 2023; root:xnu-8796.141.3~6/RELEASE_ARM64_T8103 arm64

CI Pipeline example 2 - Xcode Cloud running macOS 13.5.2, Xcode 15.1:

swift-driver version: 1.87.3 Apple Swift version 5.9.2 (swiftlang-5.9.2.2.56 clang-1500.1.0.2.5)
Target: x86_64-apple-macosx13.0
Darwin e1b43c42-7e4f-47a9-9e31-f99a59f26be5-f46ffd7c-f6l9d-vm.podset-vms.dt-skywagon-workers-prod-vm.svc.kube.us-west-3f.k8s.cloud.apple.com 22.6.0 Darwin Kernel Version 22.6.0: Wed Jul  5 22:21:56 PDT 2023; root:xnu-8796.141.3~6/RELEASE_X86_64 x86_64

[yim-lee]

Since https://github.com/apple/swift-package-manager/issues/6395#issuecomment-1639148347, using netrc to authorize does not work either (as mentioned https://github.com/apple/swift-package-manager/issues/6395#issuecomment-1830544462).

For reference, the --netrc issue is being tracked with rdar://118898849.

And for Keychain, I wonder if this might help: https://byteable.medium.com/creating-a-temporary-keychain-for-your-build-system-e598628c65fd

@sebsto might have some ideas as well?

[Joel-Bell-okcupid]

For reference, the --netrc issue is being tracked with rdar://118898849.

Good to know, thank you!

And for Keychain, I wonder if this might help: https://byteable.medium.com/creating-a-temporary-keychain-for-your-build-system-e598628c65fd

I'll give that a shot.

[sebsto]

@Joel-Bell-okcupid Thank you for the detailled report. I have a question : how do you connect to your EC2 Mac instance ? Is it SSH / SSM only ? or through a graphical session (such as Apple remote Desktop / RDP / AnyConnect etc ) ?

In the first case, there is no keychain created for your user, any application expecting a user keychain will fail (this is the case of codesign for example)

When I wrote the demo, I only tested the GUI session use case, I did not test the command line only use case.

I will find time this week to test this.

From what I understand in the thread, the --netrc option is not working, leaving you with the only possibility to create a temporary keychain.

Here is the script I use to create temporary keychain for codesigning. You can reuse most of it to create a SwiftPM's credentails keychain.

Pay attention to line 16, you probably have to add the swift executable to the authorization list. Everything on line 43 and after is related to code signing, you can probably delete all this.

[Joel-Bell-okcupid]

how do you connect to your EC2 Mac instance ? Is it SSH / SSM only ? or through a graphical session (such as Apple remote Desktop / RDP / AnyConnect etc ) ?

I can log in to our github runner machines / instances via Screen Sharing.

From what I understand in the thread, the --netrc option is not working, leaving you with the only possibility to create a temporary keychain.

Correct, I can do the following successfully:

swift package-registry --netrc login https://my_domain.d.codeartifact.my_region.amazonaws.com/swift/my_repo/login --token ${CODEARTIFACT_AUTH_TOKEN} --no-confirm

However, when I proceed to resolve package dependencies using xcodebuild, it results in this error:

xcodebuild: error: Could not resolve package dependencies:
  failed fetching my_repo releases list from https://my_domain.d.codeartifact.my_region.amazonaws.com/swift/my_repo/[:](https://my_domain.d.codeartifact.my_region.amazonaws.com/swift/my_repo/:) server error 401: Unauthenticated: request did not include an Authorization header. Please provide your credentials.

Here is the script I use to create temporary keychain for codesigning. You can reuse most of it to create a SwiftPM's credentails keychain.

Okay, thank you! I'll try this as well.

[sebsto]

Here is the script I use to create temporary keychain for codesigning. You can reuse most of it to create a SwiftPM's credentails keychain.

Okay, thank you! I'll try this as well.

If you connect through screen sharing, these steps are not necessary. The Login Window will create a session for you and configure the keychain correctly. Then, you open the Terminal app and the command line tools will have access to the keychain created for you.

How do you run the GitHub agent ? When run as a Launch Agent, it is started when the Login Window starts the session, so it should have access to the keychain.

If you run it as a LaunchDaemon (started when the system starts), he does not have access to the keychain by default.

So the question is : what is starting xcodebuild ? I guess it is the GitHub agent. How is the github agent started ?

[Joel-Bell-okcupid]

So the question is : what is starting xcodebuild ? I guess it is the GitHub agent. How is the github agent started ?

I have access to the runners via Screen Sharing but the actual work is automated. In the case where xcodebuild is failing, it's a specific job/check in a github action workflow that runs on every PR:

  swiftpackage:
    name: Swift Package
    runs-on: [m1-os13.5.2]
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Install Dependencies
        run: xcodebuild -resolvePackageDependencies -scheme Archer
      - name: Ensure Git Status Clean
        run: sh Scripts/ensure_git_status_clean.sh

In my attempt to get this working, I added a step before - name: Install Dependencies that executed the following script. I know this is not the most efficient way to do this in terms of using an up-to-date token but I was just trying to get it working first before optimizing the flow.

#!/bin/sh

echo "Start ArrowPay registry update"

cd Modules/Core

export CODEARTIFACT_AUTH_TOKEN=`aws codeartifact get-authorization-token --domain [domain] --domain-owner [owner id] --region eu-west-3 --query authorizationToken --output text --profile [profile]`

swift package-registry set --global [url]
swift package-registry login [url]/login --token ${CODEARTIFACT_AUTH_TOKEN}

[sebsto]

Thank you @Joel-Bell-okcupid Just for my understanding of your setup : what is launching the GitHub runner on your machine ?

Is this a Launch agent (with a launch script in /Users/ec2-user/Library/LaunchAgents ? or a lunch daemon (with a launch script in /Library/LaunchDaemons) ? Or do you use another method to start the runner ? (like manually typing the command in a Terminal)

[Joel-Bell-okcupid]

@sebsto I reached out to someone on our ops team (as some of this is a bit out of my wheelhouse).

From him:

Regarding the github actions runner configuration, we're using the stock service setup that comes bundled with the package, instructions are as follows here: https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-run[…]-the-self-hosted-runner-application-as-a-service?platform=mac

Looks to be using the launch agents pattern, script has it installing to there and on one of the host i see a plist in that directory:

$ cat svc.sh
...
LAUNCH_PATH="${HOME}/Library/LaunchAgents"
PLIST_PATH="${LAUNCH_PATH}/${SVC_NAME}.plist"
...

$ ls ${HOME}/Library/LaunchAgents
actions.runner.[org - repo].[runner].plist

[toheebster]

When I wrote the demo, I only tested the GUI session use case, I did not test the command line only use case.

Hi @sebsto, any update on from the command line working for this? I’m seeing that my github actions ci machine is stuck on a keychain prompt even when I create my own keychain & unlock it & set the partition list (though I’m not 100% I implemented right).

Pay attention to line 16, you probably have to add the swift executable to the authorization list.

Looking into this, it seems we can only pass authorizations when the secure item is being added & not retroactively, at least not without avoiding a keychain prompt. And also while passing an authorization flag is available when adding a cert, I don’t know if we’re able to pass authorizations when adding an Internet password.

One thing that might work is that it seems that add-internet-password in the security tool has a -A option to always allow applications to access the password without a keychain prompt. It’s not the most secure option though.

[sebsto]

@toheebster I'll take time this week to check this

[sebsto]

I managed to create a keychain and have the swift command use it (no need for --netrc) But xcodebuild fails to pickup the credentials.

On a machine with a GUI session :

✗ xcodebuild -resolvePackageDependencies       
Command line invocation:
    /Applications/Xcode-15.1.app/Contents/Developer/usr/bin/xcodebuild -resolvePackageDependencies

User defaults from command line:
    IDEPackageSupportUseBuiltinSCM = YES

Resolve Package Graph

Resolved source packages:
  swift-collections: https://github.com/apple/swift-collections @ 1.0.6
  smithy-swift: https://github.com/awslabs/smithy-swift @ 0.28.0
  swift-log: https://github.com/apple/swift-log.git @ 1.5.3
  aws-crt-swift: https://github.com/awslabs/aws-crt-swift @ 0.13.0
  aws-sdk-swift: aws.aws-sdk-swift @ 0.25.0
  XMLCoder: https://github.com/MaxDesiatov/XMLCoder.git @ 0.17.0

resolved source packages: swift-collections, smithy-swift, swift-log, aws-crt-swift, aws-sdk-swift, XMLCoder

On a machine without a GUI session:

% xcodebuild -resolvePackageDependencies 
Command line invocation:
    /Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild -resolvePackageDependencies

User defaults from command line:
    IDEPackageSupportUseBuiltinSCM = YES

Resolve Package Graph

Updating from https://github.com/awslabs/smithy-swift

Updating from https://github.com/apple/swift-collections

Updating from https://github.com/apple/swift-log.git

Updating from https://github.com/MaxDesiatov/XMLCoder.git

Updating from https://github.com/awslabs/aws-crt-swift

failed fetching aws.aws-sdk-swift releases list from https://stormacq-test-486652066693.d.codeartifact.us-west-2.amazonaws.com/swift/MySwiftRepo/: server error 401: Unauthenticated: request did not include an Authorization header. Please provide your credentials.2024-01-16 18:29:18.912 xcodebuild[8775:68889] Writing error result bundle to /var/folders/l5/09bgh0l54y7dc834kb51x3vc0000gn/T/ResultBundle_2024-16-01_18-29-0018.xcresult
xcodebuild: error: Could not resolve package dependencies:
  failed fetching aws.aws-sdk-swift releases list from https://stormacq-test-486652066693.d.codeartifact.us-west-2.amazonaws.com/swift/MySwiftRepo/: server error 401: Unauthenticated: request did not include an Authorization header. Please provide your credentials.

The token is present in the keychain (and the swift command can use it) :

 % security dump-keychain /Users/ec2-user/Library/Keychains/login.keychain-db
keychain: "/Users/ec2-user/Library/Keychains/login.keychain-db"
version: 512
class: "inet"
attributes:
    0x00000007 <blob>="stormacq-test-486652066693.d.codeartifact.us-west-2.amazonaws.com"
    0x00000008 <blob>=<NULL>
    "acct"<blob>="token"
    "atyp"<blob>=<NULL>
    "cdat"<timedate>=0x32303234303131363138303032335A00  "20240116180023Z\000"
    "crtr"<uint32>=<NULL>
    "cusi"<sint32>=<NULL>
    "desc"<blob>=<NULL>
    "icmt"<blob>=<NULL>
    "invi"<sint32>=<NULL>
    "mdat"<timedate>=0x32303234303131363138303032335A00  "20240116180023Z\000"
    "nega"<sint32>=<NULL>
    "path"<blob>=<NULL>
    "port"<uint32>=0x00000000 
    "prot"<blob>=<NULL>
    "ptcl"<uint32>="htps"
    "scrp"<sint32>=<NULL>
    "sdmn"<blob>=<NULL>
    "srvr"<blob>="stormacq-test-486652066693.d.codeartifact.us-west-2.amazonaws.com"
    "type"<uint32>=<NULL>

Steps to create the keychain :

KEYCHAIN_PASSWORD=Passw0rd
KEYCHAIN_NAME=login.keychain
SYSTEM_KEYCHAIN=/Library/Keychains/System.keychain
AUTHORISATION=(-T /usr/bin/security -T /usr/bin/codesign -T /usr/bin/xcodebuild -T /usr/bin/swift)

if [ -f $HOME/Library/Keychains/"${KEYCHAIN_NAME}"-db ]; then
    echo "Deleting old ${KEYCHAIN_NAME} keychain"
    security delete-keychain "${KEYCHAIN_NAME}"
fi
echo "Create Keychain"
security create-keychain -p "${KEYCHAIN_PASSWORD}" "${KEYCHAIN_NAME}"

EXISTING_KEYCHAINS=( $( security list-keychains | sed -e 's/ *//' | tr '\n' ' ' | tr -d '"') )
sudo security list-keychains -s "${KEYCHAIN_NAME}" "${EXISTING_KEYCHAINS[@]}"

echo "New keychain search list :"
security list-keychain 

echo "Configure keychain : remove lock timeout"
security unlock-keychain -p "${KEYCHAIN_PASSWORD}" "${KEYCHAIN_NAME}"
security set-keychain-settings "${KEYCHAIN_NAME}"

[sebsto]

The difference I found is that when there is a GUI session, xcodebuild (or the keychain) prompts the user for the account-level password to authorize access to the keychain entry, something that can't happen in a SSH session.

Solution might be to configure the keychain entry in such a way no password prompt is required.

[sebsto]

@yim-lee I turned this around in many directions and I can't find a way to create / update the token in the keychain in such a way that xcodebuild can read / access it.

This is the code from SPM that creates the entry https://github.com/apple/swift-package-manager/blob/main/Sources/Basics/AuthorizationProvider.swift#L339

And this is the code that reads it https://github.com/apple/swift-package-manager/blob/main/Sources/Basics/AuthorizationProvider.swift#L261

There is nothing special about it, no magic permission set or anything fancy.

Without having access to the code of xcodebuild I'm afraid I can't go further.

Two suggestions for next steps

• either ensure a future version of xcodebuild can read the token from the default keychain, just as swift-package does
• either implement --netrc in a future version of xcodebuild

In the meantime, I am afraid it's not possible to include a package from a third-party repository in a headless build system using xcodebuild.

@Joel-Bell-okcupid a workaround might be to isolate the package you store on CodeArtifact in a Library that can be build with swift package (because swift package command can read the token, either from the keychain, either with --netrc), then include the local build as a dependency in the xcodebuild build.

[yim-lee]

Thanks so much for looking into this @sebsto. 🙏

either implement --netrc in a future version of xcodebuild

We are tracking this with rdar://118898849.

either ensure a future version of xcodebuild can read the token from the default keychain, just as swift-package does

Does xcodebuild fail to read credentials from Keychain in all cases? From https://github.com/apple/swift-package-manager/issues/7236#issuecomment-1894255929 it seems like it works when there is GUI session?

[toheebster]

failed fetching aws.aws-sdk-swift releases list from https://stormacq-test-486652066693.d.codeartifact.us-west-2.amazonaws.com/swift/MySwiftRepo/: server error 401: Unauthenticated: request did not include an Authorization header. Please provide your credentials.

@sebsto -- I believe this error is actually due to the password not being in the keychain the build system is using. When you create the new keychain, you have to set it as the default keychain to tell xcodebuild to look there.

Does xcodebuild fail to read credentials from Keychain in all cases? From https://github.com/apple/swift-package-manager/issues/7236#issuecomment-1894255929 it seems like it works when there is GUI session?

@yim-lee -- with the credentials in the proper keychains, the build system will actually just hang due to the keychain prompt for a password. Similar to how the build system hangs in a GUI session until the user inputs a password in the Keychain prompt.

[toheebster]

@yim-lee -- can you think of any workarounds that would allow xcodebuild to get access to the credentials it needs on a CI system like github actions?

[Joel-Bell-okcupid]

Yes, thank you @sebsto for all your efforts and possible workaround! For now, we are using an xcframework.

And thank you @toheebster for your input and questions!

@yim-lee -- can you think of any workarounds that would allow xcodebuild to get access to the credentials it needs on a CI system like github actions?

🙏

[toheebster]

No problem @Joel-Bell-okcupid

I’m currently running into the same issue & would need to use codeartifact 😅🙏🏿

[yim-lee]

with the credentials in the proper keychains, the build system will actually just hang due to the keychain prompt for a password. Similar to how the build system hangs in a GUI session until the user inputs a password in the Keychain prompt.

OK, so the Keychain prompt is what's preventing xcodebuild from working (i.e., reading credentials). IIUC, to workaround this there needs to be a way to disable Keychain prompt like Sebastien mentioned. I thought security unlock-keychain would do it, but I see Sebastien does that already, so I suppose it doesn't do what I hope it would do.

can you think of any workarounds that would allow xcodebuild to get access to the credentials it needs on a CI system like github actions?

Not at this time unfortunately. If we support something like --netrc in xcodebuild then that would be a workaround, but that's not available right now.

[toheebster]

IIUC, to workaround this there needs to be a way to disable Keychain prompt like Sebastien mentioned.

Hi @yim-lee — I think a potential solution is actually passing the kSecAttrAccesible flag where you’re adding the credentials into the Keychain, that way xcode will be able to have access to the password without needing to answer a Keychain prompt.

[sebsto]

@yim-lee

Does xcodebuild fail to read credentials from Keychain in all cases? From https://github.com/apple/swift-package-manager/issues/7236#issuecomment-1894255929 it seems like it works when there is GUI session?

Correct. when there is a GUI session, there is a GUI prompt asking the user for the keychain password to allow xcodebuild to access the token. When we're connected with SSH only, there is no GUI session and xcodebuild doesn't prompt for keychain password.

I tried to massage the keychain entry in multiple ways but I couldn't find a configuration that would allow xcodebuild to access the token without prompting for a password. I tried -A and -T but with no success.

(source : man security https://ss64.com/mac/security-internet.html)

I thought security unlock-keychain would do it, but I see https://github.com/apple/swift-package-manager/issues/7236#issuecomment-1894255929, so I suppose it doesn't do what I hope it would do.

I used an unlocked keychain. The problem is not at keychain level, it is at the entry-level. i.e. when creating the entry, we must tell that xcodebuild has access to the item.
I tried to modify the entry (security add-internet-password -U -T /usr/bin/xcodebuild or even security add-internet-password -U -A to grant all app access) but it doesn't work. Maybe I missed something in my security command) I also compared the entry created by swiftPM with an entry I created manually with security add-internet-password I couldn't see the difference but the manualy created entry was not picked up by xcodebuild :-( )

The reason why it works with SwiftPM is :

By default, the application which creates an item is trusted to access its data without warning.

(source security add-internet-password -h)

I think a potential solution is actually passing the kSecAttrAccesible flag where you’re adding the credentials into the Keychain, that way xcode will be able to have access to the password without needing to answer a Keychain prompt.

This is a path to explore. It's easy to test with a local build of SwiftPM. This is the equivalent to security add-internet-password -A

However, this is not a secured solution, it grants all applications access to the the token.

A more secure option would be to create the entry with ksecAttrAccess (https://developer.apple.com/documentation/security/ksecattraccess) limited to SwiftPM and xcodebuild only.

The idea is to have the keychain entry with this permission (taken from a machine with a GUI)

[sebsto]

I further explored options to programmatically set the trust of the keychain item to xcodebuild but it looks like all API to manage ACL are deprecated https://developer.apple.com/documentation/security/1393522-secaccesscreate

@yim-lee what are the replacement API ? Can you get help from the team managing these API to get more info or a code sample ?

[toheebster]

@sebsto hmm I might be thinking about this wrong, but would it be possible to explore this route:

• after swift registry adds the credential
• we use security cli to delete the credential
• we use the security cli to re add the credential with the proper access list.

this would necessitate having access to the password. is this a viable option to explore?

[sebsto]

@toheebster that's exactly the approach I tried yesterday. I stopped after 4 hours :-) I don't know if I did something wrong or if xcodebuild is unable to pick it up.

Here are the two sets of commands I tried. the first one sets an authorization list to security swift-package and xcodebuild. The second gives permission to all (-A)

SERVER=$(echo $CODEARTIFACT_REPO | sed  's/https:\/\///g' | sed 's/.com.*$/.com/g')
AUTHORISATION=(-T /usr/bin/security -T /Library/Developer/CommandLineTools/usr/bin/swift-package -T /usr/bin/xcodebuild)
security add-internet-password -a token \
                               -s $SERVER \
                   -w $CODEARTIFACT_AUTH_TOKEN \
                   -r htps \
                   -U \
                   "${AUTHORISATION[@]}" \
                  $HOME/Library/Keychains/"${KEYCHAIN_NAME}"-db 

security add-internet-password -a token \
                                                     -s $SERVER \
                             -w $CODEARTIFACT_AUTH_TOKEN \
                             -r htps \
                             -A \
                             $HOME/Library/Keychains/"${KEYCHAIN_NAME}"-db

I also use this command to read the keychain value

security find-internet-password -s $SERVER 

I also copied (with scp) the keychain file from the EC2 Mac to my laptop to inspect it with the GUI or access the password with the -w option and observe if I'm prompted for a password or not.

scp ec2-user@$MY_IP:/Users/ec2-user/Library/Keychains/login.keychain-db ~/Desktop/test.keychain-db 

# then 

security find-internet-password -s $SERVER -w ~/Desktop/test.keychain-db  

You can force create a clean entry in the keychain with

swift package-registry login https://$SERVER/swift/MySwiftRepo/login --token $CODEARTIFACT_AUTH_TOKEN 

And to verify if xcodebuild can pick it up, I created a very simple CLI macOS project with just a dependency on the packagein CodeArtifact.

Then I type :

rm -rf ~/Library/Developer/Xcode/DerivedData/test-* && xcodebuild -resolvePackageDependencies

On the mac laptop, it prompts for the password, then I know my keychain entry is not correct. I select Denied and it usually fails with the same error as on the EC2 Mac / SSH only

failed fetching aws.aws-sdk-swift releases list from https://stormacq-test-486652066693.d.codeartifact.us-west-2.amazonaws.com/swift/MySwiftRepo/: server error 401: Unauthenticated: request did not include an Authorization header. Please provide your credentials.2024-01-17 18:34:30.076 xcodebuild[80341:5717692] Writing error result bundle to /var/folders/14/nwpsn4b504gfp02_mrbyd2jr0000gr/T/ResultBundle_2024-17-01_18-34-0030.xcresult
xcodebuild: error: Could not resolve package dependencies:
  failed fetching aws.aws-sdk-swift releases list from https://stormacq-test-486652066693.d.codeartifact.us-west-2.amazonaws.com/swift/MySwiftRepo/: server error 401: Unauthenticated: request did not include an Authorization header. Please provide your credentials.

[toheebster]

@sebsto -- do you unlock the keychain before trying to build?

that might be why the prompt for password is popping up.

[sebsto]

@toheebster yes I do. and I remove the lock timeout. I shared the script I use to (re)create the empty keychain in this comment https://github.com/apple/swift-package-manager/issues/7236#issuecomment-1894255929

[toheebster]

ahh okay, yeah then in that I usually only get the Please provide your credentials prompt if its looking in the wrong keychain. are you sure xcodebuild is looking in your temp keychain?

[yim-lee]

A more secure option would be to create the entry with ksecAttrAccess (https://developer.apple.com/documentation/security/ksecattraccess) limited to SwiftPM and xcodebuild only.

+1. I prefer that we limit access to SwiftPM and xcodebuild only (if we manage to modify entry ACL programmatically somehow)

I further explored options to programmatically set the trust of the keychain item to xcodebuild but it looks like all API to manage ACL are deprecated https://developer.apple.com/documentation/security/1393522-secaccesscreate

@yim-lee what are the replacement API ? Can you get help from the team managing these API to get more info or a code sample ?

@sebsto I will try and see if I can get more information.

[Edit] I am told we should use data protection keychain instead: https://developer.apple.com/documentation/security/ksecusedataprotectionkeychain

[sebsto]

@toheebster

ahh okay, yeah then in that I usually only get the Please provide your credentials prompt if its looking in the wrong keychain. are you sure xcodebuild is looking in your temp keychain?

I'm reasonably sure for three reasons

• I use the same name as the GUI keychain (Login.keychain) (that's the least convincing reason)
• the temp keychain is the default and is visible from the access list (security list-keychain)
• SwiftPM finds it and uses it

[toheebster]

hmm I see, @sebsto -- then the password must be wrong somehow. can you check that the original password added by the login command and the one you manually add resolve to the same value?

[sebsto]

@toheebster I told you I spent 4 hours last night on that. Of course, I verified. Using the security find-internet-password -w option on a laptop + using the diff command + copying over the working keychain from my machine to EC2 Mac ... all failed

You're redoing all my thought process from last night. Please try it yourself. I gave up after 4 hours. I will try the programmatic approach in SwiftPM source code instead.

[toheebster]

SERVER=$(echo $CODEARTIFACT_REPO | sed  's/https:\/\///g' | sed 's/.com.*$/.com/g')
AUTHORISATION=(-T /usr/bin/security -T /Library/Developer/CommandLineTools/usr/bin/swift-package -T /usr/bin/xcodebuild)
security add-internet-password -a token \
                               -s $SERVER \
                 -w $CODEARTIFACT_AUTH_TOKEN \
                 -r htps \
                 -U \
                 "${AUTHORISATION[@]}" \
                $HOME/Library/Keychains/"${KEYCHAIN_NAME}"-db 

security add-internet-password -a token \
                                                     -s $SERVER \
                           -w $CODEARTIFACT_AUTH_TOKEN \
                           -r htps \
                           -A \
                           $HOME/Library/Keychains/"${KEYCHAIN_NAME}"-db

just double checking, shouldn't -r htps be -r https @sebsto

[sebsto]

@toheebster nope, it's a 4 character string, as per the help message and the error message you receive when passing https

About

then the password must be wrong somehow

I'm sure the password (token) is correct because the command swift package resolve works correctly. Only xcodebuild fails to pick it up. That also shows that my temporary keychain is correctly unlocked and placed in the search list.

[toheebster]

You're redoing all my thought process from last night. Please try it yourself. I gave up after 4 hours. I will try the programmatic approach in SwiftPM source code instead.

@sebsto -- yep, gave it my own 4 hours. Able to recreate the password in the keychain with the right permissions such that it looks like the image below, but for some reason Xcode and xcodebuild are still triggering the Keychain password prompt.

There must be some hidden rules that we are not privvy to.

How's the programmatic approach with the SwiftPM source code going?

[sebsto]

Thank you @toheebster ! At least I feel better I didn't do something stupid during my tests :-) I haven't had the time to progress on the programmatic aspect. Just looking at the doc, the ACL part of Keychain API, which allows to define what applications are trusted, is deprecated. I asked Apple guidance about what replace these APIs.

I'll try to book time to try these API anyway, just to prove the point. If it works, we'll have to find a path to avoid usage of these deprecated APIs.

[toheebster]

@sebsto -- I actually was able to get it working via security with a minor tweak 🎉

TL;DR, instead of just including /usr/bin/xcodebuild as a preapproved application (using -T) when re-adding the password (via security add-internet-password), I saw that my CI machine (github actions) was using a different xcodebuild (Applications/Xcode_15.0.1.app/Contents/Developer/usr/bin/xcodebuild), so I included that also.

1. Create and unlock keychain

   security create-keychain -p "${MATCH_KEYCHAIN_PASSWORD}" "${MATCH_KEYCHAIN_NAME}"
   security default-keychain -s "${MATCH_KEYCHAIN_NAME}"
   security unlock-keychain -p "${MATCH_KEYCHAIN_PASSWORD}" "${MATCH_KEYCHAIN_NAME}"
   security set-keychain-settings "${MATCH_KEYCHAIN_NAME}"

2. Add new keychain to existing keychain list

   EXISTING_KEYCHAINS=( $( security list-keychains | sed -e 's/ *//' | tr '\n' ' ' | tr -d '"') )
   sudo security list-keychains -s "${MATCH_KEYCHAIN_NAME}" "${EXISTING_KEYCHAINS[@]}"
   security list-keychain

3. Log into code artifact with the proper authorizations

   export CODEARTIFACT_AUTH_TOKEN=`aws codeartifact get-authorization-token --domain $DOMAIN --domain-owner $DOMAIN_OWNER --query authorizationToken --region $REGION --output text`
   swift package-registry login "$SEVER/login" --token $CODEARTIFACT_AUTH_TOKEN

4. Delete whatever password was inserted into the keychain by the login command because that password doesn't have the proper preapproved application list and thus would prompt the Keychain password UI dialog causing the build system to hang.

   security delete-internet-password -a token -s $SERVER -r htps "${MATCH_KEYCHAIN_NAME}"

5. Add the same password back with the proper approved application list. The main emphasis here is that instead of just including /usr/bin/xcodebuild, I saw that my CI machine (github actions) was using a different xcodebuild (Applications/Xcode_15.0.1.app/Contents/Developer/usr/bin/xcodebuild), so I included that also

   export PREAPPROVED_APPLICATION_LIST=(-T /usr/bin/security -T /usr/bin/codesign -T /usr/bin/productbuild -T /usr/bin/productsign -T /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/swift-package -T /Applications/Xcode_15.0.1.app/Contents/Developer/usr/bin/xcodebuild -T /Applications/Xcode.app)
   security add-internet-password -a token -s $SERVER -w $CODEARTIFACT_AUTH_TOKEN -r htps -U "${PREAPPROVED_APPLICATION_LIST[@]}" $HOME/Library/Keychains/"${MATCH_KEYCHAIN_NAME}"
   security set-internet-password-partition-list -a token -s $SERVER -S "com.apple.swift-package,com.apple.security,com.apple.dt.Xcode,apple-tool:,apple:,codesign" -k "${MATCH_KEYCHAIN_PASSWORD}" "${MATCH_KEYCHAIN_NAME}"
   security find-internet-password -s $SERVER

[sebsto]

Thank @toheebster - you made it - I confirm this works.

The main two differences with what I tried

• you delete the item created by SwiftPM first. If you don't a second item is created in the keychain, despite the -U (update) flag. This might cause confusion to application consuming the token as we don't know which item they picked up.

• The security set-internet-password-partition-list is required. I should have known. I use that to import signing keys.

I just tested end to end with a slightly different script, but this is a matter of personal taste :-)

Prepare the keychain

KEYCHAIN_PASSWORD=$(openssl rand -base64 20)
KEYCHAIN_NAME=login.keychain
SYSTEM_KEYCHAIN=/Library/Keychains/System.keychain

if [ -f $HOME/Library/Keychains/"${KEYCHAIN_NAME}"-db ]; then
    echo "Deleting old ${KEYCHAIN_NAME} keychain"
    security delete-keychain "${KEYCHAIN_NAME}"
fi
echo "Create Keychain"
security create-keychain -p "${KEYCHAIN_PASSWORD}" "${KEYCHAIN_NAME}"

EXISTING_KEYCHAINS=( $( security list-keychains | sed -e 's/ *//' | tr '\n' ' ' | tr -d '"') )
sudo security list-keychains -s "${KEYCHAIN_NAME}" "${EXISTING_KEYCHAINS[@]}"

echo "New keychain search list :"
security list-keychain 

echo "Configure keychain : remove lock timeout"
security unlock-keychain -p "${KEYCHAIN_PASSWORD}" "${KEYCHAIN_NAME}"
security set-keychain-settings "${KEYCHAIN_NAME}"

Get CodeArtifact token and repo

export CODEARTIFACT_AUTH_TOKEN=`aws codeartifact get-authorization-token \
                                     --region us-west-2 \
                                     --domain YOUR_DOMAIN   \
                                     --domain-owner YOUR_ACCOUNT   \
                                     --query authorizationToken  \
                                     --output text`

export CODEARTIFACT_REPO=`aws codeartifact get-repository-endpoint  \
                                --region us-west-2   \
                                --domain YOUR_DOMAIN   \
                                --domain-owner YOUR_ACCOUNT    \
                                --format swift     \
                                --repository YOUR_REPO    \
                                --query repositoryEndpoint   \
                                --output text`        

Login (this creates the entry in the keychain

aws codeartifact login   \
--region us-west-2    \
--tool swift   \
--domain YOUR_DOMAIN  \
--repository YOUR_REPO  \
--namespace aws \
--domain-owner YOUR_ACCOUNT 

# verify the entry was created 
security find-internet-password   "${KEYCHAIN_NAME}"

Delete and recreate the entry, as @toheebster said :-)

SERVER=$(echo $CODEARTIFACT_REPO | sed  's/https:\/\///g' | sed 's/.com.*$/.com/g')
AUTHORISATION=(-T /usr/bin/security -T /usr/bin/codesign -T /usr/bin/xcodebuild -T /usr/bin/swift -T /Applications/Xcode-15.2.app/Contents/Developer/usr/bin/xcodebuild)

security delete-internet-password -a token -s $SERVER -r htps "${KEYCHAIN_NAME}"

security add-internet-password -a token \
                               -s $SERVER \
                               -w $CODEARTIFACT_AUTH_TOKEN \
                               -r htps \
                               -U \
                               "${AUTHORISATION[@]}" \
                               "${KEYCHAIN_NAME}"

security set-internet-password-partition-list \
             -a token \
             -s $SERVER \
             -S "com.apple.swift-package,com.apple.security,com.apple.dt.Xcode,apple-tool:,apple:,codesign" \
             -k "${KEYCHAIN_PASSWORD}" "${KEYCHAIN_NAME}"

security find-internet-password   "${KEYCHAIN_NAME}"

@Joel-Bell-okcupid now that we have one workaround (and two sets of script for your convenience) - can you test on your side and report back ?

[Joel-Bell-okcupid]

🥳🍾

@toheebster nice work!

@sebsto Will make some time today or Monday and let you know!

[Joel-Bell-okcupid]

@sebsto tried your approach and it worked! Thank you both again (@toheebster) for your efforts!

My only concern is my lack of understanding about all the unrelated items in the login keychain that get deleted (certs, app passwords, etc.). Wondering if I should/can export/move those from/to the keychain I'm creating.

I still need to test this approach in the CI pipeline using Xcode Cloud servers. Will follow up about that shortly.

[sebsto]

Thank you @Joel-Bell-okcupid to confirm it works for you (and again kudos to @toheebster for unlock the last step !)

I understand your concern about other items that might be present in the Login keychain.

1. do not use the login keychain, use a temporary keychain instead, it will preserve your existing items in Login keychain. To do so, just use another name here ->

# do not use 
KEYCHAIN_NAME=login.keychain

# use this instead
KEYCHAIN_NAME=build.keychain

2. If you use disposable EC2 Mac instances (pets vs cattle) accessible only through SSH, there should be no Login keychain at all :-) but I understand it can takes time to get to that level of automation.

[Joel-Bell-okcupid]

@sebsto I had tried that (i.e., using a different keychain name) and ran into the 25308 error again.

I think it has to do with the use of sudo in sudo security list-keychains -s "${KEYCHAIN_NAME}" "${EXISTING_KEYCHAINS[@]}". I think I need to update the server machine to allow use of sudo in my script w/o a password.

Since a password wasn't provided, the new keychain wasn't added to the list of keychains.

[sebsto]

@Joel-Bell-okcupid I just tested again with a non default keychain name (I used dev.keychain) and indeed it fails. I don't really understand why.

Error: other("Failed to save credentials for \'https://stormacq-test-486652066693.d.codeartifact.us-west-2.amazonaws.com/\' to keychain: status -60008")

The code causing the error is here, but it does not specify a keychain name, it relies on the standard keychain list mechanism to locate one and the dev.keychain is the first one in the list.

% security list-keychains                                      
    "/Users/ec2-user/Library/Keychains/dev.keychain-db"
    "/Library/Keychains/System.keychain"

Mystery for the moment, but I stop working on that :slightly_smiling_face: