[Bug] - yum/dnf update always resolving to s3.dualstack.amazonaws url

[cloudmatt]

Describe the bug

I have a freshly deployed Amazon Linux 2023 instance that is placed in a VPC without direct internet access. The VPC has the s3 endpoint enabled with private DNS which should allow me to update this instance but whenever the yum update command is ran, it attempts to resolve a DNS record for a public facing IP address.

To Reproduce

1. Connect to amazon linux 2023 EC2 instance
2. Run yum literallyanything

Expected behavior I should be able to use a private s3 endpoint without internet access to download updates from the mirror.

Screenshots

[root@ip-10-4-16-11 ~]# cat /etc/yum.repos.d/amazonlinux.repo
[amazonlinux]
name=Amazon Linux 2023 repository
mirrorlist=https://al2023-repos-us-east-1-de612dc2.s3.us-east-1.amazonaws.com/core/mirrors/$releasever/$basearch/mirror.list
priority=10
enabled=1
repo_gpgcheck=0
type=rpm
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-amazon-linux-2023

** snipped for brevity **
[root@ip-10-4-16-11 ~]# yum makecache
^CAmazon Linux 2023 repository                                                                                                          [                                                                                       ===                                                     ] ---  B/s |   0  B     --:-- ET
Amazon Linux 2023 repository                                                                                                                                                                                                                                                            0.0  B/s |   0  B     01:52
Errors during downloading metadata for repository 'amazonlinux':
  - Curl error (28): Timeout was reached for https://al2023-repos-us-east-1-de612dc2.s3.dualstack.us-east-1.amazonaws.com/core/guids/bbefcd838a04819579bd2ca79d007905ca63d53b623c38b52e5fb51bf6aa91d6/x86_64/repodata/repomd.xml [Connection timeout after 30000 ms]
Error: Failed to download metadata for repo 'amazonlinux': Librepo was interrupted by a signal
Ignoring repositories: amazonlinux
Metadata cache created.

Desktop (please complete the following information):

• OS: amazon linux 2023

Additional context Even when hardcoding the mirrorlist to use the ipv4 private vpc endpoint, when it retrieves the list it always attempts to connect out to the public IP addressed mirror. As you can see in the text box up there, even when hardcoding the URL to not include the .dualstack URL component, it's still being retrieved by something in the repolist

[nmeyerhans]

The reason that changing the mirrorlist entry doesn't help is that the content at the location it references is itself a URL, and it will also be relative to the dualstack endpoints.

In order to work around the behavior you're seeing, you'll need to hardcode the baseurl value instead of the mirrorlist. You can find the baseurl value with dnf repolist -v. Be aware that the GUID references a specific version of the repository, though, so you'll need to update it any time you want to point to a new AL2023 release.

What's actually happening in your case is that the VPC DNS server overrides the IPv4-only S3 endpoint DNS names, but does not override the corresponding dualstack name. I don't know why that is and we'll need to chase that down with the right team internally. I don't think there's anything we can do to make this work by default within the distro, since we explicitly want to default to using the dualstack endpoints for our repositories.