[3.1.3] Default profiles' settings not loaded

Bug summary

When settings the current profile to ephemeral using the CLI:

prefect profile use ephemeral

And then importing the PREFECT_SERVER_ALLOW_EPHEMERAL_MODE setting, we can see that it is not set to the profile's value:

from prefect.settings import PREFECT_SERVER_ALLOW_EPHEMERAL_MODE

print(PREFECT_SERVER_ALLOW_EPHEMERAL_MODE.value())
# > False

The profile's default value of True was expected. This resulted in being unable to run the API locally (Prefect expected PREFECT_API_URL to be set) even though the ephemeral profile was selected.

Version info

Version:             3.1.3
API version:         0.8.4
Python version:      3.10.15
Git commit:          39b6028c
Built:               Tue, Nov 19, 2024 3:25 PM
OS/Arch:             darwin/arm64
Profile:             ephemeral
Server type:         unconfigured
Pydantic version:    2.7.4
Integrations:
  prefect-docker:    0.6.2

Additional context

I looked through the source code and noticed the priority system for loading profile settings, with init variables, env filter, dotenv filter, file secret, prefect.toml, pyproject.toml and profiles.toml.

I suggest that we either have another source for the default profiles' settings as the last priority, or that the ProfileSettingsTomlLoader._load_profile_settings method loads settings from the DEFAULT_PROFILES_PATH and then fill in missing values on the profile data from self.profiles_path = _get_profiles_path().

In this code region (between lines 146 and 148) https://github.com/PrefectHQ/prefect/blob/62b100162bed359b4eede533d302125ca4c85a3f/src/prefect/settings/sources.py#L146-L148

Have something like this:

# naive but straightforward implementation...
if not active_profile:
    return {}

default_profile_data = toml.load(DEFAULT_PROFILES_PATH)
default_profiles_data: dict = default_profile_data.get("profiles", {})

active_profile_default_settings: dict
setting: str
if active_profile_default_settings := default_profiles_data.get(active_profile):
    if active_profile not in profiles_data:
        profiles_data[active_profile] = active_profile_default_settings
    else:
        for setting, setting_val in active_profile_default_settings.items():
            if setting not in profiles_data[active_profile]:
                profiles_data[active_profile][setting] = setting_val

if active_profile not in profiles_data:
    return {}

I will happily work towards a PR if this makes sense to you. Thank you.

PrefectHQ / prefect