neondatabase / neon

Neon: Serverless Postgres. We separated storage and compute to offer autoscaling, code-like database branching, and scale to zero.
https://neon.tech
Apache License 2.0
14.28k stars 408 forks source link

Changes to `neon.max_cluster_size` are not applied until restart #7628

Closed jcsp closed 2 months ago

jcsp commented 4 months ago

Internal thread: https://neondb.slack.com/archives/C04DGM6SMTM/p1715008657801279

Per Anastasia, this property is not handled during live config updates.

That's awkward, because if someone has a running compute and their max size has been increased, they could still run into their prior limit.

lubennikovaav commented 2 months ago

Tested it once again, When project's neon.max_cluster_size setting is changed via Admin UI, there is indeed no immediate effect on the endpoint.

https://api-docs.neon.tech/reference/updateprojectendpoint To apply this setting without endpoint restart, one can use apply_config operation with empty spec:

export PROJECT_ID=
export ENDPOINT_ID=
export NEON_CLIENT_API_KEY=

curl --request PATCH \
     --url https://console.neon.tech/api/v2/projects/$PROJECT_ID/endpoints/$ENDPOINT_ID \
     --header 'accept: application/json' \
     --header 'authorization: Bearer $NEON_CLIENT_API_KEY' \
     --header 'content-type: application/json' \
     --data '
{
  "endpoint": {
    "settings": {
      "pg_settings": {
      }
    }
  }
}
'

CAUTION: if endpoint had any non-default per-endpoint pg_settings, this will override them. please, correct me, if I'm wrong.

@ololobus , wdyt about adding new field to this API to explicitly force endpoint to recollect all settings from cplane database and apply them? something like this:

--data '
{
  "endpoint": {
    "reload": true
    }
}
ololobus commented 2 months ago

https://console.neon.tech/api/v2/projects/$PROJECT_ID/endpoints/$ENDPOINT_ID

This is cplane part of the public API, so users should not be allowed to change neon.max_cluster_size , for everything else it should reload the config

Regarding explicit reload, I think that API has to handle updates properly and reload automatically when it's needed

For manual config reload there is another issue https://github.com/neondatabase/cloud/issues/7931

lubennikovaav commented 2 months ago

I didn't know about https://github.com/neondatabase/cloud/issues/7931

I'll move the api discussion there. Do you think we can close this as duplicate?

lubennikovaav commented 2 months ago

This is cplane part of the public API, so users should not be allowed to change neon.max_cluster_size , for everything else it should reload the config

yes, I described a hack to trigger pg_settings reload for the endpoint, because we don't have a proper way to do that. users cannot set neon.max_cluster_size in the endpoint settings, but they can force apply_config to pick up changes in settings, i.e. made by support (see this message: https://neondb.slack.com/archives/C04DGM6SMTM/p1715012142945999?thread_ts=1715008657.801279&cid=C04DGM6SMTM)

ololobus commented 2 months ago

Yeah, let's close this in favor of https://github.com/neondatabase/cloud/issues/7931. It's also a different repo