Feedback: Multi-Region Zone Config Extensions

[jeffswenson]

Jeffrey Swenson (JeffSwenson) commented:

Jeff Swenson (JeffSwenson) commented:

Page: https://cockroachlabs.com/docs/v23.1/zone-config-extensions.html

What is the reason for your feedback?

[*] Too complicated

Additional details

The ALTER LOCALITY syntax is custom to cockroachdb and is complicated. I think zone config extensions page would really benefit from concrete examples of how to use the commands. Currently, a docs reader would need to understand the auto generated syntax diagrams to formulate an ALTER LOCALITY query.

Jira Issue: DOC-8111

[exalate-issue-sync[bot]]

shannonb (shannonbradshaw) commented: Richard Loveland I think this is you, but feel free to reassign if not.

[exalate-issue-sync[bot]]

Richard Loveland (rmloveland) commented: thanks! took a look. was confused because I know we already had docs on this.

the detailed examples were removed in the recent SQL docs refactoring

previously we definitely had a bunch of examples on a separate page for {{ALTER LOCALITY}} - see https://github.com/cockroachdb/docs/pull/16151/files#diff-0c977bf8f59568c5748b1a4879fd4b362093def46102431483eb74bcb7275c0e

agree with Jeff that this is way too complex an operation to send the user to a syntax diagram for

did some additional digging and this is also related to https://cockroachlabs.atlassian.net/browse/DOC-8067 since we used to have more detailed examples on that statement’s page as well

[exalate-issue-sync[bot]]

Richard Loveland (rmloveland) commented: Ryan Kuo let me know if you disagree but I think this one is for you, we had some usage examples for this statement that were removed in https://github.com/cockroachdb/docs/pull/16151

[exalate-issue-sync[bot]]

shannonb (shannonbradshaw) commented: Thanks, Richard Loveland ! Ryan Kuo what’s your take?

[exalate-issue-sync[bot]]

Ryan Kuo (taroface) commented: I might be wrong, but the examples that were previously on the {{ALTER LOCALITY}} doc (https://github.com/cockroachdb/docs/blob/c9638fbc9be8b60b1d6040aec56f3683a0891174/v22.2/alter-locality.md ) were all moved to the subheading “Examples > Use Zone Config Extensions”: https://www.cockroachlabs.com/docs/stable/alter-database.html#use-zone-config-extensions .

Maybe the problem is that the examples aren’t obvious enough? The connection between the {{ALTER LOCALITY}} syntax and the examples is definitely stretched, now that we have many other subcommands documented on the same page.

• The subheading could have a different name that clearly relates to {{ALTER LOCALITY}} syntax.
• The line “For examples, see Use Zone Config Extensions.” under the {{### ALTER LOCALITY}} subheading could be more obvious and be in bold, or in a callout box.

[codingconcepts]

Hey all, I should qualify that I think the docs for this are perfectly fine...

My issue (which is what started this conversation), is that in Serverless, I can't reduce the number of replicas down to a number below the constraint of 6 regions.

In my scenario, I have 6 regions and I'm grouping them into super regions containing 2 regions each. I would like to for replication to occur only within those paired regions but it doesn't seem to be working.

I think this is more a bug than a docs issue.

[codingconcepts]

Here are the steps to replicate the problem:

1. Create a multi-region Serverless cluster (I'm using 6 regions but the problem surfaces with just 3)

2. Connect via a terminal and run the following SQL:

CREATE DATABASE example;
USE example;

ALTER DATABASE example SET PRIMARY REGION "aws-eu-central-1";
ALTER DATABASE example ADD REGION "aws-ap-south-1";
ALTER DATABASE example ADD REGION "aws-ap-southeast-1";
ALTER DATABASE example ADD REGION "aws-eu-west-1";
ALTER DATABASE example ADD REGION "aws-us-east-1";
ALTER DATABASE example ADD REGION "aws-us-west-2";

SET enable_super_regions = on;
ALTER DATABASE example ADD SUPER REGION us VALUES "aws-us-east-1", "aws-us-west-2";
ALTER DATABASE example ADD SUPER REGION eu VALUES "aws-eu-central-1", "aws-eu-west-1";
ALTER DATABASE example ADD SUPER REGION ap VALUES "aws-ap-south-1", "aws-ap-southeast-1";

CREATE TABLE member (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  full_name STRING NOT NULL
) LOCALITY REGIONAL BY ROW;

INSERT INTO member (full_name, crdb_region) VALUES
  ('a', 'aws-eu-central-1'),
  ('b', 'aws-ap-south-1'),
  ('c', 'aws-ap-southeast-1'),
  ('d', 'aws-eu-west-1'),
  ('e', 'aws-us-east-1'),
  ('f', 'aws-us-west-2');

SET enable_multiregion_placement_policy = on;
ALTER DATABASE example PLACEMENT RESTRICTED;

SELECT raw_config_sql FROM [SHOW ZONE CONFIGURATION FROM TABLE member];

SELECT replicas, replica_localities FROM [SHOW RANGES FROM TABLE member]
WHERE start_key NOT LIKE '%Prefix%';

Even after a long delay (and multiple interactions with the member table, the replicas are not what I'd expect for a placement-restricted, regional by row table:

{2,8,14} | {"region=aws-us-east-1,az=aws-us-east-1a,dns=cockroachdb-qln7k.cockroachdb.us-east-1.svc.cluster.local","region=aws-us-west-2,az=aws-us-west-2c,dns=cockroachdb-c9jd6.cockroachdb.us-west-2.svc.cluster.local","region=aws-eu-west-1,az=aws-eu-west-1b,dns=cockroachdb-h8rwd.cockroachdb.eu-west-1.svc.cluster.local"}

[rmloveland]

"exalate-issue-sync[bot]" @.***> writes:

Ryan Kuo (taroface) commented: I might be wrong, but the examples that were previously on the {{ALTER LOCALITY}} doc (https://github.com/cockroachdb/docs/blob/c9638fbc9be8b60b1d6040aec56f3683a0891174/v22.2/alter-locality.md ) were all moved to the subheading “Examples > Use Zone Config Extensions”: https://www.cockroachlabs.com/docs/stable/alter-database.html#use-zone-config-extensions

Thanks for clarifying Ryan. Guess I am not used to the new page layout and totally missed this - Kinda embarrassed tbh!

Maybe the problem is that the examples aren’t obvious enough?

• The subheading could have a different name that clearly relates to {{ALTER LOCALITY}} syntax.

Since we are jumping around via anchor links inside a much bigger page now, I would vote for this option to make it more obvious that this is referring to the same thing as ALTER LOCALITY

[rmloveland]

Rob Reid @.***> writes:

Here are the steps to replicate the problem:

1 Create a multi-region Serverless cluster (I'm using 6 regions but the problem surfaces with just 3)

2 Connect via a terminal and run the following SQL:

CREATE DATABASE example; USE example;

ALTER DATABASE example SET PRIMARY REGION "aws-eu-central-1"; ALTER DATABASE example ADD REGION "aws-ap-south-1"; ALTER DATABASE example ADD REGION "aws-ap-southeast-1"; ALTER DATABASE example ADD REGION "aws-eu-west-1"; ALTER DATABASE example ADD REGION "aws-us-east-1"; ALTER DATABASE example ADD REGION "aws-us-west-2";

SET enable_super_regions = on; ALTER DATABASE example ADD SUPER REGION us VALUES "aws-us-east-1", "aws-us-west-2"; ALTER DATABASE example ADD SUPER REGION eu VALUES "aws-eu-central-1", "aws-eu-west-1"; ALTER DATABASE example ADD SUPER REGION ap VALUES "aws-ap-south-1", "aws-ap-southeast-1";

CREATE TABLE member ( id UUID PRIMARY KEY DEFAULT gen_random_uuid(), full_name STRING NOT NULL ) LOCALITY REGIONAL BY ROW;

INSERT INTO member (full_name, crdb_region) VALUES ('a', 'aws-eu-central-1'), ('b', 'aws-ap-south-1'), ('c', 'aws-ap-southeast-1'), ('d', 'aws-eu-west-1'), ('e', 'aws-us-east-1'), ('f', 'aws-us-west-2');

SET enable_multiregion_placement_policy = on; ALTER DATABASE example PLACEMENT RESTRICTED;

SELECT raw_config_sql FROM [SHOW ZONE CONFIGURATION FROM TABLE member];

SELECT replicas, replica_localities FROM [SHOW RANGES FROM TABLE member] WHERE start_key NOT LIKE '%Prefix%';

Even after a long delay (and multiple interactions with the member table, the replicas are not what I'd expect for a placement-restricted, regional by row table:

{2,8,14} | {"region=aws-us-east-1,az=aws-us-east-1a,dns=cockroachdb-qln7k.cockroachdb.us-east-1.svc.cluster.local","region=aws-us-west-2,az=aws-us-west-2c,dns=cockroachdb-c9jd6.cockroachdb.us-west-2.svc.cluster.local","region=aws-eu-west-1,az=aws-eu-west-1b,dns=cockroachdb-h8rwd.cockroachdb.eu-west-1.svc.cluster.local"}

I'm not aware of whether we support using PLACEMENT RESTRICTED and super regions at the same time. I have a docs ticket to update our domiciling docs to say that we recommend super regions instead of PLACEMENT RESTRICTED for domiciling: https://cockroachlabs.atlassian.net/browse/DOC-7812

My quick intuition is that there may be an interaction between these features?

There is also the fact that PLACEMENT RESTRICTED doesn't work for databases set to REGION survivability (per https://www.cockroachlabs.com/docs/stable/multiregion-overview.html#super-regions)

However your repro doesn't change the survival goals so I guess it's not that.

Having all of the above I think this is probably a bit over my head. You might make more progress toward a solution with a post in the

multi-region or #kv slack channels

If this is not a bug, we should document whatever this specific behavior is. Please let us know what you find out!

[exalate-issue-sync[bot]]

Richard Loveland (rmloveland) commented: In the PR https://github.com/cockroachdb/docs/pull/18337 (to address https://cockroachlabs.atlassian.net/browse/DOC-9520 )

we now have updated the Zone Config Extensions page to link directly to the zone config examples in a section called Use Zone Config Extensions

Based on the original description of this issue, I think this issue can now be closed. Ryan Kuo what do you think? I was gonna close but it’s assigned to you so did not want to barge in and do something without asking