search

yugabyte / yugabyte-db

YugabyteDB - the cloud native distributed SQL database for mission-critical applications.
https://www.yugabyte.com
Other
8.99k stars 1.07k forks source link

[docs] Document YW APIs #1581

Closed Arnav15 closed 3 years ago

Arnav15 commented 5 years ago

Write documentation for YW APIs.

RashmiPanthangi commented 5 years ago

Provide API documentation for:

  1. Universe management (Create, Edit and Delete)
  2. Enabling health checks
RashmiPanthangi commented 5 years ago
  1. Provide steps to directly call helm install calls for universe management without calling YW APIs
Arnav15 commented 5 years ago

The following APIs need to be used for universe management:

CONFIGURE This API is used to get the whole payload required for create universe.

Endpoint: POST <yugaware_ip>:9000/api/v1/customers/<customer_uuid>/universe_configure
Data Payload: {"currentClusterType":"PRIMARY","clusters":[{"clusterType":"PRIMARY","userIntent":{"universeName":"<name>","provider":"<provider_uuid>","regionList":["<list_of_region_uuid1>"],"assignPublicIP":true,"useTimeSync":false,"enableYSQL":false,"enableNodeToNodeEncrypt":false,"enableClientToNodeEncrypt":false,"numNodes":3,"instanceType":"small","ybSoftwareVersion":"1.2.12.0-b7","replicationFactor":3,"deviceInfo":{"volumeSize":100,"numVolumes":1,"mountPoints":null},"accessKeyCode":"yugabyte-default","masterGFlags":[{}],"tserverGFlags":[{}],"providerType":"kubernetes"}}],"clusterOperation":"CREATE","userAZSelected":false}

The response returned by configure API call contains all the payload to be sent to the Create call. You can modify placement by updating the AZ list in the payload and resending it to the configure API and it will update you with the new payload that should be submitted to the create call.

CREATE This API triggers the actual create universe call

Endpoint: POST <yugaware_ip>:9000/api/v1/customers/<customer_uuid>/universes
Data Payload: The response from configure call

EDIT Similar to create, update the payload you used for create according to what you want to edit: Change in AZs, adding/removing nodes or changing instance types. Send it to configure once again for verification and use the response from that call as the payload for the edit.

Endpoint: PUT <yugaware_ip>:9000/api/v1/customers/<customer_uuid>/universes/<universe_uuid>
Data payload: The response from configure call

DELETE

DELETE <yugaware_ip>:9000/api/v1/customers/<customer_uuid>/universes/<universe_uuid>?isForceDelete=true
sraghavan-eqix commented 5 years ago

For Limited Availability release, we will be requiring ability for Users to select different number of Nodes (Pods) in different availability zones. To start with we are supporting SV, DC and NY as the Availability Zones.

We will be requiring support at the API level to specify custom number of nodes on a per availabilityZone basis (Per Region). Refer to screenshot attached...

For example; User can select 2 nodes in SV, 3 in DC and 4 in NY.. How does this get translated to get to the API? Screen Shot 2019-07-29 at 11 56 41 AM

rkarthik007 commented 5 years ago

Hi @sraghavan-eqix,

While theoretically this is possible, this is not a recommended pattern because this inherently means that the three regions are not even in terms of usage. This can cause issues when there are regional outages. Can you please explain what use case you are trying to enable with this configuration?

For example - lets say the ny5 region fails, then the impact on the other regions is a lot more (because 4 nodes are lost) compared to the failure of the other regions. So the default orchestrator does not support this pattern.

sraghavan-eqix commented 5 years ago

Hi @rkarthik007

Referring to the Configure API posted by Arnav above, specifically the number Of Nodes parameter.

We need a way to map the number of Nodes to metros (availabilutyZone).

For example; if a customer chooses 6 as Number Of Nodes, does the default orchestrator assign 2 nodes to each Region (we have 3 regions to start with) based on number of Regions specified in the RegionList?

Trying to understand the relationship between regionsList and NumberOfNodes in the above API payload..

rkarthik007 commented 5 years ago

For example; if a customer chooses 6 as Number Of Nodes, does the default orchestrator assign 2 nodes to each Region (we have 3 regions to start with) based on number of Regions specified in the RegionList?

Yes, it will be 2 per region. The logic is to balance the distribution of nodes across regions evenly - so in a serial round-robin manner.

sraghavan-eqix commented 5 years ago

@rkarthik007 Thanks for the confirmation. I tried a bunch of scenarios to create Universe using the API payload above (Configure and Create). Case 1: Number of Nodes was 7 across 3 regions (SV,DC & NY). Universe Creation succeed with 3 nodes assigned to SV and 2 each to DC and NY. Replication Factor = 3

Case2: Number of Nodes was 6 across 2 Metros (SV and DC). Universe Creation Failed with below err; Replication Factor set to 3 in the Payload.

Failed to execute task java.util.concurrent.FutureTask@752ece56, hit error java.lang.RuntimeException: Error Server[YB Master - yb-master-1.yb-masters.yb-80707-svanddc-sv4.svc.sv-gcdb.edge.equinix.com:7100] INVALID_ARGUMENT[code 4]: Sum of minimum replicas per placement (6) is greater than num_replicas (3) while creating table redis.

What could be the reason?

Here is the Request Payload:

ConfigureAPI Request Payload: "currentClusterType": "PRIMARY", "clusters": [ { "clusterType": "PRIMARY", "userIntent": { "universeName": "svanddc", "provider": "890e5bf2-f483-4614-8937-d8e3805436c7", "regionList": [ "afb48419-1dbd-482d-bf5b-b41773a1c4f6", "dda42b30-5bb7-4e66-8fa2-5743d5f17cb8" ], "assignPublicIP": true, "useTimeSync": false, "enableYSQL": false, "enableNodeToNodeEncrypt": false, "enableClientToNodeEncrypt": false, "numNodes": 6, "instanceType": "Small", "ybSoftwareVersion": "1.2.11.0-b19", "replicationFactor": 3, "deviceInfo": { "volumeSize": 50, "numVolumes": 1, "mountPoints": null }, "accessKeyCode": "yugabyte-default", "masterGFlags": [ {} ], "tserverGFlags": [ {} ], "providerType": "kubernetes" } } ], "clusterOperation": "CREATE", "userAZSelected": false }'

Create Universe Request Payload:

{ "errorString": null, "deviceInfo": null, "universeUUID": "4f3945b1-e0d7-428a-923d-c38132cc176a", "expectedUniverseVersion": -1, "clusters": [ { "uuid": "028843db-2641-4bf2-a679-62ea4c01e1da", "clusterType": "PRIMARY", "userIntent": { "universeName": "svanddc", "provider": "890e5bf2-f483-4614-8937-d8e3805436c7", "providerType": "kubernetes", "replicationFactor": 3, "regionList": [ "afb48419-1dbd-482d-bf5b-b41773a1c4f6", "dda42b30-5bb7-4e66-8fa2-5743d5f17cb8" ], "preferredRegion": null, "instanceType": "Small", "numNodes": 6, "ybSoftwareVersion": "1.2.11.0-b19", "accessKeyCode": "yugabyte-default", "deviceInfo": { "volumeSize": 50, "numVolumes": 1, "diskIops": null, "storageClass": "standard", "mountPoints": null, "storageType": null }, "assignPublicIP": true, "useTimeSync": false, "enableYSQL": false, "enableNodeToNodeEncrypt": false, "enableClientToNodeEncrypt": false, "masterGFlags": {}, "tserverGFlags": {}, "instanceTags": {} }, "placementInfo": { "cloudList": [ { "uuid": "890e5bf2-f483-4614-8937-d8e3805436c7", "code": "kubernetes", "regionList": [ { "uuid": "afb48419-1dbd-482d-bf5b-b41773a1c4f6", "code": "Silicon Valley", "name": "SV", "azList": [ { "uuid": "b265d61f-b617-4add-a5a4-f42c9c6e78ce", "name": "sv4", "replicationFactor": 3, "subnet": null, "numNodesInAZ": 3, "isAffinitized": true } ] }, { "uuid": "dda42b30-5bb7-4e66-8fa2-5743d5f17cb8", "code": "DC", "name": "DC", "azList": [ { "uuid": "5e71cc73-b41c-435c-b9a6-8373bb5fc65b", "name": "dc3", "replicationFactor": 3, "subnet": null, "numNodesInAZ": 3, "isAffinitized": true } ] } ] } ] }, "index": 0 } ], "nodePrefix": "yb-80707-svanddc", "rootCA": null, "userAZSelected": false, "nodeDetailsSet": [ { "nodeIdx": 2, "nodeName": null, "nodeUuid": null, "cloudInfo": { "private_ip": null, "public_ip": null, "public_dns": null, "private_dns": null, "instance_type": "Small", "subnet_id": null, "az": "dc3", "region": "DC", "cloud": "kubernetes", "assignPublicIP": true, "useTimeSync": false }, "azUuid": "5e71cc73-b41c-435c-b9a6-8373bb5fc65b", "placementUuid": "028843db-2641-4bf2-a679-62ea4c01e1da", "state": "ToBeAdded", "isMaster": false, "masterHttpPort": 7000, "masterRpcPort": 7100, "isTserver": true, "tserverHttpPort": 9000, "tserverRpcPort": 9100, "isRedisServer": true, "redisServerHttpPort": 11000, "redisServerRpcPort": 6379, "isYqlServer": true, "yqlServerHttpPort": 12000, "yqlServerRpcPort": 9042, "isYsqlServer": true, "ysqlServerHttpPort": 13000, "ysqlServerRpcPort": 5433, "allowedActions": [ "DELETE" ] }, { "nodeIdx": 1, "nodeName": null, "nodeUuid": null, "cloudInfo": { "private_ip": null, "public_ip": null, "public_dns": null, "private_dns": null, "instance_type": "Small", "subnet_id": null, "az": "sv4", "region": "Silicon Valley", "cloud": "kubernetes", "assignPublicIP": true, "useTimeSync": false }, "azUuid": "b265d61f-b617-4add-a5a4-f42c9c6e78ce", "placementUuid": "028843db-2641-4bf2-a679-62ea4c01e1da", "state": "ToBeAdded", "isMaster": false, "masterHttpPort": 7000, "masterRpcPort": 7100, "isTserver": true, "tserverHttpPort": 9000, "tserverRpcPort": 9100, "isRedisServer": true, "redisServerHttpPort": 11000, "redisServerRpcPort": 6379, "isYqlServer": true, "yqlServerHttpPort": 12000, "yqlServerRpcPort": 9042, "isYsqlServer": true, "ysqlServerHttpPort": 13000, "ysqlServerRpcPort": 5433, "allowedActions": [ "DELETE" ] }, { "nodeIdx": 5, "nodeName": null, "nodeUuid": null, "cloudInfo": { "private_ip": null, "public_ip": null, "public_dns": null, "private_dns": null, "instance_type": "Small", "subnet_id": null, "az": "sv4", "region": "Silicon Valley", "cloud": "kubernetes", "assignPublicIP": true, "useTimeSync": false }, "azUuid": "b265d61f-b617-4add-a5a4-f42c9c6e78ce", "placementUuid": "028843db-2641-4bf2-a679-62ea4c01e1da", "state": "ToBeAdded", "isMaster": false, "masterHttpPort": 7000, "masterRpcPort": 7100, "isTserver": true, "tserverHttpPort": 9000, "tserverRpcPort": 9100, "isRedisServer": true, "redisServerHttpPort": 11000, "redisServerRpcPort": 6379, "isYqlServer": true, "yqlServerHttpPort": 12000, "yqlServerRpcPort": 9042, "isYsqlServer": true, "ysqlServerHttpPort": 13000, "ysqlServerRpcPort": 5433, "allowedActions": [ "DELETE" ] }, { "nodeIdx": 3, "nodeName": null, "nodeUuid": null, "cloudInfo": { "private_ip": null, "public_ip": null, "public_dns": null, "private_dns": null, "instance_type": "Small", "subnet_id": null, "az": "sv4", "region": "Silicon Valley", "cloud": "kubernetes", "assignPublicIP": true, "useTimeSync": false }, "azUuid": "b265d61f-b617-4add-a5a4-f42c9c6e78ce", "placementUuid": "028843db-2641-4bf2-a679-62ea4c01e1da", "state": "ToBeAdded", "isMaster": false, "masterHttpPort": 7000, "masterRpcPort": 7100, "isTserver": true, "tserverHttpPort": 9000, "tserverRpcPort": 9100, "isRedisServer": true, "redisServerHttpPort": 11000, "redisServerRpcPort": 6379, "isYqlServer": true, "yqlServerHttpPort": 12000, "yqlServerRpcPort": 9042, "isYsqlServer": true, "ysqlServerHttpPort": 13000, "ysqlServerRpcPort": 5433, "allowedActions": [ "DELETE" ] }, { "nodeIdx": 4, "nodeName": null, "nodeUuid": null, "cloudInfo": { "private_ip": null, "public_ip": null, "public_dns": null, "private_dns": null, "instance_type": "Small", "subnet_id": null, "az": "dc3", "region": "DC", "cloud": "kubernetes", "assignPublicIP": true, "useTimeSync": false }, "azUuid": "5e71cc73-b41c-435c-b9a6-8373bb5fc65b", "placementUuid": "028843db-2641-4bf2-a679-62ea4c01e1da", "state": "ToBeAdded", "isMaster": false, "masterHttpPort": 7000, "masterRpcPort": 7100, "isTserver": true, "tserverHttpPort": 9000, "tserverRpcPort": 9100, "isRedisServer": true, "redisServerHttpPort": 11000, "redisServerRpcPort": 6379, "isYqlServer": true, "yqlServerHttpPort": 12000, "yqlServerRpcPort": 9042, "isYsqlServer": true, "ysqlServerHttpPort": 13000, "ysqlServerRpcPort": 5433, "allowedActions": [ "DELETE" ] }, { "nodeIdx": 6, "nodeName": null, "nodeUuid": null, "cloudInfo": { "private_ip": null, "public_ip": null, "public_dns": null, "private_dns": null, "instance_type": "Small", "subnet_id": null, "az": "dc3", "region": "DC", "cloud": "kubernetes", "assignPublicIP": true, "useTimeSync": false }, "azUuid": "5e71cc73-b41c-435c-b9a6-8373bb5fc65b", "placementUuid": "028843db-2641-4bf2-a679-62ea4c01e1da", "state": "ToBeAdded", "isMaster": false, "masterHttpPort": 7000, "masterRpcPort": 7100, "isTserver": true, "tserverHttpPort": 9000, "tserverRpcPort": 9100, "isRedisServer": true, "redisServerHttpPort": 11000, "redisServerRpcPort": 6379, "isYqlServer": true, "yqlServerHttpPort": 12000, "yqlServerRpcPort": 9042, "isYsqlServer": true, "ysqlServerHttpPort": 13000, "ysqlServerRpcPort": 5433, "allowedActions": [ "DELETE" ] } ], "updateInProgress": false, "updateSucceeded": true, "nextClusterIndex": 1, "importedState": "NONE", "capability": "EDITS_ALLOWED" }

stevebang commented 4 years ago

Endpoints mentioned in this too: #1578 and #1579