Netflix / dynomite

A generic dynamo implementation for different k-v storage engines
Apache License 2.0
4.2k stars 533 forks source link

Redis-Dynomite HA Configuration and testing Not able To achieve HA #770

Open nabajyoti123 opened 4 years ago

nabajyoti123 commented 4 years ago

Hi All, i have created a cluster using Redis-dynomite with below configurations.

ServerA Configuration dyn.yml

dyn_o_mite: datacenter: dc-b rack: rack1 dyn_listen: x.x.x.152:7379 dyn_seeds:

dyn_o_mite: datacenter: dc-b rack: rack1 dyn_listen: x.x.x.152:7380 dyn_seeds:

ServerB Configuration dyn.yml

dyn_o_mite: datacenter: dc-a rack: rack1 dyn_listen: y.y.y.69:7379 dyn_seeds:

dyn_o_mite: datacenter: dc-a rack: rack1 dyn_listen: y.y.y.69:7380 dyn_seeds:

Here im able to acchieve replication while all nodes are up . But not able to achieve HA .

Steps im following to achieve HA

making x.x.x.152:22122 x.x.x.152:22123 down (making serverA redis-server nodes down). feeding to y.y.y.69:8380 and y.y.y.69:8379 as key value pair(using serverB redis-server nodes to feed data to redis cluster ) making x.x.x.152:22122 x.x.x.152:22123 Up.(making serverA redis-server nudes up) checking feeded data in x.x.x.152:8380 and x.x.x.152:8379(trying to retrieve serverB feeded data in serverA redis nodes.)

But not able to retrieve same .please help me with this .thanks!!

nabajyoti123 commented 4 years ago

Hi @ipapapa @timiblossom please help me with this .

ipapapa commented 4 years ago

I am not following what you mean you are not achieving HA although you are achieving replication. By default, once you have many replicas if a replica is lost then you have another replica to failover. To bring the data back, you need a feature to do a warmup. This is part of Dynomite-manager.

nabajyoti123 commented 4 years ago

Hi @ipapapa Thanks for the response . we are using normal linux machine .I have installed redis,dynomite and dynomite-manger .as this link https://github.com/Netflix/dynomite-manager says we have to implement some interfaces to run dynomite-manager .

  1. can you provide me some example of implemented interfaces for normal linux machine ?

    2.can we use couchbase/mysql for token management ? if yes how ?

  2. what is the purpose of token management ?

  3. im not able to create dynomite-manager.war file to deploy in tomcat server, installed in linux machine . how do i do that?

  4. Searched so many places only found steps for working with AWS . please let me know the steps (like installation ,if some property/file change required for installing in normal linux environment with some example ,what all steps to be followed for warming-up a node using dynomite-manager )

Basically im trying to achieve warming-up a node with normal linux machine,redis,dynomite,dynomite-manager .if some other db required ,we can use couchbase/mysql .

please help me with this .struggling a lot . Thanks in advance!!

ipapapa commented 4 years ago
  1. See the AWS bindings for the interfaces for what you need to implement
  2. It depends on whether you are deploying Dynomite in a single region or many regions. For many regions, you need a system that would do global replication. For local region, you should be OK with MySQL/Couchbase. More info here.
  3. You can read about Token Management. Fundamentally, Dynomite uses a database to store static tokens. The Cassandra paper is a good reference to get started.
  4. There are instructions on creating a .war in here.
  5. Dynomite manager was developed against AWS because that is what we use at Netflix. You can change the AWS bindings.
nabajyoti123 commented 4 years ago

Hi @ipapapa @diegopacheco , im trying to implement IMembership interface . Followed AWSMembership class and trying to implement methods . So these are methods i need to implement and im neither using Amazon nor AWS . What exact implementation should i provide for these methods ? what is expected from this methods without Amazon and AWS ?for normal linux machine currently its giving me this exception .

A............................?

============================================================ 1) Error injecting constructor, java.lang.IllegalArgumentException: hostname cannot be null at com.netflix.nfsidecar.identity.InstanceIdentity.(InstanceIdentity.java:89) at com.netflix.nfsidecar.identity.InstanceIdentity.class(InstanceIdentity.java:52) while locating com.netflix.nfsidecar.identity.InstanceIdentity for the 4th parameter of com.netflix.dynomitemanager.FloridaServer.(FloridaServer.java:66) at com.netflix.florida.startup.FloridaModule.configure(FloridaModule.java:91) (via modules: com.google.inject.util.Modules$CombinedModule -> com.netflix.florida.startup.FloridaModule) while locating com.netflix.dynomitemanager.FloridaServer Caused by: java.lang.IllegalArgumentException: hostname cannot be null at com.amazonaws.util.AwsHostNameUtils.parseRegion(AwsHostNameUtils.java:79) at com.amazonaws.util.AwsHostNameUtils.parseRegionName(AwsHostNameUtils.java:59) at com.amazonaws.AmazonWebServiceClient.computeSignerByURI(AmazonWebServiceClient.java:277) at com.amazonaws.AmazonWebServiceClient.setEndpoint(AmazonWebServiceClient.java:229) at com.netflix.nfsidecar.aws.AWSMembership.getAutoScalingClient(AWSMembership.java:322)

2) Error injecting constructor, java.lang.IllegalArgumentException: hostname cannot be null at com.netflix.nfsidecar.identity.InstanceIdentity.(InstanceIdentity.java:89) at com.netflix.nfsidecar.identity.InstanceIdentity.class(InstanceIdentity.java:52) while locating com.netflix.nfsidecar.identity.InstanceIdentity for the 3rd parameter of com.netflix.dynomitemanager.dynomite.DynomiteStandardTuner.(DynomiteStandardTuner.java:44) at com.netflix.dynomitemanager.dynomite.DynomiteStandardTuner.class(DynomiteStandardTuner.java:27) while locating com.netflix.dynomitemanager.dynomite.DynomiteStandardTuner while locating com.netflix.nfsidecar.utils.ProcessTuner for the 2nd parameter of com.netflix.dynomitemanager.dynomite.DynomiteYamlTask.(DynomiteYamlTask.java:36) at com.netflix.dynomitemanager.dynomite.DynomiteYamlTask.class(DynomiteYamlTask.java:36) while locating com.netflix.dynomitemanager.dynomite.DynomiteYamlTask for the 6th parameter of com.netflix.dynomitemanager.FloridaServer.(FloridaServer.java:66) at com.netflix.florida.startup.FloridaModule.configure(FloridaModule.java:91) (via modules: com.google.inject.util.Modules$CombinedModule -> com.netflix.florida.startup.FloridaModule) while locating com.netflix.dynomitemanager.FloridaServer Caused by: java.lang.IllegalArgumentException: hostname cannot be null at com.amazonaws.util.AwsHostNameUtils.parseRegion(AwsHostNameUtils.java:79) at com.amazonaws.util.AwsHostNameUtils.parseRegionName(AwsHostNameUtils.java:59) at com.amazonaws.AmazonWebServiceClient.computeSignerByURI(AmazonWebServiceClient.java:277) at com.amazonaws.AmazonWebServiceClient.setEndpoint(AmazonWebServiceClient.java:229) at com.netflix.nfsidecar.aws.AWSMembership.getAutoScalingClient(AWSMembership.java:322)

B....................................?

bind(IMembership.class).to(AWSMembership.class); (what implementation should i provide in its implementation class without AWS Amazon ?for normal linux machine)

IMembership.java

/**

C....................................?

bind(ICredential.class).to(IAMCredential.class);

(can i provide null for bellow method ) ? what implementation without AWS Amazon ?for normal linux machine

ICredential.java public AWSCredentialsProvider getAwsCredentialProvider();

D....................................?

bind(ICredential.class).annotatedWith(Names.named("awsroleassumption")).to(AwsRoleAssumptionCredential.class);

(what implementation should i provide in its implementation class which will accept above binding ? without AWS Amazon ?for normal linux machine)

E....................................?

bind(InstanceEnvIdentity.class).to(AwsInstanceEnvIdentity.class); for this i have changed like bellow as suggested by @diegopacheco Will it work for local without Amazon without AWS for linux machine ?

AwsInstanceEnvIdentity.class

package com.netflix.nfsidecar.aws;

import com.netflix.nfsidecar.identity.InstanceEnvIdentity; import com.netflix.nfsidecar.instance.InstanceDataRetriever; import com.netflix.nfsidecar.instance.VpcInstanceDataRetriever;

/**

}