No codis-config binary in 3.0.3

[Roycohen]

Hi Guys,

I'm installing codis with GO 1.7 and I've successfully compiled it with all dependencies. First there are issues resolving something in the Makefile but I could managed to resolve it.

Now after everything is compiled I have the follwoing files under bin directory: drwxr-xr-x. 4 root root 4096 Jul 4 16:10 assets -rwxr-xr-x. 1 root root 15041943 Jul 4 16:10 codis-admin -rwxr-xr-x. 1 root root 16271124 Jul 4 16:10 codis-dashboard -rwxr-xr-x. 1 root root 8843404 Jul 4 16:10 codis-fe -rwxr-xr-x. 1 root root 8620289 Jul 4 16:10 codis-ha -rwxr-xr-x. 1 root root 10079739 Jul 4 16:10 codis-proxy -rwxr-xr-x. 1 root root 6312957 Jul 4 16:08 codis-server -rw-r--r--. 1 root root 151 Jul 4 16:08 version

Where and how I can compile the codis-config. Do I need it in this version?

[spinlock]

codis-config has been removed from codis3.x. Please use codis-dashboard and codis-fe instead. codis-dashboard is equivalent to codis-config --dashboard and codis-fe is the frond end of codis-dashboad.

You can find a bash script named demo.sh under folder scripts. It will setup a full featured codis-cluster for demo purpose.

[Roycohen]

Thanks you very much for the detailed answer. I will go over it and Let You Know how it is working for me.

Where I can find English Documentation for the new release? I'm eager to use codis for a long time and now is the time for me to make the switch :)

[Roycohen]

Do I have to install ZooKeeper and etcd?

[spinlock]

You can choose either one of them.

I prefer etcd for demo purpose. You can clone it from github, and compile it with script build.sh (can be found under its repo). Then the binary files etcd and etcdctl will be generated to bin folder. You can copy them to any where you want (e.g. $GOPATH/bin).

[Roycohen]

Great. I already manage to solve this. Now, What is the dashboard port. I can log in to several ports but I only see JSON.

In addition, what the demo config? And what is the use of etcd in this case? To manage to configurations?

[spinlock]

sorry, these's no english document for this release yet. so I wrote some comments for you.

#!/bin/bash

PATH=$PATH:`realpath ../bin`

# preapre step 1.
# choose etcd as default coordinator
for x in etcd codis-dashboard codis-proxy codis-admin codis-server; do
    which $x >/dev/null
    if [ $? -ne 0 ]; then
        echo "missing $x"
        exit 1
    fi
done

# prepare step 2.
# create a temp folder for demo (logs & configs)
rm -rf tmp; mkdir -p tmp && pushd tmp
if [ $? -ne 0 ]; then
    echo "pushd failed"
    exit 1
fi

trap "kill 0" EXIT SIGQUIT SIGKILL SIGTERM

# setup 1.
# run etcd with default configuration (listen on port 2379)
nohup etcd --name=codis-test &>etcd.log &
lastpid=$!
pidlist=$lastpid
echo "etcd.pid=$lastpid"

# setup 2.
# create 8 codis-server instances
#   + listen on port 16379 ~ 16386
for ((i=0;i<8;i++)); do
    let p="16379+i"
    nohup codis-server --port ${p} &>redis-${p}.log &
    lastpid=$!
    pidlist="$pidlist $lastpid"
    echo "codis-server-${p}.pid=$lastpid"
done

# setup 3.
# create 4 codis-proxy instances with default config
#   + listen on port 19000 ~ 19003 for redis-client
#   + listen on port 11080 ~ 11083 for codis-cluster (restful api)
#       - can be visited by codis-dashboard or codis-admin
#   + run `codis-proxy -h` for help
#       - default config can be generated by `codis-proxy --default-config`
for ((i=0;i<4;i++)); do
    let p1="11080+i"
    let p2="19000+i"
    cat > ${p1}.toml <<EOF
product_name = "codis-test"
product_auth = ""
proto_type = "tcp4"
admin_addr = "0.0.0.0:${p1}"
proxy_addr = "0.0.0.0:${p2}"
EOF
    nohup codis-proxy -c ${p1}.toml &>${p1}.log &
    lastpid=$!
    pidlist="$pidlist $lastpid"
    echo "proxy-${p1}x${p2}.pid=$lastpid"
done

# setup 4.
# create 1 codis-dashboard instance with default config (with etcd)
#   + listen on port 18080 for codis-fe
cat > dashboard.toml <<EOF
coordinator_name = "etcd"
coordinator_addr = "127.0.0.1:2379"
product_name = "codis-test"
product_auth = ""
admin_addr = "0.0.0.0:18080"
EOF

nohup codis-dashboard -c dashboard.toml &> dashboard.log &
lastpid=$!
pidlist="$pidlist $lastpid"
echo "dashboard.pid=$lastpid"

# setup 5.
# setup codis-fe for codis-dashboard
cat > codis.json <<EOF
[
    {
        "name": "codis-test",
        "dashboard": "127.0.0.1:18080"
    }
]
EOF

nohup ../../bin/codis-fe -d codis.json --listen 0.0.0.0:8080 &> fe.log &
lastpid=$!
pidlist="$pidlist $lastpid"
echo "fe.pid=$lastpid"

sleep 3

for pid in $pidlist; do
    ps -p $pid >/dev/null
    if [ $? -ne 0 ]; then
        echo "pid=$pid not found"
        exit 1
    fi
done

codis_admin() {
    codis-admin --dashboard=127.0.0.1:18080 $@
    if [ $? -ne 0 ]; then
        echo "codis-admin error: $@"
        exit 1
    fi
}

# setup 6.
# create 4 groups (group-1 ~ group-4)
for ((i=0;i<4;i++)); do
    let g="i+1"
    codis_admin --create-group --gid $g
done

# setup 7.
# 2 codis-server for each group (1master + 1slave)
for ((i=0;i<8;i++)); do
    let p="16379+i"
    let g="i/2+1"
    codis_admin --group-add --gid $g -x 127.0.0.1:${p}
done

# setup 8.
# add codis-proxy instances to codis-cluster
for ((i=0;i<4;i++)); do
    let p1="11080+i"
    codis_admin --create-proxy -x 127.0.0.1:${p1}
done

# setup 9.
# setup slots (auto-rebalance 1024 slots into 4 groups)
codis_admin --slot-action --interval=100
codis_admin --rebalance --confirm

echo done

# setup 10.
# all done!!
# please visit codis-fe (localhost:8080) with your browser
#    + once the auto-rebalace phase has been completed, you can run redis-benchmark or anything else you want

while true; do
    date
    sleep 60
done

[spinlock]

here's the architecture of codis3, https://raw.githubusercontent.com/CodisLabs/codis/release3.0/doc/pictures/architecture.png

we use zookeeper & etcd as external storage to save the configs of codis-clusters (such as slot mappings, proxy instances and group servers, etc .,)

[Roycohen]

@spinlock THANKS YOU VERY MUCH!!! You are much appreciated. I will continue with my tests and report if I have questions or issues.

Questions that pops out are:

1. Redis Client is the application layer client? PHP client or JAVA client or any client?
2. listen on port 11080 ~ 11083 for codis-cluster (restful api)?? Whta does that means? What is codis-cluster? Whta is the purpose of that.
3. Probably i need to connect to the proxy, so what is codis-cluster? in addition, if i have several proxies i need to make round-robin or any other method myself to decide to which proxy to connect? or there is any other method?

NOTE: I've just tried to use rename method from PHP client and I'm getting an error. "read error on connection" Is rename method is supported?

[Roycohen]

I'm getting this error when trying to use: codis-admin --dashboard=127.0.0.1:18080 --create-group --gid 1 2016/07/06 17:50:12 dashboard.go:394: [PANIC] call rpc create-group to dashboard 127.0.0.1:18080 failed

It seems like everything is fine but the codis-admin command doesn't work.

When trying to log in to web I get 403 forbidden. netstat -nap | grep 8080 tcp 0 0 :::18080 :::* LISTEN 21134/codis-dashboa tcp 0 0 :::8080 :::* LISTEN 21159/codis-fe tcp 0 0 ::ffff:192.168.56.4:8080 ::ffff:192.168.56.2:49978 ESTABLISHED 21159/codis-fe

[spinlock]

Hi @Roycohen

Q1: Yep, you can choose any client that implements the RESP protocol. And here is the list of the recommended clients from the official website. Please note that, codis only supports a subset of redis commands, here is the blacklist.

Q2: codis-proxy will open 2 different tcp ports, you can find their definitions from its config file: one for your client program; and the second one, namely admin_addr, can be used to communicate with codis-dashboard or codis-admin to share proxy's status or sync slot mappings, group servers, etc.,

Q3: codis-cluster means the codis-dashbard + codis-proxy(s) + codis-server(s). If you want to use multiple proxies, jodis may be your best choice. And we'll publish a go version of jodis as soon as possible.

Q4: please check the log of codis-dashboard and the detail of the panic message, it'll tell you some reasons. I think it was probably caused by 'group-1 already exists'.

[spinlock]

@Roycohen codis supports most of the redis commands, including pipeline. but it doesn't support transactions or any other command which will block the tcp connection. once the codis-proxy detects any invalid commands from client or any failure from backend codis-server, it'll reset the client's connection. please check your proxy's log file, it will tell you more information.

[Roycohen]

I have issues reloading dashboard if I don't delete the old data in etcd.

2016/07/10 11:23:35 topom.go:121: [ERROR] store: acquire lock of aniview-realtime-analytics failed
[error]: 105: Key already exists (/codis3/aniview-realtime-analytics/topom) [8]
    4   /usr/src/go/src/github.com/CodisLabs/codis/pkg/models/etcd/etcdclient.go:133
            github.com/CodisLabs/codis/pkg/models/etcd.(*EtcdClient).Create
    3   /usr/src/go/src/github.com/CodisLabs/codis/pkg/models/store.go:63
            github.com/CodisLabs/codis/pkg/models.(*Store).Acquire
    2   /usr/src/go/src/github.com/CodisLabs/codis/pkg/topom/topom.go:120
            github.com/CodisLabs/codis/pkg/topom.(*Topom).setup
    1   /usr/src/go/src/github.com/CodisLabs/codis/pkg/topom/topom.go:95
            github.com/CodisLabs/codis/pkg/topom.New
    0   /usr/src/go/src/github.com/CodisLabs/codis/cmd/dashboard/main.go:116
            main.main
        ... ...
[stack]:
    2   /usr/src/go/src/github.com/CodisLabs/codis/pkg/topom/topom.go:121
            github.com/CodisLabs/codis/pkg/topom.(*Topom).setup
    1   /usr/src/go/src/github.com/CodisLabs/codis/pkg/topom/topom.go:95
            github.com/CodisLabs/codis/pkg/topom.New
    0   /usr/src/go/src/github.com/CodisLabs/codis/cmd/dashboard/main.go:116
            main.main
        ... ...
2016/07/10 11:23:35 main.go:118: [PANIC] create topom with config file failed
coordinator_name = "etcd"
coordinator_addr = "127.0.0.1:2379"
admin_addr = "0.0.0.0:18080"
product_name = "aniview-realtime-analytics"
product_auth = ""
[error]: store: acquire lock of aniview-realtime-analytics failed
    2   /usr/src/go/src/github.com/CodisLabs/codis/pkg/topom/topom.go:122
            github.com/CodisLabs/codis/pkg/topom.(*Topom).setup
    1   /usr/src/go/src/github.com/CodisLabs/codis/pkg/topom/topom.go:95
            github.com/CodisLabs/codis/pkg/topom.New
    0   /usr/src/go/src/github.com/CodisLabs/codis/cmd/dashboard/main.go:116
            main.main
        ... ...
[stack]:
    0   /usr/src/go/src/github.com/CodisLabs/codis/cmd/dashboard/main.go:118
            main.main
        ... ...

[spinlock]

@Roycohen you can remove the lock node manually with command codis-admin --remove-lock.

[Roycohen]

@spinlock thank you very much. Do you know why I might have 403 forbidden? From the logs it seems like everything was initialized as it should.

[spinlock]

@Roycohen what's your error message?

It may be caused by some failures while loading the css resources. Please make sure that codis-fe and the folder assets are under the same directory and try again.

[Roycohen]

@spinlock, I think this is the issue. OK, After using the codis-fe in the binary the page loads.

Basically what I'm doing is: After compiling Codis I'm taking the binaries to the /usr/local/bin so I'm not using the codis-fe from the /usr/local/bin directory. So basically, all the binaries that I'm using are not in the bin folder under the Codis library. Is that something that might make things wrong?

Is there a way to make it work? In addition another several questions:

1. Are there any plans to upgrade to the latest Redis? What will happen if I'll use another version of Redis?
2. Are there plans to support the unsupported commands such as Rename and transaction or these commands are not supported due to a limitation in the infrastructure of Codis?
3. How can I iterate over keys if scan and keys are not supported?

BTW, after this long thread I will probably make a PR with Q&A So you guys will be able to use :)

[spinlock]

@Roycohen, I think this problem may be caused by running codis-fe without any path prefix.

Here's the source code and some comments for you:

    // calculate the location with your command line arguments
    binpath, err := filepath.Abs(filepath.Dir(os.Args[0]))
    if err != nil {
        log.PanicErrorf(err, "get path of binary failed")
    }
    // calculate the absolute path of `assets`
    assets := filepath.Join(binpath, "assets")

    fi, err := os.Stat(assets)
    if err != nil {
        log.PanicErrorf(err, "get stat of %s failed", assets)
    }

After putting those binary files to /usr/local/bin, you can run codis-fe without any path prefix. But the calculated value of binpath should be always the _current path_ which is wrong.

[Roycohen]

So I suggest to you guys to maybe add a '--assets-dir' argument where you will be able to decide the assets directory :) - It is pretty easy to add this.

Can yo please answer my questions above.. How can you iterate over keys. It is something that is really important in most applications.

[spinlock]

Thank you for your suggestions, I'll add this argument as soon as possible. 😄

[spinlock]

@Roycohen, SCAN is not supported by codis-proxy.

[Roycohen]

@spinlock So how do you suggest to iterate over keys? Is there a way or something you know.

It can't be that companies are using Codis without the ability to iterate over keys. How do you suggest doing so?

[Roycohen]

It will be great if you can share with me a thought on how to achieve this. I must iterate over the keys...

[Roycohen]

@spinlock, any comment? 👍

[springlie]

775

[Roycohen]

@springlie, thank you very much. If someone implemented it, why you are not adding it to the official release? How and where should i put it? And probably I need to recompile Codis again?

[Roycohen]

@springlie, please help me to integrate it. It is a must in our env. And you could please answer the above.

[spinlock]

@Roycohen it's very difficult to implement such command in codis. I'll consider about your requirements and suggestions, and I'll try to solve this problem in next release of codis3, but I can't give you any promise.

[Roycohen]

@spinlock thanks for the reply. I understand that you can't promise this kind of thing. But it is really a must feature in your amazing project. Seems like this is not a big issue for someone who knows the Codis system. Basically you need to iterate over the slots and then over the relevant redis's. Or you can just iterate over the redis servers and keeping a pointer in Codis where you stopped and from where to continue.. 💯

Will be awesome if you could add it to the next release. Even something that is not 100% optimized but stable will be enough for now.

BTW, You guys are doing a great work!

[spinlock]

Hi @Roycohen ,

Thank you very much for your suggestions.

Now Codis3.1 implemented a new command named 'SLOTSSCAN' which meets your requirements. This command has a similar style to 'SCAN': slotsscan slotnum cursor [COUNT count]. Here is the patch file 0001-slotsscan.patch, you can find more detail in it.

Please notice that dev3.1 is an unstable branch, and it will be released ASAP.