search

cloudnativecube / octopus

14 stars 2 forks source link

clickhouse operator调研 #81

Open Cas-pian opened 3 years ago

Cas-pian commented 3 years ago

候选项目：

  1. https://github.com/Altinity/clickhouse-operator
  2. https://github.com/mackwong/clickhouse-operator

测试发现第二个项目依赖于sensetime（商汤）的私有代码，无法下载，故放弃

Cas-pian commented 3 years ago

  1. 下载代码：git clone https://github.com/Altinity/clickhouse-operator，进入到该目录；

  2. 创建namespace：kubectl create ns ck

  3. 创建clickhosue operator： kubectl apply -f deploy/operator/clickhouse-operator-install.yaml，在kube-system里可以看到clickhouse operator的pod； image

  4. 准备两个operator相关镜像和ck镜像，步骤可参考这里；

  5. 准备单副本单实例的ClickhouseInstalltion CRD的yaml：

    apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
name: "volume-hostpath"
spec:
defaults:
templates:
  podTemplate: clickhouse-per-host-on-servers-with-ssd
configuration:
clusters:
  - name: local-storage
    layout:
      shardsCount: 1
templates:
podTemplates:
  # Specify Pod Templates with affinity

  - name: clickhouse-per-host-on-servers-with-ssd
    #zone:
    #  key: "disktype"
    #  values:
    #    - "ssd"
    podDistribution:
      - type: ClickHouseAntiAffinity
    spec:
      volumes:
        # Specify volume as path on local filesystem as a directory which will be created, if need be
        - name: local-path
          hostPath:
            path: /mnt/podvolume
            type: DirectoryOrCreate
      containers:
        - name: clickhouse-pod
          image: yandex/clickhouse-server:21.4.4.30
          volumeMounts:
            # Specify reference to volume on local filesystem
            - name: local-path
              mountPath: /var/lib/clickhouse

  6. 部署ck集群：kubectl -n ck apply -f example-01.yaml

  7. 验证CK是否可用：

    $ kubectl -n ck get svc
chi-volume-hostpath-local-storage-0-0   ClusterIP      None          <none>        8123/TCP,9000/TCP,9009/TCP      54m
clickhouse-volume-hostpath              LoadBalancer   10.1.41.160   <pending>     8123:32307/TCP,9000:31321/TCP   54m

    image 进入容器访问可以，在外面访问容器IP有点问题

Cas-pian commented 3 years ago

检查发现ck operator配置的用户都是通过k8s的configmap挂载到users.d下来实现的 image 这里有这几个影响：

  1. 生成的default账号只能用于在CK集群间访问，集群外无法访问，会报错DB::Exception: default: Authentication failed: password is incorrect or there is no user with such name，实际是网段限制了：DB::Exception: Connections from 10.0.0.13 are not allowed； image

  2. 配置文件只读，所以通过SQL去修改这些用户的权限是不能修改的（k8s的configmap是readonly，ck在users.xml里的用户都是readonly），关联issue1, issue2；

  3. 这种账号并不存储CK的存储上，迁移时需要注意到这一点（其实理解可以放到CK里存的)；

Cas-pian commented 3 years ago

mackwong有私有库忽略，这个ch-operator可以作为备选，我等会再调研下这个 cc @mdianjun

Cas-pian commented 3 years ago

搭建minio单机版：

# 跑minio server，数据挂出来
docker run -d --name minio-server -e MINIO_ROOT_USER=admin -e MINIO_ROOT_PASSWORD=iamadmin --network host --restart always -v $(pwd)/data:/data minio/minio:RELEASE.2021-04-22T15-44-28Z server --address=:9001 /data
# 配置client
mc alias set c4admin http://10.0.0.14:9001  admin iadmadmin
# 配置新用户
mc admin user add myminio just thisiskey
mc alias set c4 http://10.0.0.14:9001 just thisiskey
# 配置的policy，mycktest00.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
                "s3:*"
      ],
      "Resource": [
        "arn:aws:s3:::testck00/*"
      ]
    }
  ]
}

# 创建policy
mc admin policy add c4admin mycktest00 mycktest00.json
# 新用户创建bucket
mc mb c4/testck00
# 新用户测试权限
mc ls c4
[2021-04-26 10:11:49 CST]     0B testck00/

在ck server里新增配置如下：

    <storage_configuration>
     <disks>
        <s3>
            <type>s3</type>
            <endpoint>http://10.0.0.14:9001/testck00/ck-0/</endpoint>
            <access_key_id>just</access_key_id>
            <secret_access_key>thisiskey</secret_access_key>
            <connect_timeout_ms>10000</connect_timeout_ms>
            <request_timeout_ms>5000</request_timeout_ms>
            <retry_attempts>10</retry_attempts>
            <min_bytes_for_seek>1000</min_bytes_for_seek>
            <metadata_path>/var/lib/clickhouse/disks/s3/</metadata_path>
            <cache_enabled>true</cache_enabled>
            <cache_path>/var/lib/clickhouse/disks/s3/cache/</cache_path>
            <skip_access_check>false</skip_access_check>
        </s3>
   </disks>
   <policies>
     <s3>
       <volumes>
         <v1>
           <disk>s3</disk>
         </v1>
       </volumes>
     </s3>
   </policies>
  </storage_configuration>
Cas-pian commented 3 years ago
  1. 更新镜像后，重新部署到k8里： image
  2. 使用提供S3接口的minio作为存储建表，并写数： 
    
-- add local table
CREATE TABLE IF NOT EXISTS default.test on CLUSTER 'all-sharded' (
id     UInt32,
t     DateTime Codec(Delta, LZ4)
)
ENGINE = ReplicatedMergeTree('/clickhouse/tables/default/test/shard-{shard}', '{replica}')
PARTITION BY toYYYYMM(t)
ORDER BY (id, t)
SETTINGS index_granularity = 8192, storage_policy = 's3';

-- add distributed table CREATE TABLE IF NOT EXISTS default.test_d ON CLUSTER 'all-sharded' AS default.test ENGINE = Distributed('all-sharded', 'default', 'test', rand());

-- insert data INSERT INTO default.test_d(id, t) SELECT toUInt32(rand(1)%1000+1) AS id, toDateTime('2020-01-01 00:00:00') + rand(2)%(36002430)*12 as timestamp FROM numbers(10000);



1. 确认数据存储情况：
![image](https://user-images.githubusercontent.com/5690854/116022127-2ad9bb80-a67c-11eb-94b0-2b7d05ff67e8.png)
检查minio存储目录
![image](https://user-images.githubusercontent.com/5690854/116022244-65dbef00-a67c-11eb-890d-76e067d50d00.png)
Cas-pian commented 3 years ago

ch-operator 刚才测试了下，也可以跑，但是它有些代码并未完成（比如指定镜像、存储等等），容易panic，基本还在开发中。

image

Cas-pian commented 3 years ago

目前在ck上配置了一个新用户user: test, password: 1：

apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
  name: "volume-hostpath"
spec:
  defaults:
    templates:
      podTemplate: clickhouse-per-host-on-servers-with-ssd
  configuration:
    clusters:
      - name: local-storage
        layout:
          shardsCount: 2
    users:
      default/access_management: 1
      test/password: 1
      test/networks/ip: "::/0"
      test/access_management: 1
.......

使用这个用户去连接OK，并且和ZK和S3一起使用也可以打通。

Cas-pian commented 3 years ago

目前这个operator是一个实例用一个statefulset来实现的，比如2分片2副本，共有4个statefulset，按照官方说法是为了支持不同版本的ck以及相关的配置，参考：comment1, comment2

mdianjun commented 3 years ago

问题：k8s环境下，怎么实现ck server从零重建？

每个ck server的本地元数据是在写数据的过程中创建的，元数据保存的是ck data part文件与s3文件的映射关系；replica之间同步数据时也会创建元数据。如果ck server所在的机器宕了，那么这些元数据就不存在了；极端情况是所有ck server的机器全宕了，无法利用replica来恢复元数据。 一种办法是把元数据也保存到s3上，比如使用mount方式。

mdianjun commented 3 years ago

https://github.com/cloudnativecube/octopus/blob/master/deploy/clickhouse-operator.md

mdianjun commented 3 years ago

radondb-clickhouse on k8s