synch -c synch.yaml not working

[ericchuawc]

When I ran this

[eric@localhost mysetup]$ synch -c synch.yaml 
Traceback (most recent call last):
  File "/home/wenching/.local/bin/synch", line 11, in <module>
    sys.exit(cli())
  File "/home/wenching/.local/lib/python3.6/site-packages/synch/cli.py", line 78, in cli
    parse_args.run(parse_args)
AttributeError: 'Namespace' object has no attribute 'run'

If I look thru the codes, I believe it's expecting the command line parameters

parse_args = parser.parse_args()
parse_args.run(parse_args)

Any help? Thanks.

[long2ice]

You can install lastet version from dev branch or install 0.6.5 from pypi with python3.7

[ericchuawc]

I have installed Python3.7.8 on Centos 7 and ran this

pip3 install -e git+https://github.com/long2ice/synch.git@dev#egg=synch --user

But I got this

Obtaining synch from git+https://github.com/long2ice/synch.git@dev#egg=synch
  Cloning https://github.com/long2ice/synch.git (to dev) to ./src/synch
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/usr/lib64/python3.6/tokenize.py", line 452, in open
        buffer = _builtin_open(filename, 'rb')
    FileNotFoundError: [Errno 2] No such file or directory: '/home/eric/mysetup/src/synch/setup.py'

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /home/eric/mysetup/src/synch/

Any idea?

[long2ice]

Try install from pypi

[ericchuawc]

I managed to install 0.65 (but need to use Python3.8 - which I installed 3.8.4)

I also need to install this

pip3.8 install kakfa-python

But I still see this error

synch -c synch.yaml 
Usage: synch [OPTIONS] COMMAND [ARGS]...
Try 'synch -h' for help.

Error: Missing option '--alias'.

So the synch.yaml config can't work?

[long2ice]

As the log said, you missed option --alias, you can run synch --alias some_db, and you can ignore -c synch.yaml since synch.yaml is default.

[long2ice]

And you can run synch -h for help.

[ericchuawc]

I tried

synch --alias mysql_db

Usage: synch [OPTIONS] COMMAND [ARGS]...
Try 'synch -h' for help.

Error: Missing command.

In my synch.yaml file

source_dbs:
  - db_type: mysql
    alias: mysql_db # must be unique
    broker_type: redis # current support redis and kafka
    server_id: 2

[long2ice]

synch --alias mysql_db produce

[ericchuawc]

I managed to get the program running.

...
2020-07-16 17:27:11 - synch.reader.mysql:125 - DEBUG - success set binlog pos:bin.000051:719

But,

Do i need to create the database and tables in CH like below?

create database db01;

CREATE TABLE table1 ENGINE = MergeTree ORDER BY (Id) AS SELECT * FROM mysql('192.168.1.22:3306', 'db01', 'table1', 'chuser', 'pass');

When I ran this

clickhouse-client -h 127.0.0.1

and

show databases

the db01 isn't in there

[long2ice]

There is a config auto_create: true for database.

[ericchuawc]

I have that

...
databases:
  - database: db01
    # optional, default true, auto create database when database in clickhouse not exists
    auto_create: true
    tables:
      - table: tb01,tb02
        # optional, default false, if your table has decimal column with nullable, there is a bug with full $
        skip_decimal: false # set it true will replace decimal with string type.
        # optional, default true
...

Weird, how come it doesn't create the database hmm...

For table, what if I want all tables, do I left it blank?

- table:

or

- table: tb01,tb02,tb03 (all the 100s tables' names)

[long2ice]

- table: test1
- table: test2

[ericchuawc]

so if i have 100 tables to sync, i need to do this?

- table: test1
...
- table: test100

[long2ice]

Yes,that's it.