howto attach one mysql server to another with seed\receive data over orchestrator-agent?

[Slach]

i just start two separate empty mysql 5.7 servers on separate hosts mysql1 and mysql2, and try attach mysql2 as slave into mysql1 and see following messages:

    mysql-proxy: ++ orchestrator -c discover -i mysql1
    mysql-proxy: mysql1:3306
    mysql-proxy: ++ orchestrator -c discover -i mysql2
    mysql-proxy: mysql2:3306
    mysql-proxy: ++ orchestrator -c relocate -i mysql2 -d mysql1
    mysql-proxy: 2018-03-12 15:56:59 ERROR Relocating mysql2:3306 below mysql1:3306 turns to be too complex; please do it manually
    mysql-proxy: 2018-03-12 15:56:59 FATAL 2018-03-12 15:56:59 ERROR Relocating mysql2:3306 below mysql1:3306 turns to be too complex; please do it manually

i setup orchestrator-agent on mysql hosts and use following /etc/orchestrator.conf.json

{
  "BackendDB": "sqlite",
  "SQLite3DataFile": "/usr/local/orchestrator/orchestrator.sqlite3",
  "MySQLTopologyUser": "orchestrator",
  "MySQLTopologyPassword": "orch_topology_password",
  "AutoPseudoGTID": true,
  "ServeAgentsHttp": true
}

and use following /etc/ochestrator-agent.conf.json

{
  "SnapshotMountPoint": "/tmp",
  "AgentsServer": "http://mysql-proxy",
  "AgentsServerPort": ":3001",
  "ContinuousPollSeconds" : 60,
  "ResubmitAgentIntervalMinutes": 60,
  "CreateSnapshotCommand": "echo 'no action'",
  "AvailableLocalSnapshotHostsCommand": "echo 127.0.0.1",
  "AvailableSnapshotHostsCommand": "echo localhost\n127.0.0.1",
  "SnapshotVolumesFilter": "-my-snapshot-",
  "MySQLDatadirCommand": "mysql -B -e 'SELECT @@datadir'",
  "MySQLPortCommand": "echo '3306'",
  "MySQLDeleteDatadirContentCommand": "export DATADIR=$(mysql -B -e 'SELECT @@datadir'); systemctl mysql stop; rm --preserve-root -rf $DATADIR/*",
  "MySQLServiceStopCommand":      "systemctl mysql stop",
  "MySQLServiceStartCommand":     "systemctl mysql start",
  "MySQLServiceStatusCommand":    "systemctl mysql status",
  "ReceiveSeedDataCommand":       "/usr/local/orchestrator-agent/mysql-receive-data.sh",
  "SendSeedDataCommand":          "/usr/local/orchestrator-agent/mysql-send-data.sh",
  "PostCopyCommand":              "/usr/local/orchestrator-agent/mysql-post-receive.sh",
  "HTTPPort": 3002,
  "HTTPAuthUser": "",
  "HTTPAuthPassword": "",
  "UseSSL": false,
  "SSLCertFile": "",
  "SSLPrivateKeyFile": "",
  "HttpTimeoutSeconds": 10,
  "ExecWithSudo": true,
  "CustomCommands": {
    "true": "/bin/true"
  },
  "TokenHintFile": ""
}

how i can use orchestrator-cli or orchestrator-client or direct api calls for CLONE mysql1 to mysql2 and attach mysql2 to mysql1 as a slave?

[shlomi-noach]

@Slach with regard the error message:

mysql-proxy: 2018-03-12 15:56:59 FATAL 2018-03-12 15:56:59 ERROR Relocating mysql2:3306 below mysql1:3306 turns to be too complex; please do it manually

The reason orchestrator says that is that there's no prior connection between the boxes. Are you running GTID, by the way? Or Pseudo-GTID?

With regard cloning:

https://github.com/github/orchestrator-agent/ is a side-project I haven't touched in a while, and at this time it isn't on our focus. At the time, it was created to run LVM based snapshots. If you have LVM running, then you're in luck.

Otherwise, we leave backup/restore/cloning to your own infrastructure. and orchestrator avoids the question. You should use your own infra to boostrap your initial topology.

If you're unsure how to do that, well, it depends whether you're using GTID or not. Setting up replication for the first time is easy if your databases are not populated, but out of scope of this issue.

[Slach]

https://github.com/github/orchestrator-agent/ is a side-project I haven't touched in a while

yep i see could you merge my pull request ? https://github.com/github/orchestrator-agent/pull/18

time it isn't on our focus. At the time, it was created to run LVM based snapshots. If you have LVM running, then you're in luck.

could orchestrator-agent use custom commands for seeding data? i plan use xtrabackup + nc for transfer database between host it's theoretically possible?

[shlomi-noach]

Yep. There's a /api/custom-commands/:cmd API endpoint, though I have to leave it to you to explore. This code was contributed by SquareUp.

orchestrator-agent already supports LVM+nc, so adding extrabackup is certainly possible.

You may go the path you suggested, by custom commands, or, if you're interested in implementing real xtrabackup support to orchestrator-agent, I'm happy.

[Slach]

also i created new PR in orchestrator-agent @shlomi-noach could you merge this PR? https://github.com/github/orchestrator-agent/pull/19

[Slach]

now, i have test stand on my fork
it worked "data seed process" over innobackupex + netcat see https://github.com/github/orchestrator/pull/445