go-mysql-org / go-mysql-elasticsearch

Sync MySQL data into elasticsearch
MIT License
4.11k stars 798 forks source link

How to set river.toml ,and parse the json format . #54

Open qsbaq opened 8 years ago

qsbaq commented 8 years ago

Hi: I have a mysql table like this: table name: torrent0

field infohash : 0F850D075A59522FD975404CF6D205530F066243

field data : {"Infohash":"0F850D075A59522FD975404CF6D205530F066243","Name":"John.Wick.2014.DUAL.BDRip-AVC.x264.mkv","Length":2484494781,"Heat":0,"FileCount":1,"Files":[{"Name":"John.Wick.2014.DUAL.BDRip-AVC.x264.mkv","Length":2484494781}],"CreateTime":"2016-08-23T03:17:55.97859726-04:00"}

field create_time : 2016-08-23 03:17:55

Now how to do the river.toml to set the elasticsearch like this .

curl -XGET 10.25.166.191:9200/torrent/_search?pretty

{
"_index" : "torrent", "_type" : "0", "_id" : "0F850D075A59522FD975404CF6D205530F066243", "_score" : 1.0, "_source" : { "Name" : "Beshenye.2015.P.HDRip.1400MB.nnmclub.avi", "Length" : 1467906048, "Heat" : 0, "CreateTime" : "2016-09-02T14:40:53.779099802+08:00" } },

I've tryed setted this below: [rule.field] _id="infohash,string" _source="data,list" Result: { "_index" : "torrent", "_type" : "0", "_id" : "800910", "_score" : 1.0, "_source" : { "create_time" : "2016-08-23 03:17:55", "data" : "{\"Infohash\":\"02A5B835DF1831C808CDDBDEEBBC6CBBA2AC3478\",\"Name\":\"Tokyo_MarbleChocolate(2007)[720p,BluRay,x264]-_THORA\",\"Length\":1049992650,\"Heat\":0,\"FileCount\":5,\"Files\":[{\"Name\":\"Tokyo_Marble_Chocolate_part1_MataAimashou[720p,BluRay,x264]_-_THORA.mkv\",\"Length\":525121923},{\"Name\":\"Tokyo_Marble_Chocolate_part2_ZenryokuShounen[720p,BluRay,x264]_-_THORA.mkv\",\"Length\":524839477},{\"Name\":\"Tokyo_MarbleChocolate(2007)[720p,BluRay,x264]-_THORA.nfo\",\"Length\":16634},{\"Name\":\"한글 자막 - 보실 때 압축풀고 압축파일은 보관해주세요.7z\",\"Length\":14395},{\"Name\":\"Tokyo_MarbleChocolate(2007)[720p,BluRay,x264]-_THORA.md5\",\"Length\":221}],\"CreateTime\":\"2016-08-23T03:17:55.967615474-04:00\"}", "id" : 800910, "infohash" : "02A5B835DF1831C808CDDBDEEBBC6CBBA2AC3478" } } ......

siddontang commented 8 years ago

Here go-mysql-elasticsearch can only sync your data from MySQL table to ES index with same table field name (or you can define a mapping for it).