Closed chandanbansal closed 9 years ago
Hi Chandan,
This looks like more in the OR library.
Is the version of MySQL 5.6.4 or higher ? Can you please check if it has a fractional part http://dev.mysql.com/doc/refman/5.6/en/datetime.html ?
You may consider submitting a patch to OpenReplicator
https://code.google.com/p/open-replicator/
Thanks Phani
On Thu, Apr 3, 2014 at 8:34 AM, Chandan Bansal notifications@github.comwrote:
open replicator is working fine with mysql 5.5 But not able to parse Timestamp in 5.6. As there are some changes in bin log for timestamp in 5.6. Is there any plan to update that.
Reply to this email directly or view it on GitHubhttps://github.com/linkedin/databus/issues/25 .
Thanks for the reply Phanindra, Yes, i am using 5.6.9 on my local machine. will update OpenReplicator and as per Mysql changes and will submit the patch.
canal is working fine with mysql 5.5 & mysql5.6 & mariadb 5/10.
canal is a opensource by alibaba corp , a simplified version of databus.
canal : https://github.com/alibaba/canal
we already fixed OpenReplicator for same and release the patch ASAP. will also look into canal & otter
Thanks for the note. I went through the wiki for Canal ( thanks for adding notes in English as well ).
Agapple / Chandan,
If possible, please share some thoughts on Canal vs OpenReplicator. Is
there a wiki page for otter ?
Thanks Phani
On Tue, Apr 15, 2014 at 8:33 PM, agapple notifications@github.com wrote:
canal is working fine with mysql 5.5 & mysql5.6 & mariadb 5/10.
canal : https://github.com/alibaba/canal
— Reply to this email directly or view it on GitHubhttps://github.com/linkedin/databus/issues/25#issuecomment-40559708 .
Are you able to share the O/R patch here ?
Thanks Phani
On Wed, Apr 16, 2014 at 12:44 AM, Chandan Bansal notifications@github.comwrote:
we already fixed OpenReplicator for same and release the patch ASAP. will also look into canal & otter
— Reply to this email directly or view it on GitHubhttps://github.com/linkedin/databus/issues/25#issuecomment-40571782 .
otter wiki : https://github.com/alibaba/otter/wiki
what is canal & otter ?
canal vs O/R :
2014-04-16 23:41 GMT+08:00 Phanindra Ganti notifications@github.com:
Thanks for the note. I went through the wiki for Canal ( thanks for adding notes in English as well ).
Agapple / Chandan,
If possible, please share some thoughts on Canal vs OpenReplicator. Is there a wiki page for otter ?
Thanks Phani
On Tue, Apr 15, 2014 at 8:33 PM, agapple notifications@github.com wrote:
canal is working fine with mysql 5.5 & mysql5.6 & mariadb 5/10.
canal : https://github.com/alibaba/canal
— Reply to this email directly or view it on GitHub< https://github.com/linkedin/databus/issues/25#issuecomment-40559708> .
— Reply to this email directly or view it on GitHubhttps://github.com/linkedin/databus/issues/25#issuecomment-40614545 .
Hi, Agapple !
Thanks for sharing the links. I took a cursory look at the
code-base, and was able to get some context on parts of binlog processing (dbsync) and client-side event processing ( example, sink directories ). I still have a couple of gaps in my understanding, which I am listing below.
Are there guarantees about timeline consistency ( or any form of eventual consistency ) ? In other words, can we guarantee that a client never misses events when failover happens ?
The timeline seems to be (serverId, timestampAtServer, start(logOffset ?)) based on LogEvent.java. If that is the case, when failover happens to another node, we are not guaranteed that we will have a monotically increasing timeline for subsequent events on that particular database.
Very interested to learn more about (1) how quick random lookups from within the event buffer (2) How efficient streaming (i.e., contiguous seek from within a buffer ) is
If I understand correctly, (2) is quick, but (1) may be slow, based on implementation in "store" directory.
Cheers Phani
On Thu, Apr 17, 2014 at 1:53 AM, agapple notifications@github.com wrote:
otter wiki : https://github.com/alibaba/otter/wiki
what is canal & otter ?
canal vs O/R :
- O/R just an binlog parse lib, not even deal with encoding .
- canal more like a product ,server/client mode, support server HA / client HA / mysql switch master&slave etc.
2014-04-16 23:41 GMT+08:00 Phanindra Ganti notifications@github.com:
Thanks for the note. I went through the wiki for Canal ( thanks for adding notes in English as well ).
Agapple / Chandan,
If possible, please share some thoughts on Canal vs OpenReplicator. Is there a wiki page for otter ?
Thanks Phani
On Tue, Apr 15, 2014 at 8:33 PM, agapple notifications@github.com wrote:
canal is working fine with mysql 5.5 & mysql5.6 & mariadb 5/10.
canal : https://github.com/alibaba/canal
— Reply to this email directly or view it on GitHub< https://github.com/linkedin/databus/issues/25#issuecomment-40559708> .
— Reply to this email directly or view it on GitHub< https://github.com/linkedin/databus/issues/25#issuecomment-40614545> .
— Reply to this email directly or view it on GitHubhttps://github.com/linkedin/databus/issues/25#issuecomment-40693982 .
hi phanindraganti https://github.com/phanindraganti : my native language is not English, so ppt/wiki is not available at the moment English version.
2014-04-19 8:08 GMT+08:00 Phanindra Ganti notifications@github.com:
Hi, Agapple !
Thanks for sharing the links. I took a cursory look at the code-base, and was able to get some context on parts of binlog processing (dbsync) and client-side event processing ( example, sink directories ). I still have a couple of gaps in my understanding, which I am listing below.
- Are there guarantees about timeline consistency ( or any form of eventual consistency ) ? In other words, can we guarantee that a client never misses events when failover happens ?
The timeline seems to be (serverId, timestampAtServer, start(logOffset ?)) based on LogEvent.java. If that is the case, when failover happens to another node, we are not guaranteed that we will have a monotically increasing timeline for subsequent events on that particular database.
- Very interested to learn more about (1) how quick random lookups from within the event buffer (2) How efficient streaming (i.e., contiguous seek from within a buffer ) is
If I understand correctly, (2) is quick, but (1) may be slow, based on implementation in "store" directory.
- Also, otter seems like a library to handle client side functionality - including usecases like ETL. If able to share details about how incremental ingestion from these sources to offline processing (Hadoop?) happens, will be very interesting.
- Any chance the presentations at https://github.com/alibaba/otter/wiki/%E7%9B%B8%E5%85%B3ppt%26pdf are accessible in English ? I was able to translate most of the wiki pages into English, and there was a good deal of useful information there. Thanks !
Cheers Phani
On Thu, Apr 17, 2014 at 1:53 AM, agapple notifications@github.com wrote:
otter wiki : https://github.com/alibaba/otter/wiki
what is canal & otter ?
canal vs O/R :
- O/R just an binlog parse lib, not even deal with encoding .
- canal more like a product ,server/client mode, support server HA / client HA / mysql switch master&slave etc.
2014-04-16 23:41 GMT+08:00 Phanindra Ganti notifications@github.com:
Thanks for the note. I went through the wiki for Canal ( thanks for adding notes in English as well ).
Agapple / Chandan,
If possible, please share some thoughts on Canal vs OpenReplicator. Is there a wiki page for otter ?
Thanks Phani
On Tue, Apr 15, 2014 at 8:33 PM, agapple notifications@github.com wrote:
canal is working fine with mysql 5.5 & mysql5.6 & mariadb 5/10.
canal : https://github.com/alibaba/canal
— Reply to this email directly or view it on GitHub< https://github.com/linkedin/databus/issues/25#issuecomment-40559708> .
— Reply to this email directly or view it on GitHub< https://github.com/linkedin/databus/issues/25#issuecomment-40614545> .
— Reply to this email directly or view it on GitHub< https://github.com/linkedin/databus/issues/25#issuecomment-40693982> .
— Reply to this email directly or view it on GitHubhttps://github.com/linkedin/databus/issues/25#issuecomment-40854832 .
we have paid great attention to LinkedIn databus, we have to solve business are the same. You automatically switch the recent and historical quene is worth using for reference.
2014-04-19 8:08 GMT+08:00 Phanindra Ganti notifications@github.com:
Hi, Agapple !
Thanks for sharing the links. I took a cursory look at the code-base, and was able to get some context on parts of binlog processing (dbsync) and client-side event processing ( example, sink directories ). I still have a couple of gaps in my understanding, which I am listing below.
- Are there guarantees about timeline consistency ( or any form of eventual consistency ) ? In other words, can we guarantee that a client never misses events when failover happens ?
The timeline seems to be (serverId, timestampAtServer, start(logOffset ?)) based on LogEvent.java. If that is the case, when failover happens to another node, we are not guaranteed that we will have a monotically increasing timeline for subsequent events on that particular database.
- Very interested to learn more about (1) how quick random lookups from within the event buffer (2) How efficient streaming (i.e., contiguous seek from within a buffer ) is
If I understand correctly, (2) is quick, but (1) may be slow, based on implementation in "store" directory.
- Also, otter seems like a library to handle client side functionality - including usecases like ETL. If able to share details about how incremental ingestion from these sources to offline processing (Hadoop?) happens, will be very interesting.
- Any chance the presentations at https://github.com/alibaba/otter/wiki/%E7%9B%B8%E5%85%B3ppt%26pdf are accessible in English ? I was able to translate most of the wiki pages into English, and there was a good deal of useful information there. Thanks !
Cheers Phani
On Thu, Apr 17, 2014 at 1:53 AM, agapple notifications@github.com wrote:
otter wiki : https://github.com/alibaba/otter/wiki
what is canal & otter ?
canal vs O/R :
- O/R just an binlog parse lib, not even deal with encoding .
- canal more like a product ,server/client mode, support server HA / client HA / mysql switch master&slave etc.
2014-04-16 23:41 GMT+08:00 Phanindra Ganti notifications@github.com:
Thanks for the note. I went through the wiki for Canal ( thanks for adding notes in English as well ).
Agapple / Chandan,
If possible, please share some thoughts on Canal vs OpenReplicator. Is there a wiki page for otter ?
Thanks Phani
On Tue, Apr 15, 2014 at 8:33 PM, agapple notifications@github.com wrote:
canal is working fine with mysql 5.5 & mysql5.6 & mariadb 5/10.
canal : https://github.com/alibaba/canal
— Reply to this email directly or view it on GitHub< https://github.com/linkedin/databus/issues/25#issuecomment-40559708> .
— Reply to this email directly or view it on GitHub< https://github.com/linkedin/databus/issues/25#issuecomment-40614545> .
— Reply to this email directly or view it on GitHub< https://github.com/linkedin/databus/issues/25#issuecomment-40693982> .
— Reply to this email directly or view it on GitHubhttps://github.com/linkedin/databus/issues/25#issuecomment-40854832 .
Thanks Agapple for sharing your thoughts !
Cheers Phani
On Sat, Apr 19, 2014 at 2:34 AM, agapple notifications@github.com wrote:
hi phanindraganti https://github.com/phanindraganti : my native language is not English, so ppt/wiki is not available at the moment English version.
- If the MySQL master crash, slave hasn't fully catch up with master binlog , this situation may lose data (can be considered to optimize semi-sync). In addition, the company DBA has a MySQL master/slave switching tools, will be the first to set the master to read-only, waiting for slave to complete after master binlog, the master/slave data consistency, set the slave can be written, only through the mainmanagement system to the business and canal , this switch is without loss of data.
- random search in the design of eventBuffer occurs only at the first time,subsequent get requests was a results based on the nearest search. At presentthe open source version also does not provide a persistent store implementation,has been doing similar to Kafka(a message queue system) WAL, can provide a plurality of subscription.
- Otter is mainly to solve the synchronization data across the china<->us, has its own specific scene, synchronization mechanism and MySQL will be different, consider the network factors, will use the batch. One of the biggest characteristics, using a simple and effective way to solve the data consistency of MySQL double activestructure. Before participated in Hadoop, the follow-up to have the opportunity totry
2014-04-19 8:08 GMT+08:00 Phanindra Ganti notifications@github.com:
Hi, Agapple !
Thanks for sharing the links. I took a cursory look at the code-base, and was able to get some context on parts of binlog processing (dbsync) and client-side event processing ( example, sink directories ). I still have a couple of gaps in my understanding, which I am listing below.
- Are there guarantees about timeline consistency ( or any form of eventual consistency ) ? In other words, can we guarantee that a client never misses events when failover happens ?
The timeline seems to be (serverId, timestampAtServer, start(logOffset ?)) based on LogEvent.java. If that is the case, when failover happens to another node, we are not guaranteed that we will have a monotically increasing timeline for subsequent events on that particular database.
- Very interested to learn more about (1) how quick random lookups from within the event buffer (2) How efficient streaming (i.e., contiguous seek from within a buffer ) is
If I understand correctly, (2) is quick, but (1) may be slow, based on implementation in "store" directory.
3. Also, otter seems like a library to handle client side functionality
including usecases like ETL. If able to share details about how incremental ingestion from these sources to offline processing (Hadoop?) happens, will be very interesting.
- Any chance the presentations at https://github.com/alibaba/otter/wiki/%E7%9B%B8%E5%85%B3ppt%26pdf are accessible in English ? I was able to translate most of the wiki pages into English, and there was a good deal of useful information there. Thanks !
Cheers Phani
On Thu, Apr 17, 2014 at 1:53 AM, agapple notifications@github.com wrote:
otter wiki : https://github.com/alibaba/otter/wiki
what is canal & otter ?
canal vs O/R :
- O/R just an binlog parse lib, not even deal with encoding .
- canal more like a product ,server/client mode, support server HA / client HA / mysql switch master&slave etc.
2014-04-16 23:41 GMT+08:00 Phanindra Ganti notifications@github.com:
Thanks for the note. I went through the wiki for Canal ( thanks for adding notes in English as well ).
Agapple / Chandan,
If possible, please share some thoughts on Canal vs OpenReplicator. Is there a wiki page for otter ?
Thanks Phani
On Tue, Apr 15, 2014 at 8:33 PM, agapple notifications@github.com wrote:
canal is working fine with mysql 5.5 & mysql5.6 & mariadb 5/10.
canal : https://github.com/alibaba/canal
— Reply to this email directly or view it on GitHub< https://github.com/linkedin/databus/issues/25#issuecomment-40559708>
.
— Reply to this email directly or view it on GitHub< https://github.com/linkedin/databus/issues/25#issuecomment-40614545> .
— Reply to this email directly or view it on GitHub< https://github.com/linkedin/databus/issues/25#issuecomment-40693982> .
— Reply to this email directly or view it on GitHub< https://github.com/linkedin/databus/issues/25#issuecomment-40854832> .
— Reply to this email directly or view it on GitHubhttps://github.com/linkedin/databus/issues/25#issuecomment-40865029 .
We have fixed open-replicator to support Mysql version 5.6. Below is the github link. https://github.com/Flipkart/open-replicator.
Support has been added for
Really sorry was busy with something didn't got time to look into canal & otter. OR Patch link is already shared by jagadeesh.
Jagadeesh,
Thanks for the link. There seem to be quite a few files that have been
changed in the branch. Are they are changes over 1.0.5 OopenReplicator ? Or is there just a diff that we can look at ?
--Phani
On Thu, May 1, 2014 at 12:13 AM, jagadeesh-huliyar <notifications@github.com
wrote:
We have fixed open-replicator to support Mysql version 5.6. Below is the github link. https://github.com/Flipkart/open-replicator.
Support has been added for
- Checksum.
- datetime2 and time2 datatypes
— Reply to this email directly or view it on GitHubhttps://github.com/linkedin/databus/issues/25#issuecomment-41886593 .
you can check diff for 06c87542..61ff163824
open replicator is working fine with mysql 5.5 But not able to parse Timestamp in 5.6. As there are some changes in bin log for timestamp in 5.6. Is there any plan to update open replicator.