Bug: errors while connecting

[nickchomey]

Bug description

When I try to start conduit with a mysql pipeline, i get the following error:

2024-07-21T11:45:04+00:00 INF starting fetcher for table "wp_posts" component=plugin.standalone connector_id=mysql2:mysql plugin_name=conduit-connector-mysql
2024-07-21T11:45:04+00:00 INF source connector plugin successfully started component=connector.Source connector_id=mysql2:mysql
2024-07-21T11:45:04+00:00 INF plugin process exited component=plugin.standalone connector_id=mysql2:mysql id=58336 plugin=/home/nick/tool-dev/conduit-binary/connectors/conduit-connector-mysql
2024-07-21T11:45:04+00:00 INF source connector plugin successfully torn down component=connector.Source connector_id=mysql2:mysql
2024-07-21T11:45:04+00:00 ERR node stopped error="node mysql2:mysql stopped with error: source stream was stopped unexpectedly: error reading from source: read plugin error: snapshot complete" component=pipeline.Service node_id=mysql2:mysql stack=[{"file":"/home/runner/work/conduit/conduit/pkg/pipeline/lifecycle.go","func":"github.com/conduitio/conduit/pkg/pipeline.(*Service).runPipeline.func2","line":583},{"file":"/home/runner/work/conduit/conduit/pkg/pipeline/stream/source.go","func":"github.com/conduitio/conduit/pkg/pipeline/stream.(*SourceNode).Run","line":136},{"file":"/home/runner/work/conduit/conduit/pkg/pipeline/stream/source.go","func":"github.com/conduitio/conduit/pkg/pipeline/stream.(*SourceNode).Run.func1","line":87},{"file":"/home/runner/work/conduit/conduit/pkg/plugin/connector/standalone/v1/client.go","func":"github.com/conduitio/conduit/pkg/plugin/connector/standalone/v1.unwrapGRPCError","line":165}]

Steps to reproduce

my pipeline config yaml is as follows:

version: 2.2
pipelines:
  - id: mysql2
    # run pipeline on startup
    status: running
    description: >
      Mysql wp_posts and writing into file "example2.out".

    connectors:
      - id: mysql
        type: source
        # use the built-in file plugin as the source
        plugin: standalone:mysql@87234d0
        settings:
          # read data from example.in
          url: root:root@tcp(127.0.0.1:32784)/db
          tables: wp_posts

      - id: example2.out
        type: destination
        # use the built-in file plugin as the destination
        plugin: builtin:file
        settings:
          # write data to example.out
          path: ./example2.out
          # output the raw payload as a string
          sdk.record.format: template
          sdk.record.format.options: '{{ printf "%s" .Payload.After }}'

I took it from the Getting Started doc

The db is running in a docker container, but I can successfully read the binlogs from the host shell with this command:

./mariadb-binlog --read-from-remote-server --host=127.0.0.1 --port=32784 --user=root --password=root --database=db master.000006
/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=1*/;
/*!40019 SET @@session.max_delayed_threads=0*/;
/*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/;
DELIMITER /*!*/;
# at 4
#240721 11:29:10 server id 1  end_log_pos 256 CRC32 0x44454670  Start: binlog v 4, server v 10.11.8-MariaDB-ubu2204-log created 240721 11:29:10 at startup
ROLLBACK/*!*/;
BINLOG '
ZkWdZg8BAAAA/AAAAAABAAAAAAQAMTAuMTEuOC1NYXJpYURCLXVidTIyMDQtbG9nAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAABmRZ1mEzgNAAgAEgAEBAQEEgAA5AAEGggAAAAICAgCAAAACgoKAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAEEwQADQgICAoKCgFwRkVE
'/*!*/;
# at 256
#240721 11:29:10 server id 1  end_log_pos 299 CRC32 0x4794d920  Gtid lis

etc....

Version

built from current master branch source standalone:mysql@87234d0

[nickchomey]

I just tried connecting maxwell's daemon, and it worked seamlessly. It also connects to nats (which is the destination I want to use) with 1 config line.

https://maxwells-daemon.io/quickstart/

So, I have to figure that there's something just not working with this new mysql connector.

But I would very much prefer to use Conduit, for all sorts of reasons, so I'd be happy to help debug this - and even step debug it if you can point me in the right direction for how to do that (I'm generally familiar with step debugging and developing go applications, but not sure where to start with Conduit in particular).

[alarbada]

Hello @nickchomey. First of all, thanks for trying out the conduit mysql connector! What you are experiencing is an expected error.

The source connector works in 2 modes: snapshot and cdc. The error you see means that the snapshot was complete, and the cdc mode should start, but it is not yet fully merged. You can try it out at this branch: https://github.com/conduitio-labs/conduit-connector-mysql/tree/consistent-snapshot

The cdc mode is coming soon, probably this week.

[nickchomey]

Ah ok! I didn't realize the connector was very much still a work in progress.

I'll try the branch later and let you know how it goes. I've also subscribed to all activity in this repo and will be happy to do further testing, suggest feature etc.

I'm VERY keen for this to be a fantastic connector, as mysql is at the root of my legacy application and I'd like to leave it all as-is, while being able to seamlessly move and transform the data to the new system. I have to figure many other people are in a similar situation.

[nickchomey]

Its working on the consistent-snapshot branch! I'll try to use it over the coming days and report any issues that I find.