cashapp / spirit

Online Schema Change Tool for MySQL 8.0+
Apache License 2.0
113 stars 20 forks source link

spirit: error: wait position #343

Closed DevMasterSoonToBe closed 2 months ago

DevMasterSoonToBe commented 2 months ago

Hi,

We are trying to resume Spirit from checkpoint after _sentinel is dropped and it does checksum but it fails after a long time with this output:

INFO[80525] migration status: state=checksum checksum-progress=“8a868fff8dc26038018dd6d8aa8f546c”/“8a868fff8fe55566018fe590b55c09ef” binlog-deltas=0 total-time=22h22m6s checksum-time=22h21m55s conns-in-use=5 INFO[80530] finished periodic flush of binary log: total-duration=4.51µs batch-size=1000 INFO[80545] checksum passed INFO[80545] starting to flush changeset INFO[80555] closing canal INFO[80555] syncer is closing... WARN[80555] releasing metadata lock: spirit_xxxxxxx_xxxxxxx INFO[80555] kill last connection id 79927 INFO[80555] syncer is closed spirit: error: wait position (mysql-bin-changelog.033214, 157) too long > 10s

Previously we have deleted _sentinel table but it didn't do cutover because spirit was not running. We are thinking it should do cutover after we run spirit again without sentinel table in the database. However it fails with this **spirit: error: wait position (mysql-bin-changelog.033214, 157) too long > 10s_** error and could you advise what might be the issue that spirit informs about here?

morgo commented 2 months ago

I believe this is a duplicate of https://github.com/cashapp/spirit/issues/337 (fixed)

We introduced this regression about 1 month ago in https://github.com/cashapp/spirit/pull/332 and then worked around it in https://github.com/cashapp/spirit/pull/339

The final fix was just merged though with https://github.com/cashapp/spirit/pull/340

Please check it out and let me know if you are experiencing issues with it.

morgo commented 2 months ago

I'm going to close this as fixed. Please re-open if you experience otherwise, thanks!