search

dimitri / pgloader

Migrate to PostgreSQL in a single command!
http://pgloader.io
Other
5.42k stars 546 forks source link

ERROR A thread failed with error: MySQL ERROR: Partial Read of 769 bytes, expected 1000 #999

Open githubhs17 opened 5 years ago

githubhs17 commented 5 years ago

Using docker:

sonarqube_db = mysql docker container
db_postgres = postgresql docker container
pgloader_ccl:latest = pgloader docker container build with Dockerfile.ccl

Running pgloader finished (after ~ 5 min) with ERROR

docker run --security-opt seccomp=unconfined --rm --net net_postgres --name pgloader pgloader_ccl:latest pgloader --debug  mysql://root:mysqlroot4u@sonarqube_db:3306/sonar postgresql://sonar:sonar@db_postgres:5432/sonar
2019-07-09T08:37:10.002255Z LOG pgloader version "3.6.2~devel"
2019-07-09T08:37:10.017239Z LOG Migrating from #<MYSQL-CONNECTION mysql://root@sonarqube_db:3306/sonar #x3020014F257D>
2019-07-09T08:37:10.017593Z LOG Migrating into #<PGSQL-CONNECTION pgsql://sonar@db_postgres:5432/sonar #x3020014EAAFD>
2019-07-09T08:42:46.032796Z ERROR A thread failed with error: MySQL ERROR: Partial Read of 769 bytes, expected 1000
Detail: check MySQL logs for (Got timeout writing communication packets)
Hint: adjust net_read_timeout and net_write_timeout
2019-07-09T08:42:46.036826Z ERROR Date/time: 2019-07-09-08:42An unhandled error condition has been signalled: MySQL ERROR: Partial Read of 769 bytes, expected 1000
Detail: check MySQL logs for (Got timeout writing communication packets)
Hint: adjust net_read_timeout and net_write_timeout

 (7F1C255FA200) : 0 (PRINT-CALL-HISTORY :CONTEXT NIL :PROCESS NIL :ORIGIN NIL :DETAILED-P NIL :COUNT 1152921504606846975 :START-FRAME-NUMBER 0 :STREAM #<STRING-OUTPUT-STREAM  #x30201A909B7D> :PRINT-LEVEL 2 :PRINT-LENGTH 5 :PRINT-STRING-LENGTH :DEFAULT :SHOW-INTERNAL-FRAMES NIL :FORMAT :TRADITIONAL) 941
 (7F1C255FA398) : 1 (PRINT-BACKTRACE-TO-STREAM #<STRING-OUTPUT-STREAM  #x30201A909B7D>) 85
 (7F1C255FA3C8) : 2 (PRINT-BACKTRACE #<QMYND:PARTIAL-READ #x30201A90A48D> :OUTPUT NIL :IF-EXISTS :APPEND :VERBOSE NIL) 797
 (7F1C255FA460) : 3 (FUNCALL #'#<(:INTERNAL (PGLOADER.LOAD:COPY-FROM (PGLOADER.SOURCES:COPY)))> #<QMYND:PARTIAL-READ #x30201A90A48D>) 197
 (7F1C255FA488) : 4 (CONDITION-HANDLER #<QMYND:PARTIAL-READ #x30201A90A48D>) 493
 (7F1C255FA500) : 5 (SIGNAL #<QMYND:PARTIAL-READ #x30201A90A48D>) 989
 (7F1C255FA558) : 6 (%ERROR #<QMYND:PARTIAL-READ #x30201A90A48D> NIL 17469857854640) 117
 (7F1C255FA580) : 7 (READ-WHOLE-CHUNK 1000 #<MY-PACKET-STREAM 0/1000 [241]>) 317
 (7F1C255FA5B0) : 8 (PREPARE-NEXT-CHUNK #<MY-PACKET-STREAM 0/1000 [241]>) 701
 (7F1C255FA5E8) : 9 (FUNCALL #'#<#<STANDARD-METHOD QMYND-IMPL:MYSQL-CONNECTION-READ-PACKET (QMYND:MYSQL-BASE-CONNECTION)>> #<QMYND-IMPL::MYSQL-INET-CONNECTION #x302002EDA95D>) 133
 (7F1C255FA608) : 10 (PARSE-RESULTSET-ROW-AS-TEXT-VECTOR 37 #(#S(QMYND-IMPL::COLUMN-DEFINITION-V41-PACKET :CATALOG "def" :SCHEMA "sonar" ...) #S(QMYND-IMPL::COLUMN-DEFINITION-V41-PACKET :CATALOG "def" :SCHEMA "sonar" ...) #S(QMYND-IMPL::COLUMN-DEFINITION-V41-PACKET :CATALOG "def" :SCHEMA "sonar" ...) #S(QMYND-IMPL::COLUMN-DEFINITION-V41-PACKET :CATALOG "def" :SCHEMA "sonar" ...) #S(QMYND-IMPL::COLUMN-DEFINITION-V41-PACKET :CATALOG "def" :SCHEMA "sonar" ...) ...)) 45
 (7F1C255FA648) : 11 (MAP-RESULTSET-ROWS #<COMPILED-LEXICAL-CLOSURE (:INTERNAL #) #x302002EDAD2F> 37 #(#S(QMYND-IMPL::COLUMN-DEFINITION-V41-PACKET :CATALOG "def" :SCHEMA "sonar" ...) #S(QMYND-IMPL::COLUMN-DEFINITION-V41-PACKET :CATALOG "def" :SCHEMA "sonar" ...) #S(QMYND-IMPL::COLUMN-DEFINITION-V41-PACKET :CATALOG "def" :SCHEMA "sonar" ...) #S(QMYND-IMPL::COLUMN-DEFINITION-V41-PACKET :CATALOG "def" :SCHEMA "sonar" ...) #S(QMYND-IMPL::COLUMN-DEFINITION-V41-PACKET :CATALOG "def" :SCHEMA "sonar" ...) ...) :AS-TEXT T :RESULT-TYPE VECTOR) 405
 (7F1C255FA6B8) : 12 (SEND-COMMAND-QUERY "SELECT `id`, `name`, `description`, `enabled`, `scope`, `qualifier`, `kee`, `language`, `long_name`, `created_at`, `path`, `deprecated_kee`, `uuid`, `project_uuid`, `module_uuid`, `module_uuid_path`, `authorization_updated_at`, `root_uuid`, `copy_component_uuid`, `developer_uuid`, `uuid_path`, `b_changed`, `b_copy_component_uuid`, `b_description`, `b_enabled`, `b_language`, `b_long_name`, `b_module_uuid`, `b_module_uuid_path`, `b_name`, `b_path`, `b_qualifier`, `b_uuid_path`, `organization_uuid`, `tags`, `private`, `main_branch_project_uuid` FROM `projects`" :ROW-FN #<COMPILED-LEXICAL-CLOSURE (:INTERNAL #) #x302002EDAD2F> :AS-TEXT T :RESULT-TYPE VECTOR) 1149
 (7F1C255FA738) : 13 (MYSQL-QUERY #<QMYND-IMPL::MYSQL-INET-CONNECTION #x302002EDA95D> "SELECT `id`, `name`, `description`, `enabled`, `scope`, `qualifier`, `kee`, `language`, `long_name`, `created_at`, `path`, `deprecated_kee`, `uuid`, `project_uuid`, `module_uuid`, `module_uuid_path`, `authorization_updated_at`, `root_uuid`, `copy_component_uuid`, `developer_uuid`, `uuid_path`, `b_changed`, `b_copy_component_uuid`, `b_description`, `b_enabled`, `b_language`, `b_long_name`, `b_module_uuid`, `b_module_uuid_path`, `b_name`, `b_path`, `b_qualifier`, `b_uuid_path`, `organization_uuid`, `tags`, `private`, `main_branch_project_uuid` FROM `projects`" :ROW-FN #<COMPILED-LEXICAL-CLOSURE (:INTERNAL #) #x302002EDAD2F> :AS-TEXT T :RESULT-TYPE VECTOR) 285
 (7F1C255FA7C0) : 14 (%%CHECK-KEYWORDS #(2 #(:RESULT-TYPE :AS-TEXT :ROW-FN) #<METHOD-FUNCTION PGLOADER.CONNECTION:QUERY #>) 17469857854724) 229
 (7F1C255FA868) : 15 (CALL-WITH-ENCODING-HANDLER #<PGLOADER.SOURCE.MYSQL:COPY-MYSQL #x3020016BB46D> "projects" #<COMPILED-LEXICAL-CLOSURE (:INTERNAL #) #x302002ED93DF>) 693
 (7F1C255FA8D0) : 16 (FUNCALL #'#<#<STANDARD-METHOD PGLOADER.SOURCES:MAP-ROWS (PGLOADER.SOURCE.MYSQL:COPY-MYSQL)>> #<PGLOADER.SOURCE.MYSQL:COPY-MYSQL #x3020016BB46D> :PROCESS-ROW-FN #<COMPILED-LEXICAL-CLOSURE (:INTERNAL #) #x302002EDAD2F>) 2813
 (7F1C255FA9A8) : 17 (%%CHECK-KEYWORDS #(1 #(:PROCESS-ROW-FN) #<METHOD-FUNCTION PGLOADER.SOURCES:MAP-ROWS #>) 17469857854784) 229
 (7F1C255FAA20) : 18 (FUNCALL #'#<#<STANDARD-METHOD PGLOADER.LOAD:QUEUE-RAW-DATA (PGLOADER.SOURCES:COPY T T)>> #<PGLOADER.SOURCE.MYSQL:COPY-MYSQL #x3020016BB46D> #S(LPARALLEL.VECTOR-QUEUE:VECTOR-QUEUE :IMPL #S(LPARALLEL.VECTOR-QUEUE::RAW-QUEUE :DATA # :START 1382 ...) :LOCK #<RECURSIVE-LOCK "Anonymous lock" [ptr @ #x7F1C20037C40] #x3020016BBBFD> ...) 1) 725
 (7F1C255FAA70) : 19 (FUNCALL #'#<(:INTERNAL #:BODY-FN64274 LPARALLEL.KERNEL::MAKE-CHANNELED-TASK)>) 245
 (7F1C255FAAD8) : 20 (FUNCALL #'#<(:INTERNAL LPARALLEL.KERNEL::MAKE-CHANNELED-TASK)>) 117
 (7F1C255FAB28) : 21 (EXEC-TASK/WORKER (#<COMPILED-LEXICAL-CLOSURE # #x3020016BBB3F> . :DEFAULT) #S(LPARALLEL.KERNEL::WORKER :HANDSHAKE/FROM-WORKER #S(LPARALLEL.CONS-QUEUE:CONS-QUEUE :IMPL # :LOCK #<# # [ptr @ #x7F1C20033400] #x3020014C54AD> ...) :HANDSHAKE/TO-WORKER #S(LPARALLEL.CONS-QUEUE:CONS-QUEUE :IMPL # :LOCK #<# # [ptr @ #x7F1C20033480] #x3020014C53DD> ...) ...)) 197
 (7F1C255FAB90) : 22 (WORKER-LOOP #<KERNEL :NAME "lparallel" :WORKER-COUNT 4 :USE-CALLER NIL :ALIVE T #x3020014C6CAD> #S(LPARALLEL.KERNEL::WORKER :HANDSHAKE/FROM-WORKER #S(LPARALLEL.CONS-QUEUE:CONS-QUEUE :IMPL # :LOCK #<# # [ptr @ #x7F1C20033400] #x3020014C54AD> ...) :HANDSHAKE/TO-WORKER #S(LPARALLEL.CONS-QUEUE:CONS-QUEUE :IMPL # :LOCK #<# # [ptr @ #x7F1C20033480] #x3020014C53DD> ...) ...)) 237
 (7F1C255FAC00) : 23 (%CALL-WITH-TASK-HANDLER #<COMPILED-LEXICAL-CLOSURE (:INTERNAL LPARALLEL.KERNEL::ENTER-WORKER-LOOP) #x3020014CF82F>) 789
 (7F1C255FACA8) : 24 (FUNCALL #'#<(:INTERNAL LPARALLEL.KERNEL::CALL-WITH-WORKER-CONTEXT)>) 869
 (7F1C255FAD28) : 25 (FUNCALL #<COMPILED-LEXICAL-CLOSURE (:INTERNAL LPARALLEL.KERNEL::CALL-WITH-WORKER-CONTEXT) #x3020014CF77F>) 149
 (7F1C255FAD58) : 26 (CALL-WITH-WORKER-CONTEXT #<COMPILED-LEXICAL-CLOSURE (:INTERNAL LPARALLEL.KERNEL::ENTER-WORKER-LOOP) #x3020014CF82F> #<Compiled-function FUNCALL #x30000007FEFF> #<KERNEL :NAME "lparallel" :WORKER-COUNT 4 :USE-CALLER NIL :ALIVE T #x3020014C6CAD> #S(LPARALLEL.KERNEL::WORKER :HANDSHAKE/FROM-WORKER #S(LPARALLEL.CONS-QUEUE:CONS-QUEUE :IMPL # :LOCK #<# # [ptr @ #x7F1C20033400] #x3020014C54AD> ...) :HANDSHAKE/TO-WORKER #S(LPARALLEL.CONS-QUEUE:CONS-QUEUE :IMPL # :LOCK #<# # [ptr @ #x7F1C20033480] #x3020014C53DD> ...) ...)) 341
 (7F1C255FADC0) : 27 (FUNCALL #'#<(:INTERNAL LPARALLEL.KERNEL::MAKE-WORKER-THREAD)>) 157
 (7F1C255FAE18) : 28 (FUNCALL #'#<(:INTERNAL BORDEAUX-THREADS::BINDING-DEFAULT-SPECIALS)>) 797
 (7F1C255FAE60) : 29 (RUN-PROCESS-INITIAL-FORM #<PROCESS lparallel(8) [Active] #x3020014C4C3D> (#<COMPILED-LEXICAL-CLOSURE # #x3020014C49CF>)) 709
 (7F1C255FAEF0) : 30 (FUNCALL #'#<(:INTERNAL (CCL::%PROCESS-PRESET-INTERNAL (PROCESS)))> #<PROCESS lparallel(8) [Active] #x3020014C4C3D> (#<COMPILED-LEXICAL-CLOSURE # #x3020014C49CF>)) 573
 (7F1C255FAF98) : 31 (FUNCALL #'#<(:INTERNAL CCL::THREAD-MAKE-STARTUP-FUNCTION)>) 277
KABOOM!
FATAL error: MySQL ERROR: Partial Read of 769 bytes, expected 1000
Detail: check MySQL logs for (Got timeout writing communication packets)
Hint: adjust net_read_timeout and net_write_timeout
An unhandled error condition has been signalled: MySQL ERROR: Partial Read of 769 bytes, expected 1000
Detail: check MySQL logs for (Got timeout writing communication packets)
Hint: adjust net_read_timeout and net_write_timeout

What I am doing here?

MySQL ERROR: Partial Read of 769 bytes, expected 1000
Detail: check MySQL logs for (Got timeout writing communication packets)
Hint: adjust net_read_timeout and net_write_timeout
dimitri commented 5 years ago

Hi, did you check your MySQL logs? did you have timeout error messages there? did you try adjusting the parameters as instructed: you can do that from the pgloader command itself.

Detail: check MySQL logs for (Got timeout writing communication packets) Hint: adjust net_read_timeout and net_write_timeout

taoxinyi commented 4 years ago

I have a similiar problem

lock" (free)> :NOTIFY-PUSH NIL :NOTIFY-POP #<SB-THREAD:WAITQUEUE Anonymous condition variable {10173D1DB3}>) 1) [fast-method]
16: ((FLET "BODY-FN0" :IN LPARALLEL.KERNEL::MAKE-CHANNELED-TASK))
17: ((LAMBDA NIL :IN LPARALLEL.KERNEL::MAKE-CHANNELED-TASK))
18: (LPARALLEL.KERNEL::EXEC-TASK/WORKER #<unavailable argument> #S(LPARALLEL.KERNEL::WORKER :HANDSHAKE/FROM-WORKER #S(LPARALLEL.CONS-QUEUE:CONS-QUEUE :IMPL #S(LPARALLEL.RAW-QUEUE:RAW-QUEUE :HEAD NIL :TAIL NIL) :LOCK #<SB-THREAD:MUTEX #1="Anonymous lock" (free)> :CVAR #<SB-THREAD:WAITQUEUE #2=Anonymous condition variable {1005A8AF93}>) :HANDSHAKE/TO-WORKER #S(LPARALLEL.CONS-QUEUE:CONS-QUEUE :IMPL #S(LPARALLEL.RAW-QUEUE:RAW-QUEUE :HEAD NIL :TAIL NIL) :LOCK #<SB-THREAD:MUTEX #1# (free)> :CVAR #<SB-THREAD:WAITQUEUE #2# {1005A8AFF3}>) :EXIT-NOTIFICATION #S(LPARALLEL.CONS-QUEUE:CONS-QUEUE :IMPL #S(LPARALLEL.RAW-QUEUE:RAW-QUEUE :HEAD NIL :TAIL NIL) :LOCK #<SB-THREAD:MUTEX #1# (free)> :CVAR NIL) :THREAD #<SB-THREAD:THREAD "lparallel" RUNNING {1006AB2263}> :RUNNING-CATEGORY :DEFAULT :INDEX 1 :TASKS #S(LPARALLEL.SPIN-QUEUE:SPIN-QUEUE :HEAD #3=(LPARALLEL.SPIN-QUEUE::DUMMY) :TAIL #3#)))
19: (LPARALLEL.KERNEL::WORKER-LOOP #<LPARALLEL.KERNEL:KERNEL :NAME "lparallel" :WORKER-COUNT 4 :USE-CALLER NIL :ALIVE T :SPIN-COUNT 2000 {1006AB1833}> #S(LPARALLEL.KERNEL::WORKER :HANDSHAKE/FROM-WORKER #S(LPARALLEL.CONS-QUEUE:CONS-QUEUE :IMPL #S(LPARALLEL.RAW-QUEUE:RAW-QUEUE :HEAD NIL :TAIL NIL) :LOCK #<SB-THREAD:MUTEX #1="Anonymous lock" (free)> :CVAR #<SB-THREAD:WAITQUEUE #2=Anonymous condition variable {1005A8AF93}>) :HANDSHAKE/TO-WORKER #S(LPARALLEL.CONS-QUEUE:CONS-QUEUE :IMPL #S(LPARALLEL.RAW-QUEUE:RAW-QUEUE :HEAD NIL :TAIL NIL) :LOCK #<SB-THREAD:MUTEX #1# (free)> :CVAR #<SB-THREAD:WAITQUEUE #2# {1005A8AFF3}>) :EXIT-NOTIFICATION #S(LPARALLEL.CONS-QUEUE:CONS-QUEUE :IMPL #S(LPARALLEL.RAW-QUEUE:RAW-QUEUE :HEAD NIL :TAIL NIL) :LOCK #<SB-THREAD:MUTEX #1# (free)> :CVAR NIL) :THREAD #<SB-THREAD:THREAD "lparallel" RUNNING {1006AB2263}> :RUNNING-CATEGORY :DEFAULT :INDEX 1 :TASKS #S(LPARALLEL.SPIN-QUEUE:SPIN-QUEUE :HEAD #3=(LPARALLEL.SPIN-QUEUE::DUMMY) :TAIL #3#)))
20: (LPARALLEL.KERNEL::%CALL-WITH-TASK-HANDLER #<unavailable argument>)
21: ((LAMBDA NIL :IN LPARALLEL.KERNEL::CALL-WITH-WORKER-CONTEXT))
22: (LPARALLEL.KERNEL::CALL-WITH-WORKER-CONTEXT #<CLOSURE (LAMBDA NIL :IN LPARALLEL.KERNEL::ENTER-WORKER-LOOP) {1006527E6B}> #<FUNCTION FUNCALL> #<LPARALLEL.KERNEL:KERNEL :NAME "lparallel" :WORKER-COUNT 4 :USE-CALLER NIL :ALIVE T :SPIN-COUNT 2000 {1006AB1833}> #S(LPARALLEL.KERNEL::WORKER :HANDSHAKE/FROM-WORKER #S(LPARALLEL.CONS-QUEUE:CONS-QUEUE :IMPL #S(LPARALLEL.RAW-QUEUE:RAW-QUEUE :HEAD NIL :TAIL NIL) :LOCK #<SB-THREAD:MUTEX #1="Anonymous lock" (free)> :CVAR #<SB-THREAD:WAITQUEUE #2=Anonymous condition variable {1005A8AF93}>) :HANDSHAKE/TO-WORKER #S(LPARALLEL.CONS-QUEUE:CONS-QUEUE :IMPL #S(LPARALLEL.RAW-QUEUE:RAW-QUEUE :HEAD NIL :TAIL NIL) :LOCK #<SB-THREAD:MUTEX #1# (free)> :CVAR #<SB-THREAD:WAITQUEUE #2# {1005A8AFF3}>) :EXIT-NOTIFICATION #S(LPARALLEL.CONS-QUEUE:CONS-QUEUE :IMPL #S(LPARALLEL.RAW-QUEUE:RAW-QUEUE :HEAD NIL :TAIL NIL) :LOCK #<SB-THREAD:MUTEX #1# (free)> :CVAR NIL) :THREAD #<SB-THREAD:THREAD "lparallel" RUNNING {1006AB2263}> :RUNNING-CATEGORY :DEFAULT :INDEX 1 :TASKS #S(LPARALLEL.SPIN-QUEUE:SPIN-QUEUE :HEAD #3=(LPARALLEL.SPIN-QUEUE::DUMMY) :TAIL #3#)))
23: ((LAMBDA NIL :IN LPARALLEL.KERNEL::MAKE-WORKER-THREAD))
24: ((LAMBDA NIL :IN BORDEAUX-THREADS::BINDING-DEFAULT-SPECIALS))
25: ((FLET SB-UNIX::BODY :IN SB-THREAD::INITIAL-THREAD-FUNCTION-TRAMPOLINE))
26: ((FLET "WITHOUT-INTERRUPTS-BODY-4" :IN SB-THREAD::INITIAL-THREAD-FUNCTION-TRAMPOLINE))
27: ((FLET SB-THREAD::WITH-MUTEX-THUNK :IN SB-THREAD::INITIAL-THREAD-FUNCTION-TRAMPOLINE))
28: ((FLET "WITHOUT-INTERRUPTS-BODY-1" :IN SB-THREAD::CALL-WITH-MUTEX))
29: (SB-THREAD::CALL-WITH-MUTEX #<CLOSURE (FLET SB-THREAD::WITH-MUTEX-THUNK :IN SB-THREAD::INITIAL-THREAD-FUNCTION-TRAMPOLINE) {7FFFF4DBED4B}> #<SB-THREAD:MUTEX "thread result lock" owner: #<SB-THREAD:THREAD "lparallel" RUNNING {1006AB2263}>> NIL T NIL)
30: (SB-THREAD::INITIAL-THREAD-FUNCTION-TRAMPOLINE #<SB-THREAD:THREAD "lparallel" RUNNING {1006AB2263}> NIL #<CLOSURE (LAMBDA NIL :IN BORDEAUX-THREADS::BINDING-DEFAULT-SPECIALS) {1006AB220B}> NIL)
31: ("foreign function: call_into_lisp")
32: ("foreign function: new_thread_trampoline")

2020-04-18T20:30:30.698000Z INFO Stopping monitor

What I am doing here?

MySQL ERROR: Partial Read of 157 bytes, expected 240
Detail: check MySQL logs for (Got timeout writing communication packets)
Hint: adjust net_read_timeout and net_write_timeout

This happens when I load from local MySQL to AWS RDS Aurora Postgresql. I have changed net_read_timeout and net_write_timeout to 600 but still not working.

iswarezwp commented 3 years ago

same problem

20: (LPARALLEL.KERNEL::%CALL-WITH-TASK-HANDLER #<unavailable argument>)
21: ((LAMBDA NIL :IN LPARALLEL.KERNEL::CALL-WITH-WORKER-CONTEXT))
22: (LPARALLEL.KERNEL::CALL-WITH-WORKER-CONTEXT #<CLOSURE (LAMBDA NIL :IN LPARALLEL.KERNEL::ENTER-WORKER-LOOP) {10077AFDCB}> #<FUNCTION FUNCALL> #<LPARALLEL.KERNEL:KERNEL :NAME "lparallel" :WORKER-COUNT 4 :USE-CALLER NIL :ALIVE T :SPIN-COUNT 2000 {1007D7A513}> #S(LPARALLEL.KERNEL::WORKER :HANDSHAKE/FROM-WORKER #S(LPARALLEL.CONS-QUEUE:CONS-QUEUE :IMPL #S(LPARALLEL.RAW-QUEUE:RAW-QUEUE :HEAD NIL :TAIL NIL) :LOCK #<SB-THREAD:MUTEX #1="Anonymous lock" (free)> :CVAR #<SB-THREAD:WAITQUEUE #2=Anonymous condition variable {100715DB23}>) :HANDSHAKE/TO-WORKER #S(LPARALLEL.CONS-QUEUE:CONS-QUEUE :IMPL #S(LPARALLEL.RAW-QUEUE:RAW-QUEUE :HEAD NIL :TAIL NIL) :LOCK #<SB-THREAD:MUTEX #1# (free)> :CVAR #<SB-THREAD:WAITQUEUE #2# {10077AFDE3}>) :EXIT-NOTIFICATION #S(LPARALLEL.CONS-QUEUE:CONS-QUEUE :IMPL #S(LPARALLEL.RAW-QUEUE:RAW-QUEUE :HEAD NIL :TAIL NIL) :LOCK #<SB-THREAD:MUTEX #1# (free)> :CVAR NIL) :THREAD #<SB-THREAD:THREAD "lparallel" RUNNING {1007D7AF53}> :RUNNING-CATEGORY :DEFAULT :INDEX 1 :TASKS #S(LPARALLEL.SPIN-QUEUE:SPIN-QUEUE :HEAD (LPARALLEL.SPIN-QUEUE::DUMMY (#<CLOSURE #3=(LAMBDA NIL :IN LPARALLEL.KERNEL::MAKE-CHANNELED-TASK) {100728ED3B}> . :DEFAULT) (#<CLOSURE #3# {100728ED6B}> . :DEFAULT) (#<CLOSURE #3# {100728ED9B}> . :DEFAULT) (#<CLOSURE #3# {100728EDCB}FAULT) (#<CLOSURE #3# {100728EEEB}> . :DEFAULT) (#<CLOSURE #3# {100728EF1B}> . :DEFAULT) (#<CLOSURE #3# {100728EF4B}> . :DEFAULT) (#<CLOSURE #3# {100728EF7B}> . :DEFAULT) (#<CLOSURE #3# {100728EFAB}> . :DEFAULT) (#<CLOSURE #3# {100728EFDB}> . :DEFAULT) (#<CLOSURE #3# {100728F00B}> . :DEFAULT) (#<CLOSURE #3# {100728F03B}> . :DEFAULT) (#<CLOSURE #3# {100728F06B}> . :DEFAULT) (#<CLOSURE #3# {100728F09B}> . :DEFAULT) (#<CLOSURE #3# {100728F0CB}> . :DEFAULT) . #4=((#<CLOSURE #3# {100728F0FB}> . :DEFAULT))) :TAIL #4#)))23: ((LAMBDA NIL :IN LPARALLEL.KERNEL::MAKE-WORKER-THREAD))
24: ((LAMBDA NIL :IN BORDEAUX-THREADS::BINDING-DEFAULT-SPECIALS))
25: ((FLET "WITHOUT-INTERRUPTS-BODY-4" :IN SB-THREAD::INITIAL-THREAD-FUNCTION-TRAMPOLINE))
26: ((FLET SB-THREAD::WITH-MUTEX-THUNK :IN SB-THREAD::INITIAL-THREAD-FUNCTION-TRAMPOLINE))
27: ((FLET "WITHOUT-INTERRUPTS-BODY-1" :IN SB-THREAD::CALL-WITH-MUTEX))
28: (SB-THREAD::CALL-WITH-MUTEX #<CLOSURE (FLET SB-THREAD::WITH-MUTEX-THUNK :IN SB-THREAD::INITIAL-THREAD-FUNCTION-TRAMPOLINE) {7FFFEDE9ED2B}> #<SB-THREAD:MUTEX "thread result lock" owner: #<SB-THREAD:THREAD "lparallel" RUNNING {1007D7AF53}>> NIL T NIL)29: (SB-THREAD::INITIAL-THREAD-FUNCTION-TRAMPOLINE #<SB-THREAD:THREAD "lparallel" RUNNING {1007D7AF53}> NIL #<CLOSURE (LAMBDA NIL :IN BORDEAUX-THREADS::BINDING-DEFAULT-SPECIALS) {1007D7AEFB}> (#<SB-THREAD:THREAD #1="lparallel" RUNNING {1007D7AF53}> #<SB-THREAD:THREAD #1# RUNNING {1007D7AA03}> #<SB-THREAD:THREAD #1# RUNNING {1006C70163}> #<SB-THREAD:THREAD "main thread" RUNNING {1006C3E3C3}>) NIL NIL NIL NIL)30: ("foreign function: call_into_lisp")
31: ("foreign function: new_thread_trampoline")

What I am doing here?

MySQL ERROR: Partial Read of 9 bytes, expected 49
Detail: check MySQL logs for (Got timeout writing communication packets)
Hint: adjust net_read_timeout and net_write_timeout
parthjoshi-pc commented 2 years ago

I'm facing the same issue and I'm running pgloader over a local UNIX socket connection to MySQL. The command:

pgloader --client-min-messages info migration-script.load

with migration-script.load containing:

LOAD DATABASE
  FROM mysql:/my_user:password@unix:/var/run/mysqld/mysqld.sock:/source_db
  INTO postgresql://pg_user:password@pg_db_server:pg_port/target_db?sslmode=require
WITH no truncate, create tables, include drop, create indexes, reset sequences,
     foreign keys, downcase identifiers, uniquify index names, workers = 8, concurrency = 2,
     multiple readers per thread, rows per range = 20000
SET MySQL PARAMETERS net_read_timeout = '1800', net_write_timeout = '1800';

As you can see, I've tried increasing net_read_timeout and net_write_timeout to 1800 starting from 600 and I still keep getting errors like this:

2022-02-15T16:39:20.987008+05:30 ERROR A thread failed with error: MySQL ERROR: Partial Read of 76 bytes, expected 145
Detail: check MySQL logs for (Got timeout writing communication packets)
Hint: adjust net_read_timeout and net_write_timeout
2022-02-15T16:39:20.993859+05:30 ERROR A thread failed with error: Unexpected end of file on #<FUNDAMENTAL-FILE-SOCKET-STREAM :UTF-8 (SOCKET/5) #x302014F007FD>
2022-02-15T16:39:20.994096+05:30 ERROR A thread failed with error: MySQL ERROR: Partial Read of 40 bytes, expected 153
Detail: check MySQL logs for (Got timeout writing communication packets)
Hint: adjust net_read_timeout and net_write_timeout
2022-02-15T16:39:21.023208+05:30 ERROR Date/time: 2022-02-15-16:39!
An unhandled error condition has been signalled: MySQL ERROR: Partial Read of 40 bytes, expected 153
Detail: check MySQL logs for (Got timeout writing communication packets)
Hint: adjust net_read_timeout and net_write_timeout

I've compiled pgloader using CCL with a DYNSIZE set to 8192.

$ pgloader --version
pgloader version "3.6.2"
compiled with Clozure Common Lisp Version 1.12.1 (v1.12.1) LinuxX8664

I tried running originally without the workers, concurrency, multiple readers per thread and rows_per_range parameters but that kept failing with the same error too (it ran faster and broke earlier).

I'm not tool familiar with MySQL internals and was looking for a simple way to migrate my fairly large MySQL database (~207GB) to Postgres but at this point, having spent more than a week without making much headway, I'm almost ready to give up on pgloader. I'm unsure of what parameters and settings to tune apart from retrying with a larger net_read_timeout and net_write_timeout as suggested in the log.

@dimitri Any help here would be greatly appreciated.

rogerhu commented 2 years ago

I got things to work for migrating a 50GB DB from MySQL to Postgres with the following requirements:

I did use SET MySQL PARAMETERS net_read_timeout = '1800', net_write_timeout = '1800' as well but I noticed the EC2 machine running pgloader was capping out on CPU. I was able to get the job using the default concurrency options to complete in 10 hours.

andyfcx commented 1 year ago

Still stucked in this issue.

philsmy commented 1 year ago

I'm trying to move over a 1.2TB (yes, TB) database from MySQL and hitting this same error. I wonder if this will ever work.

Marcnuth commented 1 year ago

stucked in this issue too

mpneyer commented 1 year ago

I was stuck with same and/or similar issues. I was not able to convert a large mariaDB database to postgreSQL and wasted days. Neither on Intel-Mac, M1-Mac nor on any docker container i was able to convert data reliably, trying out many pgloader options. Only about half of my attempts were successful, which was not enough for the big bang day with production data.

The only solution which successfully and reliably worked 100% was renting a cloud root server with Debian 11 and converting the database there. This cost a few bucks, but is no relation to the hours spent in debugging why pgloader crashes

cmdreay commented 1 month ago

stucked in this issue too