search

facebook / wdt

Warp speed Data Transfer (WDT) is an embeddedable library (and command line tool) aiming to transfer data between 2 systems as fast as possible over multiple TCP paths.
https://www.facebook.com/WdtOpenSource
Other
2.86k stars 391 forks source link

Always fails to transfer the whole files #138

Closed nxtreaming closed 7 years ago

nxtreaming commented 7 years ago

I get the following log: root@DQ5606-C12-2:/home/wdt_test# wdt --directory /home/wdt_test | ssh cn.wdthost.com wdt --directory /data/dev/wdt_test - I1207 02:50:52.842162 21692 WdtFlags.cpp:63] wdt> Running WDT 1.27.1612021 p 27 I1207 02:50:52.842989 21692 EncryptionUtils.cpp:83] wdt> Openssl library initialized I1207 02:50:52.843101 21692 WdtFlags.cpp:63] wdt> Running WDT 1.27.1612021 p 27 I1207 02:50:52.843622 21692 WdtResourceController.cpp:29] wdt> Updated max number of senders for root controller to 0 I1207 02:50:52.843740 21692 WdtResourceController.cpp:22] wdt> Updated max number of receivers for root controller to 0 I1207 02:50:52.843777 21692 Throttler.cpp:73] wdt> No average rate specified I1207 02:50:52.843806 21692 Throttler.cpp:80] wdt> No peak rate specified I1207 02:50:52.843847 21692 Wdt.cpp:24] wdt> One time initialization of WDT for wdt I1207 02:50:52.843894 21692 Throttler.cpp:98] wdt> Updating the rates avgRateBytesPerSec : -1.04858e+06 bucketRateBytesPerSec : 0 bytesTokenBucketLimit : 0 I1207 02:50:52.843991 21692 Receiver.cpp:46] wdt> WDT Receiver 1.27.1612021 p 27 I1207 02:50:52.844125 21692 WdtBase.cpp:82] wdt> Setting transfer id 540054088 I1207 02:50:52.844192 21692 WdtBase.cpp:94] wdt> using wdt protocol version 27 I1207 02:50:52.844244 21692 FileCreator.cpp:319] wdt> dir already exists / I1207 02:50:52.844298 21692 FileCreator.cpp:319] wdt> dir already exists /home/ I1207 02:50:52.844334 21692 FileCreator.cpp:319] wdt> dir already exists /home/wdt_test/ I1207 02:50:52.844372 21692 Receiver.cpp:170] wdt> aes128gcm encryption is enabled for this transfer I1207 02:50:52.844410 21692 Receiver.cpp:173] wdt> Receiver generating encryption key for type aes128gcm I1207 02:50:52.844635 21692 EncryptionUtils.cpp:115] wdt> New encryption params 0x7ffc1d2991b0 2:...11920925460756788275... I1207 02:50:52.845219 21692 ReceiverThread.cpp:987] wdt> Thread[0, port: 47908] Listening on port 47908 I1207 02:50:52.845518 21692 ReceiverThread.cpp:987] wdt> Thread[1, port: 49590] Listening on port 49590 I1207 02:50:52.845783 21692 ReceiverThread.cpp:987] wdt> Thread[2, port: 45411] Listening on port 45411 I1207 02:50:52.846050 21692 ReceiverThread.cpp:987] wdt> Thread[3, port: 35384] Listening on port 35384 I1207 02:50:52.846314 21692 ReceiverThread.cpp:987] wdt> Thread[4, port: 58549] Listening on port 58549 I1207 02:50:52.846580 21692 ReceiverThread.cpp:987] wdt> Thread[5, port: 49513] Listening on port 49513 I1207 02:50:52.846845 21692 ReceiverThread.cpp:987] wdt> Thread[6, port: 54616] Listening on port 54616 I1207 02:50:52.847108 21692 ReceiverThread.cpp:987] wdt> Thread[7, port: 47603] Listening on port 47603 I1207 02:50:52.847178 21692 Receiver.cpp:212] wdt> Registered 8 successful sockets I1207 02:50:52.847273 21692 wdtCmdLine.cpp:321] wdt> Starting receiver with connection url wdt://DQ5606-C12-2?dir=%2fhome%2fwdt_test&enc=2:...11920925460756788275...&id=540054088&ports=47908,49590,45411,35384,58549,49513,54616,47603&recpv=27 I1207 02:50:52.847404 21692 Receiver.cpp:455] wdt> Starting (receiving) server on ports [ 47908 49590 45411 35384 58549 49513 54616 47603 ] Target dir : /home/wdt_test I1207 02:50:52.847502 21692 Throttler.cpp:73] wdt> No average rate specified I1207 02:50:52.847534 21692 Throttler.cpp:80] wdt> No peak rate specified I1207 02:50:52.847564 21692 WdtBase.cpp:183] wdt> Enabling throttling avgRate: -1 Mbytes/sec, peakRate: 0 Mbytes/sec, bucketLimit: 0 Mbytes, throttlerLogTimeMillis: 0 I1207 02:50:52.847981 21702 Receiver.cpp:396] wdt> Progress reporter updating every 200 ms I1207 02:50:53.048244 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:50:53.248508 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:50:53.448701 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:50:53.648888 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:50:53.849079 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:50:54.049265 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:50:54.249464 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:50:54.449650 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:50:54.649834 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:50:54.850025 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:50:55.050211 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:50:55.250371 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:50:55.450556 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:50:55.650743 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:50:55.850953 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:50:56.051161 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 18:50:44.761514 13436 WdtFlags.cpp:63] wdt> Running WDT 1.27.1612021 p 27 I1207 18:50:44.762029 13436 EncryptionUtils.cpp:83] wdt> Openssl library initialized I1207 18:50:44.762055 13436 WdtFlags.cpp:63] wdt> Running WDT 1.27.1612021 p 27 I1207 18:50:44.762279 13436 WdtResourceController.cpp:29] wdt> Updated max number of senders for root controller to 0 I1207 18:50:44.762300 13436 WdtResourceController.cpp:22] wdt> Updated max number of receivers for root controller to 0 I1207 18:50:44.762308 13436 Throttler.cpp:73] wdt> No average rate specified I1207 18:50:44.762310 13436 Throttler.cpp:80] wdt> No peak rate specified I1207 18:50:44.762320 13436 Wdt.cpp:24] wdt> One time initialization of WDT for wdt I1207 18:50:44.762336 13436 Throttler.cpp:98] wdt> Updating the rates avgRateBytesPerSec : -1.04858e+06 bucketRateBytesPerSec : 0 bytesTokenBucketLimit : 0 I1207 18:50:44.762446 13436 wdtCmdLine.cpp:282] wdt> Parsed url as wdt://DQ5606-C12-2?dir=%2fdata%2fdev%2fwdt_test&enc=2:...11920925460756788275...&id=540054088&ports=47908,49590,45411,35384,58549,49513,54616,47603&recpv=27 I1207 18:50:44.762462 13436 wdtCmdLine.cpp:365] wdt> Making Sender with encryption set = 1 I1207 18:50:44.762475 13436 WdtResourceController.cpp:357] wdt> First time (default) is seen, creating. I1207 18:50:44.762483 13436 WdtResourceController.cpp:29] wdt> Updated max number of senders for to 0 I1207 18:50:44.762488 13436 WdtResourceController.cpp:22] wdt> Updated max number of receivers for to 1 I1207 18:50:44.762498 13436 Throttler.cpp:98] wdt> Updating the rates avgRateBytesPerSec : -1.04858e+06 bucketRateBytesPerSec : 0 bytesTokenBucketLimit : 0 I1207 02:50:56.251382 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:50:56.269986 21699 Receiver.cpp:93] wdt> Starting new transfer, peerIp 120.132.7.182 , transfer id 540054088 I1207 02:50:56.270177 21699 WdtSocket.cpp:67] wdt> Successfully read encryption settings 49513 aes128gcm I1207 02:50:56.274632 21694 WdtSocket.cpp:67] wdt> Successfully read encryption settings 47908 aes128gcm I1207 18:50:44.762518 13436 Sender.cpp:47] wdt> WDT Sender 1.27.1612021 p 27 I1207 18:50:44.762532 13436 WdtResourceController.cpp:379] wdt> Successfully added a sender for identifier DQ5606-C12-2 I1207 18:50:44.762543 13436 WdtBase.cpp:94] wdt> using wdt protocol version 27 I1207 18:50:44.762549 13436 Sender.cpp:84] wdt> Encryption is enabled for this transfer I1207 18:50:44.762572 13436 DirectorySourceQueue.cpp:127] wdt> Root dir now /data/dev/wdt_test/ I1207 18:50:44.762593 13436 Sender.cpp:311] wdt> Client (sending) to DQ5606-C12-2, Using ports [ 47908 49590 45411 35384 58549 49513 54616 47603 ] I1207 18:50:44.762599 13436 Sender.cpp:325] wdt> Skipping throttler setup. External throttler set.Throttler details : avgRate: -1 Mbytes/sec, peakRate: 0 Mbytes/sec, bucketLimit: 0 Mbytes, throttlerLogTimeMillis: 0 I1207 18:50:44.763731 13439 Sender.cpp:41] wdt> Starting a new transfer 540054088 to DQ5606-C12-2 I1207 18:50:44.763741 13437 DirectorySourceQueue.cpp:240] wdt> Exploring root dir /data/dev/wdt_test/ include_pattern : exclude_pattern : prune_dir_pattern : I1207 18:50:44.764354 13437 DirectorySourceQueue.cpp:389] wdt> Number of files explored: 1 opened 0 with direct 0 errors false I1207 18:50:44.764436 13446 Sender.cpp:418] wdt> Progress reporter tracking every 200 ms I1207 18:50:44.964593 13446 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete) I1207 18:50:44.973379 13443 SenderThread.cpp:85] wdt> Thread[5, port: 49513] Connection took 1 attempt(s) and 0.209555 seconds. port 49513 I1207 18:50:44.976073 13438 SenderThread.cpp:85] wdt> Thread[0, port: 47908] Connection took 1 attempt(s) and 0.212263 seconds. port 47908 I1207 18:50:44.984274 13444 SenderThread.cpp:85] wdt> Thread[6, port: 54616] Connection took 1 attempt(s) and 0.220449 seconds. port 54616 I1207 18:50:44.986655 13439 SenderThread.cpp:85] wdt> Thread[1, port: 49590] Connection took 1 attempt(s) and 0.222851 seconds. port 49590 I1207 02:50:56.283303 21695 WdtSocket.cpp:67] wdt> Successfully read encryption settings 49590 aes128gcm I1207 18:50:44.988518 13442 SenderThread.cpp:85] wdt> Thread[4, port: 58549] Connection took 1 attempt(s) and 0.224649 seconds. port 58549 I1207 02:50:56.287132 21698 WdtSocket.cpp:67] wdt> Successfully read encryption settings 58549 aes128gcm I1207 02:50:56.291884 21700 WdtSocket.cpp:67] wdt> Successfully read encryption settings 54616 aes128gcm I1207 18:50:45.000504 13440 SenderThread.cpp:85] wdt> Thread[2, port: 45411] Connection took 1 attempt(s) and 0.236658 seconds. port 45411 I1207 02:50:56.311111 21696 WdtSocket.cpp:67] wdt> Successfully read encryption settings 45411 aes128gcm I1207 02:50:56.354485 21701 ReceiverThread.cpp:195] wdt> Thread[7, port: 47603] entered ACCEPT_WITH_TIMEOUT state I1207 02:50:56.354607 21697 ReceiverThread.cpp:195] wdt> Thread[3, port: 35384] entered ACCEPT_WITH_TIMEOUT state I1207 02:50:56.451607 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0.00102392 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 18:50:45.164758 13446 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete) I1207 02:50:56.651818 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0.000487037 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 18:50:45.364905 13446 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete) I1207 02:50:56.852028 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0.000319506 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 18:50:45.565044 13446 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete) I1207 02:50:57.052247 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0.000237731 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 18:50:45.765179 13446 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete) I1207 02:50:57.252461 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0.000189284 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 18:50:45.965308 13446 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete) I1207 02:50:57.452673 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0.000157241 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 18:50:46.165446 13446 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete) I1207 02:50:57.652895 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0.000134475 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 18:50:46.365588 13446 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete) I1207 02:50:57.853124 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0.000117467 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 18:50:46.565747 13446 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete) I1207 02:50:58.053350 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0.000104279 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 18:50:46.765897 13446 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete) I1207 02:50:58.253563 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 9.37532e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 18:50:46.966025 13446 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete) I1207 02:50:58.453788 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 8.51572e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 18:50:47.166177 13446 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete) I1207 02:50:58.654011 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 7.80053e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 18:50:47.366334 13446 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete) I1207 02:50:58.854233 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 7.19616e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 18:50:47.566481 13446 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete) I1207 02:50:59.054456 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 6.67871e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 18:50:47.766621 13446 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete) I1207 02:50:59.254686 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 6.23066e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 18:50:47.966785 13446 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete) I1207 02:50:59.454839 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 5.8391e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 18:50:48.166957 13446 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete) I1207 02:50:59.655055 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 5.49373e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 18:50:48.367097 13446 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete) I1207 02:50:59.855269 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 5.18694e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 18:50:48.567258 13446 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete) I1207 02:51:00.055480 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 4.91261e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 18:50:48.767405 13446 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete) I1207 02:51:00.255697 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 4.66583e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 18:50:48.967564 13446 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete) I1207 02:51:00.455852 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 4.44272e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 18:50:49.167713 13446 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete) I1207 02:51:00.656059 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 4.23993e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 18:50:49.367856 13446 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete) I1207 02:51:00.856274 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 4.05484e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 18:50:49.567999 13446 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete) I1207 02:51:01.056495 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 3.88522e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 18:50:49.768143 13446 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete) I1207 02:51:01.256732 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 3.72922e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 18:50:49.968300 13446 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete) I1207 02:51:01.456986 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 3.58525e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 18:50:50.168464 13446 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete) I1207 02:51:01.657230 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 3.45198e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 18:50:50.368628 13446 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete) I1207 02:51:01.857601 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 3.3282e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 18:50:50.568780 13446 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete) I1207 02:51:02.057842 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 3.21306e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 18:50:50.765522 13441 WdtSocket.cpp:282] wdt> socket io timed out after 5000 ms, retries 8754349 fd 6 doneBytes 0 I1207 18:50:50.765550 13445 WdtSocket.cpp:282] wdt> socket io timed out after 5000 ms, retries 7521722 fd 8 doneBytes 0 E1207 18:50:50.765683 13441 WdtSocket.cpp:324] wdt> Unable to read tag at end of stream got -1 needed 16 0x00000000000000000000000000000000 E1207 18:50:50.765702 13445 WdtSocket.cpp:324] wdt> Unable to read tag at end of stream got -1 needed 16 0x00000000000000000000000000000000 E1207 02:51:02.064534 21697 WdtSocket.cpp:29] wdt> Failed to read encryption settings 0 35384 E1207 02:51:02.064535 21701 WdtSocket.cpp:29] wdt> Failed to read encryption settings 0 47603 E1207 02:51:02.064581 21697 ReceiverThread.cpp:44] Read error on 35384 after 0: Resource temporarily unavailable [11] E1207 02:51:02.064587 21701 ReceiverThread.cpp:44] Read error on 47603 after 0: Resource temporarily unavailable [11] E1207 02:51:02.064631 21697 ReceiverThread.cpp:265] wdt> Thread[3, port: 35384] socket read failure 256 -1 E1207 02:51:02.064643 21701 ReceiverThread.cpp:265] wdt> Thread[7, port: 47603] socket read failure 256 -1 I1207 02:51:02.064666 21701 ReceiverThread.cpp:195] wdt> Thread[7, port: 47603] entered ACCEPT_WITH_TIMEOUT state I1207 02:51:02.064647 21697 ReceiverThread.cpp:195] wdt> Thread[3, port: 35384] entered ACCEPT_WITH_TIMEOUT state I1207 18:50:50.768913 13446 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete) I1207 02:51:02.258024 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 3.10564e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 18:50:50.969048 13446 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete) Error in `wdt': free(): invalid pointer: 0x00007fd614000930 I1207 02:51:02.333204 21697 ReceiverThread.cpp:237] wdt> Thread[3, port: 35384] entered SEND_LOCAL_CHECKPOINT state checkpoint-port: 35384 num-blocks: 0 seq-id: 0 block-offset: 0 received-bytes: 0 I1207 02:51:02.333727 21701 ReceiverThread.cpp:237] wdt> Thread[7, port: 47603] entered SEND_LOCAL_CHECKPOINT state checkpoint-port: 47603 num-blocks: 0 seq-id: 0 block-offset: 0 received-bytes: 0 I1207 02:51:02.458220 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 3.00516e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) E1207 02:51:02.570698 21697 WdtSocket.cpp:256] non-retryable error encountered during socket io 31 0 1: Connection reset by peer [104] E1207 02:51:02.570750 21697 WdtSocket.cpp:29] wdt> Failed to read encryption settings -1 35384 E1207 02:51:02.570780 21697 ReceiverThread.cpp:44] Read error on 35384 after 0: Connection reset by peer [104] E1207 02:51:02.570806 21697 ReceiverThread.cpp:265] wdt> Thread[3, port: 35384] socket read failure 256 -1 I1207 02:51:02.570835 21697 ReceiverThread.cpp:195] wdt> Thread[3, port: 35384] entered ACCEPT_WITH_TIMEOUT state I1207 02:51:02.570884 21697 EncryptionUtils.cpp:366] wdt> Encryption finish tag = 0x5ca08bb044c63b00118e0fde4f84db1b E1207 02:51:02.571048 21701 WdtSocket.cpp:256] non-retryable error encountered during socket io 32 0 1: Connection reset by peer [104] E1207 02:51:02.571104 21701 WdtSocket.cpp:29] wdt> Failed to read encryption settings -1 47603 E1207 02:51:02.571120 21701 ReceiverThread.cpp:44] Read error on 47603 after 0: Connection reset by peer [104] E1207 02:51:02.571135 21701 ReceiverThread.cpp:265] wdt> Thread[7, port: 47603] socket read failure 256 -1 I1207 02:51:02.571148 21701 ReceiverThread.cpp:195] wdt> Thread[7, port: 47603] entered ACCEPT_WITH_TIMEOUT state I1207 02:51:02.571177 21701 EncryptionUtils.cpp:366] wdt> Encryption finish tag = 0xea11379ebf080cedf46543d84682b1fc I1207 02:51:02.658404 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 2.91099e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) W1207 02:51:02.667558 21700 ReceiverThread.cpp:75] wdt> Eof on 27 E1207 02:51:02.667672 21700 ReceiverThread.cpp:537] wdt> Thread[6, port: 54616] could not read entire content for ffmpeg.tar.bz2 port 54616 I1207 02:51:02.667717 21700 ReceiverThread.cpp:195] wdt> Thread[6, port: 54616] entered ACCEPT_WITH_TIMEOUT state E1207 02:51:02.667752 21700 EncryptionUtils.cpp:473] wdt> Need tag for gcm mode W1207 02:51:02.719172 21695 ReceiverThread.cpp:75] wdt> Eof on 23 E1207 02:51:02.719282 21695 ReceiverThread.cpp:537] wdt> Thread[1, port: 49590] could not read entire content for ffmpeg.tar.bz2 port 49590 I1207 02:51:02.719326 21695 ReceiverThread.cpp:195] wdt> Thread[1, port: 49590] entered ACCEPT_WITH_TIMEOUT state E1207 02:51:02.719358 21695 EncryptionUtils.cpp:473] wdt> Need tag for gcm mode E1207 02:51:02.757189 21699 ReceiverThread.cpp:572] wdt> Thread[5, port: 49513] socket read failure 256 0 I1207 02:51:02.757382 21699 ReceiverThread.cpp:195] wdt> Thread[5, port: 49513] entered ACCEPT_WITH_TIMEOUT state E1207 02:51:02.757423 21699 EncryptionUtils.cpp:473] wdt> Need tag for gcm mode W1207 02:51:02.780213 21698 ReceiverThread.cpp:75] wdt> Eof on 25 E1207 02:51:02.780335 21698 ReceiverThread.cpp:537] wdt> Thread[4, port: 58549] could not read entire content for ffmpeg.tar.bz2 port 58549 I1207 02:51:02.780380 21698 ReceiverThread.cpp:195] wdt> Thread[4, port: 58549] entered ACCEPT_WITH_TIMEOUT state E1207 02:51:02.780412 21698 EncryptionUtils.cpp:473] wdt> Need tag for gcm mode I1207 02:51:02.858681 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 2.82251e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) W1207 02:51:02.864729 21694 ReceiverThread.cpp:75] wdt> Eof on 21 E1207 02:51:02.864847 21694 ReceiverThread.cpp:537] wdt> Thread[0, port: 47908] could not read entire content for ffmpeg.tar.bz2 port 47908 I1207 02:51:02.864890 21694 ReceiverThread.cpp:195] wdt> Thread[0, port: 47908] entered ACCEPT_WITH_TIMEOUT state E1207 02:51:02.864923 21694 EncryptionUtils.cpp:473] wdt> Need tag for gcm mode I1207 02:51:03.058962 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 2.73924e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:03.259237 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 2.66075e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:03.459511 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 2.58663e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:03.659790 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 2.51653e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:03.860057 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 2.45013e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:04.060333 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 2.38714e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:04.260581 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 2.32732e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) W1207 02:51:04.331140 21696 ReceiverThread.cpp:75] wdt> Eof on 29 E1207 02:51:04.331253 21696 ReceiverThread.cpp:537] wdt> Thread[2, port: 45411] could not read entire content for ffmpeg.tar.bz2 port 45411 I1207 02:51:04.331310 21696 ReceiverThread.cpp:195] wdt> Thread[2, port: 45411] entered ACCEPT_WITH_TIMEOUT state E1207 02:51:04.331352 21696 EncryptionUtils.cpp:473] wdt> Need tag for gcm mode I1207 02:51:04.460783 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 2.27043e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) E1207 02:51:04.571967 21697 ReceiverThread.cpp:218] wdt> Thread[3, port: 35384] accept() failed with error CONN_ERROR timeout 2000 I1207 02:51:04.572010 21697 ReceiverThread.cpp:866] wdt> Thread[3, port: 35384] entered FINISH_WITH_ERROR state I1207 02:51:04.572070 21697 Receiver.cpp:30] wdt> Adding global checkpoint 35384 0 0 E1207 02:51:04.572098 21701 ReceiverThread.cpp:218] wdt> Thread[7, port: 47603] accept() failed with error CONN_ERROR timeout 2000 I1207 02:51:04.572106 21697 ReceiverThread.cpp:964] wdt> Thread[3, port: 35384] Transfer status (local) = CONN_ERROR, (remote) = OK. Number of blocks transferred = 0. Data Mbytes = 0. Header Kbytes = 0.0585938 (100% overhead). Total bytes = 60. Wasted bytes due to failure = 0 (100% overhead). Encryption type = aes128gcm. I1207 02:51:04.572131 21701 ReceiverThread.cpp:866] wdt> Thread[7, port: 47603] entered FINISH_WITH_ERROR state I1207 02:51:04.572213 21701 Receiver.cpp:30] wdt> Adding global checkpoint 47603 0 0 I1207 02:51:04.572237 21701 ReceiverThread.cpp:964] wdt> Thread[7, port: 47603] Transfer status (local) = CONN_ERROR, (remote) = OK. Number of blocks transferred = 0. Data Mbytes = 0. Header Kbytes = 0.0585938 (100% overhead). Total bytes = 60. Wasted bytes due to failure = 0 (100% overhead). Encryption type = aes128gcm. I1207 02:51:04.661051 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 2.21624e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:04.861266 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 2.1646e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:05.061517 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 2.11529e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:05.261729 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 2.06819e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:05.461952 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 2.02314e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:05.662168 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.98001e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:05.862378 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.93869e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:06.062590 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.89905e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:06.262804 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.861e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:06.463014 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.82445e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:06.663224 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.7893e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:06.863433 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.75549e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:07.063647 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.72292e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:07.263859 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.69155e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:07.464063 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.66129e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:07.664271 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.6321e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:07.864480 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.60392e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:08.064704 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.57669e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:08.264933 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.55037e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:08.465144 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.52492e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:08.665357 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.50029e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) E1207 02:51:08.668330 21700 ReceiverThread.cpp:218] wdt> Thread[6, port: 54616] accept() failed with error CONN_ERROR timeout 6000 I1207 02:51:08.668367 21700 ReceiverThread.cpp:866] wdt> Thread[6, port: 54616] entered FINISH_WITH_ERROR state I1207 02:51:08.668419 21700 Receiver.cpp:30] wdt> Adding global checkpoint 54616 0 0 I1207 02:51:08.668440 21700 ReceiverThread.cpp:964] wdt> Thread[6, port: 54616] Transfer status (local) = CONN_ERROR, (remote) = OK. Number of blocks transferred = 0. Data Mbytes = 0. Header Kbytes = 0.0322266 (100% overhead). Total bytes = 7222578. Wasted bytes due to failure = 7222545 (100% overhead). Encryption type = aes128gcm. E1207 02:51:08.719959 21695 ReceiverThread.cpp:218] wdt> Thread[1, port: 49590] accept() failed with error CONN_ERROR timeout 6000 I1207 02:51:08.719993 21695 ReceiverThread.cpp:866] wdt> Thread[1, port: 49590] entered FINISH_WITH_ERROR state I1207 02:51:08.720039 21695 Receiver.cpp:30] wdt> Adding global checkpoint 49590 0 0 I1207 02:51:08.720057 21695 ReceiverThread.cpp:964] wdt> Thread[1, port: 49590] Transfer status (local) = CONN_ERROR, (remote) = OK. Number of blocks transferred = 0. Data Mbytes = 0. Header Kbytes = 0.0322266 (100% overhead). Total bytes = 8882978. Wasted bytes due to failure = 8882945 (100% overhead). Encryption type = aes128gcm. E1207 02:51:08.758036 21699 ReceiverThread.cpp:218] wdt> Thread[5, port: 49513] accept() failed with error CONN_ERROR timeout 6000 I1207 02:51:08.758093 21699 ReceiverThread.cpp:866] wdt> Thread[5, port: 49513] entered FINISH_WITH_ERROR state I1207 02:51:08.758157 21699 Receiver.cpp:30] wdt> Adding global checkpoint 49513 0 0 I1207 02:51:08.758193 21699 ReceiverThread.cpp:964] wdt> Thread[5, port: 49513] Transfer status (local) = CONN_ERROR, (remote) = OK. Number of blocks transferred = 0. Data Mbytes = 0. Header Kbytes = 0.0292969 (100% overhead). Total bytes = 16777246. Wasted bytes due to failure = 16777216 (100% overhead). Encryption type = aes128gcm. E1207 02:51:08.781049 21698 ReceiverThread.cpp:218] wdt> Thread[4, port: 58549] accept() failed with error CONN_ERROR timeout 6000 I1207 02:51:08.781083 21698 ReceiverThread.cpp:866] wdt> Thread[4, port: 58549] entered FINISH_WITH_ERROR state I1207 02:51:08.781129 21698 Receiver.cpp:30] wdt> Adding global checkpoint 58549 0 0 I1207 02:51:08.781147 21698 ReceiverThread.cpp:964] wdt> Thread[4, port: 58549] Transfer status (local) = CONN_ERROR, (remote) = OK. Number of blocks transferred = 0. Data Mbytes = 0. Header Kbytes = 0.0322266 (100% overhead). Total bytes = 10327465. Wasted bytes due to failure = 10327432 (100% overhead). Encryption type = aes128gcm. I1207 02:51:08.865571 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.47644e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) E1207 02:51:08.865777 21694 ReceiverThread.cpp:218] wdt> Thread[0, port: 47908] accept() failed with error CONN_ERROR timeout 6000 I1207 02:51:08.865810 21694 ReceiverThread.cpp:866] wdt> Thread[0, port: 47908] entered FINISH_WITH_ERROR state I1207 02:51:08.865855 21694 Receiver.cpp:30] wdt> Adding global checkpoint 47908 0 0 I1207 02:51:08.865874 21694 ReceiverThread.cpp:964] wdt> Thread[0, port: 47908] Transfer status (local) = CONN_ERROR, (remote) = OK. Number of blocks transferred = 0. Data Mbytes = 0. Header Kbytes = 0.0322266 (100% overhead). Total bytes = 1342578. Wasted bytes due to failure = 1342545 (100% overhead). Encryption type = aes128gcm. I1207 02:51:09.065784 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.45334e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:09.266005 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.43095e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:09.466218 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.40924e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:09.666429 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.38818e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:09.866642 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.36774e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:10.066854 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.34789e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:10.267066 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.32861e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) E1207 02:51:10.331954 21696 ReceiverThread.cpp:218] wdt> Thread[2, port: 45411] accept() failed with error CONN_ERROR timeout 6000 I1207 02:51:10.331990 21696 ReceiverThread.cpp:866] wdt> Thread[2, port: 45411] entered FINISH_WITH_ERROR state I1207 02:51:10.332042 21696 Receiver.cpp:30] wdt> Adding global checkpoint 45411 0 0 I1207 02:51:10.332068 21696 Receiver.cpp:107] wdt> Ending the transfer 540054088 I1207 02:51:10.332085 21696 ReceiverThread.cpp:964] wdt> Thread[2, port: 45411] Transfer status (local) = CONN_ERROR, (remote) = OK. Number of blocks transferred = 0. Data Mbytes = 0. Header Kbytes = 0.0322266 (100% overhead). Total bytes = 2661065. Wasted bytes due to failure = 2661032 (100% overhead). Encryption type = aes128gcm. E1207 02:51:10.332516 21692 Reporting.cpp:200] wdt> Did not receive all the blocks sent by the sender -1 0 I1207 02:51:10.332576 21692 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.32242e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (incomplete) W1207 02:51:10.332618 21692 Receiver.cpp:331] wdt> WDT receiver's transfer has been finished I1207 02:51:10.332646 21692 Receiver.cpp:332] wdt> Transfer status (local) = CONN_ERROR, (remote) = OK. Number of blocks transferred = 0. Data Mbytes = 0. Header Kbytes = 0.307617 (100% overhead). Total bytes = 47214030. Wasted bytes due to failure = 47213715 (100% overhead). Encryption type = none. Previously sent bytes : 0. W1207 02:51:10.332726 21692 Receiver.cpp:300] wdt> Threads have already been joined. Returning the transfer report E1207 02:51:10.332757 21692 Reporting.cpp:200] wdt> Did not receive all the blocks sent by the sender -1 0 E1207 02:51:10.333192 21692 wdtCmdLine.cpp:374] wdt> Returning with code 3 CONN_ERROR I1207 02:51:10.333254 21692 WdtResourceController.cpp:294] wdt> Shutting down the controller (0 senders 0 receivers)

ldemailly commented 7 years ago

I will look more later but it seems you get a crash (free error) can you get the stack trace for that (run with gdb) ?

ldemailly commented 7 years ago

It looks like your sender failed to connect to 2 of the 8 ports

can you try a few things:

a) reduce the number of ports ( -num_ports=4 for instance) b) use gdb as mentioned above c) run with -v=1 and maybe a small file see if the error/crash happens also with small and verbose d) you may have some ipv4/ipv6 issue - try passing -ipv4 or -ipv6 depending on which you have e) maybe some ports are blocked, if you know ports that are not blocked use those, for instance wdt -start_port 12345 f) which OS is this on ? (uname -a) g) to rule out openssl version issues, try -encryption_type none

please let me know more, in particular the crash, that shouldn't happen even if caused by port blocking etc

thanks!

nxtreaming commented 7 years ago

I use the same command, it seems fine!!!

I will try to find the reason.

Thanks for your prompt feedback.

ldemailly commented 7 years ago

yes if you can reproduce the *** Error in 'wdt': free(): invalid pointer: 0x00007fd614000930 *** part I would really like to know / fix it

nxtreaming commented 7 years ago

I fail to get the stack by using gdb

How to compile wdt in DEBUG mode?

ldemailly commented 7 years ago

in CMakeLists.txt change

#set(CMAKE_BUILD_TYPE Debug)
set(CMAKE_BUILD_TYPE Release)

to

set(CMAKE_BUILD_TYPE Debug)
#set(CMAKE_BUILD_TYPE Release)

btw did you run cmake with -DBUILD_TESTING=1 and ran make -j && CTEST_OUTPUT_ON_FAILURE=1 make test and that passes ? (if not send the output and also the output of cmake)

nxtreaming commented 7 years ago

Still fail to get the crash back stack

root@host5833c749:/data/dev/wdt# make install [ 11%] Built target folly4wdt [ 85%] Built target wdt_min [ 91%] Built target wdt [100%] Built target wdtbin Install the project... -- Install configuration: "Debug" -- Installing: /usr/local/bin/wdt -- Set runtime path of "/usr/local/bin/wdt" to "/usr/local/lib" -- Installing: /usr/local/lib/libwdt.so -- Set runtime path of "/usr/local/lib/libwdt.so" to "/usr/local/lib" -- Installing: /usr/local/lib/libwdt_min.so -- Set runtime path of "/usr/local/lib/libwdt_min.so" to "/usr/local/lib" -- Installing: /usr/local/lib/libfolly4wdt.so -- Set runtime path of "/usr/local/lib/libfolly4wdt.so" to "/usr/local/lib" -- Installing: /usr/local/include/wdt/AbortChecker.h -- Installing: /usr/local/include/wdt/WdtConfig.h -- Installing: /usr/local/include/wdt/SenderThread.h -- Installing: /usr/local/include/wdt/Protocol.h -- Installing: /usr/local/include/wdt/Reporting.h -- Installing: /usr/local/include/wdt/Writer.h -- Installing: /usr/local/include/wdt/WdtBase.h -- Installing: /usr/local/include/wdt/Throttler.h -- Installing: /usr/local/include/wdt/Receiver.h -- Installing: /usr/local/include/wdt/WdtOptions.h -- Installing: /usr/local/include/wdt/ErrorCodes.h -- Installing: /usr/local/include/wdt/Wdt.h -- Installing: /usr/local/include/wdt/WdtThread.h -- Installing: /usr/local/include/wdt/Sender.h -- Installing: /usr/local/include/wdt/WdtResourceController.h -- Installing: /usr/local/include/wdt/WdtTransferRequest.h -- Installing: /usr/local/include/wdt/ReceiverThread.h -- Installing: /usr/local/include/wdt/ByteSource.h -- Installing: /usr/local/include/wdt/SourceQueue.h -- Up-to-date: /usr/local/include/wdt/util -- Installing: /usr/local/include/wdt/util/WdtSocket.h -- Installing: /usr/local/include/wdt/util/WdtFlagsMacros.h -- Installing: /usr/local/include/wdt/util/TransferLogManager.h -- Installing: /usr/local/include/wdt/util/ServerSocket.h -- Installing: /usr/local/include/wdt/util/DirectorySourceQueue.h -- Installing: /usr/local/include/wdt/util/ClientSocket.h -- Installing: /usr/local/include/wdt/util/EncryptionUtils.h -- Installing: /usr/local/include/wdt/util/FileWriter.h -- Installing: /usr/local/include/wdt/util/SerializationUtil.h -- Installing: /usr/local/include/wdt/util/ThreadsController.h -- Installing: /usr/local/include/wdt/util/FileCreator.h -- Installing: /usr/local/include/wdt/util/CommonImpl.h -- Installing: /usr/local/include/wdt/util/ThreadTransferHistory.h -- Installing: /usr/local/include/wdt/util/WdtFlags.h -- Installing: /usr/local/include/wdt/util/FileByteSource.h -- Installing: /usr/local/bin/wcp root@host5833c749:/data/dev/wdt# gdb wdb GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1 Copyright (C) 2014 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/. Find the GDB manual and other documentation resources online at: http://www.gnu.org/software/gdb/documentation/. For help, type "help". Type "apropos word" to search for commands related to "word"... wdb: No such file or directory. (gdb) set args -num_ports 4 -progress_report_interval_millis 0 -directory /home/wdt_test | ssh upcn.fakecn.com wdt -num_ports 4 -progress_report_interval_millis 0 -directory /data/dev/wdt_test - (gdb) r Starting program: -num_ports 4 -progress_report_interval_millis 0 -directory /home/wdt_test | ssh upcn.fakecn.com wdt -num_ports 4 -progress_report_interval_millis 0 -directory /data/dev/wdt_test - No executable file specified. Use the "file" or "exec-file" command. (gdb) quit root@host5833c749:/data/dev/wdt# gdb /usr/local/bin/wdt GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1 Copyright (C) 2014 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/. Find the GDB manual and other documentation resources online at: http://www.gnu.org/software/gdb/documentation/. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/local/bin/wdt...done. (gdb) set args -num_ports 4 -progress_report_interval_millis 0 -directory /home/wdt_test | ssh upcn.fakecn.com wdt -num_ports 4 -progress_report_interval_millis 0 -directory /data/dev/wdt_test - (gdb) r Starting program: /usr/local/bin/wdt -num_ports 4 -progress_report_interval_millis 0 -directory /home/wdt_test | ssh upcn.fakecn.com wdt -num_ports 4 -progress_report_interval_millis 0 -directory /data/dev/wdt_test - I1208 08:08:23.462692 30988 WdtFlags.cpp:63] wdt> Running WDT 1.27.1612070 p 27 I1208 08:08:23.463438 30988 EncryptionUtils.cpp:83] wdt> Openssl library initialized I1208 08:08:23.463565 30988 WdtFlags.cpp:63] wdt> Running WDT 1.27.1612070 p 27 I1208 08:08:23.464022 30988 WdtResourceController.cpp:29] wdt> Updated max number of senders for root controller to 0 I1208 08:08:23.464136 30988 WdtResourceController.cpp:22] wdt> Updated max number of receivers for root controller to 0 I1208 08:08:23.464241 30988 Throttler.cpp:73] wdt> No average rate specified I1208 08:08:23.464335 30988 Throttler.cpp:80] wdt> No peak rate specified I1208 08:08:23.464469 30988 Wdt.cpp:24] wdt> One time initialization of WDT for wdt I1208 08:08:23.464591 30988 Throttler.cpp:98] wdt> Updating the rates avgRateBytesPerSec : -1.04858e+06 bucketRateBytesPerSec : 0 bytesTokenBucketLimit : 0 I1208 08:08:23.464889 30988 Receiver.cpp:46] wdt> WDT Receiver 1.27.1612070 p 27 I1208 08:08:23.465102 30988 WdtBase.cpp:82] wdt> Setting transfer id 1270292567 I1208 08:08:23.465209 30988 WdtBase.cpp:94] wdt> using wdt protocol version 27 I1208 08:08:23.465374 30988 FileCreator.cpp:319] wdt> dir already exists / I1208 08:08:23.465509 30988 FileCreator.cpp:319] wdt> dir already exists /home/ I1208 08:08:23.465617 30988 FileCreator.cpp:319] wdt> dir already exists /home/wdt_test/ I1208 08:08:23.465735 30988 Receiver.cpp:170] wdt> aes128gcm encryption is enabled for this transfer I1208 08:08:23.465837 30988 Receiver.cpp:173] wdt> Receiver generating encryption key for type aes128gcm I1208 08:08:23.466081 30988 EncryptionUtils.cpp:115] wdt> New encryption params 0x7fffffffdb90 2:...16031797015489660015... I1208 08:08:23.467111 30988 ReceiverThread.cpp:987] wdt> Thread[0, port: 49275] Listening on port 49275 I1208 08:08:23.467490 30988 ReceiverThread.cpp:987] wdt> Thread[1, port: 46859] Listening on port 46859 I1208 08:08:23.467769 30988 ReceiverThread.cpp:987] wdt> Thread[2, port: 55467] Listening on port 55467 I1208 08:08:23.468034 30988 ReceiverThread.cpp:987] wdt> Thread[3, port: 57551] Listening on port 57551 I1208 08:08:23.468154 30988 Receiver.cpp:212] wdt> Registered 4 successful sockets I1208 08:08:23.468406 30988 wdtCmdLine.cpp:321] wdt> Starting receiver with connection url wdt://host5833c749?dir=%2fhome%2fwdt_test&enc=2:...16031797015489660015...&id=1270292567&ports=49275,46859,55467,57551&recpv=27 I1208 08:08:23.468585 30988 Receiver.cpp:455] wdt> Starting (receiving) server on ports [ 49275 46859 55467 57551 ] Target dir : /home/wdt_test I1208 08:08:23.468708 30988 Throttler.cpp:73] wdt> No average rate specified I1208 08:08:23.468804 30988 Throttler.cpp:80] wdt> No peak rate specified I1208 08:08:23.468902 30988 WdtBase.cpp:183] wdt> Enabling throttling avgRate: -1 Mbytes/sec, peakRate: 0 Mbytes/sec, bucketLimit: 0 Mbytes, throttlerLogTimeMillis: 0 I1208 08:08:20.337579 607 WdtFlags.cpp:63] wdt> Running WDT 1.27.1612021 p 27 I1208 08:08:20.338167 607 EncryptionUtils.cpp:83] wdt> Openssl library initialized I1208 08:08:20.338199 607 WdtFlags.cpp:63] wdt> Running WDT 1.27.1612021 p 27 I1208 08:08:20.338446 607 WdtResourceController.cpp:29] wdt> Updated max number of senders for root controller to 0 I1208 08:08:20.338469 607 WdtResourceController.cpp:22] wdt> Updated max number of receivers for root controller to 0 I1208 08:08:20.338475 607 Throttler.cpp:73] wdt> No average rate specified I1208 08:08:20.338479 607 Throttler.cpp:80] wdt> No peak rate specified I1208 08:08:20.338488 607 Wdt.cpp:24] wdt> One time initialization of WDT for wdt I1208 08:08:20.338501 607 Throttler.cpp:98] wdt> Updating the rates avgRateBytesPerSec : -1.04858e+06 bucketRateBytesPerSec : 0 bytesTokenBucketLimit : 0 I1208 08:08:20.338599 607 wdtCmdLine.cpp:282] wdt> Parsed url as wdt://host5833c749?dir=%2fdata%2fdev%2fwdt_test&enc=2:...16031797015489660015...&id=1270292567&ports=49275,46859,55467,57551&recpv=27 I1208 08:08:20.338614 607 wdtCmdLine.cpp:365] wdt> Making Sender with encryption set = 1 I1208 08:08:20.338627 607 WdtResourceController.cpp:357] wdt> First time (default) is seen, creating. I1208 08:08:20.338635 607 WdtResourceController.cpp:29] wdt> Updated max number of senders for to 0 I1208 08:08:20.338639 607 WdtResourceController.cpp:22] wdt> Updated max number of receivers for to 1 I1208 08:08:20.338649 607 Throttler.cpp:98] wdt> Updating the rates avgRateBytesPerSec : -1.04858e+06 bucketRateBytesPerSec : 0 bytesTokenBucketLimit : 0 I1208 08:08:24.971940 30992 Receiver.cpp:93] wdt> Starting new transfer, peerIp 120.132.7.182 , transfer id 1270292567 I1208 08:08:24.972123 30992 WdtSocket.cpp:67] wdt> Successfully read encryption settings 55467 aes128gcm I1208 08:08:20.338665 607 Sender.cpp:47] wdt> WDT Sender 1.27.1612021 p 27 I1208 08:08:20.338678 607 WdtResourceController.cpp:379] wdt> Successfully added a sender for identifier host5833c749 I1208 08:08:20.338688 607 WdtBase.cpp:94] wdt> using wdt protocol version 27 I1208 08:08:20.338692 607 Sender.cpp:84] wdt> Encryption is enabled for this transfer I1208 08:08:20.338713 607 DirectorySourceQueue.cpp:127] wdt> Root dir now /data/dev/wdt_test/ I1208 08:08:20.338732 607 Sender.cpp:311] wdt> Client (sending) to host5833c749, Using ports [ 49275 46859 55467 57551 ] I1208 08:08:20.338737 607 Sender.cpp:325] wdt> Skipping throttler setup. External throttler set.Throttler details : avgRate: -1 Mbytes/sec, peakRate: 0 Mbytes/sec, bucketLimit: 0 Mbytes, throttlerLogTimeMillis: 0 I1208 08:08:20.339030 608 DirectorySourceQueue.cpp:240] wdt> Exploring root dir /data/dev/wdt_test/ include_pattern : exclude_pattern : prune_dir_pattern : I1208 08:08:20.339032 609 Sender.cpp:41] wdt> Starting a new transfer 1270292567 to host5833c749 I1208 08:08:20.339226 608 DirectorySourceQueue.cpp:389] wdt> Number of files explored: 3 opened 0 with direct 0 errors false I1208 08:08:20.433791 611 SenderThread.cpp:85] wdt> Thread[2, port: 55467] Connection took 1 attempt(s) and 0.0946553 seconds. port 55467 I1208 08:08:24.972959 30991 WdtSocket.cpp:67] wdt> Successfully read encryption settings 46859 aes128gcm I1208 08:08:24.973100 30993 WdtSocket.cpp:67] wdt> Successfully read encryption settings 57551 aes128gcm I1208 08:08:20.437222 612 SenderThread.cpp:85] wdt> Thread[3, port: 57551] Connection took 1 attempt(s) and 0.0980262 seconds. port 57551 I1208 08:08:20.437528 610 SenderThread.cpp:85] wdt> Thread[1, port: 46859] Connection took 1 attempt(s) and 0.0984208 seconds. port 46859 I1208 08:08:20.445560 609 SenderThread.cpp:85] wdt> Thread[0, port: 49275] Connection took 1 attempt(s) and 0.106469 seconds. port 49275 I1208 08:08:24.983455 30990 WdtSocket.cpp:67] wdt> Successfully read encryption settings 49275 aes128gcm Error in `wdt': free(): invalid pointer: 0x00007f2134000930 W1208 08:08:49.343605 30992 ReceiverThread.cpp:75] wdt> Eof on 11 E1208 08:08:49.343678 30992 ReceiverThread.cpp:537] wdt> Thread[2, port: 55467] could not read entire content for ffmpeg.tar.bz2 port 55467 I1208 08:08:49.343755 30992 ReceiverThread.cpp:195] wdt> Thread[2, port: 55467] entered ACCEPT_WITH_TIMEOUT state E1208 08:08:49.343812 30992 EncryptionUtils.cpp:473] wdt> Need tag for gcm mode W1208 08:08:49.343605 30990 ReceiverThread.cpp:75] wdt> Eof on 17 E1208 08:08:49.344055 30990 ReceiverThread.cpp:537] wdt> Thread[0, port: 49275] could not read entire content for ffmpeg.tar.bz2 port 49275 I1208 08:08:49.344243 30990 ReceiverThread.cpp:195] wdt> Thread[0, port: 49275] entered ACCEPT_WITH_TIMEOUT state E1208 08:08:49.344416 30990 EncryptionUtils.cpp:473] wdt> Need tag for gcm mode E1208 08:08:49.433836 30991 ReceiverThread.cpp:572] wdt> Thread[1, port: 46859] socket read failure 256 0 I1208 08:08:49.433939 30991 ReceiverThread.cpp:195] wdt> Thread[1, port: 46859] entered ACCEPT_WITH_TIMEOUT state E1208 08:08:49.433972 30991 EncryptionUtils.cpp:473] wdt> Need tag for gcm mode W1208 08:08:49.783360 30993 ReceiverThread.cpp:75] wdt> Eof on 14 E1208 08:08:49.783558 30993 ReceiverThread.cpp:537] wdt> Thread[3, port: 57551] could not read entire content for ffmpeg.tar.bz2 port 57551 I1208 08:08:49.783723 30993 ReceiverThread.cpp:195] wdt> Thread[3, port: 57551] entered ACCEPT_WITH_TIMEOUT state E1208 08:08:49.783846 30993 EncryptionUtils.cpp:473] wdt> Need tag for gcm mode E1208 08:08:55.344427 30992 ReceiverThread.cpp:218] wdt> Thread[2, port: 55467] accept() failed with error CONN_ERROR timeout 6000 I1208 08:08:55.344540 30992 ReceiverThread.cpp:866] wdt> Thread[2, port: 55467] entered FINISH_WITH_ERROR state I1208 08:08:55.344689 30992 Receiver.cpp:30] wdt> Adding global checkpoint 55467 0 0 I1208 08:08:55.344818 30992 ReceiverThread.cpp:964] wdt> Thread[2, port: 55467] Transfer status (local) = CONN_ERROR, (remote) = OK. Number of blocks transferred = 0. Data Mbytes = 0. Header Kbytes = 0.0292969 (100% overhead). Total bytes = 16654377. Wasted bytes due to failure = 16654347 (100% overhead). Encryption type = aes128gcm. E1208 08:08:55.345752 30990 ReceiverThread.cpp:218] wdt> Thread[0, port: 49275] accept() failed with error CONN_ERROR timeout 6000 I1208 08:08:55.345778 30990 ReceiverThread.cpp:866] wdt> Thread[0, port: 49275] entered FINISH_WITH_ERROR state I1208 08:08:55.345823 30990 Receiver.cpp:30] wdt> Adding global checkpoint 49275 0 0 I1208 08:08:55.345854 30990 ReceiverThread.cpp:964] wdt> Thread[0, port: 49275] Transfer status (local) = CONN_ERROR, (remote) = OK. Number of blocks transferred = 0. Data Mbytes = 0. Header Kbytes = 0.0322266 (100% overhead). Total bytes = 16629177. Wasted bytes due to failure = 16629144 (100% overhead). Encryption type = aes128gcm. E1208 08:08:55.434906 30991 ReceiverThread.cpp:218] wdt> Thread[1, port: 46859] accept() failed with error CONN_ERROR timeout 6000 I1208 08:08:55.435003 30991 ReceiverThread.cpp:866] wdt> Thread[1, port: 46859] entered FINISH_WITH_ERROR state I1208 08:08:55.435086 30991 Receiver.cpp:30] wdt> Adding global checkpoint 46859 0 0 I1208 08:08:55.435163 30991 ReceiverThread.cpp:964] wdt> Thread[1, port: 46859] Transfer status (local) = CONN_ERROR, (remote) = OK. Number of blocks transferred = 0. Data Mbytes = 0. Header Kbytes = 0.0322266 (100% overhead). Total bytes = 16777249. Wasted bytes due to failure = 16777216 (100% overhead). Encryption type = aes128gcm. E1208 08:08:55.784584 30993 ReceiverThread.cpp:218] wdt> Thread[3, port: 57551] accept() failed with error CONN_ERROR timeout 6000 I1208 08:08:55.784656 30993 ReceiverThread.cpp:866] wdt> Thread[3, port: 57551] entered FINISH_WITH_ERROR state I1208 08:08:55.784710 30993 Receiver.cpp:30] wdt> Adding global checkpoint 57551 0 0 I1208 08:08:55.784751 30993 Receiver.cpp:107] wdt> Ending the transfer 1270292567 I1208 08:08:55.784806 30993 ReceiverThread.cpp:964] wdt> Thread[3, port: 57551] Transfer status (local) = CONN_ERROR, (remote) = OK. Number of blocks transferred = 0. Data Mbytes = 0. Header Kbytes = 0.0322266 (100% overhead). Total bytes = 7450777. Wasted bytes due to failure = 7450744 (100% overhead). Encryption type = aes128gcm. E1208 08:08:55.784953 30988 Reporting.cpp:200] wdt> Did not receive all the blocks sent by the sender -1 0 W1208 08:08:55.785006 30988 Receiver.cpp:331] wdt> WDT receiver's transfer has been finished I1208 08:08:55.785019 30988 Receiver.cpp:332] wdt> Transfer status (local) = CONN_ERROR, (remote) = OK. Number of blocks transferred = 0. Data Mbytes = 0. Header Kbytes = 0.125977 (100% overhead). Total bytes = 57511580. Wasted bytes due to failure = 57511451 (100% overhead). Encryption type = none. Previously sent bytes : 0. W1208 08:08:55.785070 30988 Receiver.cpp:300] wdt> Threads have already been joined. Returning the transfer report E1208 08:08:55.785085 30988 Reporting.cpp:200] wdt> Did not receive all the blocks sent by the sender -1 0 E1208 08:08:55.785622 30988 wdtCmdLine.cpp:374] wdt> Returning with code 3 CONN_ERROR I1208 08:08:55.785656 30988 WdtResourceController.cpp:294] wdt> Shutting down the controller (0 senders 0 receivers) During startup program exited with code 255. (gdb) bt No stack. (gdb)

ldemailly commented 7 years ago

looks like it is your other side that crashes then (you can tell that by redirecting too:

if you run

( wdt --directory /home/wdt_test 2> /tmp/receiver.log ) | ( ssh cn.wdthost.com wdt --directory /data/dev/wdt_test - 2> /tmp/sender.log )

which one has the Error in `wdt': free(): invalid pointer: 0x00007f2134000930 ? gdb that one

you may need to have to use

run --connection_url $URL ... on the remote side where URL is what the receiver outputs on stdout (which is what - means, read that url from stdin, which works well with ssh but you can also just pass it on the command line)

Also what about my other questions about the tests passing ?

nxtreaming commented 7 years ago

TEST passed

root@host5833c749:/data/dev/wdt# CTEST_OUTPUT_ON_FAILURE=1 make test Running tests... Test project /data/dev/wdt Start 1: AllTestsInGenTest 1/23 Test #1: AllTestsInGenTest ................ Passed 0.33 sec Start 2: AllTestsInProtocolTest 2/23 Test #2: AllTestsInProtocolTest ........... Passed 0.15 sec Start 3: test_encdeci64_func 3/23 Test #3: test_encdeci64_func .............. Passed 1.86 sec Start 4: ResourceControllerTests 4/23 Test #4: ResourceControllerTests .......... Passed 0.02 sec Start 5: WdtUrlTests 5/23 Test #5: WdtUrlTests ...................... Passed 0.05 sec Start 6: WdtMiscTests 6/23 Test #6: WdtMiscTests ..................... Passed 10.43 sec Start 7: WdtFdTests 7/23 Test #7: WdtFdTests ....................... Passed 0.06 sec Start 8: EncryptionTests 8/23 Test #8: EncryptionTests .................. Passed 0.01 sec Start 9: FileReaderTests 9/23 Test #9: FileReaderTests .................. Passed 0.08 sec Start 10: WdtRandGenTest 10/23 Test #10: WdtRandGenTest ................... Passed 0.04 sec Start 11: WdtBasicE2E 11/23 Test #11: WdtBasicE2E ...................... Passed 14.85 sec Start 12: WdtLockFailFast 12/23 Test #12: WdtLockFailFast .................. Passed 3.03 sec Start 13: WdtBasicE2Exfs 13/23 Test #13: WdtBasicE2Exfs ................... Passed 0.01 sec Start 14: WdtOptionsTypeTests 14/23 Test #14: WdtOptionsTypeTests .............. Passed 0.10 sec Start 15: hostname_override_test 15/23 Test #15: hostname_override_test ........... Passed 1.36 sec Start 16: WdtPortBlockTest 16/23 Test #16: WdtPortBlockTest ................. Passed 25.09 sec Start 17: WdtProtocolNegotiationTest 17/23 Test #17: WdtProtocolNegotiationTest ....... Passed 0.09 sec Start 18: WdtSimpleOdirectTest 18/23 Test #18: WdtSimpleOdirectTest ............. Passed 13.64 sec Start 19: WdtFileListTest 19/23 Test #19: WdtFileListTest .................. Passed 3.65 sec Start 20: WdtOverwriteTest 20/23 Test #20: WdtOverwriteTest ................. Passed 6.11 sec Start 21: WdtBadServerTest 21/23 Test #21: WdtBadServerTest ................. Passed 12.51 sec Start 22: ReceiverThrottlerRefCountTest 22/23 Test #22: ReceiverThrottlerRefCountTest .... Passed 1.02 sec Start 23: WdtLongRunningTest 23/23 Test #23: WdtLongRunningTest ............... Passed 1.48 sec

100% tests passed, 0 tests failed out of 23

Total Test time (real) = 96.00 sec

nxtreaming commented 7 years ago

It seems that the "free() error" comes from wdt sender. wdt -num_ports 4 -progress_report_interval_millis 0 -directory /home/wdt_test 2>rec.log | ssh upcn.fakecn.com wdt -num_ports 4 -progress_report_interval_millis 0 -directory /data/dev/wdt_test - 2>send.log

rec.log

I1208 08:29:31.663635 3275 WdtFlags.cpp:63] wdt> Running WDT 1.27.1612070 p 27 I1208 08:29:31.664383 3275 EncryptionUtils.cpp:83] wdt> Openssl library initialized I1208 08:29:31.664424 3275 WdtFlags.cpp:63] wdt> Running WDT 1.27.1612070 p 27 I1208 08:29:31.664906 3275 WdtResourceController.cpp:29] wdt> Updated max number of senders for root controller to 0 I1208 08:29:31.665000 3275 WdtResourceController.cpp:22] wdt> Updated max number of receivers for root controller to 0 I1208 08:29:31.665074 3275 Throttler.cpp:73] wdt> No average rate specified I1208 08:29:31.665089 3275 Throttler.cpp:80] wdt> No peak rate specified I1208 08:29:31.665136 3275 Wdt.cpp:24] wdt> One time initialization of WDT for wdt I1208 08:29:31.665172 3275 Throttler.cpp:98] wdt> Updating the rates avgRateBytesPerSec : -1.04858e+06 bucketRateBytesPerSec : 0 bytesTokenBucketLimit : 0 I1208 08:29:31.665591 3275 Receiver.cpp:46] wdt> WDT Receiver 1.27.1612070 p 27 I1208 08:29:31.665725 3275 WdtBase.cpp:82] wdt> Setting transfer id 1149010876 I1208 08:29:31.665755 3275 WdtBase.cpp:94] wdt> using wdt protocol version 27 I1208 08:29:31.665863 3275 FileCreator.cpp:319] wdt> dir already exists / I1208 08:29:31.666018 3275 FileCreator.cpp:319] wdt> dir already exists /home/ I1208 08:29:31.666100 3275 FileCreator.cpp:319] wdt> dir already exists /home/wdt_test/ I1208 08:29:31.666137 3275 Receiver.cpp:170] wdt> aes128gcm encryption is enabled for this transfer I1208 08:29:31.666151 3275 Receiver.cpp:173] wdt> Receiver generating encryption key for type aes128gcm I1208 08:29:31.666398 3275 EncryptionUtils.cpp:115] wdt> New encryption params 0x7fff78255d50 2:...2824705704520940625... I1208 08:29:31.667999 3275 ReceiverThread.cpp:987] wdt> Thread[0, port: 56035] Listening on port 56035 I1208 08:29:31.668364 3275 ReceiverThread.cpp:987] wdt> Thread[1, port: 43664] Listening on port 43664 I1208 08:29:31.668514 3275 ReceiverThread.cpp:987] wdt> Thread[2, port: 37284] Listening on port 37284 I1208 08:29:31.668642 3275 ReceiverThread.cpp:987] wdt> Thread[3, port: 47271] Listening on port 47271 I1208 08:29:31.668659 3275 Receiver.cpp:212] wdt> Registered 4 successful sockets I1208 08:29:31.668961 3275 wdtCmdLine.cpp:321] wdt> Starting receiver with connection url wdt://host5833c749?dir=%2fhome%2fwdt_test&enc=2:...2824705704520940625...&id=1149010876&ports=56035,43664,37284,47271&recpv=27 I1208 08:29:31.669066 3275 Receiver.cpp:455] wdt> Starting (receiving) server on ports [ 56035 43664 37284 47271 ] Target dir : /home/wdt_test I1208 08:29:31.669100 3275 Throttler.cpp:73] wdt> No average rate specified I1208 08:29:31.669111 3275 Throttler.cpp:80] wdt> No peak rate specified I1208 08:29:31.669123 3275 WdtBase.cpp:183] wdt> Enabling throttling avgRate: -1 Mbytes/sec, peakRate: 0 Mbytes/sec, bucketLimit: 0 Mbytes, throttlerLogTimeMillis: 0 I1208 08:29:33.146258 3277 Receiver.cpp:93] wdt> Starting new transfer, peerIp 120.132.7.182 , transfer id 1149010876 I1208 08:29:33.146520 3277 WdtSocket.cpp:67] wdt> Successfully read encryption settings 56035 aes128gcm I1208 08:29:33.147394 3279 WdtSocket.cpp:67] wdt> Successfully read encryption settings 37284 aes128gcm I1208 08:29:33.150241 3278 WdtSocket.cpp:67] wdt> Successfully read encryption settings 43664 aes128gcm I1208 08:29:33.151386 3280 WdtSocket.cpp:67] wdt> Successfully read encryption settings 47271 aes128gcm W1208 08:29:51.033403 3279 ReceiverThread.cpp:75] wdt> Eof on 12 E1208 08:29:51.033457 3279 ReceiverThread.cpp:537] wdt> Thread[2, port: 37284] could not read entire content for ffmpeg.tar.bz2 port 37284 I1208 08:29:51.033524 3279 ReceiverThread.cpp:195] wdt> Thread[2, port: 37284] entered ACCEPT_WITH_TIMEOUT state E1208 08:29:51.033573 3279 EncryptionUtils.cpp:473] wdt> Need tag for gcm mode E1208 08:29:51.047396 3280 ReceiverThread.cpp:572] wdt> Thread[3, port: 47271] socket read failure 256 0 I1208 08:29:51.047440 3280 ReceiverThread.cpp:195] wdt> Thread[3, port: 47271] entered ACCEPT_WITH_TIMEOUT state E1208 08:29:51.047456 3280 EncryptionUtils.cpp:473] wdt> Need tag for gcm mode W1208 08:29:51.223481 3278 ReceiverThread.cpp:75] wdt> Eof on 13 E1208 08:29:51.223542 3278 ReceiverThread.cpp:537] wdt> Thread[1, port: 43664] could not read entire content for ffmpeg.tar.bz2 port 43664 I1208 08:29:51.223598 3278 ReceiverThread.cpp:195] wdt> Thread[1, port: 43664] entered ACCEPT_WITH_TIMEOUT state E1208 08:29:51.223631 3278 EncryptionUtils.cpp:473] wdt> Need tag for gcm mode W1208 08:29:51.315354 3277 ReceiverThread.cpp:75] wdt> Eof on 11 E1208 08:29:51.315399 3277 ReceiverThread.cpp:537] wdt> Thread[0, port: 56035] could not read entire content for ffmpeg.tar.bz2 port 56035 I1208 08:29:51.315456 3277 ReceiverThread.cpp:195] wdt> Thread[0, port: 56035] entered ACCEPT_WITH_TIMEOUT state E1208 08:29:51.315481 3277 EncryptionUtils.cpp:473] wdt> Need tag for gcm mode E1208 08:29:57.033998 3279 ReceiverThread.cpp:218] wdt> Thread[2, port: 37284] accept() failed with error CONN_ERROR timeout 6000 I1208 08:29:57.034076 3279 ReceiverThread.cpp:866] wdt> Thread[2, port: 37284] entered FINISH_WITH_ERROR state I1208 08:29:57.034209 3279 Receiver.cpp:30] wdt> Adding global checkpoint 37284 0 0 I1208 08:29:57.034307 3279 ReceiverThread.cpp:964] wdt> Thread[2, port: 37284] Transfer status (local) = CONN_ERROR, (remote) = OK. Number of blocks transferred = 0. Data Mbytes = 0. Header Kbytes = 0.0322266 (100% overhead). Total bytes = 16763577. Wasted bytes due to failure = 16763544 (100% overhead). Encryption type = aes128gcm. E1208 08:29:57.048619 3280 ReceiverThread.cpp:218] wdt> Thread[3, port: 47271] accept() failed with error CONN_ERROR timeout 6000 I1208 08:29:57.048651 3280 ReceiverThread.cpp:866] wdt> Thread[3, port: 47271] entered FINISH_WITH_ERROR state I1208 08:29:57.048704 3280 Receiver.cpp:30] wdt> Adding global checkpoint 47271 0 0 I1208 08:29:57.048749 3280 ReceiverThread.cpp:964] wdt> Thread[3, port: 47271] Transfer status (local) = CONN_ERROR, (remote) = OK. Number of blocks transferred = 0. Data Mbytes = 0. Header Kbytes = 0.0322266 (100% overhead). Total bytes = 16777249. Wasted bytes due to failure = 16777216 (100% overhead). Encryption type = aes128gcm. E1208 08:29:57.224597 3278 ReceiverThread.cpp:218] wdt> Thread[1, port: 43664] accept() failed with error CONN_ERROR timeout 6000 I1208 08:29:57.224670 3278 ReceiverThread.cpp:866] wdt> Thread[1, port: 43664] entered FINISH_WITH_ERROR state I1208 08:29:57.224745 3278 Receiver.cpp:30] wdt> Adding global checkpoint 43664 0 0 I1208 08:29:57.224791 3278 ReceiverThread.cpp:964] wdt> Thread[1, port: 43664] Transfer status (local) = CONN_ERROR, (remote) = OK. Number of blocks transferred = 0. Data Mbytes = 0. Header Kbytes = 0.0322266 (100% overhead). Total bytes = 3746377. Wasted bytes due to failure = 3746344 (100% overhead). Encryption type = aes128gcm. E1208 08:29:57.316146 3277 ReceiverThread.cpp:218] wdt> Thread[0, port: 56035] accept() failed with error CONN_ERROR timeout 6000 I1208 08:29:57.316222 3277 ReceiverThread.cpp:866] wdt> Thread[0, port: 56035] entered FINISH_WITH_ERROR state I1208 08:29:57.316298 3277 Receiver.cpp:30] wdt> Adding global checkpoint 56035 0 0 I1208 08:29:57.316341 3277 Receiver.cpp:107] wdt> Ending the transfer 1149010876 I1208 08:29:57.316411 3277 ReceiverThread.cpp:964] wdt> Thread[0, port: 56035] Transfer status (local) = CONN_ERROR, (remote) = OK. Number of blocks transferred = 0. Data Mbytes = 0. Header Kbytes = 0.0292969 (100% overhead). Total bytes = 4219577. Wasted bytes due to failure = 4219547 (100% overhead). Encryption type = aes128gcm. E1208 08:29:57.317193 3275 Reporting.cpp:200] wdt> Did not receive all the blocks sent by the sender -1 0 W1208 08:29:57.317240 3275 Receiver.cpp:331] wdt> WDT receiver's transfer has been finished I1208 08:29:57.317255 3275 Receiver.cpp:332] wdt> Transfer status (local) = CONN_ERROR, (remote) = OK. Number of blocks transferred = 0. Data Mbytes = 0. Header Kbytes = 0.125977 (100% overhead). Total bytes = 41506780. Wasted bytes due to failure = 41506651 (100% overhead). Encryption type = none. Previously sent bytes : 0. W1208 08:29:57.317322 3275 Receiver.cpp:300] wdt> Threads have already been joined. Returning the transfer report E1208 08:29:57.317340 3275 Reporting.cpp:200] wdt> Did not receive all the blocks sent by the sender -1 0 E1208 08:29:57.317687 3275 wdtCmdLine.cpp:374] wdt> Returning with code 3 CONN_ERROR I1208 08:29:57.317725 3275 WdtResourceController.cpp:294] wdt> Shutting down the controller (0 senders 0 receivers)

send.log

I1208 08:29:28.490911 3679 WdtFlags.cpp:63] wdt> Running WDT 1.27.1612021 p 27 I1208 08:29:28.491353 3679 EncryptionUtils.cpp:83] wdt> Openssl library initialized I1208 08:29:28.491379 3679 WdtFlags.cpp:63] wdt> Running WDT 1.27.1612021 p 27 I1208 08:29:28.491629 3679 WdtResourceController.cpp:29] wdt> Updated max number of senders for root controller to 0 I1208 08:29:28.491652 3679 WdtResourceController.cpp:22] wdt> Updated max number of receivers for root controller to 0 I1208 08:29:28.491662 3679 Throttler.cpp:73] wdt> No average rate specified I1208 08:29:28.491668 3679 Throttler.cpp:80] wdt> No peak rate specified I1208 08:29:28.491683 3679 Wdt.cpp:24] wdt> One time initialization of WDT for wdt I1208 08:29:28.491700 3679 Throttler.cpp:98] wdt> Updating the rates avgRateBytesPerSec : -1.04858e+06 bucketRateBytesPerSec : 0 bytesTokenBucketLimit : 0 I1208 08:29:28.491801 3679 wdtCmdLine.cpp:282] wdt> Parsed url as wdt://host5833c749?dir=%2fdata%2fdev%2fwdt_test&enc=2:...2824705704520940625...&id=1149010876&ports=56035,43664,37284,47271&recpv=27 I1208 08:29:28.491821 3679 wdtCmdLine.cpp:365] wdt> Making Sender with encryption set = 1 I1208 08:29:28.491838 3679 WdtResourceController.cpp:357] wdt> First time (default) is seen, creating. I1208 08:29:28.491850 3679 WdtResourceController.cpp:29] wdt> Updated max number of senders for to 0 I1208 08:29:28.491858 3679 WdtResourceController.cpp:22] wdt> Updated max number of receivers for to 1 I1208 08:29:28.491873 3679 Throttler.cpp:98] wdt> Updating the rates avgRateBytesPerSec : -1.04858e+06 bucketRateBytesPerSec : 0 bytesTokenBucketLimit : 0 I1208 08:29:28.491907 3679 Sender.cpp:47] wdt> WDT Sender 1.27.1612021 p 27 I1208 08:29:28.491930 3679 WdtResourceController.cpp:379] wdt> Successfully added a sender for identifier host5833c749 I1208 08:29:28.491943 3679 WdtBase.cpp:94] wdt> using wdt protocol version 27 I1208 08:29:28.491961 3679 Sender.cpp:84] wdt> Encryption is enabled for this transfer I1208 08:29:28.491993 3679 DirectorySourceQueue.cpp:127] wdt> Root dir now /data/dev/wdt_test/ I1208 08:29:28.492019 3679 Sender.cpp:311] wdt> Client (sending) to host5833c749, Using ports [ 56035 43664 37284 47271 ] I1208 08:29:28.492032 3679 Sender.cpp:325] wdt> Skipping throttler setup. External throttler set.Throttler details : avgRate: -1 Mbytes/sec, peakRate: 0 Mbytes/sec, bucketLimit: 0 Mbytes, throttlerLogTimeMillis: 0 I1208 08:29:28.492280 3680 DirectorySourceQueue.cpp:240] wdt> Exploring root dir /data/dev/wdt_test/ include_pattern : exclude_pattern : prune_dir_pattern : I1208 08:29:28.492321 3681 Sender.cpp:41] wdt> Starting a new transfer 1149010876 to host5833c749 I1208 08:29:28.492409 3680 DirectorySourceQueue.cpp:389] wdt> Number of files explored: 3 opened 0 with direct 0 errors false I1208 08:29:28.595293 3681 SenderThread.cpp:85] wdt> Thread[0, port: 56035] Connection took 1 attempt(s) and 0.102942 seconds. port 56035 I1208 08:29:28.595520 3682 SenderThread.cpp:85] wdt> Thread[1, port: 43664] Connection took 1 attempt(s) and 0.103048 seconds. port 43664 I1208 08:29:28.595544 3683 SenderThread.cpp:85] wdt> Thread[2, port: 37284] Connection took 1 attempt(s) and 0.102781 seconds. port 37284 I1208 08:29:28.599361 3684 SenderThread.cpp:85] wdt> Thread[3, port: 47271] Connection took 1 attempt(s) and 0.106493 seconds. port 47271 Error in `wdt': free(): invalid pointer: 0x00007f5af4000930

ldemailly commented 7 years ago

does it still fail with -encryption_type none ?

if so just open a window on each computer and start the receiver and sender separately and for the sender run it with gdb and add -connection_url "$URL" where URL is what you copy from what the receiver prints after it starts (the line not starting with I1208 )

also run the sender with -v=1

if you want you can use gitter to chat interactively

nxtreaming commented 7 years ago

does it still fail with -encryption_type none ?

NO, it works with ' -encryption_type none'

I will fail without this option.

nxtreaming commented 7 years ago

I also found an issue: If a sub-directory is empty, it will not be sent to the remote receiver ( the receiver has no the empty sub-directory).

ldemailly commented 7 years ago

what is the stack trace of the crash with encryption on ?

which version of openssl do you have ? I suspect that's maybe the issue

it's meant to send files not directories (though when/if we have a rsync mode it would make sense to also deal with directories as such)

ldemailly commented 7 years ago

@nxtreaming ping ^

nxtreaming commented 7 years ago

I will debug this issue today.

On Tue, Dec 13, 2016 at 2:38 PM, Laurent Demailly notifications@github.com wrote:

@nxtreaming https://github.com/nxtreaming ping ^

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/facebook/wdt/issues/138#issuecomment-266659454, or mute the thread https://github.com/notifications/unsubscribe-auth/AC9tCxKK0NOHA7gK2BozWRtHeYovTnM2ks5rHj13gaJpZM4LGdW5 .

ldemailly commented 7 years ago

@nxtreaming happy new year ! any update ? I'm guessing you have a buggy openssl like #141 but it would be nice to confirm

ldemailly commented 7 years ago

closing this but if you do have more info please reopen