Closed arekm closed 11 years ago
lavv17
commented
11 years ago I could not reproduce the problem. But in ec49c021ef83e70ab1d2d111d6b63eddf9cd3e93 I have made Fish disconnect in more cases, which should make protocol desync less likely. Please test.
arekm
commented
11 years ago Was able to reproduce with 4.4.6 and ec49c02. lftp over fish, queue get onefile; queue mirror -c dir-with-55-files-of-15MB-size. Below is part of "debug 10 -o logfile". Also seems that garbage is much more often logged inside of debug logfile than it is on stdout
<--- ### 100 0:0 translated to pair 0:0 (0,0) 0 translated to pair 0:0 (0,0) copy: put confirmed store copy: get is finished - all done copy: put confirmed store copy: get is finished - all done ---> #RETR something.r05 ls -lLd something.r05; echo '### 100'; cat something.r05; echo '### 200' ---> #RETR something.r04 ls -lLd something.r04; echo '### 100'; cat something.r04; echo '### 200' <--- -rw-r--r-- 1 arekm users 15000000 May 20 15:00 something.r03 <--- -rw-r--r-- 1 arekm users 15000000 May 20 15:00 something.r05 <--- ### 100 <--- ### 100 <--- -rw-r--r-- 1 arekm users 15000000 May 20 15:00 something.r04 <--- ### 100 copy: get hit eof copy: waiting for put confirmation copy: put confirmed store copy: get is finished - all done copy: get hit eof copy: waiting for put confirmation copy: put confirmed store copy: get is finished - all done <--- ### 200 copy: get hit eof copy: waiting for put confirmation copy: put confirmed store copy: get is finished - all done ---> #RETR something.r06 ls -lLd something.r06; echo '### 100'; cat something.r06; echo '### 200' <--- -rw-r--r-- 1 arekm users 15000000 May 20 15:00 something.r06 <--- ### 100 <---� �F2yZ�0��l�2�{�MEh�Z��Ś~Q��^0��muF1%ʇ���nj��rԨ[�3����}ά?��8�:��od��Ddj� I]t�'ґ�:�αi�ni� ��9��#<�Q�TA(C��;�K,)�t�ED$<x!D�g��1z�f٤�e��J'iʯ���w�˽�����▒5�y▒d�p����P��{C VZ�.�_���joAq����rE����N��2� copy: get hit eof copy: waiting for put confirmation copy: put confirmed store copy: get is finished - all done <--- ### 200 copy: get hit eof copy: waiting for put confirmation copy: put confirmed store copy: get is finished - all done
OR
copy: get hit eof copy: waiting for put confirmation copy: put confirmed store copy: get is finished - all done ---> #RETR something.r19 ls -lLd something.r19; echo '### 100'; cat something.r19; echo '### 200' <--- -rw-r--r-- 1 arekm users 15000000 May 20 15:00 something.r19 <--- ### 100 <--- �Z�yq lb��F�OI�0Lܧ���b�~��ɡ2M���y��#�$'��1��'reF���▒}b�k�ra�� <--- ### 200 copy: get hit eof copy: waiting for put confirmation copy: put confirmed store copy: get is finished - all done ---> #RETR something.r20 ls -lLd something.r20; echo '### 100'; cat something.r20; echo '### 200' <--- -rw-r--r-- 1 arekm users 15000000 May 20 15:00 something.r20 <--- ### 100 <--- ### 200 copy: get hit eof copy: waiting for put confirmation copy: put confirmed store
OR
copy: get hit eof copy: waiting for put confirmation copy: put confirmed store copy: get is finished - all done ---> #RETR something.r52 ls -lLd something.r52; echo '### 100'; cat something.r52; echo '### 200' <--- -rw-r--r-- 1 arekm users 15000000 May 20 15:00 something.r52 <--- ### 100 <--- �n��J���!�qS���ܑm�wf[�(�����m�0)���%�"? ���iǨ���8ϰ��t�P#���w�a{��l��s�^�Z�V�Ns <--- ### 200 copy: get hit eof copy: waiting for put confirmation copy: put confirmed store copy: get is finished - all done ---> #RETR something.r53
lavv17
commented
11 years ago Should be fixed in d3cf34f, please try.
arekm
commented
11 years ago Now it doesn't see contents of directory (probably parses it incorrectly). It sees things like " total 77064192", so files are there:
lftp arekm@somehost:~> cd t/download cd ok, cwd=/home/users/arekm/t/download lftp arekm@somehost:~/t/download> ls ls: Access failed: total 77064192 drwxr-xr-x 354 arekm users 28672 May 21 08:38 . lftp arekm@somehost:~/t/download>
and in debug log:
---- attempt number 0
---- attempt number 1
---- Running connect program (ssh -a -x -l arekm somehost echo FISH:;/bin/sh)
<--- Password: XXXX
<--- FISH:
---> #FISH
TZ=GMT;export TZ;LC_ALL=C;export LC_ALL;exec 2>&1;echo;start_fish_server;echo '### 200'
---> #VER 0.0.2
echo '### 000'
---> #PWD
pwd; echo '### 200'
<---
<--- /bin/sh:
arekm
commented
11 years ago Note: using 4.4.6 + all new changes from git, not only d3cf34f.
lavv17
commented
11 years ago Please try 0f6f620 (the previous change was overdone)
arekm
commented
11 years ago No garbage and no corruption in last few hours with these fixes.
I think the bug can be closed and if corruption becomes visible again I'll open new one with new details.
Thanks!
lavv17
commented
11 years ago Thank you!
When downloading (queue mirror -c xyz; queue mirror -c abc and so on) bunch of directories with files over fish:// then lftp (tested on 4.3.1 , 4.4.5) sometimes it doesn't download one or two files correctly
Without "set mirror:parallel-transfer-count 10" I didn't have this problem, so something looks broken in parallel handling.
"Brak dostępu" means "Permission denied" but I have full access to all files and directories on both sides.
lftp prints on console:
mirror: Brak dostępu: H�b���ᖚ/�+9��?@���W4�z�TgR���kԁ�?� ���ގ�"7G�82'�N��\/o @+�Ē�J��': �T�~/���8��t�5OP x͜y�Ԗ�t@!৫%2�b��>< T?�DheΓ�� ���>�t}�S$�[Zr�N]�KS�G>��/���mp�o��~�Q?f�Y~�� ���AS4����P�rZ�@e�J�]@ʗ�4P&l��إ��S�a#Ä ���]�aآq�٧R�u�tU� �o��W/RLH ��C�ς5�^���y���Yض۩�M�I�y�S��a�O00dc�
M$[Q}}����]0�T<4�^�!����5<��kԈ�WeE�(m��0�ۍ��AШ} */�ZyUr���<��I=�P���Pt�z%�£�2���j+��]Ω,T0-q| z�U�����D������� B|2�;јl1����@����RyVp
:9,�JYTt�f�d���.�DU bIv�A�Z(�˚�or
mirror: Brak dostępu: �= -rw-r--r-- 1 arekm users 50000000 May 18 01:00 somefile.r12 mirror: Brak dostępu: �5�5�w�7j K䣖�-�H@�O��d�?,&{�U��g�+�'�s0�xb8 (somefile.r13) lftp arekm@somehost:~/download>
~/.lftp/rc: set net:timeout 30 set cmd:move-background false set cmd:queue-parallel 4 set mirror:parallel-transfer-count 6
Linux, openssh 6.1p1, bash 4.2.45 on remote side