search

lavv17 / lftp

sophisticated command line file transfer program (ftp, http, sftp, fish, torrent)
http://lftp.yar.ru
GNU General Public License v3.0
1.11k stars 162 forks source link

Continue mirror in pget mode #356

Open AndrewSav opened 7 years ago

AndrewSav commented 7 years ago

I wrote this stack exchange question, but then I had a look at the source code.

It appears that it's impossible to continue a mirror job in pget mode if it was initially run in pget mode.

It would be very nice piece of functionality, if mirror mode could use lftp-pget-status when continuing the same way pget itself can.

j4ckp0t85 commented 5 years ago

hi. i have found a way to resume the download (at least it works in my environment). look on my repo. notice that i use the same number of chunks for mirror and pget. also this must not be set file:use-fallocate true

AndrewSav commented 5 years ago

@j4ckp0t85 sorry, not sure what you mean.

j4ckp0t85 commented 5 years ago

See his question here https://superuser.com/questions/1220626/resuming-segmented-file-transfer

Basically a misunderstood of the behavior of - c flag in mirror command (looking the man page it was not 100% clear to be honest).

AndrewSav commented 5 years ago

Ah, so you misunderstood something, I'm glad you got whatever that was sorted out.

ldzhjn commented 5 years ago

So what's the conclusion for this issue here? Is it possible to fix?

AndrewSav commented 5 years ago

I believe in programming everything is possible ;) We just need someone willing and able to fix this!

j4ckp0t85 commented 5 years ago

sorry what's the problem?

the resume is already possible.

from the man page:

pget [OPTS] rfile [-o lfile]

   Gets the specified file using several connections. This can speed up transfer, but loads  the
   net  and  server  heavily  impacting other users. Use only if you really have to transfer the
   file ASAP.  Options:

        -c           continue transfer. Requires lfile.lftp-pget-status file.
        -n maxconn   set maximum number of connections (default  is  taken  from  pget:default-n
                     setting)

so supposing you have an interrupted download for somehow reason, you can resume it this way notice i'm using the parameter -n 10 because on my lftp script is set to use max 10 connections

pget -c -n 10 </remote/ftp/folder>/$remotepath/$resumefile

ldzhjn commented 5 years ago

sorry what's the problem?

the resume is already possible.

from the man page:

pget [OPTS] rfile [-o lfile]

   Gets the specified file using several connections. This can speed up transfer, but loads  the
   net  and  server  heavily  impacting other users. Use only if you really have to transfer the
   file ASAP.  Options:

        -c           continue transfer. Requires lfile.lftp-pget-status file.
        -n maxconn   set maximum number of connections (default  is  taken  from  pget:default-n
                     setting)

so supposing you have an interrupted download for somehow reason, you can resume it this way notice i'm using the parameter -n 10 because on my lftp script is set to use max 10 connections

pget -c -n 10 </remote/ftp/folder>/$remotepath/$resumefile

The problem is when using mirror command with pget to download folder, it does not support resume.

j4ckp0t85 commented 5 years ago

why not? overall, a subfolder of a mirrored path is only a logic matter to handle.

the concept is the same

AndrewSav commented 5 years ago

Yep, it just needs to be implemented in lftp. Currently it is not working.

j4ckp0t85 commented 5 years ago

sw engineering meme

Welcome to software engineering lifecycle :)

Maybe what you want is not the behaviour the dev want

lavv17 commented 5 years ago

Mirror does not know the pget-status-file yet, so it does not run pget with -c option or even removes the file as not present on the remote side.

пт, 29 мар. 2019 г. в 11:51, Michele Granato notifications@github.com:

[image: sw engineering meme] https://camo.githubusercontent.com/2ec9b02a8044d7e596a0d7d84d0990aa88944251/68747470733a2f2f692e6962622e636f2f786a516e30674b2f6664382e6a7067

Welcome to software engineering lifecycle :)

Maybe what you want is not the behaviour the dev want

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/lavv17/lftp/issues/356#issuecomment-477918704, or mute the thread https://github.com/notifications/unsubscribe-auth/AA67XGaxGmgqvB6F0-lHths4v3VjsWgsks5vbdQRgaJpZM4N9twI .

-- Alexander.

j4ckp0t85 commented 5 years ago

Of course the pget status file is stored on local-side.

Maybe he wants that the continue function look for and, if present, handle (resume) the partial download first.

ldzhjn commented 5 years ago

Mirror does not know the pget-status-file yet, so it does not run pget with -c option or even removes the file as not present on the remote side. пт, 29 мар. 2019 г. в 11:51, Michele Granato notifications@github.com: … [image: sw engineering meme] https://camo.githubusercontent.com/2ec9b02a8044d7e596a0d7d84d0990aa88944251/68747470733a2f2f692e6962622e636f2f786a516e30674b2f6664382e6a7067 Welcome to software engineering lifecycle :) Maybe what you want is not the behaviour the dev want — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#356 (comment)>, or mute the thread https://github.com/notifications/unsubscribe-auth/AA67XGaxGmgqvB6F0-lHths4v3VjsWgsks5vbdQRgaJpZM4N9twI . -- Alexander.

Can you explain the logic behind this design? Is mirror not knowing pget-status-file reasonable?

If someone is going to fix this issue, what do you suggest?

lavv17 commented 5 years ago

it can be fixed, someone has to invest several working hours to that. Usually it's me, but I have little free time lately.

сб, 30 марта 2019, 6:01 ldzhjn notifications@github.com:

Mirror does not know the pget-status-file yet, so it does not run pget with -c option or even removes the file as not present on the remote side. пт, 29 мар. 2019 г. в 11:51, Michele Granato notifications@github.com: … <#m8603425284100019027> [image: sw engineering meme] https://camo.githubusercontent.com/2ec9b02a8044d7e596a0d7d84d0990aa88944251/68747470733a2f2f692e6962622e636f2f786a516e30674b2f6664382e6a7067 Welcome to software engineering lifecycle :) Maybe what you want is not the behaviour the dev want — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#356 (comment) https://github.com/lavv17/lftp/issues/356#issuecomment-477918704>, or mute the thread https://github.com/notifications/unsubscribe-auth/AA67XGaxGmgqvB6F0-lHths4v3VjsWgsks5vbdQRgaJpZM4N9twI . -- Alexander.

Can you explain the logic behind this design? Is mirror not knowing pget-status-file reasonable?

If someone is going to fix this issue, what do you suggest?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/lavv17/lftp/issues/356#issuecomment-478200377, or mute the thread https://github.com/notifications/unsubscribe-auth/AA67XAn_-oow9MXombGw2FRCWefp2BqZks5vbtOigaJpZM4N9twI .