Open danhamill opened 2 years ago
I tried setting something up like:
remote_stat = await client.stat(path)
remote_size = int(remote_stat['size'])
if await aos.path.exists(dest):
stat = await aos.stat(dest)
size = stat.st_size
else:
size = 0
while not remote_size == size:
async with client.download_stream(path, offset=size) as stream:
But that will only work if the local file is exactly the size of the remote file. How can I check stream
to see the current postiion to test within async with client.download_stream
block?
In first post I see some problems:
async for block in stream.iter_by_block(512):
if not block:
break
await local_file.write(block)
Unfortunately aioftp is a little chaotic project, so you wont get empty block on iteration end. Maybe this should be made as you expected. So your check and break call will never happend. Iteration internaly ends when empty block comes from socket, so you need just to iterate as much as you can.
Thanks for your reply. I had a long debugging session yesterday and can confirm
if not block:
break
is never evaluated True
. In case somebody else has a similar issue, I will post my working solution here
while max_attempts<30:
try:
logging.info(f'Starting Attempt {max_attempts}')
async with async_timeout.timeout(20):
async with aioftp.Client.context('192.12.137.7', user='anonymous', port=21) as client:
if remote_size is None:
logging.info(f'Getting remote stats for file {path}')
remote_stat = await client.stat(path)
remote_size = int(remote_stat['size'])
logging.info(f' Remote file has size {remote_size}')
async with aiofiles.open(dest, mode='ab', ) as local_file:
#Check to see if local_file exists
if await aos.path.exists(dest):
stat = await aos.stat(dest)
size = stat.st_size
else:
size = 0
logging.info(f'Starting at postition {size}')
local_file.seek(size)
if remote_size == size:
break
async with client.download_stream(path, offset=size) as stream:
async for block in stream.iter_by_block():
await local_file.write(block)
except aioftp.StatusCodeError as ftp_e:
max_attempts +=1
logging.info(f'Found aioftp error, trying another attempt')
if ftp_e.received_codes ==( '426',):
logging.info(f'Forced timeout error, trying another attempt')
if ftp_e.received_codes != ( '426',):
logging.info('new code')
asyncio.sleep(1)
continue
except asyncio.exceptions.TimeoutError as asy_e:
logging.info(f'found time out exception')
max_attempts +=1
continue
Adding a time out of 20 seconds creates a lot of 426 failed to write block
errors and only a few Timeout Errors
, After a little tinkering with the maximum allowable attempts, this seems to be a reasonable workflow to ensure the complete files are downloaded.
This issue should be opened, since I don't understand the nature of failure is ot ftp client or server problelm.
All I can say is during my debugging when I got to a point where remote_size == size
, as async for block in stream.iter_by_block()
proceeded to await local_file.write(block)
with out any variable named block
. It did not return a block
with a value of None
When remote_size > size
I could see the bytes for each block
.
@danhamill
I see that if remote_size == size
break
will be called. And I see no remote_size
starting definition.
In my comment, remote_size
is defined inside the top of the download routine. e.g.,
while max_attempts<30 ; try: async with async_timeout.timeout(20): async with aioftp.Client.context('192.12.137.7', user='anonymous', port=21) as client: if remote_size is None: logging.info(f'Getting remote stats for file {path}') remote_stat = await client.stat(path) remote_size = int(remote_stat['size']) logging.info(f' Remote file has size {remote_size}')
This loop sits inside an async def
function in my main script. At first pass remote_size==None
, and the file size is determined from server. This is skipped for subsequent attempts.
I have been extensively using this routine and have found some of the files I download locally are larger than what is provided on the server. Do you have any idea why this could happen?
Thanks!
Not enough info to make predictions. You should try to reproduce this on some big text file and then compare local «bigger» and remote «smaller» files with diff
or something. Also, maybe add logs on failures and try to find out is there any correlation between attempts and different file sizes.
I am trying to implement a the advanced-download-and-upload-abort-restart example for my use and seem to have come across an infinte loop when my first file completes the download:
Logging Info
As you can see the code appears to be working properly, and I am getting a complete transfer. However the code does not seem to move on to the next file.
How can I write a conditional statement that checks for
[aioftp.client] 226 Transfer complete
? Which bit of code is that message coming from?