search

enram / data-repository

Data quality assessment
https://enram.github.io/data-repository/
MIT License
3 stars 1 forks source link

Make FTP connector more resilient by retrying after a timeout? #60

Closed niconoe closed 1 year ago

niconoe commented 4 years ago

I recently found the following in the log file (and it seems less files were subsequently transferred at the time):

Traceback (most recent call last):
  File "/home/ubuntu/data-repository/file_transfer/baltrad_to_s3.py", line 54, in <module>
    sys.exit(main())
  File "/home/ubuntu/data-repository/file_transfer/baltrad_to_s3.py", line 23, in main
    btos.transfer(name_match="_vp_", overwrite=False, limit=None)
  File "/home/ubuntu/data-repository/file_transfer/datamover/transporters.py", line 92, in transfer
    self.ftp._ftp.retrbinary('RETR ' + filename, f.write)
  File "/home/ubuntu/miniconda2/envs/enramtransfer/lib/python3.5/ftplib.py", line 441, in retrbinary
    with self.transfercmd(cmd, rest) as conn:
  File "/home/ubuntu/miniconda2/envs/enramtransfer/lib/python3.5/ftplib.py", line 398, in transfercmd
    return self.ntransfercmd(cmd, rest)[0]
  File "/home/ubuntu/miniconda2/envs/enramtransfer/lib/python3.5/ftplib.py", line 360, in ntransfercmd
    source_address=self.source_address)
  File "/home/ubuntu/miniconda2/envs/enramtransfer/lib/python3.5/socket.py", line 712, in create_connection
    raise err
  File "/home/ubuntu/miniconda2/envs/enramtransfer/lib/python3.5/socket.py", line 703, in create_connection
    sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out
peterdesmet commented 1 year ago

This should be investigated a part of vptstools, closing here.