Set up new Refinitiv SFTP server

iangow / se_core

Core code for StreetEvents data

7 stars 5 forks source link

Set up new Refinitiv SFTP server #21

Closed iangow closed 3 years ago

iangow commented 4 years ago

Existing files not included in new feed should be moved to subdirectory.

iangow commented 4 years ago

Ran this command on Z640:

rsync -avz -e 'ssh -p 2020' $SE_DIR iangow.me:/home/igow/se_data_old/

iangow commented 4 years ago

Command used to get data from Refinitiv:

igow@igow-ubuntu-mate:~$ lftp -e 'mirror --parallel=12 / ~/se_data' -u  UniMelbourne_T\&B sftp://ftp-setranscripts.refinitiv.com 
Password: 
Total: 24 directories, 452734 files, 0 symlinks                                  
New: 2517 files, 0 symlinks
Modified: 703 files, 0 symlinks
144716708 bytes transferred in 306 seconds (462.2 KiB/s)
To be removed: 0 directories, 2916 files, 0 symlinks
lftp UniMelbourne_T&B@ftp-setranscripts.refinitiv.com:/> exit
igow@igow-ubuntu-mate:~$

iangow commented 3 years ago

Research Platform Services sorted this issue out.