🐛 FTP data transfer fails

petrolep commented 2 years ago

Describe the bug

Transferring files via FTP over WARP client and cloudflared causes data loss.

Using Cloudflare Zero Trust + WARP. On Cloudflare Teams web I created a new tunnel with defined private network, installed cloudflared on a local server and WARP client on a local client machine. The client machine (Windows) is connecting via WARP to cloudflared on the local server which proxies the traffic to a third server.

Other services (HTTP, HTTPS, RDP, MSSQL) seems to be working correctly except FTP data transfers. The FTP connection is successfully established but file transfers produce connection errors and files are transferred incomplete or with a corrupted content.

Expected behavior

Files are transferred correctly.

Environment and versions

OS: cloudflared on Windows Server 2016 and on Amazon Linux (same behavior); WARP on Windows 10
Architecture: AMD
Version: cloudflared 2022.7.1, 2022.6.3, 2022.4.0 (same behavior); WARP 2022.7.421.0

Logs and errors

There is no error in cloudflared log. For each transfer I can see

2022-08-03T15:24:13Z DBG tcp proxy stream started flowID=d1f1c6b9-31bd-4609-b24d-56fafb24400f
2022-08-03T15:24:13Z DBG tcp proxy stream finished successfully flowID=d1f1c6b9-31bd-4609-b24d-56fafb24400f
2022-08-03T15:24:13Z DBG origin->tunnel copy: read tcp %local%:63576->%remote%:49314: use of closed network connection

However the same DBG message is logged also for successful transfers and also for other services (e.g. MSSQL).

I tried protocols http2 and quic, no-chunked-encoding but did not help.

Additional context

Tried several FTP clients (Total commander, WinSCP, FileZilla), FTP servers (both Windows and Linux) and client computers. Everything has the same issue.

Is there a configuration which could help? Or any other more advanced debugging option?

Thanks a lot!

sudarshan-reddy commented 2 years ago

@petrolep : I'm going to look into this. Do you have a minimal example I can try and reproduce and iterate on?

petrolep commented 2 years ago

@sudarshan-reddy Thank you. I am not sure what could be a minimal example, please ask for more info if necessary.

I was just able to simmulate the issue using following steps:

Setup

clean Ubuntu server (for cloudflared and test FTP server)
clean Windows 10 machine (for WARP)
new account on dash.teams.cloudflare.com with Free tier

Cloudflare setup (on dash.teams.cloudflare.com)

create new tunnel
choose Debian/64-bit + copy installation script + run on Ubuntu server
connector is installed
specify new private network - IP of the Ubuntu machine (same as IP of the connector)
tunnel should be up and running

Install FTP on Ubuntu machine

(You can skip if you have any other FTP you can try)

Following should install vsftpd with anonymous access

sudo apt install vsftpd
sudo cp /etc/vsftpd.conf /etc/vsftpd.conf.orig
sudo ufw allow 20,21,990/tcp
sudo ufw allow 40000:50000/tcp

manually modify sudo nano /etc/vsftpd.conf

anonymous_enable=YES
write_enable=YES
local_umask=022
allow_writeable_chroot=YES
anon_upload_enable=YES
anon_mkdir_write_enable=YES
anon_other_write_enable=YES
pasv_min_port=40000
pasv_max_port=50000
anon_root=/var/ftp

update file permissions (the configuration of the permissions to properly upload files might be more tricky, I apologize but I am not a linux admin; please adjust via SSH if needed)
```
mkdir /var/ftp
chown ftp /var/ftp
chmod 755 /var/ftp
mkdir /var/ftp/upload
chown ftp /var/ftp/upload
chmod 755 /var/ftp/upload
```
restart FTP server sudo systemctl restart vsftpd

Install FTP client on Windows

I tried Total Commander and Filezilla (use passive mode)
try connection to the FTP on Ubuntu, should work

Install WARP on Windows

on Windows 10 machine download and install WARP from https://1.1.1.1
turn the WARP connection on (using the red UI)
try connection to the FTP and upload files = should be working fine (=> WARP does not seem to be the problem)
connect WARP to your team created on dash.teams.cloudflare.com (WARP Settings / Account / Login with Cloudflare Zero Trust)
- this is more tricky, I was not able to use OTP to authenticate -- I had to connect to our IDP (hopefully you already have an account)
- this will switch WARP to blue UI "Zero Trust"
turn the Zero Trust connection on (using the blue UI)
try connection to the FTP and upload files (with a size of few MB)
- please verify that the traffic is going through the tunnel (e.g. by setting up firewall on the FTP server to allow only the tunnel IP)
- some of the transfers will fail
- Total commander will detect that the upload was not completed and will try to reupload
  - some reuploads will be successful, some will fail
  - Total commander detects the success by comparing file size = Total commander might report success even when the file is acctually corrupted, but has the correct file size - if you compare the binary content of the original file and the uploaded file, they might be different (after some correctly transferred bytes the content will start to differ)
- Filezilla will not do anything, will keep the file uncomplete/corrupted

I realize that this is quite a complex description. If you already have WARP/FTP, you can start by testing with your existing resources.

If I can provide any more info, please let me know.

Thank you very much!

Best,

Petr

sudarshan-reddy commented 2 years ago

Thanks for this @petrolep . I'll give this a go and get back to you in a bit.

petrolep commented 2 years ago

Hello @sudarshan-reddy , have you been able to simulate this behavior? Is there anything I can do? Unfortunatelly this is currently a blocker for us to deploy Zero Trust/cloudflared in our company.

Thanks a lot.

petrolep commented 2 years ago

@sudarshan-reddy Is there a chance to have a quick Teams call to show you the problem and logs?

Thank you.

kierzniak commented 2 years ago

I have similar problem. I created tunnel for my team devices to have fixed IP for my client FTP server using instruction from this post: https://community.cloudflare.com/t/static-outbound-ip-address-with-warp-for-team/306821

Tunnel is working fine and I can connect to client FTP server but I can not transfer files. Sometimes I'm getting error, sometimes only part of the file is being transferred.

petrolep commented 1 year ago

@kierzniak Were you able to solve the issue with corrupted data? I am still strugling, Cloudflare support was not helpful. Thanks.

kierzniak commented 1 year ago

Not really but I mange to fix my problem using https://github.com/sshuttle/sshuttle. On my proxy server I created linux users for each member of my team to make it possible to connect through ssh. Now they can use sshuttle to proxy all traffic through our proxy server and have static IP.

curl ifconfig.me