🐛 FTP data transfer fails

[petrolep]

Describe the bug

Transferring files via FTP over WARP client and cloudflared causes data loss.

Using Cloudflare Zero Trust + WARP. On Cloudflare Teams web I created a new tunnel with defined private network, installed cloudflared on a local server and WARP client on a local client machine. The client machine (Windows) is connecting via WARP to cloudflared on the local server which proxies the traffic to a third server.

Other services (HTTP, HTTPS, RDP, MSSQL) seems to be working correctly except FTP data transfers. The FTP connection is successfully established but file transfers produce connection errors and files are transferred incomplete or with a corrupted content.

Expected behavior

Files are transferred correctly.

Environment and versions

• OS: cloudflared on Windows Server 2016 and on Amazon Linux (same behavior); WARP on Windows 10
• Architecture: AMD
• Version: cloudflared 2022.7.1, 2022.6.3, 2022.4.0 (same behavior); WARP 2022.7.421.0

Logs and errors

There is no error in cloudflared log. For each transfer I can see

2022-08-03T15:24:13Z DBG tcp proxy stream started flowID=d1f1c6b9-31bd-4609-b24d-56fafb24400f
2022-08-03T15:24:13Z DBG tcp proxy stream finished successfully flowID=d1f1c6b9-31bd-4609-b24d-56fafb24400f
2022-08-03T15:24:13Z DBG origin->tunnel copy: read tcp %local%:63576->%remote%:49314: use of closed network connection

However the same DBG message is logged also for successful transfers and also for other services (e.g. MSSQL).

I tried protocols http2 and quic, no-chunked-encoding but did not help.

Additional context

Tried several FTP clients (Total commander, WinSCP, FileZilla), FTP servers (both Windows and Linux) and client computers. Everything has the same issue.

Is there a configuration which could help? Or any other more advanced debugging option?

Thanks a lot!

[sudarshan-reddy]

@petrolep : I'm going to look into this. Do you have a minimal example I can try and reproduce and iterate on?

[petrolep]

@sudarshan-reddy Thank you. I am not sure what could be a minimal example, please ask for more info if necessary.

I was just able to simmulate the issue using following steps:

Setup

• clean Ubuntu server (for cloudflared and test FTP server)
• clean Windows 10 machine (for WARP)
• new account on dash.teams.cloudflare.com with Free tier

Cloudflare setup (on dash.teams.cloudflare.com)

• create new tunnel
• choose Debian/64-bit + copy installation script + run on Ubuntu server
• connector is installed
• specify new private network - IP of the Ubuntu machine (same as IP of the connector)
• tunnel should be up and running

Install FTP on Ubuntu machine

(You can skip if you have any other FTP you can try)

Following should install vsftpd with anonymous access

• sudo apt install vsftpd
• sudo cp /etc/vsftpd.conf /etc/vsftpd.conf.orig
• sudo ufw allow 20,21,990/tcp
• sudo ufw allow 40000:50000/tcp
• manually modify sudo nano /etc/vsftpd.conf

  anonymous_enable=YES
  write_enable=YES
  local_umask=022
  allow_writeable_chroot=YES
  anon_upload_enable=YES
  anon_mkdir_write_enable=YES
  anon_other_write_enable=YES
  pasv_min_port=40000
  pasv_max_port=50000
  anon_root=/var/ftp

• update file permissions (the configuration of the permissions to properly upload files might be more tricky, I apologize but I am not a linux admin; please adjust via SSH if needed)

  mkdir /var/ftp
  chown ftp /var/ftp
  chmod 755 /var/ftp
  mkdir /var/ftp/upload
  chown ftp /var/ftp/upload
  chmod 755 /var/ftp/upload

• restart FTP server sudo systemctl restart vsftpd

Install FTP client on Windows

• I tried Total Commander and Filezilla (use passive mode)
• try connection to the FTP on Ubuntu, should work

Install WARP on Windows

• on Windows 10 machine download and install WARP from https://1.1.1.1
• turn the WARP connection on (using the red UI)
• try connection to the FTP and upload files = should be working fine (=> WARP does not seem to be the problem)
• connect WARP to your team created on dash.teams.cloudflare.com (WARP Settings / Account / Login with Cloudflare Zero Trust)
  • this is more tricky, I was not able to use OTP to authenticate -- I had to connect to our IDP (hopefully you already have an account)
  • this will switch WARP to blue UI "Zero Trust"
• turn the Zero Trust connection on (using the blue UI)
• try connection to the FTP and upload files (with a size of few MB)
  • please verify that the traffic is going through the tunnel (e.g. by setting up firewall on the FTP server to allow only the tunnel IP)
  • some of the transfers will fail
  • Total commander will detect that the upload was not completed and will try to reupload
    • some reuploads will be successful, some will fail
    • Total commander detects the success by comparing file size = Total commander might report success even when the file is acctually corrupted, but has the correct file size - if you compare the binary content of the original file and the uploaded file, they might be different (after some correctly transferred bytes the content will start to differ)
  • Filezilla will not do anything, will keep the file uncomplete/corrupted

I realize that this is quite a complex description. If you already have WARP/FTP, you can start by testing with your existing resources.

If I can provide any more info, please let me know.

Thank you very much!

Best,

Petr

[sudarshan-reddy]

Thanks for this @petrolep . I'll give this a go and get back to you in a bit.

[petrolep]

Hello @sudarshan-reddy , have you been able to simulate this behavior? Is there anything I can do? Unfortunatelly this is currently a blocker for us to deploy Zero Trust/cloudflared in our company.

Thanks a lot.

[petrolep]

@sudarshan-reddy Is there a chance to have a quick Teams call to show you the problem and logs?

Thank you.

[kierzniak]

I have similar problem. I created tunnel for my team devices to have fixed IP for my client FTP server using instruction from this post: https://community.cloudflare.com/t/static-outbound-ip-address-with-warp-for-team/306821

Tunnel is working fine and I can connect to client FTP server but I can not transfer files. Sometimes I'm getting error, sometimes only part of the file is being transferred.

[petrolep]

@kierzniak Were you able to solve the issue with corrupted data? I am still strugling, Cloudflare support was not helpful. Thanks.

[kierzniak]

Not really but I mange to fix my problem using https://github.com/sshuttle/sshuttle. On my proxy server I created linux users for each member of my team to make it possible to connect through ssh. Now they can use sshuttle to proxy all traffic through our proxy server and have static IP.

curl ifconfig.me

[frederikspang]

Any Update? We’re seeing similar behaviour on warp-routing tunnels.

[petrolep]

Still now working for us, same issue persists with the last version 2023.3.1

[frederikspang]

Still now working for us, same issue persists with the last version 2023.3.1

Same here, @petrolep I wonder where @sudarshan-reddy went - Did the task not get prioritised internally? Forgotten?

We found out about this error the hard way - Publishing application over tunnel, corrupted the virtually all files... Quite a large headache for us, and blocker in rolling out to all employees.

[sudarshan-reddy]

Apologies for the delay. Got derailed by some priorities. I will circle back to this in a couple of days.

[frederikspang]

@sudarshan-reddy Thanks! Appreciate it. I wonder if this affects other services. It "should", right? Since cloudflared just communicates the TCP through, it should in theory not be protocol-specific, for FTP only?

I have not tested on FTP Active mode, though - But it could be something with the specifics of how Passive mode works in FTP?

[chihangc]

I believe this issue is related to FTP PASSIVE mode. In passive mode, the server define a specific random port each time back to client. So tunnel may not able to forward that random port number agreed during the passive mode handshake

[frederikspang]

I believe this issue is related to FTP PASSIVE mode. In passive mode, the server define a specific random port each time back to client. So tunnel may not able to forward that random port number agreed during the passive mode handshake

This seems pretty likely. I have put in internal resources in trying out configurations about AV Scanning and using protocol: quic in our config etc, and we are about to set up a test scenario with active/passive mode as well.

Hopefully @sudarshan-reddy is able to provide some more insight soon :)

[frederikspang]

So update on debug Newest versions still fail:

Warp on MacOS: Version: 2023.3.415.0 (20230417.6) Cloudflared on Ubuntu 22.04: cloudflared version 2023.4.1 (built 2023-04-17-1941 UTC)

Reproduction flow:

1. Connect to FTP (Passive mode).
2. Upload file file-v1.0.3-before
3. Rename file to -after
4. Download file file-v1.0.3-after
5. Assert file hash, and size.

$ ls -al
-rwxr-xr-x    1 fst  staff  12793106 Jul 26  2021 file-v1.0.3-before
-rwxr-xr-x    1 fst  staff  11747328 Apr 27 13:45 file-v1.0.3-after

$ md5 file-v1.0.3-before
MD5 (file-v1.0.3-before) = ac594f1031d68b115091ec53710630a9

$ md5 file-v1.0.3-after
MD5 (file-v1.0.3-after) = bc055d0503b7ef3a5e9201db177e6889

These are not equivalent.

This debug has been done in both FileZilla, and some other FTP client my colleague uses, so it is not specific to the client either.

FTP Active Mode:

Currently FileZilla will issue following.

Command:    PORT 172,16,0,2,230,104
Response:   200 PORT command successful
Command:    MLSD
Response:   425 Unable to build data connection: Network is unreachable
Error:          Failed to retrieve directory listing

The IP Address: 172.16.0.2, is bound to MacOS utun7 interface, which is Warp's interface (as confirmed by closing connection, and seeing it disappear, as well as checking netstat routing table). That kills the FTP connection, as the FTP server cannot reach the 172.16.0.2 address either.

Config setup in cloudflare (Anonymised, obviously)

tunnel: 00000000-0000-0000-0000-000000000000
credentials-file: /root/.cloudflared/00000000-0000-0000-0000-000000000000.json
protocol: quic
warp-routing:
  enabled: true

cc @sudarshan-reddy

[dusansusic]

+1 for this issue, we have corrupted files when transferring them over the cloudflared tunnel

[ghost]

@sudarshan-reddy Any update on this?

[reavpC6K]

Yet to have it. I'd wireshark port 21 on client and server VM's to find the fault. SFTP works fine.