search

grke / burp

burp - backup and restore program
http://burp.grke.net
Other
485 stars 77 forks source link

[NOT A BUG] burp does not backup with 4G #782

Closed deajan closed 5 years ago

deajan commented 5 years ago

Hello,

I have stumbled upon a very strange issue on a computer, where burp 2.2.12 protocol 1 works when backing up via ethernet or wifi, but does not backup via 4G (LTE via integrated USB card). I've tried different french 4G providers (orange & bouygues). I've also tried to do thethering via my phone, connecting the computer via wifi to my phone which acts as 4G router, does not work either, using bouygues or SFR 4G providers. Using a standard wifi ADSL connection or Fiber works.

I get the following message on the client:

2018-12-07 15:59:00 +0100: burp[18885] main socket: Got network write error
2018-12-07 15:59:00 +0100: burp[18885] main socket: network write problem in asfd_do_write_ssl: 5 - 0=no error
2018-12-07 15:59:00 +0100: burp[18885] This is probably caused by the peer exiting.

And the following on the server logs:

2018-12-07 15:59:00 +0100: burp[4494] Begin phase1 (read previous file system scan)
2018-12-07 15:59:00 +0100: burp[4494] Setting up resume positions...
2018-12-07 15:59:00 +0100: burp[4494]   nothing previously transferred
2018-12-07 15:59:00 +0100: burp[4494] End phase1 (read previous file system scan)
2018-12-07 15:59:55 +0100: burp[4494] main socket 4: Got network read error
2018-12-07 15:59:55 +0100: burp[4494] main socket 4: network read problem in asfd_do_read_ssl: 5 - 104=Connection reset by peer
2018-12-07 15:59:55 +0100: burp[4494] This is probably caused by the peer exiting.
2018-12-07 15:59:55 +0100: burp[4494] Please check the peer's logs.

I have tested another computer / server couple with the same phone thetering with 2.2.12 and it works flawlessly.

One the computer that does not work, I disabled AV / firewall software, checked for malware etc, tried to download/upload big files and check their SHA256 sum in order to make sure the links work okay. One the server, I checked the firewalls for any prohibited traffic, without finding anything. Other clients backup to that server without problems (not using 3G). Since the computer is a customer's who can't leave it for a long diag, I just need some tips from you guys, to see whether I missed something obvious.

The server itself listens on port 443 instead of 4971 in order to pass most firewalls. Can it be that 4G providers have some fancy 443 filtering that would make burp fail but big https downloads work ?

Thanks.

ziirish commented 5 years ago

Hello, there are some rumors for some times that suggest most 4G providers do some DPI on their networks.

There have also been some proofs that plain HTTP answers got altered by something in those providers networks (mostly to insert ads and stuff like that).

So if you mix up both those proxies and DPIs stories, it is indeed possible that running non HTTPS traffic through port 443 on such networks result in unexpected behavior.

deajan commented 5 years ago

Yes, that could indeed explain alot. I guess I'll try to reproduce the problem by having having a burp server listen on port 443 and make 4G tests again.

ziirish commented 5 years ago

Sorry, this article is in French but explains that Buygues Telecom probably do some MITM in their 3G (so I guess it's true for 4G as well) network.

pablodav commented 5 years ago

what could be the workaround? a vpn could help?

ziirish commented 5 years ago

I'm not sure there are any viable workarounds because from what I understand, the fact that the burp server is listening on port 443 is to bypass some firewall restrictions.

Now if you want to to use a vpn you'll have to expose it on port 443 too (I guess), and you'll hit the same problem: running non HTTPS traffic may cause you some troubles.

The only solution is to avoid ports 25, 443, 80, 53 (which are well known standards and more prone to MITM/DPI)

P.S.: I personally haven't studied those networks, these are just readings I used to find a while back.

deajan commented 5 years ago

I'll do some tests to confirm this shortly.

grke commented 5 years ago

Am I able to close this?

deajan commented 5 years ago

Humm.... Sorry, I totally forgot to make the corresponding tests. Please give me some time (like until this weekend) so I can have the adequate tests done.

deajan commented 5 years ago

I could successfully reproduce the error:

Here's what I got when trying to backup via 4G mobile network on port 443.

C:\Program Files\Burp\bin>burp -a b -c ../burp-sp-p1.conf
2019-02-21 09:39:06: burp[1460] windows drives detected: CD
2019-02-21 09:39:06: burp[1460] Connecting to sp.compress.to:443
2019-02-21 09:39:07: burp[1460] WARNING: w:0077:Client 'sp.protocol1.local' version '2.3.0' does not match server version '2.3.1'. An upgrade is recommended.

2019-02-21 09:39:07: burp[1460] auth ok
2019-02-21 09:39:07: burp[1460] Server version: 2.3.1
2019-02-21 09:39:07: burp[1460] nocsr ok
2019-02-21 09:39:07: burp[1460] SSL is using cipher: DHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=DH       Au=RSA  Enc=AESGCM(256) Mac=AEAD

2019-02-21 09:39:07: burp[1460] extra_comms_begin ok:autoupgrade:incexc:orig_client:uname:sincexc:counters_json:msg:csetproto:rshash=blake2:
2019-02-21 09:39:07: burp[1460] Server is setting includes/excludes.
2019-02-21 09:39:07: burp[1460] Client accepts.
2019-02-21 09:39:07: burp[1460] windows drives detected: CD
2019-02-21 09:39:08: burp[1460] Server has protocol=0 (auto)
2019-02-21 09:39:08: burp[1460] Using protocol=1
2019-02-21 09:39:08: burp[1460] Server is overriding the configuration
2019-02-21 09:39:08: burp[1460] with the following settings:
2019-02-21 09:39:08: burp[1460] include = C:/SomeData
2019-02-21 09:39:08: burp[1460] nobackup = .nobackup
2019-02-21 09:39:08: burp[1460] exclude_ext = crdownload
[...]
2019-02-21 09:39:08: burp[1460] exclude_comp = zip
2019-02-21 09:39:08: burp[1460] cross_all_filesystems = 0
2019-02-21 09:39:08: burp[1460] read_all_fifos = 0
2019-02-21 09:39:08: burp[1460] read_all_blockdevs = 0
2019-02-21 09:39:08: burp[1460] min_file_size = 0
2019-02-21 09:39:08: burp[1460] max_file_size = 0
2019-02-21 09:39:08: burp[1460] split_vss = 0
2019-02-21 09:39:08: burp[1460] strip_vss = 1
2019-02-21 09:39:08: burp[1460] acl = 1
2019-02-21 09:39:08: burp[1460] xattr = 1
2019-02-21 09:39:08: burp[1460] atime = 0
2019-02-21 09:39:08: burp[1460] scan_problem_raises_error = 0
2019-02-21 09:39:08: burp[1460] overwrite = 0
2019-02-21 09:39:08: burp[1460] strip = 0
2019-02-21 09:39:18: burp[1460] Compression level: 9
2019-02-21 09:39:18: burp[1460] do backup client
2019-02-21 09:39:18: burp[1460] Using librsync hash blake2
2019-02-21 09:39:18: burp[1460] Control handler registered.
2019-02-21 09:39:21: burp[1460] Generate VSS snapshots.
2019-02-21 09:39:21: burp[1460] Driver="VSS Vista", Drive(s)="C"
2019-02-21 09:39:46: burp[1460] VSS drive letters: 0
2019-02-21 09:39:46: burp[1460] VSS Writer (PrepareForBackup): "Task Scheduler Writer", State: 0x1 (VSS_WS_STABLE)
2019-02-21 09:39:46: burp[1460] VSS Writer (PrepareForBackup): "VSS Metadata Store Writer", State: 0x1 (VSS_WS_STABLE)
2019-02-21 09:39:46: burp[1460] VSS Writer (PrepareForBackup): "Performance Counters Writer", State: 0x1 (VSS_WS_STABLE)
2019-02-21 09:39:46: burp[1460] VSS Writer (PrepareForBackup): "System Writer", State: 0x1 (VSS_WS_STABLE)
2019-02-21 09:39:46: burp[1460] VSS Writer (PrepareForBackup): "ASR Writer", State: 0x1 (VSS_WS_STABLE)
2019-02-21 09:39:46: burp[1460] VSS Writer (PrepareForBackup): "Shadow Copy Optimization Writer", State: 0x1 (VSS_WS_STABLE)
2019-02-21 09:39:46: burp[1460] VSS Writer (PrepareForBackup): "MSSearch Service Writer", State: 0x1 (VSS_WS_STABLE)
2019-02-21 09:39:46: burp[1460] VSS Writer (PrepareForBackup): "Registry Writer", State: 0x1 (VSS_WS_STABLE)
2019-02-21 09:39:46: burp[1460] VSS Writer (PrepareForBackup): "WMI Writer", State: 0x1 (VSS_WS_STABLE)
2019-02-21 09:39:46: burp[1460] VSS Writer (PrepareForBackup): "COM+ REGDB Writer", State: 0x1 (VSS_WS_STABLE)
2019-02-21 09:39:46: burp[1460] VSS Writer (PrepareForBackup): "BITS Writer", State: 0x1 (VSS_WS_STABLE)
2019-02-21 09:39:46: burp[1460] Phase 1 begin (file system scan)

ddffffffdfddfffffdffddffdffdfdfffffffdfdfdffffddffddffdfdfffdddf 64
[...]
ffffdfffffffdfffffffffd 5911

2019-02-21 09:39:49: burp[1460] Phase 1 end (file system scan)
2019-02-21 09:39:49: burp[1460] Phase 2 begin (send backup data)

2019-02-21 09:40:12: burp[1460] main socket 632: Got network write error
2019-02-21 09:40:12: burp[1460] main socket 632: network write problem in asfd_do_write_ssl: 5 - 0=No error
2019-02-21 09:40:12: burp[1460] This is probably caused by the peer exiting.
2019-02-21 09:40:12: burp[1460] Please check the peer's logs.

--------------------------------------------------------------------------------
Start time: 2019-02-21 09:39:06
  End time: 2019-02-21 09:40:12
Time taken: 01:06
                         New   Changed Duplicate   Deleted     Total |  Scanned
                   ------------------------------------------------------------
             Files:        0         0         0         0         0 |     5641
       Directories:        0         0         0         0         0 |      270
       Grand total:        0         0         0         0         0 |     5911
                   ------------------------------------------------------------

             Messages:             0
             Warnings:             1

      Bytes estimated:     364701561 (347.81 MB)
      Bytes in backup:             0
       Bytes received:          5353 (5.23 KB)
           Bytes sent:       9863242 (9.41 MB)
--------------------------------------------------------------------------------
2019-02-21 09:40:12: burp[1460] Error in phase 2
2019-02-21 09:40:12: burp[1460] Phase 2 end (send file data)
2019-02-21 09:40:15: burp[1460] VSS Writer (BackupComplete): "Task Scheduler Writer", State: 0x1 (VSS_WS_STABLE)
2019-02-21 09:40:15: burp[1460] VSS Writer (BackupComplete): "VSS Metadata Store Writer", State: 0x1 (VSS_WS_STABLE)
2019-02-21 09:40:15: burp[1460] VSS Writer (BackupComplete): "Performance Counters Writer", State: 0x1 (VSS_WS_STABLE)
2019-02-21 09:40:15: burp[1460] VSS Writer (BackupComplete): "System Writer", State: 0x1 (VSS_WS_STABLE)
2019-02-21 09:40:15: burp[1460] VSS Writer (BackupComplete): "Shadow Copy Optimization Writer", State: 0x1 (VSS_WS_STABLE)
2019-02-21 09:40:15: burp[1460] VSS Writer (BackupComplete): "ASR Writer", State: 0x1 (VSS_WS_STABLE)
2019-02-21 09:40:15: burp[1460] VSS Writer (BackupComplete): "MSSearch Service Writer", State: 0x1 (VSS_WS_STABLE)
2019-02-21 09:40:15: burp[1460] VSS Writer (BackupComplete): "WMI Writer", State: 0x1 (VSS_WS_STABLE)
2019-02-21 09:40:15: burp[1460] VSS Writer (BackupComplete): "Registry Writer", State: 0x1 (VSS_WS_STABLE)
2019-02-21 09:40:15: burp[1460] VSS Writer (BackupComplete): "BITS Writer", State: 0x1 (VSS_WS_STABLE)
2019-02-21 09:40:15: burp[1460] VSS Writer (BackupComplete): "COM+ REGDB Writer", State: 0x1 (VSS_WS_STABLE)

Of course doing the backup on port 444 instead of 443 does work without any further problems. The issue is replicable without any problems on both French SFR and Bouygues networks. I also know of identical problems with Orange, but I couldn't it myself.

I've setup ratelimit = 0.1 in both client and server, just to make sure it's not a bandwidth limitation, but the problem remains.

Now of course changing the port number from 443 to anything else resolves the issue, but I have that question that remains: Why does initial communication before phase 2 work ? I've made an insanely long phase 1 test, which never got interrupted, even if it sent loads of data.

So why would phase 2 get killed if phase 1 can send big amounts of data ? @grke I know this isn't a burp bug, but I am trying to understand what can be the issue here, and maybe ask you if there may be a workaround, since this mobile filter bullshit is going to be more and more present.

I've played with the only 'flow' settings I know in burp, eg ssl_compression = zlib0-9, but it didn't do anything. At last, I also changed ssl_ciphers hoping to get away from content inspection, without success.

Dou you have any idea ?

Ohh and.. dear mobile operators, you're pricks for not being transparent about what is filtered / altered !

grke commented 5 years ago

There are a few differences in the data that goes across the wire between phase1 and phase2.

In phase1, the client sends the file scan data to the server, and it is one-way traffic, and it is mostly not binary-looking traffic. That is, it looks like the burp manifest files that you see in the completed backups. Something like this:

r0040gB ogJr EHt Pr Po Po A RAA BAA CQ BcZQ2i BcZQ2a BcZQ2a A A J A A
d0015/home/graham/testdir5
t0018t/home/graham/testdir5/0
r003CgB ogT3 IGk B Po Po A A BAA I BcHKzI BcHKzI BcHK0s A A J A A
f0017/home/graham/testdir5/0
x00220:d41d8cd98f00b204e9800998ecf8427e
t00110000/0000/0001.gz
r003CgB ogT3 IGk B Po Po A A BAA I BcHKzI BcHKzI BcHK0s A A J A A
m0017/home/graham/testdir5/0
x0024206:1c3b0e514f60f06c8bfd70e11cf86485

In phase2, the server requests files from the client, and the client sends back the data, which could be anything. The server might send librsync signatures back to the client also.

So possibly, the mobile operators don't like what the phase2 part looks like (ie, more server to client traffic), or they are doing some kind of content manipulation that gets triggered occasionally by the binary data. Since the comms are SSL encrypted, it seems more likely that the additional server to client traffic is what does it. But, I don't really know.

I would recommend not running burp over port 443.

deajan commented 5 years ago

The problem also happens with protocol 2, so it's not librsync related. You're probably right about the server to client traffic. Also, running over port 80 does the same behavior, so both 80 and 443 ports are proxified.

I understand running burp over port 443 is bad, but sometimes there's no other solution when firewall policies block everything.

I would like to make a feature request that could workaround this kind of problem:

A fallback connection address for the client, ie in burp.conf:

server = burp.server
port = 4971
fallback_server = burp.server
fallback_port = 443

This would allow using official IANA registered burp port whenever it is not blocked, and try a second port on connection failure. What do you think about this solution ?

grke commented 5 years ago

I am not saying that it is librsync related.

Protocol2 phase1 is the same as protocol1 phase1. Protocol2 phase2 is different to protocol1 phase2, but they both have the server sending data to the client, which doesn't happen in phase1. And they both have the client sending binary data to the server.

For the fallback port idea, I think it would be better just to have a list:

server = burp.server:4971 server = burp.server:443 server = prub.server:4971

deajan commented 5 years ago

Sure, I just tried to keep backwards compatibility, but this indeed easier.

grke commented 5 years ago

Failover feature is now in master, and deajan says it solves his problem. Going to to an additional logging tweak on top before release.

deajan commented 5 years ago

Just to add my two cents :)

Since 4G providers don't block port 4971, solution is to set:

server = burpserver:4971 failover_server = burpserver:443

And have server listen on both ports. This way, when on 4G networks, backup proceeds via non filtered 4971 port, and falls back to port 443 on restricted networks. This should work unless port 443 communications are not altered on non-4G networks too, in which case the internet provider sucks :)

deajan commented 4 years ago

@grke just to let you know, I have coded some app that uses a CA + a client certificate in HTTPS, and of course, all 4G networks I cited above refuse to let my app work since the HTTPS proxy tries to stick between my client certificate and server certificate. So there's no way around this with burp (and other apps using client certs) than using a different port than 443.

grke commented 4 years ago

OK, thank you for the update.