Egypt and UAE HTTP Repository Manipulation/Poison

[Nurmagoz]

Qubes OS version:

Qubes 4

Affected component(s):

Repositories with HTTP

---

Steps to reproduce the behavior:

Welcome to middle east

Expected behavior:

To update/upgrade without interference from the ISP.

Actual horror behavior:

I have discovered that Etisalat (ISP of UAE, so as it has branch in Egypt) names popped up in the middle of the apt upgrade inside debian.

user@user:~$ sudo apt update && sudo apt dist-upgrade && sudo apt autoremove --purge
Hit:1 https://deb.i2p2.de stretch InRelease
Ign:2 https://cdn-aws.deb.debian.org/debian stretch InRelease
Hit:3 https://cdn-aws.deb.debian.org/debian stretch-backports InRelease
Hit:4 https://cdn-aws.deb.debian.org/debian stretch Release
Get:6 https://onlineservices.etisalat.ae/scp/open/osdpages/billPayment.jsp?internetusername=etisalat&ref=http://security.debian.org stretch/updates InRelease [6,505 B]
Err:6 https://onlineservices.etisalat.ae/scp/open/osdpages/billPayment.jsp?internetusername=etisalat&ref=http://security.debian.org stretch/updates InRelease
  Clearsigned file isn't valid, got 'NOSPLIT' (does the network require authentication?)
Get:7 https://onlineservices.etisalat.ae/scp/open/osdpages/billPayment.jsp?internetusername=etisalat&ref=http://deb.qubes-os.org/r4.0/vm stretch InRelease [6,505 B]
Err:7 https://onlineservices.etisalat.ae/scp/open/osdpages/billPayment.jsp?internetusername=etisalat&ref=http://deb.qubes-os.org/r4.0/vm stretch InRelease
  Clearsigned file isn't valid, got 'NOSPLIT' (does the network require authentication?)
Fetched 13.0 kB in 10s (1,198 B/s)
Reading package lists... Done
E: Failed to fetch http://security.debian.org/dists/stretch/updates/InRelease  Clearsigned file isn't valid, got 'NOSPLIT' (does the network require authentication?)
E: Failed to fetch http://deb.qubes-os.org/r4.0/vm/dists/stretch/InRelease  Clearsigned file isn't valid, got 'NOSPLIT' (does the network require authentication?)
E: Some index files failed to download. They have been ignored, or old ones used instead.

as we see , all HTTPS requests went through except Qubes and debian security repo which has an issue with their ssl or port configuration (thats why im using the default debian http repo):

user@user:~$ sudo apt update
Err:1 https://security.debian.org stretch/updates InRelease
  Failed to connect to security.debian.org port 443: Connection refused
W: Failed to fetch https://security.debian.org/dists/stretch/updates/InRelease  Failed to connect to security.debian.org port 443: Connection refused
W: Some index files failed to download. They have been ignored, or old ones used instead.
user@user:~$ 

The way that etisalat done the attack:

there is a payment page which is pushed from etisalat effecting only the HTTP request:

the HTTP URL used to manipulate firefox connection:

https://onlineservices.etisalat.ae/scp/open/osdpages/billPayment.jsp?internetusername=etisalat&ref=http://detectportal.firefox.com/success.txt

The page will show something like this if you will request any HTTP website through firefox:

Luckily i have used ooni-probe to detect if there is an network tampering for http manipulation and yep there was:

0.17s Runtime
Location: ZZ ‪(AS0)‬

Evidence of possible network tampering
When contacting our control servers we noticed that network traffic was manipulated. This means that there could be a “middle box” which could be responsible for censorship and/or traffic manipulation.
Technical measurement data
    *       ▶
annotations:{} 4 keys
        *       engine_name:"libmeasurement_kit"
        *       engine_version:"0.8.3"
        *       engine_version_full:"v0.8.3"
        *       platform:"ios"
    *       data_format_version:"0.2.0"
    *       id:"04cdfb30-8f78-4ac4-8bb1-77ee33dbdd1b"
    *       input:null
    *       input_hashes:[] 0 items
    *       
    *       measurement_start_time:"2018-10-21 00:00:00"
    *       options:[] 0 items
    *       
    *       probe_asn:"AS0"
    *       probe_cc:"ZZ"
    *       probe_city:null
    *       probe_ip:"127.0.0.1"
    *       report_id:"20181021T080301Z_AS0_Zc73b7lwkHuJwB2x1FOcp6Pf9HLXzPDX34IcrQ1lbD2NyajFNx"
    *       software_name:"ooniprobe-ios"
    *       software_version:"1.3.2"
    *       ▶
test_helpers:{} 1 key
        *       backend:"http://37.218.247.95:80"
    *       ▶
test_keys:{} 6 keys
        *       agent:"agent"
        *       client_resolver:"31.171.251.118"
        *       failure:null
        *       ▶
requests:[] 1 item
            *       ▶
0:{} 3 keys
                *       failure:null
                *       ▶
request:{} 5 keys
                *       
                *       ▶
response:{} 4 keys
                *       
        *       socksproxy:null
        *       ▶
tampering:{} 4 keys
            *       header_field_name:null
            *       header_name_diff:null
            *       request_line_capitalization:true
            *       total:true
    *       test_name:"http_header_field_manipulation"
    *       test_runtime:0.171327114105225
    *       test_start_time:"2018-10-21 00:00:00"
    *       test_version:"0.0.1"

What we learn from this:

We need to use only HTTPS or Onion (or any better if there is alternative) for all the repos inside Qubes OS, from Qubes repos to fedora to debian to ...etc.

[lunarthegrey]

Although this isn't really a Qubes specific issue I have seen Egyptian ISPs inject malicious content into HTTP requests before. Maybe Qubes should push an update to use only HTTPS repos? Users could of course edit the repo files themselves, but some might not know how.

[marmarek]

Although deb.qubes-os.org is reachable over HTTPS, security.debian.org is not (sic!). Also, I remember there was some recommendation against using apt-transport-https, but I believe it was about many mirrors not supporting it. @adrelanos what do you recommend? We could at least deb.qubes-os.org switch to https by default.

As for onion by default - no, that's out of the question, See https://github.com/QubesOS/qubes-issues/issues/2604#issuecomment-330423579 (and next comments) for discussion about it.

[Nurmagoz]

i asked in debian security mailing list , one of them answered to use another mirror of ssl debian security repo:

deb https://deb.debian.org/debian-security stretch/updates main

The question remain if we can shift all of the repos to use ssl by default for the sake of users security and avoid harming them through malicious ISP.

[adrelanos]

Don't remember in the top of my head.

Marek Marczykowski-Górecki:> As for onion by default - no, that's out of the question, See

https://github.com/QubesOS/qubes-issues/issues/2604#issuecomment-330423579 (and next comments) for discussion about it.

Agreed.

We'd most likely welcome any stability and usability improvement patches. Ideally we'd make all as easy as an on/off switch gui buttons.

[adrelanos]

https://github.com/QubesOS/qubes-core-agent-linux/blob/master/misc/qubes-r4.list.in still needs update.

[unman]

@TNTBOMBOM @andrewdavidwong This issue could be closed? All merges set https by default for repositories

[Nurmagoz]

from my side yes , all repos are https by default currently!.

ThX!

unman:

@TNTBOMBOM @andrewdavidwong This issue could be closed? All merges set https by default for repositories