search

ooni / probe

OONI Probe network measurement tool for detecting internet censorship
https://ooni.org/install
BSD 3-Clause "New" or "Revised" License
761 stars 142 forks source link

http_requests returns exit_ip: null and exit_name: null #509

Closed anadahz closed 8 years ago

anadahz commented 8 years ago

ooniprobe version: 1.4.2 test: blocking/http_requests ooniprobe -f /etc/ooniprobe/ooniprobe.conf blocking/http_requests --url http://cnn.com

- failure: null
  request:
    body: null
    headers: {User-Agent: 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.7)
        Gecko/20091221 Firefox/3.5.7'}
    method: GET
    tor: {exit_ip: null, exit_name: null, is_tor: true}
    url: http://cnn.com
  response:
    body: "<html>\r\n<head><title>301 Moved Permanently</title></head>\r\n<body bgcolor=\"\
      white\">\r\n<center><h1>301 Moved Permanently</h1></center>\r\n<hr><center>nginx</center>\r\
      \n</body>\r\n</html>\r\n"
    code: 301
    headers: {Content-Type: text/html, Date: 'Thu, 19 May 2016 21:18:08 GMT', Location: 'http://www.cnn.com/',
      Server: nginx, Set-Cookie: 'CG=RO:--:--; path=/', X-UA-Profile: desktop}
socksproxy: null
test_runtime: 1.227489948272705
[D] Adding {'url': 'http://cnn.com', 'headers': {'User-Agent': ['Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6']}, 'body': None, 'method': 'GET', 'tor': {'is_tor': False, 'exit_name': None, 'exit_ip': None}} to report
[D] Successfully completed measurement: <ooni.tasks.Measurement object at 0x743b7250>
[D] Successfully performed measurement <ooni.tasks.Measurement object at 0x743b7250>
[D] <twisted.web._newclient.Response object at 0x746b1050>
[D] Got response <twisted.web._newclient.Response object at 0x7439de10>
[D] Processing response body
[D] Adding {'url': 'http://cnn.com', 'headers': {'User-Agent': ['Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7']}, 'body': None, 'method': 'GET', 'tor': {'is_tor': True, 'exit_name': None, 'exit_ip': None}} to report
[D] Successfully completed measurement: <ooni.tasks.Measurement object at 0x746b1e10>
The two body lengths appear to match
censorship is probably not happening
Headers appear to match
[D] Starting this task <ooni.tasks.ReportEntry object at 0x746b1bf0>
[D] Writing report with YAML reporter
[D] Successfully performed report <ooni.tasks.ReportEntry object at 0x746b1bf0>
[D] None
[D] Starting this task <ooni.tasks.ReportEntry object at 0x746b1f50>
[D] Writing report with OONIB reporter
[D] Updating report with id 20160520T224227Z_AS1916_nPGz3MtmxQCqzmlZQbmLKJHZ5MBoEfdRrlnOis8JdDuCA3jCk8 (http://ihiderha53f36lsd.onion/report/20160520T224227Z_AS1916_nPGz3MtmxQCqzmlZQbmLKJHZ5MBoEfdRrlnOis8JdDuCA3jCk8)
[D] Sending {"content": {"test_keys": {"control_failure": null, "socksproxy": null, "headers_diff": [], "agent": "agent", "experiment_failure": null, "control_cloudflare": false, "body_proportion": 1.0, "factor": 0.8, "body_length_match": true, "requests": [{"failure": null, "request": {"body": null, "headers": {"User-Agent": "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6"}, "tor": {"is_tor": false, "exit_name": null, "exit_ip": null}, "url": "http://cnn.com", "method": "GET"}, "response": {"body": "<html>\\r\\n<head><title>301 Moved Permanently</title></head>\\r\\n<body bgcolor=\\"white\\">\\r\\n<center><h1>301 Moved Permanently</h1></center>\\r\\n<hr><center>nginx</center>\\r\\n</body>\\r\\n</html>\\r\\n", "headers": {"Set-Cookie": "CG=BR:--:--; path=/", "X-UA-Profile": "desktop", "Server": "nginx", "Location": "http://www.cnn.com/", "Date": "Fri, 20 May 2016 22:43:13 GMT", "Content-Type": "text/html"}, "code": 301}}, {"failure": null, "request": {"body": null, "headers": {"User-Agent": "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7"}, "tor": {"is_tor": true, "exit_name": null, "exit_ip": null}, "url": "http://cnn.com", "method": "GET"}, "response": {"body": "<html>\\r\\n<head><title>301 Moved Permanently</title></head>\\r\\n<body bgcolor=\\"white\\">\\r\\n<center><h1>301 Moved Permanently</h1></center>\\r\\n<hr><center>nginx</center>\\r\\n</body>\\r\\n</html>\\r\\n", "headers": {"Set-Cookie": "CG=FR:--:--; path=/", "X-UA-Profile": "desktop", "Server": "nginx", "Location": "http://www.cnn.com/", "Date": "Fri, 20 May 2016 22:43:14 GMT", "Content-Type": "text/html"}, "code": 301}}], "headers_match": true}, "software_version": "1.4.2", "measurement_start_time": "2016-05-20 22:43:13", "test_start_time": "2016-05-20 22:43:08", "probe_asn": "AS1916", "test_name": "http_requests", "software_name": "ooniprobe", "annotations": null, "data_format_version": "0.2.0", "test_runtime": 1.1328721046447754, "options": ["--url", "cnn.com"], "test_version": "0.2.5", "input_hashes": [], "input": "http://cnn.com", "probe_ip": "127.0.0.1", "report_id": "20160520T224227Z_AS1916_nPGz3MtmxQCqzmlZQbmLKJHZ5MBoEfdRrlnOis8JdDuCA3jCk8", "probe_city": null, "id": "f5557378-baf6-485b-9201-630598834084", "test_helpers": {}, "probe_cc": "BR"}, "format": "json"}
[D] Successfully performed measurement <ooni.tasks.Measurement object at 0x746b1e10>
[D] <twisted.web._newclient.Response object at 0x7439de10>
[D] Checking all tasks for completion 1 == 2
[D] Checking all tasks for completion 2 == 2
[...]
[D] None
Main loop terminated.
hellais commented 8 years ago

We need access to the Tor Control port in order to determine the exit ip address. Are you using system tor as opposed to allowing ooniprobe to start Tor? When using system tor you need to specify the control port in order to get this information.

anadahz commented 8 years ago

start_toris set to true. Contents of /etc/ooniprobe/ooniprobe.conf:

# This is the configuration file for OONIProbe
# This file follows the YAML markup format: http://yaml.org/spec/1.2/spec.html
# Keep in mind that indentation matters.

basic:
    # Where OONIProbe should be writing it's log file
    logfile: /var/log/ooni/ooniprobe.log
privacy:
    # Should we include the IP address of the probe in the report?
    includeip: false
    # Should we include the ASN of the probe in the report?
    includeasn: true
    # Should we include the country as reported by GeoIP in the report?
    includecountry: true
    # Should we include the city as reported by GeoIP in the report?
    includecity: false
    # Should we collect a full packet capture on the client?
    includepcap: false
reports:
    # Should we place a unique ID inside of every report
    unique_id: true
    # This is a prefix for each packet capture file (.pcap) per test:
    pcap: null
    collector: null
advanced:
    geoip_data_dir: /usr/share/GeoIP
    debug: true
    # enable if auto detection fails
    #tor_binary: /usr/sbin/tor
    #obfsproxy_binary: /usr/bin/obfsproxy 
    # For auto detection
    interface: auto
    # Of specify a specific interface
    #interface: wlan0
    # If you do not specify start_tor, you will have to have Tor running and
    # explicitly set the control port and SOCKS port
    start_tor: true
    # After how many seconds we should give up on a particular measurement
    measurement_timeout: 120
    # After how many retries we should give up on a measurement
    measurement_retries: 5
    # How many measurements to perform concurrently
    measurement_concurrency: 10
    # After how may seconds we should give up reporting
    reporting_timeout: 360
    # After how many retries to give up on reporting
    reporting_retries: 5
    # How many reports to perform concurrently
    reporting_concurrency: 15
    oonid_api_port: 8042
    report_log_file: null
    inputs_dir: null
    decks_dir: null
tor:
    #socks_port: 8801
    #control_port: 8802
    # Specify the absolute path to the Tor bridges to use for testing
    #bridges: bridges.list
    # Specify path of the tor datadirectory.
    # This should be set to something to avoid having Tor download each time
    # Should we include the country as reported by GeoIP in the report?
    includecountry: true
    # Should we include the city as reported by GeoIP in the report?
    includecity: false
    # Should we collect a full packet capture on the client?
    includepcap: false
reports:
    # Should we place a unique ID inside of every report
    unique_id: true
    # This is a prefix for each packet capture file (.pcap) per test:
    pcap: null
    collector: null
advanced:
    geoip_data_dir: /usr/share/GeoIP
    debug: true
    # enable if auto detection fails
    #tor_binary: /usr/sbin/tor
    #obfsproxy_binary: /usr/bin/obfsproxy 
    # For auto detection
    interface: auto
    # Of specify a specific interface
    #interface: wlan0
    # If you do not specify start_tor, you will have to have Tor running and
    # explicitly set the control port and SOCKS port
    start_tor: true
    # After how many seconds we should give up on a particular measurement
    measurement_timeout: 120
    # After how many retries we should give up on a measurement
    measurement_retries: 5
    # How many measurements to perform concurrently
    measurement_concurrency: 10
    # After how may seconds we should give up reporting
    reporting_timeout: 360
    # After how many retries to give up on reporting
    reporting_retries: 5
    # How many reports to perform concurrently
    reporting_concurrency: 15
    oonid_api_port: 8042
    report_log_file: null
    inputs_dir: null
    decks_dir: null
tor:
    #socks_port: 8801
    #control_port: 8802
    # Specify the absolute path to the Tor bridges to use for testing
    #bridges: bridges.list
    # Specify path of the tor datadirectory.
    # This should be set to something to avoid having Tor download each time
    # the descriptors and consensus data.
    #data_dir: ~/.tor/
torrc:
#    UseBridges 1
#    Bridge meeklite 0.0.2.0:1 url=https://meek-reflect.appspot.com/ front=www.google.com
#    Bridge meeklite 0.0.2.0:2 url=https://d2zfqthxsdq309.cloudfront.net/ front=a0.awsstatic.com
#    Bridge meeklite 0.0.2.0:3 url=https://az786092.vo.msecnd.net/ front=ajax.aspnetcdn.com
#    ClientTransportPlugin meeklite exec /usr/bin/obfs4proxy
    #HTTPProxy: host:port
    #HTTPProxyAuthenticator: user:password
    #HTTPSProxy: host:port
    #HTTPSProxyAuthenticator: user:password
hellais commented 8 years ago

Yes you are right this is actually a regression bug introduced in 1.4.2.

I pushed a fix for it in #511.