full integration test on pto-XX

britram commented 8 years ago

pathspider (clean-mplane) doesn't quite yet run on mplane-sdk (master). Need to figure out where the nulls are coming from.

britram commented 8 years ago

suspect this is still basically a problem with mplane: first, the component seems to think it's returning records (in ecnspider2/__init__.py we see the "ecnspider2: returning {} records" print). The client seems to be re-retrieving capabilities every 5 seconds, which might indicate it's not keeping state properly.

britram commented 8 years ago

Integration tests fail when using the web front-end.

Attempts to use BitTorrent DHT resolution fail with the following backtrace:

  File "/Users/briant/work/pathspider/pathspider/client/resolver.py", line 47, in _invoke
    spec = self.client.invoke_capability(label, "now ... future", params)
  File "/Users/briant/work/mplane-sdk/mplane/client.py", line 545, in invoke_capability
    (cap, spec) = self._spec_for(cap_tol, when, params, relabel)
  File "/Users/briant/work/mplane-sdk/mplane/client.py", line 199, in _spec_for
    cap = self.capability_for(cap_tol)
  File "/Users/briant/work/mplane-sdk/mplane/client.py", line 155, in capability_for
    raise KeyError("no capability for token or label "+token_or_label)
2016-02-14 12:26:53,261 [pathspider: webserver ] [ERROR]  Probe does not support 'btdhtresolver-ip4' capability.

britram commented 8 years ago

Integration tests fail when using the Web front-end.

Attempts to do domain resolution of a small number of domains results in the following backtraces.

The first one has to do with a missing column during chunk merge: this column should probably be added with NAs to the DataFrame on creation:

  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/briant/work/pathspider/pathspider/client/ecnclient.py", line 427, in analyzer_func
    analysis = EcnAnalysis(compiled_chunk, self.ipv, sites=self.sites)
  File "/Users/briant/work/pathspider/pathspider/client/ecnclient.py", line 177, in __init__
    self._analyze(compiled_chunk, ipv)
  File "/Users/briant/work/pathspider/pathspider/client/ecnclient.py", line 290, in _analyze
    merged, self.incomplete = self._merge_results(compiled_chunk, ipv)
  File "/Users/briant/work/pathspider/pathspider/client/ecnclient.py", line 249, in _merge_results
    for ip, result in chunk.groupby('destination.'+ipv):
  File "/Users/briant/.pathspider/lib/python3.5/site-packages/pandas/core/generic.py", line 3436, in groupby
    sort=sort, group_keys=group_keys, squeeze=squeeze)
  File "/Users/briant/.pathspider/lib/python3.5/site-packages/pandas/core/groupby.py", line 1311, in groupby
    return klass(obj, by, **kwds)
  File "/Users/briant/.pathspider/lib/python3.5/site-packages/pandas/core/groupby.py", line 418, in __init__
    level=level, sort=sort)
  File "/Users/briant/.pathspider/lib/python3.5/site-packages/pandas/core/groupby.py", line 2264, in _get_grouper
    in_axis, name, gpr = True, gpr, obj[gpr]
  File "/Users/briant/.pathspider/lib/python3.5/site-packages/pandas/core/frame.py", line 1969, in __getitem__
    return self._getitem_column(key)
  File "/Users/briant/.pathspider/lib/python3.5/site-packages/pandas/core/frame.py", line 1976, in _getitem_column
    return self._get_item_cache(key)
  File "/Users/briant/.pathspider/lib/python3.5/site-packages/pandas/core/generic.py", line 1091, in _get_item_cache
    values = self._data.get(item)
  File "/Users/briant/.pathspider/lib/python3.5/site-packages/pandas/core/internals.py", line 3211, in get
    loc = self.items.get_loc(item)
  File "/Users/briant/.pathspider/lib/python3.5/site-packages/pandas/core/index.py", line 1759, in get_loc
    return self._engine.get_loc(key)
  File "pandas/index.pyx", line 137, in pandas.index.IndexEngine.get_loc (pandas/index.c:3979)
  File "pandas/index.pyx", line 157, in pandas.index.IndexEngine.get_loc (pandas/index.c:3843)
  File "pandas/hashtable.pyx", line 668, in pandas.hashtable.PyObjectHashTable.get_item (pandas/hashtable.c:12265)
  File "pandas/hashtable.pyx", line 676, in pandas.hashtable.PyObjectHashTable.get_item (pandas/hashtable.c:12216)
KeyError: 'destination.ip4'

The second one has to do with improper stringification of IP addresses (see #14):

  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/briant/work/pathspider/pathspider/__init__.py", line 387, in state_func
    self.resolver.process()
  File "/Users/briant/work/pathspider/pathspider/client/resolver.py", line 93, in process
    self._process_result(label, token, result_sink, result)
  File "/Users/briant/work/pathspider/pathspider/client/resolver.py", line 69, in _process_result
    result_sink(label=label, token=token, result=addrs)
  File "/Users/briant/work/pathspider/pathspider/__init__.py", line 442, in resolve_sink
    self.order_ecn(result, flavor)
  File "/Users/briant/work/pathspider/pathspider/__init__.py", line 449, in order_ecn
    self.subjects_map[str(ip)]['ecn'] = None
KeyError: '91.198.174.192'

britram commented 8 years ago

(assigned to @gubser in case you have any time until I get back to this one, perhaps on Monday. We're down to the wire though.)

gubser commented 8 years ago

Cannot reproduce your exceptions using mplane-sdk master 27e6f1b and pathspider new-mplane-final-demo aa41535

standalone: when clicking Run! Neither Domains nor BitTorrent mode works. It just hangs. I think it's because in standalone mode, the webserver and mplane listener share the same tornado ioloop. So if webserver issues an mplane request and blocks, the request itself never gets handled.
client: using this client.json (deleted all probes except de):

{
    "Access" : {
        "Roles" : {
            "unauthenticated" : ["default"]
        },
        "Authorizations" : {
            "default" : ["ecnspider", "webresolver", "bhdhtresolver"]
        }
    },

    "Registries": {
        "default": "http://mami-project.eu/mplane/ecnregistry.json",
        "preload": [ "registry.json", "ecnregistry.json" ]
    },

    "Pathspider" : { 
        "use_tracebox" : true,
        "Probes" : {
            "Frankfurt" :     "http://pto-de.observatory.mami-project.eu:18888/"
        },
        "Resolver" : "http://pto-de.observatory.mami-project.eu:18888/"
    }
}

I get a lot of these exceptions. I assume the client doesn't receive the same registry threatment as the service/standalone.

Exception in thread EcnImp-Frankfurt:
Traceback (most recent call last):
  File "/usr/lib/python3.4/threading.py", line 920, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.4/threading.py", line 868, in run
    self._target(*self._args, **self._kwargs)
  File "/home/gubser/work/pathspider/pathspider/client/ecnclient.py", line 85, in worker
    self.client.retrieve_capabilities(self.url)
  File "/home/gubser/work/mplane-sdk/mplane/client.py", line 607, in retrieve_capabilities
    pool=pool, identity=identity)
  File "/home/gubser/work/mplane-sdk/mplane/client.py", line 607, in retrieve_capabilities
    pool=pool, identity=identity)
  File "/home/gubser/work/mplane-sdk/mplane/client.py", line 598, in retrieve_capabilities
    mplane.model.parse_json(res.data.decode("utf-8")), identity)
  File "/home/gubser/work/mplane-sdk/mplane/model.py", line 3070, in parse_json
    return message_from_dict(json.loads(jstr))
  File "/home/gubser/work/mplane-sdk/mplane/model.py", line 3062, in message_from_dict
    return classmap[k](dictval = d)
  File "/home/gubser/work/mplane-sdk/mplane/model.py", line 2502, in __init__
    super().__init__(dictval=dictval, verb=verb, label=label, token=token, when=when, reguri=registry_uri)
  File "/home/gubser/work/mplane-sdk/mplane/model.py", line 2102, in __init__
    self._from_dict(dictval)
  File "/home/gubser/work/mplane-sdk/mplane/model.py", line 2472, in _from_dict
    self._params_from_dict(d[KEY_PARAMETERS])
  File "/home/gubser/work/mplane-sdk/mplane/model.py", line 2536, in _params_from_dict
    self.add_parameter(k, constraint=v)
  File "/home/gubser/work/mplane-sdk/mplane/model.py", line 2145, in add_parameter
    self._params[elem_name] = Parameter(element(elem_name, reguri=self._reguri),
  File "/home/gubser/work/mplane-sdk/mplane/model.py", line 1645, in element
    raise KeyError("Element name "+name+" not present in current registry")
KeyError: 'Element name scamper.trace.loops not present in current registry'

What is your setup?

britram commented 8 years ago

you'll need to pull the latest new-mplane-final-demo, eb3b2b51ae9a4c58b48eec0f393706348cbc2e2f fixes this exception.

gubser commented 8 years ago

as of 14617e6f0290089abd919947114807e63cffc7e4 by running pathspider service and pathspider client on the same node Frankfurt I can:

successfully run ecnspider2 on
- a few domains,
- domains which don't exist (= no results but no exceptions either)
- bittorrent addresses.
successfully run tracebox on them and get a nice graph.

please test using all probes now :-)

britram commented 8 years ago

Works. Tagged 14617e6f0290089abd919947114807e63cffc7e4 as milan.

mami-project / pathspider

full integration test on pto-XX #11