heron-tracker AttributeError: 'NoneType' object has no attribute 'type'

[samek]

While trying to run heron-tracker with zookeeper I get the above mentioned error. Anyone else experiencing this as well?

Sytem configuration: heron Version 0.14.0 (release) Python version 2.7.10 zookeeper versions tried 3.4.6-1569965 and 3.4.8--1 I've tried it on 2 macs (El Capitan) and ubuntu 14.04. - same error.

heron_tracker.yaml is default.

# Configs for heron-tracker

## Contains the sources where the states are stored.
# Each source has these attributes:
#1. type - type of state manager (zookeeper or file, etc.)
#2. name - name to be used for this source
#3. hostport - only used to connect to zk, must be of the form 'host:port'
#4. rootpath - where all the states are stored
#5. tunnelhost - if ssh tunneling needs to be established to connect to it
statemgrs:
  -
    type: "file"
    name: "local"
    rootpath: "~/.herondata/repository/state/local"
    tunnelhost: "localhost"
  -
    type: "zookeeper"
    name: "zookeeper"
    hostport: "localhost:2181"
    rootpath: "/heron/cluster"
    tunnelhost: "localhost"

# The URL that points to a topology's metrics dashboard.
# This value can use following parameters to create a valid
# URL based on the topology. All parameters are self-explanatory.
# These are found in the execution state of the topology.
#
#   ${CLUSTER}
#   ${ENVIRON}
#   ${TOPOLOGY}
#   ${ROLE}
#   ${USER}
#
# This is a sample, and should be changed to point to corresponding dashboard.
#
# viz.url.format: "http://localhost/${CLUSTER}/${ENVIRON}/${TOPOLOGY}/${ROLE}/${USER}"

heron-tracker output:

$ heron-tracker 
27 May 2016 21:31:40-INFO:main.py:183: Running on port: 8888
27 May 2016 21:31:40-INFO:main.py:184: Using config file: /Users/samek/.herontools/conf/heron_tracker.yaml
27 May 2016 21:31:40-INFO:statemanagerfactory.py:61: Connecting to zk hostport: localhost:2181 rootpath: /heron/cluster
27 May 2016 21:31:40-INFO:connection.py:567: Connecting to localhost:2181
27 May 2016 21:31:40-INFO:connection.py:277: Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=10000, session_id=0, passwd='\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None)
27 May 2016 21:31:40-INFO:client.py:378: Zookeeper connection established, state: CONNECTED
27 May 2016 21:31:40-INFO:statemanagerfactory.py:81: Connecting to file state with rootpath: /Users/samek/.herondata/repository/state/local
27 May 2016 21:31:40-INFO:zkstatemanager.py:93: Adding children watch for path: /heron/cluster/topologies
27 May 2016 21:31:40-INFO:connection.py:277: Sending request(xid=1): GetChildren(path='/heron/cluster/topologies', watcher=<bound method ChildrenWatch._watcher of <kazoo.recipe.watchers.ChildrenWatch object at 0x10e490450>>)
27 May 2016 21:31:40-INFO:connection.py:345: Received error(xid=1) NoNodeError((), {})
Traceback (most recent call last):
  File "/Users/samek/bin/heron-tracker/.bootstrap/_pex/pex.py", line 319, in execute
  File "/Users/samek/bin/heron-tracker/.bootstrap/_pex/pex.py", line 254, in _wrap_coverage
  File "/Users/samek/bin/heron-tracker/.bootstrap/_pex/pex.py", line 286, in _wrap_profiling
  File "/Users/samek/bin/heron-tracker/.bootstrap/_pex/pex.py", line 362, in _execute
  File "/Users/samek/bin/heron-tracker/.bootstrap/_pex/pex.py", line 420, in execute_entry
  File "/Users/samek/bin/heron-tracker/.bootstrap/_pex/pex.py", line 425, in execute_module
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 180, in run_module
    fname, loader, pkg_name)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/Users/samek/bin/heron-tracker/heron/tracker/src/python/main.py", line 194, in <module>
  File "/Users/samek/bin/heron-tracker/heron/tracker/src/python/main.py", line 189, in main
  File "/Users/samek/bin/heron-tracker/heron/tracker/src/python/main.py", line 42, in __init__
  File "/Users/samek/bin/heron-tracker/heron/tracker/src/python/tracker.py", line 79, in synch_topologies
  File "/Users/samek/bin/heron-tracker/heron/statemgrs/src/python/zkstatemanager.py", line 80, in get_topologies
  File "/Users/samek/bin/heron-tracker/heron/statemgrs/src/python/zkstatemanager.py", line 95, in _get_topologies_with_watch
  File "/Users/samek/.pex/install/kazoo-1.3.1-py2.7.egg.9e0018919913c23864670dfb4abcbd86b8867ea6/kazoo-1.3.1-py2.7.egg/kazoo/recipe/watchers.py", line 273, in __call__
    self._get_children()
  File "/Users/samek/.pex/install/kazoo-1.3.1-py2.7.egg.9e0018919913c23864670dfb4abcbd86b8867ea6/kazoo-1.3.1-py2.7.egg/kazoo/recipe/watchers.py", line 22, in wrapper
    return func(*args, **kwargs)
  File "/Users/samek/.pex/install/kazoo-1.3.1-py2.7.egg.9e0018919913c23864670dfb4abcbd86b8867ea6/kazoo-1.3.1-py2.7.egg/kazoo/recipe/watchers.py", line 283, in _get_children
    self._path, self._watcher)
  File "/Users/samek/.pex/install/kazoo-1.3.1-py2.7.egg.9e0018919913c23864670dfb4abcbd86b8867ea6/kazoo-1.3.1-py2.7.egg/kazoo/retry.py", line 122, in __call__
    return func(*args, **kwargs)
  File "/Users/samek/.pex/install/kazoo-1.3.1-py2.7.egg.9e0018919913c23864670dfb4abcbd86b8867ea6/kazoo-1.3.1-py2.7.egg/kazoo/client.py", line 945, in get_children
    return self.get_children_async(path, watch, include_data).get()
  File "/Users/samek/.pex/install/kazoo-1.3.1-py2.7.egg.9e0018919913c23864670dfb4abcbd86b8867ea6/kazoo-1.3.1-py2.7.egg/kazoo/handlers/threading.py", line 107, in get
    raise self._exception
kazoo.exceptions.NoNodeError: ((), {})
27 May 2016 21:31:43-ERROR:connection.py:559: Unhandled exception in connection loop
Traceback (most recent call last):
  File "/Users/samek/.pex/install/kazoo-1.3.1-py2.7.egg.9e0018919913c23864670dfb4abcbd86b8867ea6/kazoo-1.3.1-py2.7.egg/kazoo/protocol/connection.py", line 527, in _connect_attempt
    self._send_ping(connect_timeout)
  File "/Users/samek/.pex/install/kazoo-1.3.1-py2.7.egg.9e0018919913c23864670dfb4abcbd86b8867ea6/kazoo-1.3.1-py2.7.egg/kazoo/protocol/connection.py", line 446, in _send_ping
    self._submit(PingInstance, connect_timeout, PING_XID)
  File "/Users/samek/.pex/install/kazoo-1.3.1-py2.7.egg.9e0018919913c23864670dfb4abcbd86b8867ea6/kazoo-1.3.1-py2.7.egg/kazoo/protocol/connection.py", line 273, in _submit
    if request.type:
AttributeError: 'NoneType' object has no attribute 'type'

zookeeper logs:

2016-05-27 21:32:40,475 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted socket connection from /0:0:0:0:0:0:0:1:52963
2016-05-27 21:32:40,536 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted socket connection from /0:0:0:0:0:0:0:1:52964
2016-05-27 21:32:40,538 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x0, likely client has closed socket
    at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:230)
    at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
    at java.lang.Thread.run(Thread.java:745)
2016-05-27 21:32:40,539 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1008] - Closed socket connection for client /0:0:0:0:0:0:0:1:52963 (no session established for client)
2016-05-27 21:32:40,543 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@900] - Client attempting to establish new session at /0:0:0:0:0:0:0:1:52964
2016-05-27 21:32:40,544 [myid:] - INFO  [SyncThread:0:FileTxnLog@199] - Creating new log file: log.1d
2016-05-27 21:32:40,651 [myid:] - INFO  [SyncThread:0:ZooKeeperServer@645] - Established session 0x154f3b42c7b0000 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:52964
2016-05-27 21:32:43,808 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x154f3b42c7b0000, likely client has closed socket
    at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:230)
    at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
    at java.lang.Thread.run(Thread.java:745)
2016-05-27 21:32:43,809 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1008] - Closed socket connection for client /0:0:0:0:0:0:0:1:52964 which had sessionid 0x154f3b42c7b0000
2016-05-27 21:32:52,004 [myid:] - INFO  [SessionTracker:ZooKeeperServer@355] - Expiring session 0x154f3b42c7b0000, timeout of 10000ms exceeded
2016-05-27 21:32:52,004 [myid:] - INFO  [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@489] - Processed session termination for sessionid: 0x154f3b42c7b0000

[nlu90]

@samek For now, what I've found is the heron_tracker.yaml you pasted is different from the 0.14.0 release as shown below:

statemgrs:
  -
    type: "file"
    name: "local"
    rootpath: "~/.herondata/repository/state/local"
    tunnelhost: "localhost"
  -
    type: "zookeeper"
    name: "localzk"
    hostport: "localhost:2181"
    rootpath: "/heron"
    tunnelhost: "localhost"

Notice the name localzk and the root path /heron are different from what you provided. And I'm testing it on Mac OS X Yosemite. So can you double check if the correct version of heron-tool is installed?

[samek]

@nlu90 I've probably changed it in the process of trying it to work. Anyway same error with localzk as name and rootpath changed to /heron :(

[nlu90]

Interesting. Can you paste the content of ~/.herontools/release.yaml here?

[samek]

sure

cat ~/.herontools/release.yaml 
heron.build.version : 0.14.0
heron.build.time : Tue May 24 22:44:01 PDT 2016
heron.build.timestamp : 1464155053000
heron.build.host : tw-mbp-kramasamy
heron.build.user : kramasamy
heron.build.git.revision : be87b09f348e0ed05f45503340a2245a4ef68a35
heron.build.git.status : Clean

[saileshmittal]

@samek Tracker does not create the nodes automatically if they are not already present. The latest version of config file states that: (https://github.com/twitter/heron/blob/master/heron/config/src/yaml/tracker/heron_tracker.yaml)

#
# To use 'localzk', launch a zookeeper server locally
# and create the following paths:
#   1. /heron/topologies
#   2. /heron/executionstate
#   3. /heron/pplans
#   4. /heron/tmasters
#   5. /heron/schedulers
#
#  -
#    type: "zookeeper"
#    name: "localzk"
#    hostport: "localhost:2181"
#    rootpath: "/heron"
#    tunnelhost: "localhost"

So you would need to create those nodes to have Tracker work. One of the reasons we do it this way is that Tracker is completely readonly and does not modify any of the states.

[samek]

I'll give it a go. What should the values of nodes be ? null ?

[nlu90]

I tested on my mac, in order to make the local zk work, you need to manually create the znode "/heron/topologies" in your zk. Once it's created, everything should be fine.

[nlu90]

@saileshmittal should we add this into our doc for future reference?

[samek]

It would definitely help me :) Thanks

[storysj]

It would be nice to also have this note visible in the Quick Start guide: http://twitter.github.io/heron/docs/getting-started/