"TypeError: unorderable types: NoneType() > NoneType()" with latest luigi-td v0.6.6

[beniyama]

Great to see the latest release, as it supports luigi 2.0!

But it fails with unorderable type comparison error as below:

test@vagrant:~$ python3 tasks.py MyTask --local-scheduler
ERROR: Luigi unexpected framework error while scheduling MyTask(date=2015-12-15)
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/luigi/worker.py", line 490, in add
    for next in self._add(item, is_complete):
  File "/usr/local/lib/python3.4/dist-packages/luigi/worker.py", line 573, in _add
    module=task.task_module)
  File "/usr/local/lib/python3.4/dist-packages/luigi/worker.py", line 388, in _add_task
    self._scheduler.add_task(*args, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/luigi/scheduler.py", line 655, in add_task
    self._update_priority(task, priority, worker_id)
  File "/usr/local/lib/python3.4/dist-packages/luigi/scheduler.py", line 588, in _update_priority
    task.priority = prio = max(prio, task.priority)
TypeError: unorderable types: NoneType() > NoneType()
INFO: Skipping error email. Set `error-email` in the `core` section of the luigi config file or override `owner_email`in the task to receive error emails.
INFO: Done scheduling tasks

The task itself is pretty simple like this

class MyTask(luigi_td.Query):
    type = 'hive'
    database = MyLuigiConfig().db
    source = 'queries/hive/daily/my_simple_query.sql'
    date = luigi.DateParameter(default=datetime.date.today())

And here is my runtime environment :

test@vagrant:~$ python3 --version
Python 3.4.3

test@vagrant:~$ pip3 list | egrep 'td|luigi'
luigi (2.0.0)
luigi-td (0.6.6)
td-client (0.3.2)

Once I set priority=2 into MyTask, the error message changes to TypeError: unorderable types: int() > NoneType() but still be not comparable.

Do you have any suggestion here? I haven't tried with td-client-python 0.4.0 as it haven't appeared at pip repository yet.

[beniyama]

Looks like default priority should be numeric, instead of None?

https://github.com/treasure-data/luigi-td/commit/e58d377f304ad2ace269c4e53cadf21750a30462#diff-b55e7b8044ec19e1074f36173ced51f4R58

[k24d]

Oops, sorry I missed this issue. Let me check it in this weekend.

[qyaman4]

I'm getting the same error.

-my env:

test-server $ pip list | egrep "td|luigi"
luigi (2.0.1)
luigi-td (0.6.6)
td-client (0.4.1)
test-server $
test-server $ python -V
Python 3.4.4
test-server $

-script:

#tasks.py
import luigi
import luigi_td

class MyQuery(luigi_td.Query):
    type = 'presto'
    database = 'qyaman_test'

    def query(self):
        return "SELECT count(1) cnt FROM test_tbl"

if __name__ == '__main__':
    luigi.run()

-error details:

test-server $ python tasks.py MyQuery --local-scheduler
DEBUG: Checking if MyQuery() is complete
/usr/local/python/lib/python3.4/site-packages/luigi/worker.py:279: UserWarning: Task MyQuery() without outputs has no custom complete() method
  is_complete = task.complete()
ERROR: Luigi unexpected framework error while scheduling MyQuery()
Traceback (most recent call last):
  File "/usr/local/python/lib/python3.4/site-packages/luigi/worker.py", line 525, in add
    for next in self._add(item, is_complete):
  File "/usr/local/python/lib/python3.4/site-packages/luigi/worker.py", line 616, in _add
    module=task.task_module)
  File "/usr/local/python/lib/python3.4/site-packages/luigi/worker.py", line 409, in _add_task
    self._scheduler.add_task(*args, **kwargs)
  File "/usr/local/python/lib/python3.4/site-packages/luigi/scheduler.py", line 661, in add_task
    self._update_priority(task, priority, worker_id)
  File "/usr/local/python/lib/python3.4/site-packages/luigi/scheduler.py", line 591, in _update_priority
    task.priority = prio = max(prio, task.priority)
TypeError: unorderable types: NoneType() > NoneType()
INFO: Sending warning email to ['kuyama@treasure-data.com']
INFO: Not sending email when running from a tty
INFO: Done scheduling tasks
INFO: Running Worker with 1 processes
DEBUG: Asking scheduler for work...
INFO: Done
INFO: There are no more tasks to run at this time
INFO: Worker Worker(salt=895341295, workers=1, host=ip-172-31-12-212, username=root, pid=6663) was stopped. Shutting down Keep-Alive thread
INFO:
===== Luigi Execution Summary =====

Scheduled 1 tasks of which:
* 1 were left pending, among these:
    * 1 were left pending because of unknown reason:
        - 1 MyQuery()

Did not run any tasks
This progress looks :) because there were no failed tasks or missing external dependencies

===== Luigi Execution Summary =====

test-server $

And it occurred on Python 3.5 too.

The above was just a quick report.