"Calculating dependencies" documentation should warn about JSON-encodable result

[gdemonet]

Attempting to return pathlib.Path objects in file_dep for a task used as calc_dep of another task ended up failing (exit with return code 3).

I assume this is because of how doit uses saved values to pass the result of the original task. This behaviour, is not explicit in documentation, and for some reason (still under investigation, likely related to our custom reporter), there were no explicit errors except the non-zero exit code when we ran this in our project (scality/metalk8s). Trying to provide a minimal reproducing example led me to understand the issue came from the values returned in the calc_dep task (see gdemonet/doit-calc-deps-issue#1).

[schettino72]

Sorry, I dont understand what is the bug. I run your example and it exits with code 0 as expected.

[gdemonet]

Thanks for your quick response.

Maybe the example was not super obvious, my bad :confused:. Actually, the main branch works as expected, and the broken branch fails (each branch shows what I see as output in the README.md file), and the difference can be seen in the PR I linked.

Anyway, I managed to strip it down to a much simpler example:

# `dodo.py`

from pathlib import Path
DEP_FILE = Path(".dep")

def main_action():
    print("Hello world!")

def task_main():
    return {"actions": [main_action], "calc_dep": ["compute_main_deps"]}

def dep_action():
    DEP_FILE.touch()

def task_dep():
    return {"actions": [dep_action], "targets": [DEP_FILE]}

def task_compute_main_deps():
    deps = {"file_dep": [DEP_FILE], "task_dep": ["dep"]}
    return {"actions": [lambda: deps]}

Running this example yields:

❯ doit -v 2 main
.  compute_main_deps
.  dep
.  main
Hello world!
Traceback (most recent call last):
  File "/home/$USER/repos/doit-calc-deps-issue/.venv/lib/python3.8/site-packages/doit/doit_cmd.py", line 190, in run
    return command.parse_execute(args)
  File "/home/$USER/repos/doit-calc-deps-issue/.venv/lib/python3.8/site-packages/doit/cmd_base.py", line 150, in parse_execute
    return self.execute(params, args)
  File "/home/$USER/repos/doit-calc-deps-issue/.venv/lib/python3.8/site-packages/doit/cmd_base.py", line 601, in execute
    return self._execute(**exec_params)
  File "/home/$USER/repos/doit-calc-deps-issue/.venv/lib/python3.8/site-packages/doit/cmd_run.py", line 264, in _execute
    return runner.run_all(self.control.task_dispatcher())
  File "/home/$USER/repos/doit-calc-deps-issue/.venv/lib/python3.8/site-packages/doit/runner.py", line 261, in run_all
    self.finish()
  File "/home/$USER/repos/doit-calc-deps-issue/.venv/lib/python3.8/site-packages/doit/runner.py", line 240, in finish
    self.dep_manager.close()
  File "/home/$USER/repos/doit-calc-deps-issue/.venv/lib/python3.8/site-packages/doit/dependency.py", line 514, in close
    self.backend.dump()
  File "/home/$USER/repos/doit-calc-deps-issue/.venv/lib/python3.8/site-packages/doit/dependency.py", line 179, in dump
    self._dbm[task_id] = self.codec.encode(self._db[task_id])
  File "/home/$USER/repos/doit-calc-deps-issue/.venv/lib/python3.8/site-packages/doit/dependency.py", line 57, in encode
    return self.encoder.encode(data)
  File "/home/$USER/.pyenv/versions/3.8.6/lib/python3.8/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/home/$USER/.pyenv/versions/3.8.6/lib/python3.8/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/home/$USER/.pyenv/versions/3.8.6/lib/python3.8/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type PosixPath is not JSON serializable
❯ echo $?
3

Some specifics about my setup:

❯ python --version
Python 3.8.6
❯ doit --version
0.33.1
lib @ /home/$USER/repos/doit-calc-deps-issue/.venv/lib/python3.8/site-packages/doit
❯ pip freeze
cloudpickle==1.6.0
doit==0.33.1
pyinotify==0.9.6
❯ uname -srvmo
Linux 5.8.11-050811-generic #202009230858 SMP Wed Sep 23 13:06:55 UTC 2020 x86_64 GNU/Linux

Hope this helps.

[schettino72]

Ok, got it. But the most serious issue was "there were no explicit errors". Can you reproduce that?

[schettino72]

Pathlib is supported on file_dep but not on return values from tasks. Not sure this conversion should be done by default. doit already supports users to specify their own Json codec: https://pydoit.org/cmd_run.html?highlight=decode#codec-cls