search

mpastell / Pweave

Pweave is a scientific report generator and a literate programming tool for Python. It can capture the results and plots from data analysis and works well with numpy, scipy and matplotlib.
http://mpastell.com/pweave
Other
435 stars 64 forks source link

Failure on chunk option source #109

Open piccolbo opened 6 years ago

piccolbo commented 6 years ago

When using the source option as in 

#+ label, source=module.function
z = module.function()

In a python file I get the following trace

Traceback (most recent call last):
  File "/Users/antonio/.local/share/virtualenvs/multitask-learn-XZ3C3iqa/bin/pweave", line 11, in <module>
    sys.exit(weave())
  File "/Users/antonio/.local/share/virtualenvs/multitask-learn-XZ3C3iqa/lib/python3.6/site-packages/pweave/scripts.py", line 53, in weave
    pweave.weave(infile, **opts_dict)
  File "/Users/antonio/.local/share/virtualenvs/multitask-learn-XZ3C3iqa/lib/python3.6/site-packages/pweave/__init__.py", line 59, in weave
    doc.weave()
  File "/Users/antonio/.local/share/virtualenvs/multitask-learn-XZ3C3iqa/lib/python3.6/site-packages/pweave/pweb.py", line 192, in weave
    self.run()
  File "/Users/antonio/.local/share/virtualenvs/multitask-learn-XZ3C3iqa/lib/python3.6/site-packages/pweave/pweb.py", line 129, in run
    proc.run()
  File "/Users/antonio/.local/share/virtualenvs/multitask-learn-XZ3C3iqa/lib/python3.6/site-packages/pweave/processors/base.py", line 46, in run
    res = self._runcode(chunk)
  File "/Users/antonio/.local/share/virtualenvs/multitask-learn-XZ3C3iqa/lib/python3.6/site-packages/pweave/processors/base.py", line 118, in _runcode
    chunk["content"] = module_text[0]["text"].rstrip()
KeyError: 'text'

This is consistent within a document (meaning, if I comment out that source, the next fails and so on) but it is not fully reproducible, even if I am running this in a virtual environment. For instance, yesterday it worked all day, but if I run pweave inside a script, I would get this error. That is, if I typed pweave at shell prompt in the virtual env, it would work. If I typed at the prompt the script with the same pweave invocation, it would fail with this error. module can be either in the current dir or installed with pip install, it doesn't seem to make a difference.

piccolbo commented 6 years ago

Painfully debugging this (painfully because one can't run it in ipython, how's that even possible), I got this

In : pdb.pm()
> /Users/antonio/.local/share/virtualenvs/multitask-learn-XZ3C3iqa/lib/python3.6/site-packages/pweave/processors/base.py(118)_runcode()
-> chunk["content"] = module_text[0]["text"].rstrip()
(Pdb) module_text
[{'output_type': 'error', 'ename': 'NameError', 'evalue': "name 'mtl' is not defined", 'traceback': ['\x1b[0;31m---------------------------------------------------------------------------\x1b[0m', '\x1b[0;31mNameError\x1b[0m                                 Traceback (most recent call last)', '\x1b[0;32m<ipython-input-1-3e510b227e0c>\x1b[0m in \x1b[0;36m<module>\x1b[0;34m()\x1b[0m\n\x1b[1;32m      1\x1b[0m \x1b[0;32mimport\x1b[0m \x1b[0minspect\x1b[0m\x1b[0;34m\x1b[0m\x1b[0m\n\x1b[0;32m----> 2\x1b[0;31m \x1b[0mprint\x1b[0m\x1b[0;34m(\x1b[0m\x1b[0minspect\x1b[0m\x1b[0;34m.\x1b[0m\x1b[0mgetsource\x1b[0m\x1b[0;34m(\x1b[0m\x1b[0mmtl\x1b[0m\x1b[0;34m.\x1b[0m\x1b[0mmake_data_hetero\x1b[0m\x1b[0;34m)\x1b[0m\x1b[0;34m)\x1b[0m\x1b[0;34m\x1b[0m\x1b[0m\n\x1b[0m', "\x1b[0;31mNameError\x1b[0m: name 'mtl' is not defined"]}]

mtl is the module being sourced from, which is a custom working dir module. It seems the problem exists with any module. I will try to get a stack trace for a system module.

piccolbo commented 6 years ago

I was wrong. The failure moves to the next chunk sourcing from a local module. Maybe this is a new variant of #102

piccolbo commented 6 years ago

As I was preparing a minimal repro, the problem disappeared, in the repro and in the original example, in the same virtualenv. Puzzling. Frustrating. Maddening.