Pweave does not recognize imported module

[stefanproell]

Hello,

I am trying to produce a report including plots from Pandas. The plots are generated based on data that is provided by a custom module that I wrote. These are the steps I use:

• Create a new environment and activate it
• Install dependencies (pandas, numpy, etc)
• Install the custom module with pip install -e /path/custom_module
• Render the report: pypublish -f html report.py

The report gets rendered with this error:

pypublish -f html report.py 
status
Processing chunk 1 named None from line 7
--- Logging error ---
Traceback (most recent call last):
  File "/home/stefan/anaconda3/lib/python3.6/logging/__init__.py", line 994, in emit
    stream.write(msg)
  File "/home/stefan/anaconda3/lib/python3.6/site-packages/ipykernel/iostream.py", line 384, in write
    self._schedule_flush()
  File "/home/stefan/anaconda3/lib/python3.6/site-packages/ipykernel/iostream.py", line 325, in _schedule_flush
    self.pub_thread.schedule(_schedule_in_thread)
  File "/home/stefan/anaconda3/lib/python3.6/site-packages/ipykernel/iostream.py", line 205, in schedule
    f()
  File "/home/stefan/anaconda3/lib/python3.6/site-packages/ipykernel/iostream.py", line 324, in _schedule_in_thread
    self._io_loop.call_later(self.flush_interval, self._flush)
  File "/home/stefan/anaconda3/lib/python3.6/site-packages/tornado/ioloop.py", line 520, in call_later
    return self.call_at(self.time() + delay, callback, *args, **kwargs)
  File "/home/stefan/anaconda3/lib/python3.6/site-packages/tornado/ioloop.py", line 921, in call_at
    heapq.heappush(self._timeouts, timeout)
TypeError: heap argument must be a list
Call stack:
  File "/home/stefan/anaconda3/bin/pypublish", line 11, in <module>
    sys.exit(publish())
  File "/home/stefan/anaconda3/lib/python3.6/site-packages/pweave/scripts.py", line 78, in publish
    pweave.publish(infile, options.format, options.theme, options.latex_engine, options.output)
  File "/home/stefan/anaconda3/lib/python3.6/site-packages/pweave/__init__.py", line 97, in publish
    doc.format()
  File "/home/stefan/anaconda3/lib/python3.6/site-packages/pweave/pweb.py", line 162, in format
    self.formatter.format()
  File "/home/stefan/anaconda3/lib/python3.6/site-packages/pweave/formatters/base.py", line 60, in format
    self.formatted.append(self.format_docchunk(chunk))
  File "/home/stefan/anaconda3/lib/python3.6/site-packages/pweave/formatters/publish.py", line 127, in format_docchunk
    chunk["content"] = markdown.markdown(chunk["content"], extensions=[MathExtension()])
  File "/home/stefan/anaconda3/lib/python3.6/site-packages/markdown/__init__.py", line 493, in markdown
    md = Markdown(*args, **kwargs)
  File "/home/stefan/anaconda3/lib/python3.6/site-packages/markdown/__init__.py", line 159, in __init__
    configs=kwargs.get('extension_configs', {}))
  File "/home/stefan/anaconda3/lib/python3.6/site-packages/markdown/__init__.py", line 190, in registerExtensions
    % (ext.__class__.__module__, ext.__class__.__name__)
Message: 'Successfully loaded extension "pweave.formatters.markdownmath.MathExtension".'
Arguments: ()
Weaved report.py to report.html

I can compile and run the same file with Python without any errors.

It seems as if the tool does not respect the environment it is executed in. I can use imports regularly, also without errors in the report, but the resulting HTML file shows this error:

Traceback (most recent call last)<ipython-input-1-64650aaac77f> in <module>()
     11 from pandas.plotting._core import boxplot_frame_groupby
     12
---> 13 import custom_module as c
     14
     15

ModuleNotFoundError: No module named
'custom_module'

A work around is adding said custom library manually to the path:

import sys
sys.path.insert(0,'/home/stefan/git/custom_module/custom_module')
import data_helpers.data_helpers as h

Any ideas what is wrong here? Thank you.

[mpastell]

Did you also install Pweave in the new environment?

[stefanproell]

Hello, yes I did. I verified with pip list. (Removed a few entries for brevity)

arrow (0.12.1)
...
-->          custom_module (0.0.1.4, /home/stefan/git/custom_module)
...
numpy (1.14.0)
pandas (0.22.0)
ptyprocess (0.5.2)
-->          Pweave (0.30.2)
...
setuptools (38.5.1)
...
wheel (0.30.0)

If Pweave would not be there, I would expect a different error message, I guess. Please note that I was referring to the command line execution tool, not the Pweave module.

I can produce a report using the Pweave module using this code:

import pweave as pw
pw.publish(file='/home/stefan/git/python_reports/report.py', doc_format='html', theme='skeleton')

However, this gives the following error:

TypeError: heap argument must be a list
Call stack:
  File "/home/stefan/git/python_reports/render_report.py", line 9, in <module>
    pw.publish(file='/home/stefan/git/python_reports/bbcc_report.py', doc_format='html', theme='skeleton')
  File "/home/stefan/Development/python-virtual-env/lib/python3.5/site-packages/pweave/__init__.py", line 97, in publish
    doc.format()
  File "/home/stefan/Development/python-virtual-env/lib/python3.5/site-packages/pweave/pweb.py", line 162, in format
    self.formatter.format()
  File "/home/stefan/Development/python-virtual-env/lib/python3.5/site-packages/pweave/formatters/base.py", line 60, in format
    self.formatted.append(self.format_docchunk(chunk))
  File "/home/stefan/Development/python-virtual-env/lib/python3.5/site-packages/pweave/formatters/publish.py", line 127, in format_docchunk
    chunk["content"] = markdown.markdown(chunk["content"], extensions=[MathExtension()])
  File "/home/stefan/Development/python-virtual-env/lib/python3.5/site-packages/markdown/__init__.py", line 493, in markdown
    md = Markdown(*args, **kwargs)
  File "/home/stefan/Development/python-virtual-env/lib/python3.5/site-packages/markdown/__init__.py", line 159, in __init__
    configs=kwargs.get('extension_configs', {}))
  File "/home/stefan/Development/python-virtual-env/lib/python3.5/site-packages/markdown/__init__.py", line 190, in registerExtensions
    % (ext.__class__.__module__, ext.__class__.__name__)
Message: 'Successfully loaded extension "pweave.formatters.markdownmath.MathExtension".'
Arguments: ()

[piccolbo]

I may have run into a variant of this problem, where I have a multitask.py and multitask_lib.py in the same directory, and I try to import multitask_lib form multitask.py. It works just fine loading multitask.py in ipython with %load, but a failure occurs with pypublish multitask.py.

ModuleNotFoundError: No module named
'multitask_lib'

Maybe it's a different enough problem that I should create a separate issue, please let me know.

[piccolbo]

Same workaround works as @stefanproell get me going as well.

[mpastell]

I don't really know why this happens, but it seems like a bug. Does this work in Jupyter as expected?

[piccolbo]

Works for me in Jupyter.

[stefcarpi]

Hello! I confirm the bug, pweaving a file.texw does not find a custom module in the same folder, raising an ImportError. I'm using a virtual environment with pweave installed in it.

For now, I have to inelegantly import the module by using:

import importlib.util spec = importlib.util.spec_from_file_location("myModule", "./myModule.py") mymodule = importlib.util.module_from_spec(spec) spec.loader.exec_module(myModule)

myModule.myClass()

[sbuller]

I dealt with importing a library from the same directory by adding the following at the top of my .pmd file. It seems, according to the docs that python doesn't check the source directory since it's not reading the source from a file. This adds the cwd to the search path.

import sys
sys.path.insert(0, '')

[lars20070]

I encountered the same problem. This fix helped. Thanks @sbuller.