pytest - ImportError: cannot import name 'main'

[sonatav2]

Attempting to install and test d3ploy on a Ubuntu 16.04 LTS machine gives the following errors, regardless of which version of python is used.

running install
running build
running build_py
package init file 'd3ploy/ __ init__.py' not found (or not a regular file)
package init file 'd3ploy/ __ init __ .py' not found (or not a regular file)
running install_lib
running install_egg_info
Removing /usr/local/lib/python3.5/dist-packages/d3ploy-0.0.1.egg-info
Writing /usr/local/lib/python3.5/dist-packages/d3ploy-0.0.1.egg-info
sonata@gwenchee:~/fuelcycle/d3ploy$ sudo pytest
============================== test session starts ===============================
platform linux -- Python 3.5.2, pytest-4.5.0, py-1.8.0, pluggy-0.12.0
rootdir: /home/sonata/fuelcycle/d3ploy
collected 0 items / 6 errors                                                     

===================================== ERRORS =====================================
________ ERROR collecting tests/integration_tests/back_deployment_test.py ________
ImportError while importing test module '/home/sonata/fuelcycle/d3ploy/tests/integration_tests/back_deployment_test.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/integration_tests/back_deployment_test.py:15: in <module>
    import d3ploy.tester as functions
/usr/local/lib/python3.5/dist-packages/d3ploy/__init__.py:1: in <module>
    from d3ploy import main
E   ImportError: cannot import name 'main'
____________ ERROR collecting tests/integration_tests/buffer_test.py _____________
ImportError while importing test module '/home/sonata/fuelcycle/d3ploy/tests/integration_tests/buffer_test.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/integration_tests/buffer_test.py:16: in <module>
    import d3ploy.tester as functions
/usr/local/lib/python3.5/dist-packages/d3ploy/__init__.py:1: in <module>
    from d3ploy import main
E   ImportError: cannot import name 'main'
__________ ERROR collecting tests/integration_tests/constraint_test.py ___________
ImportError while importing test module '/home/sonata/fuelcycle/d3ploy/tests/integration_tests/constraint_test.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/integration_tests/constraint_test.py:15: in <module>
    import d3ploy.tester as functions
/usr/local/lib/python3.5/dist-packages/d3ploy/__init__.py:1: in <module>
    from d3ploy import main
E   ImportError: cannot import name 'main'
____ ERROR collecting tests/integration_tests/continuous_integration_test.py _____
ImportError while importing test module '/home/sonata/fuelcycle/d3ploy/tests/integration_tests/continuous_integration_test.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/integration_tests/continuous_integration_test.py:10: in <module>
    import d3ploy.tester as functions
/usr/local/lib/python3.5/dist-packages/d3ploy/ __ init __ .py:1: in <module>
    from d3ploy import main
E   ImportError: cannot import name 'main'
_________ ERROR collecting tests/integration_tests/installedcap_test.py __________
ImportError while importing test module '/home/sonata/fuelcycle/d3ploy/tests/integration_tests/installedcap_test.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/integration_tests/installedcap_test.py:16: in <module>
    import d3ploy.tester as functions
/usr/local/lib/python3.5/dist-packages/d3ploy/ __ init __ .py:1: in <module>
    from d3ploy import main
E   ImportError: cannot import name 'main'
___________ ERROR collecting tests/integration_tests/tech_pref_test.py ___________
ImportError while importing test module '/home/sonata/fuelcycle/d3ploy/tests/integration_tests/tech_pref_test.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/integration_tests/tech_pref_test.py:15: in <module>
    import d3ploy.tester as functions
/usr/local/lib/python3.5/dist-packages/d3ploy/ __ init __ .py:1: in <module>
    from d3ploy import main
E   ImportError: cannot import name 'main'
!!!!!!!!!!!!!!!!!!!! Interrupted: 6 errors during collection !!!!!!!!!!!!!!!!!!!!!
============================ 6 error in 0.16 seconds =============================

The`"package init file 'd3ploy/__init__.py' not found` and `ImportError: cannot import name 'main' ` messages led me to this article: http://python-notes.curiousefficiency.org/en/latest/python_concepts/import_traps.html which seems to indicate it's an issue related to Python 3.5's module import system. The same errors/outputs occur whether I'm using Python 2.7 or 3.5.```

[gwenchee]

can you try running a d3ploy input file. So go into the input directory and run cyclus linear_power_demand.xml. I think your d3ploy did install fine. Because when I run python setup.py install, this is my output:

gwenchee ~/github/d3ploy (transitionscenarios) $ python setup.py install running install running build running build_py package init file 'd3ploy/__init__.py' not found (or not a regular file) package init file 'd3ploy/__init__.py' not found (or not a regular file) running install_lib running install_egg_info Removing /home/gwenchee/anaconda3/lib/python3.6/site-packages/d3ploy-0.0.1-py3.6.egg-info Writing /home/gwenchee/anaconda3/lib/python3.6/site-packages/d3ploy-0.0.1-py3.6.egg-info

I usually ignore the 'd3ploy/__init__.py' not found (or not a regular file) warning.

[sonatav2]

cyclus linear_power_demand.xml gives:

terminate called after throwing an instance of 'cyclus::IOError' what(): The file 'linear_pow_demand' could not be loaded. Aborted (core dumped)

But running cyclus recycle.xml from within cycamore/input results in a successful cyclus run.

[katyhuff]

did you run cyclus linear_power_demand.xml or cyclus linear_pow_demand.xml? The error message would indicate you ran the latter.

[sonatav2]

There's no linear_power_demand.xml file in d3ploy/input, just linear_pow_demand.xml- so my command was cyclus linear_pow_demand.xml

[gwenchee]

Just to check, are you on the master branch of arfc/d3ploy?

[sonatav2]

Yes

sonata@gwenchee:~/fuelcycle/d3ploy$ git diff sonata@gwenchee:~/fuelcycle/d3ploy$ git branch * master sonata@gwenchee:~/fuelcycle/d3ploy$ git pull upstream master From https://github.com/arfc/d3ploy * branch master -> FETCH_HEAD Already up-to-date.

[sonatav2]

I'm also having an issue where cycamore seems to install correctly, then fails cycamore_unit_tests with a seg fault. https://github.com/cyclus/cycamore/issues/488. Could this be related?

[gwenchee]

Yeah it is possible since we use cycamore archetypes in these d3ploy tests.

[sonatav2]

I installed cyclus and cycamore (passing all unit tests) on a MacOS system rather than Ubuntu. After installing d3ploy, the d3ploy tests still seem to be having trouble importing the d3ploy python modules. cyclus linear_pow_demand.xml returns ERROR(core ):No module found for path libd3ploy.demand_driven_deployment_inst.dylib. Running python deploy_solver_unit_tests.py from tests/unit_tests returns * thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x10) frame #0: 0x0000000111b503d4 libpython3.6m.dylibPyImport_AddModuleObject + 20 libpython3.6m.dylibPyImport_AddModuleObject

And pytest returns errors like

`===================================================================== FAILURES ======================================================================
________________________________________________________________ test_backdeployment ________________________________________________________________

    def test_backdeployment():
        output_ = 'test_backdeployment.sqlite'
        input_path = os.path.abspath(__file__)
        find = 'd3ploy/'
        indx = input_path.rfind('d3ploy/')
        input_ = input_path.replace(
            input_path[indx + len(find):], 'input/linear_pow_demand_backdeployment.xml')
        s = subprocess.check_output(['cyclus', '-o', output_, input_],
>                                   universal_newlines=True, env=ENV)

tests/integration_tests/back_deployment_test.py:37: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/anaconda3/lib/python3.6/subprocess.py:336: in check_output
    **kwargs).stdout
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

input = None, timeout = None, check = True
popenargs = (['cyclus', '-o', 'test_backdeployment.sqlite', '/Users/sonata/fuelcycle/d3ploy/input/linear_pow_demand_backdeployment.xml'],)
kwargs = {'env': {'Apple_PubSub_Socket_Render': '/private/tmp/com.apple.launchd.2e5eHuiyYA/Render', 'CMAKE_MODULE_PATH': '/opt/...ude:', 'CMAKE_PREFIX_PATH': '/opt/local:', 'CONDA_DEFAULT_ENV': 'base', ...}, 'stdout': -1, 'universal_newlines': True}
process = <subprocess.Popen object at 0x118083320>
stdout = '              :                                                               \n          .CL:CC CC             _Q   ...                             \n ERROR(core  ):No module found for path libd3ploy.demand_driven_deployment_inst.dylib\n'
stderr = None, retcode = 1

    def run(*popenargs, input=None, timeout=None, check=False, **kwargs):
        """Run command with arguments and return a CompletedProcess instance.

        The returned instance will have attributes args, returncode, stdout and
        stderr. By default, stdout and stderr are not captured, and those attributes
        will be None. Pass stdout=PIPE and/or stderr=PIPE in order to capture them.

        If check is True and the exit code was non-zero, it raises a
        CalledProcessError. The CalledProcessError object will have the return code
        in the returncode attribute, and output & stderr attributes if those streams
        were captured.

        If timeout is given, and the process takes too long, a TimeoutExpired
        exception will be raised.

        There is an optional argument "input", allowing you to
        pass a string to the subprocess's stdin.  If you use this argument
        you may not also use the Popen constructor's "stdin" argument, as
        it will be used internally.

        The other arguments are the same as for the Popen constructor.

        If universal_newlines=True is passed, the "input" argument must be a
        string and stdout/stderr in the returned object will be strings rather than
        bytes.
        """
        if input is not None:
            if 'stdin' in kwargs:
                raise ValueError('stdin and input arguments may not both be used.')
            kwargs['stdin'] = PIPE

        with Popen(*popenargs, **kwargs) as process:
            try:
                stdout, stderr = process.communicate(input, timeout=timeout)
            except TimeoutExpired:
                process.kill()
                stdout, stderr = process.communicate()
                raise TimeoutExpired(process.args, timeout, output=stdout,
                                     stderr=stderr)
            except:
                process.kill()
                process.wait()
                raise
            retcode = process.poll()
            if check and retcode:
                raise CalledProcessError(retcode, process.args,
>                                        output=stdout, stderr=stderr)
E               subprocess.CalledProcessError: Command '['cyclus', '-o', 'test_backdeployment.sqlite', '/Users/sonata/fuelcycle/d3ploy/input/linear_pow_demand_backdeployment.xml']' returned non-zero exit status 1.

/anaconda3/lib/python3.6/subprocess.py:418: CalledProcessError`

[FlanFlanagan]

At first glance it looks like d3ploy is not installed properly. Since the I don't think there is an install issue on current branches my guess is you are missing a dependency. Do you have both statsmodels and arch?

[FlanFlanagan]

Also pmdarima need to be installed as well. All three are python packages, and have done a installs

[sonatav2]

Yes, all three dependencies have been installed.

[sonatav2]

Just an update - essentially the same issue is coming up with Python2.7 on a clean ubuntu system. I was using Python3.6 on the Mac, so I don't think it's a Python version issue. d3ploy modules aren't being loaded because shared libraries (".so" files in this case) aren't being found.

sonata@sonata-VirtualBox:~/fuelcycle/d3ploy$ pytest
=========================================================== test session starts ===========================================================
platform linux2 -- Python 2.7.15rc1, pytest-3.3.2, py-1.5.2, pluggy-0.6.0
rootdir: /home/sonata/fuelcycle/d3ploy, inifile:
collected 19 items                                                                                                                        

tests/integration_tests/back_deployment_test.py F                                                                                   [  5%]
tests/integration_tests/buffer_test.py FFFF^C

================================================================ FAILURES =================================================================
___________________________________________________________ test_backdeployment ___________________________________________________________

    def test_backdeployment():
        output_ = 'test_backdeployment.sqlite'
        input_path = os.path.abspath(__file__)
        find = 'd3ploy/'
        indx = input_path.rfind('d3ploy/')
        input_ = input_path.replace(
            input_path[indx + len(find):], 'input/linear_pow_demand_backdeployment.xml')
        s = subprocess.check_output(['cyclus', '-o', output_, input_],
>                                   universal_newlines=True, env=ENV)

tests/integration_tests/back_deployment_test.py:37: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

popenargs = (['cyclus', '-o', 'test_backdeployment.sqlite', '/home/sonata/fuelcycle/d3ploy/input/linear_pow_demand_backdeployment.xml'],)
kwargs = {'env': {'BASH_ENV': '/usr/share/lmod/lmod/init/bash', 'CLUTTER_IM_MODULE': 'xim', 'COLORTERM': 'truecolor', 'DBUS_SESSION_BUS_ADDRESS': 'unix:path=/run/user/1000/bus', ...}, 'universal_newlines': True}
process = <subprocess.Popen object at 0x7f6ab64cc050>
output = '              :                                                               \n          .CL:CC CC             _Q   ...                                \n ERROR(core  ):No module found for path libd3ploy.demand_driven_deployment_inst.so\n'
unused_err = None, retcode = 1
cmd = ['cyclus', '-o', 'test_backdeployment.sqlite', '/home/sonata/fuelcycle/d3ploy/input/linear_pow_demand_backdeployment.xml']

    def check_output(*popenargs, **kwargs):
        r"""Run command with arguments and return its output as a byte string.

        If the exit code was non-zero it raises a CalledProcessError.  The
        CalledProcessError object will have the return code in the returncode
        attribute and output in the output attribute.

        The arguments are the same as for the Popen constructor.  Example:

        >>> check_output(["ls", "-l", "/dev/null"])
        'crw-rw-rw- 1 root root 1, 3 Oct 18  2007 /dev/null\n'

        The stdout argument is not allowed as it is used internally.
        To capture standard error in the result, use stderr=STDOUT.

        >>> check_output(["/bin/sh", "-c",
        ...               "ls -l non_existent_file ; exit 0"],
        ...              stderr=STDOUT)
        'ls: non_existent_file: No such file or directory\n'
        """
        if 'stdout' in kwargs:
            raise ValueError('stdout argument not allowed, it will be overridden.')
        process = Popen(stdout=PIPE, *popenargs, **kwargs)
        output, unused_err = process.communicate()
        retcode = process.poll()
        if retcode:
            cmd = kwargs.get("args")
            if cmd is None:
                cmd = popenargs[0]
>           raise CalledProcessError(retcode, cmd, output=output)
E           CalledProcessError: Command '['cyclus', '-o', 'test_backdeployment.sqlite', '/home/sonata/fuelcycle/d3ploy/input/linear_pow_demand_backdeployment.xml']' returned non-zero exit status 1

/usr/lib/python2.7/subprocess.py:223: CalledProcessError

[FlanFlanagan]

That's just how cyclus builds modules. They are built into shared objects.

[sonatav2]

We fixed the d3ploy module import error by downgrading scipy to version 1.2 (for future reference). Running d3ploy/input files still results in a seg fault. Result of backtrace (Python 3.6 on the mac) from cyclus linear_pow_demand.xml within input:

Process 53857 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x68)
    frame #0: 0x00007fff76848fbf libc++.1.dylib`std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::assign(char const*, unsigned long) + 19
libc++.1.dylib`std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::assign:
->  0x7fff76848fbf <+19>: movb   (%rbx), %cl
    0x7fff76848fc1 <+21>: testb  $0x1, %cl
    0x7fff76848fc4 <+24>: jne    0x7fff76848fcd            ; <+33>
    0x7fff76848fc6 <+26>: movl   $0x16, %esi
Target 0: (cyclus) stopped.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x68)
  * frame #0: 0x00007fff76848fbf libc++.1.dylib`std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::assign(char const*, unsigned long) + 19
    frame #1: 0x00007fff76848f9b libc++.1.dylib`std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::operator=(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) + 51
    frame #2: 0x00000001089a2559 libcyclus.dylib`cyclus::Agent::spec(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >) + 25
    frame #3: 0x00000001089a2356 libcyclus.dylib`cyclus::DynamicModule::Make(cyclus::Context*, cyclus::AgentSpec) + 1350
    frame #4: 0x000000010941cb2f libcyclus.dylib`cyclus::BuildMasterSchema(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >) + 3375
    frame #5: 0x0000000109422536 libcyclus.dylib`cyclus::XMLFileLoader::master_schema() + 134
    frame #6: 0x00000001094225f1 libcyclus.dylib`cyclus::XMLFileLoader::LoadSim() + 65
    frame #7: 0x000000010000e161 cyclus`main + 13601
    frame #8: 0x00007fff788bf015 libdyld.dylib`start + 1
    frame #9: 0x00007fff788bf015 libdyld.dylib`start + 1

[FlanFlanagan]

This is that same EXC_BAD_ACCESS.

Can you try it on your virtual machine of ubuntu with python 3.6?

[sonatav2]

The module import error re-occurs on the ubuntu virtual machine where everything has been installed with Python 3.6 and cyclus/cycamore pass their unit tests. cyclus linear_pow_demand.xml produces: ERROR(core ):No module found for path libd3ploy.demand_driven_deployment_inst.so. The installed scipy is version 1.2.0.

[FlanFlanagan]

If this is fixed can you close the issue @sonatav2 .

Thanks

[katyhuff]

Maybe close with an explanation of the fix...?

On Thu, Jun 20, 2019, 6:45 PM FlanFlanagan notifications@github.com wrote:

If this is fixed can you close the issue @sonatav2 https://github.com/sonatav2 .

Thanks

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/arfc/d3ploy/issues/260?email_source=notifications&email_token=AADAFK2LRBHIPKNMCSKIXJTP3QJAZA5CNFSM4HQEPOH2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYHA4XI#issuecomment-504237661, or mute the thread https://github.com/notifications/unsubscribe-auth/AADAFKZ7PQCWHHEFZ2JOE33P3QJAZANCNFSM4HQEPOHQ .

[FlanFlanagan]

Good idea!

[sonatav2]

There are still no .so or .dylib files in the given directories, but Python 3.6 (not downloaded through conda) on Ubuntu with scipy 1.2.0 and all of the dependencies installed using pip3 is able to successfully run the xml files in the input directory. One test in pytest fails (tests/integration_tests/continuous_integration_test), but that's a separate issue:

        for prototype in ['reactor', 'source']:
            row_count = cur.execute(query.replace('_', prototype)).fetchone()[0]
>           assert (row_count >= 1)
E           assert 0 >= 1