failure with pytest

[keith6014]

Describe the bug Able to run htmap with python. However, when I use pytest I keep getting ModuleNotFoundError

To Reproduce This works

import htmap
htmap.settings['DELIVERY_METHOD']='shared'

def f(x):
  return x**2
def main():
  m = htmap.map(f,range(10))
  print(list(m))

if __name__=="__main__":
  main()

python run.py

This fails

import htmap,unittest
htmap.settings['DELIVERY_METHOD']='shared'

def f(x):
  return x**2

class TestMe(unittest.TestCase):
  def test_map(self):
    m=htmap.map(f,range(10))
   print(list(m))

if __name__=="__main__":
  unittest.main()

pytest test_me.py

Expected behavior Both python and pytest should work.

Software Versions: Include the results of running each of the following from the command line: 0.5.1 htmap

The error message I get is ModuleNotFoundError: No module named 'test_me'

[JoshKarpel]

@keith6014 Could you clarify where you're getting the error (submit-side or execute-side)? Could you include the entire traceback you get when running pytest test_me.py?

[keith6014]

The error happens on the console. The job gets submitted it fails on _def_load_output().

Python executable information not avaliable.

Its hard for me to get the full stack because I am on a VPN. Security restrictions for Copy/Paste :-(

[keith6014]

Curious, if you are able to reproduce it with pytest.

[JoshKarpel]

We actually use pytest for the HTMap test suite (see https://github.com/htcondor/htmap/tree/master/tests), but we run them inside a single Docker container with assume delivery (see https://github.com/htcondor/htmap/blob/master/tests/_inf/Dockerfile).

Could you try running a pure-pytest version of your test_me.py, like

# test_me.py
import htmap

htmap.settings['DELIVERY_METHOD'] = 'shared'

def f(x):
    return x ** 2

def test_map():
    m = htmap.map(f, range(3))

    assert list(m) == [0, 1, 4]

(i.e., without whatever pytest is doing to run unittest-style tests.) This test passes when I run it inside our test environment with pytest test_me.py (as does your original version with unitest).

The critical piece of information I need from the traceback is whether the exception actually occurred submit-side or execute-side. If it occurred execute-side, the actual error you get will be a MapComponentError with an error report attached (probably failing in https://github.com/htcondor/htmap/blob/49e28424b91d08880f6e64dad37ec4e5649ac00d/htmap/run/run.py#L122). If it failed submit-side, the actual error will be the ModuleNotFoundError, pointing to a specific line in the HTMap source code. Please let me know which it is.

Getting the full output of running pytest test_me.py would be the most helpful, if at all possible. Could you pipe the output (stdout and stderr) into a file and attach that to the issue?

[keith6014]

I called it 'test1.py'

Its happening on the execute side. I see MapComponentError. The job does get submitted because when I do condor_q i see it idle.

File "run.py", line 129, load_func
   return load_object(Path('func'))
Local variables:

File "run.py" line 125, in load_object
  return cloudpickle.load(file)

Local variables:
  cloudpickle = <module 'cloudpickle ... pickle/__init__.py'>
  file = <gzip on ...>
  path= PosixPath('func')

ModuleNotFoundError: No module named 'test1'

htmap/maps.py:350: MapComponentError

in the DEBUG...
I see DEBUG htmap.maps:maps.py:103 Failed to read existing map state for map tall-something because: FileNotFound(2,'No such file or directory')

[JoshKarpel]

Thanks for the extra information! I'll see if I can figure out what's going wrong from my end, but I suspect that I won't be able to replicate your test environment closely enough to see the error. Is there any way I can get access to the environment you're running on?

[keith6014]

sadly no. I have to put in many requests to even submit to github. I placed an approval and have to wait for my management to approve. Good thing is, its not working with pytest but working with python. I put it out because it may cause other issues elsewhere... i am willing to try whatever you like...

[JoshKarpel]

Hmmm. If your goal is just to run tests, and if you're alright with running them on a single machine, you might try using the same Docker image we do for tests. Please feel free to poke around the HTMap test infrastructure and see if that would work for you (clone the repository and look at the dr script for an example of how to build the container).

[keith6014]

Closing. Thanks for your help!