search

CivicSpleen / ambry

A comprehensive data package manager
BSD 2-Clause "Simplified" License
4 stars 5 forks source link

Many broken tests with 'NoSysPathError: No mapping to OS filesystem: /code/casters/simple.py' error. #133

Closed nmb10 closed 8 years ago

nmb10 commented 8 years ago

test_pipe_config for example.

py.test test/test_etl/test_etl.py::Test::test_pipe_config --verbose --capture=no

Error:

test/test_etl/test_etl.py:247: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
ambry/bundle/bundle.py:746: in run
    if not self.build(sources=sources, stage=stage, force=force):
ambry/bundle/bundle.py:1557: in build
    return self.run_phase('build', sources=sources, stage=stage, force=force)
ambry/bundle/bundle.py:1462: in run_phase
    self.phase_main(phase, stage=stage, sources=sources)
ambry/bundle/bundle.py:1411: in phase_main
    pl.run(count=rows_count)
ambry/etl/pipeline.py:1832: in run
    self.sink.run()
ambry/etl/pipeline.py:287: in run
    for i, row in enumerate(self._source_pipe):
ambry/etl/pipeline.py:140: in __iter__
    self.headers = self.process_header(next(rg))
ambry/etl/pipeline.py:140: in __iter__
    self.headers = self.process_header(next(rg))
ambry/etl/pipeline.py:140: in __iter__
    self.headers = self.process_header(next(rg))
ambry/etl/pipeline.py:140: in __iter__
    self.headers = self.process_header(next(rg))
ambry/etl/pipeline.py:140: in __iter__
    self.headers = self.process_header(next(rg))
ambry/etl/pipeline.py:1064: in process_header
    self.row_processors = self.bundle.build_caster_code(self.source, self)
ambry/bundle/bundle.py:1596: in build_caster_code
    abs_path = self.build_fs.getsyspath(path)
../../.virtualenvs/ambry/local/lib/python2.7/site-packages/fs/wrapfs/__init__.py:34: in wrapper
    return func(self,*args,**kwds)
../../.virtualenvs/ambry/local/lib/python2.7/site-packages/fs/wrapfs/__init__.py:142: in getsyspath
    return self.wrapped_fs.getsyspath(self._encode(path), allow_none)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = MemoryFS(), path = '/build/code/casters/simple.py', allow_none = False

    def getsyspath(self, path, allow_none=False):
        """Returns the system path (a path recognized by the OS) if one is present.

            If the path does not map to a system path (and `allow_none` is False)
            then a NoSysPathError exception is thrown.  Otherwise, the system
            path will be returned as a unicode string.

            :param path: a path within the filesystem
            :param allow_none: if True, this method will return None when there is no system path,
                rather than raising NoSysPathError
            :type allow_none: bool
            :raises `fs.errors.NoSysPathError`: if the path does not map on to a system path, and allow_none is set to False (default)
            :rtype: unicode

            """
        if not allow_none:
>           raise NoSysPathError(path=path)
E           NoSysPathError: No mapping to OS filesystem: /code/casters/simple.py
nmb10 commented 8 years ago

Done. Using memory path instead of syspath for memoryfs while compiling casters code.

ericbusboom commented 8 years ago

On Nov 5, 2015, at 10:11 AM, Kazbek notifications@github.com wrote:

test_pipe_config for example.

These particular NoSysPath errors are due to a bad interaction with some improvements to imported code ( bundle.py and some others ) that expect the code to be on a real file system. But, there is a deeper problem with tests; they’ve gotten really message and disjointed.

Was was working on adding a test feature to the bundle builds, to test that a bundle built correctly, and I realized that nearly all of our functional tests could be implemented in it. So, I’m adding a new bundle file, test.py, which, if it exists, holds a unittest.TestCase that gets the currentl bundle injected, with decorators to mark tests to be run at certain times. For instance:

class Test(BundleTest):

@before_ingest()
def test_before_ingest(self):
    print 'BEFORE INGEST', self.bundle.identity

@before_ingest()
def test_before_ingest(self):
    self.assertTrue(False)

@after_ingest()
def test_after_ingest(self):
    print 'AFTER INGEST', self.bundle.identity
    x = 1/0

I’m writing up a guide to the system, and I’d like to move all of the functional tests to it. Then, I’d like to separate out the unit tests, so they are run independently from the functional tests. Because the functional tests are integrated into the bundles, the process for running them will be to load all of the test bundles into a library and run them in the library. In production, you’d do the same thing to ensure that data is correctly loaded into a bundle.

I’ll probably have that all done by the weekend.

eric.

Eric Busboom (619) 363 2607
eric@busboom.org http://www.busboom.org