search

counsyl / stor

A cross-compatible CLI and Python API for accessing block and object storage
https://counsyl.github.io/stor/
Other
35 stars 12 forks source link

Different behavior with non-existent target directories in POSIX vs. OBS #48

Open jtratner opened 7 years ago

jtratner commented 7 years ago

I was looking into consolidating tests and I realized that this test has different behavior on swift vs. local filesystem. Specifically, if you use copytree or copy or open, parent directories are auto-created on OBS but not on POSIX:

        folder = stor.join(container, 'analysis')
        subfolder = stor.join(container, 'analysis', 'alignments')
        file_in_folder = stor.join(container, 'analysis', 'alignments',
                                   'bam.bam')
        self.assertFalse(stor.exists(file_in_folder))
        self.assertFalse(stor.isdir(folder))
        self.assertFalse(stor.isdir(folder + '/'))
        with stor.open(file_with_prefix, 'w') as fp:
            fp.write('data\n')
        self.assertFalse(stor.isdir(folder))
        self.assertTrue(stor.isfile(file_with_prefix))

If we wanted to actually have cross-compatible behavior, I'm thinking we should make an API break and then have open(), copy(), copytree() all create parent directories for you (again, to provide consistency).

Thoughts @pkaleta @kyleabeauchamp @wesleykendall ?

jtratner commented 7 years ago

Similarly, might be nice to support use_manifest=True in copytree for POSIX paths as well - so you don't have to think about the behavior whatsoever.

jtratner commented 7 years ago

turns out copytree already handles this, so it's just adding support for copy() and open()