Various problems when running under MacOS X

[GoogleCodeExporter]

Running tests on OS X produces the following failure:

Note that OS X does not build directly see issue #441.

tests/t4_adm.py:73: AdmTests.test_authinfo FAILED
tests/t4_adm.py:73: AdmTests.test_authinfo ERROR

================================================================================
 ERRORS 
================================================================================
_____________________________________________________________ ERROR at teardown 
of AdmTests.test_authinfo 
______________________________________________________________

self = <CallInfo when='teardown' exception: Suspicious output to stderr>, func 
= <function call_runtest_hook.<locals>.<lambda> at 0x106a16170>, when = 
'teardown'

>   ???

_pytest.runner:129: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
_ _ _ _ 

>   ???

_pytest.runner:116: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
_ _ _ _ 

kwargs = {'item': <TestCaseFunction 'test_authinfo'>, 'nextitem': 
<TestCaseFunction 'test_passphrase'>}
plugins = [<Session 's3ql-2.5'>, <_pytest.core.PluginManager object at 
0x106ac3150>, <module '_pytest.config' from 'runtests.py/... '_pytest.main' 
from 'runtests.py/_pytest.main'>, <module '_pytest.terminal' from 
'runtests.py/_pytest.terminal'>, ...]

>   ???

_pytest.main:159: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
_ _ _ _ 

self = <HookCaller 'pytest_runtest_teardown'>
plugins = [<Session 's3ql-2.5'>, <_pytest.core.PluginManager object at 
0x106ac3150>, <module '_pytest.config' from 'runtests.py/... '_pytest.main' 
from 'runtests.py/_pytest.main'>, <module '_pytest.terminal' from 
'runtests.py/_pytest.terminal'>, ...]
kwargs = {'item': <TestCaseFunction 'test_authinfo'>, 'nextitem': 
<TestCaseFunction 'test_passphrase'>}
methods = [<function pytest_runtest_teardown at 0x106dff7a0>, <function pdbitem 
at 0x107017320>, <function pytest_runtest_teardo...0d0>>, <bound method 
CaptureManager.pytest_runtest_teardown of <_pytest.capture.CaptureManager 
object at 0x106cc9c10>>]

>   ???

_pytest.core:445: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
_ _ _ _ 

self = <HookCaller 'pytest_runtest_teardown'>
methods = [<function pytest_runtest_teardown at 0x106dff7a0>, <function pdbitem 
at 0x107017320>, <function pytest_runtest_teardo...0d0>>, <bound method 
CaptureManager.pytest_runtest_teardown of <_pytest.capture.CaptureManager 
object at 0x106cc9c10>>]
kwargs = {'item': <TestCaseFunction 'test_authinfo'>, 'nextitem': 
<TestCaseFunction 'test_passphrase'>}

>   ???

_pytest.core:452: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
_ _ _ _ 

self = <MultiCall 0 results, 0 meths, kwargs={'item': <TestCaseFunction 
'test_authinfo'>, 'nextitem': <TestCaseFunction 'test_passphrase'>}>

>   ???

_pytest.core:370: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
_ _ _ _ 

item = <TestCaseFunction 'test_authinfo'>, nextitem = <TestCaseFunction 
'test_passphrase'>

>   ???

_pytest.runner:89: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
_ _ _ _ 

self = <_pytest.runner.SetupState object at 0x107284110>, item = 
<TestCaseFunction 'test_authinfo'>, nextitem = <TestCaseFunction 
'test_passphrase'>

>   ???

_pytest.runner:340: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
_ _ _ _ 

self = <_pytest.runner.SetupState object at 0x107284110>
needed_collectors = [<Session 's3ql-2.5'>, <Module 'tests/t4_adm.py'>, 
<UnitTestCase 'AdmTests'>, <TestCaseFunction 'test_passphrase'>]

>   ???

_pytest.runner:346: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
_ _ _ _ 

self = <_pytest.runner.SetupState object at 0x107284110>

>   ???

_pytest.runner:315: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
_ _ _ _ 

self = <_pytest.runner.SetupState object at 0x107284110>, colitem = 
<TestCaseFunction 'test_authinfo'>

>   ???

_pytest.runner:324: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
_ _ _ _ 

self = <_pytest.runner.SetupState object at 0x107284110>, colitem = 
<TestCaseFunction 'test_authinfo'>

>   ???

_pytest.runner:321: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
_ _ _ _ 

self = <FixtureDef name='check_test_output' scope='function'>

>   ???

_pytest.python:1638: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
_ _ _ _ 

    def raise_on_exception_in_out():
        # Peek at captured output
        (stdout, stderr) = capfd.readouterr()
        sys.stdout.write(stdout)
        sys.stderr.write(stderr)

        if ('exception' in stderr.lower()
            or 'exception' in stdout.lower()):
>           raise AssertionError('Suspicious output to stderr')
E           AssertionError: Suspicious output to stderr

tests/conftest.py:30: AssertionError
--------------------------------------------------------------------------- 
Captured stderr 
----------------------------------------------------------------------------
Warning: maximum object sizes less than 1 MiB will seriously degrade 
performance.
Uncaught top-level exception:
Traceback (most recent call last):
  File "/Users/john/Tmp/s/s3ql-2.5/bin/fsck.s3ql", line 26, in <module>
    s3ql.fsck.main(sys.argv[1:])
  File "/Users/john/Tmp/s/s3ql-2.5/src/s3ql/fsck.py", line 1079, in main
    with open('/proc/mounts', 'r') as fh:
FileNotFoundError: [Errno 2] No such file or directory: '/proc/mounts'
=============================================================================== 
FAILURES 
===============================================================================
________________________________________________________________________ 
AdmTests.test_authinfo 
________________________________________________________________________
Traceback (most recent call last):
  File "/Users/john/Tmp/s/s3ql-2.5/tests/t4_adm.py", line 93, in test_authinfo
    self.assertEqual(proc.wait(), 0)
  File "/usr/local/Cellar/python3/3.3.3/Frameworks/Python.framework/Versions/3.3/lib/python3.3/unittest/case.py", line 641, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/local/Cellar/python3/3.3.3/Frameworks/Python.framework/Versions/3.3/lib/python3.3/unittest/case.py", line 634, in _baseAssertEqual
    raise self.failureException(msg)
AssertionError: 1 != 0
--------------------------------------------------------------------------- 
Captured stderr 
----------------------------------------------------------------------------
Warning: maximum object sizes less than 1 MiB will seriously degrade 
performance.
Uncaught top-level exception:
Traceback (most recent call last):
  File "/Users/john/Tmp/s/s3ql-2.5/bin/fsck.s3ql", line 26, in <module>
    s3ql.fsck.main(sys.argv[1:])
  File "/Users/john/Tmp/s/s3ql-2.5/src/s3ql/fsck.py", line 1079, in main
    with open('/proc/mounts', 'r') as fh:
FileNotFoundError: [Errno 2] No such file or directory: '/proc/mounts'
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 
stopping after 2 failures 
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
====================================================== 1 failed, 162 passed, 61 
skipped, 1 error in 3.86 seconds 
=======================================================

Original issue reported on code.google.com by john@nachtimwald.com on 29 Nov 2013 at 2:38

[GoogleCodeExporter]

Thanks for the report! Do you know if OS X has something similar to 
/proc/mounts that could be used to check if an S3QL file system is currently 
mounted?

Original comment by Nikolaus@rath.org on 29 Nov 2013 at 3:01

• Changed title: OS X tests fail due to missing /proc/mounts
• Changed state: Accepted
• Added labels: Type-Bug

[GoogleCodeExporter]

After googling for a while, it seems the best S3QL can do is look at the output 
of the "mount" command.

Could you mount an S3QL volume and post the output of "mount"? Then I can try 
to add an appropriate check.

Original comment by Nikolaus@rath.org on 29 Nov 2013 at 5:45

• Changed state: NeedsInfo

[GoogleCodeExporter]

I haven't been able to get s3ql to mount on OS X yet. I can mkfs.s3ql but mount 
is failing. There are a few issues. I've attached a patch that should fix the 
/proc/mounts issue as well as a few others.

Right now I've run into an issue with fuse4x not being loaded which I'm 
currently looking into.

Original comment by john@nachtimwald.com on 29 Nov 2013 at 4:06

Attachments:

• s3ql-osx.path

[GoogleCodeExporter]

Fuse4x not loading was my mistake. I forgot to install the kernel extension 
which needs to happen manually.

I'm closer but I still haven't been able to mount an s3ql volume. The issue is 
Fuse4x does not implement nonempty on OS X. So I'm getting an error due to that 
option being passed.

ALso, Fuse4x in Oct 2012 announced they were merging into OSXFuse. OSXFuse 
cannot currently be used because they only implement fuse version 2.7.3 while 
llfuse needs at least 2.8.0. OSXFuse is targeting 2.9.x in master but it's 
currently in an unusable state and no telling how long before they make a 
stable release from it.

I opened an issue with Fuse4x about the missing option but due to the above I 
doubt they'll have a release which adds support for nonempty. What do you think 
about removing this option on OS X?

Original comment by john@nachtimwald.com on 29 Nov 2013 at 5:42

[GoogleCodeExporter]

Another patch which includes removing the nonempty option on OS X. It also 
includes some umount fixes from the previous patch.

Mount optput. This should be handled in the patch but it doesn't hurt to double 
check
$ mount
/dev/disk1 on / (hfs, local, journaled)
devfs on /dev (devfs, local, nobrowse)
map -hosts on /net (autofs, nosuid, automounted, nobrowse)
map auto_home on /home (autofs, automounted, nobrowse)
s3://niw-j-backup on /private/tmp/m (fuse4x, nodev, nosuid, synchronous, 
mounted by john)

Original comment by john@nachtimwald.com on 29 Nov 2013 at 6:04

Attachments:

• s3ql-osx.path

[GoogleCodeExporter]

With the patch I attached previous I can mount, unmount, read and write files 
to an s3ql mount. I can also open files. However, I cannot copy using Finder. 
If I use Finder to copy the file I get an error and the mount shows as mounted 
but I can no longer access it.

Original comment by john@nachtimwald.com on 29 Nov 2013 at 6:13

[GoogleCodeExporter]

What does ~/.s3ql/mount.log say when the mountpoint becomes inaccessible?

Original comment by Nikolaus@rath.org on 29 Nov 2013 at 6:49

[GoogleCodeExporter]

I think a better replacement for /proc/%d/cmdline is to uses the output of "ps 
-p <pid> -o args=" (from 
http://superuser.com/questions/27748/how-to-get-command-line-of-unix-process). 
Could you check if that works? I don't have a Mac...

This would make the patch a lot simpler, and the same code could be used for 
Linux and OS-X.

Original comment by Nikolaus@rath.org on 29 Nov 2013 at 7:07

[GoogleCodeExporter]

> What does ~/.s3ql/mount.log say when the mountpoint becomes inaccessible?

I'll get you this shortly.

> I think a better replacement for /proc/%d/cmdline is to uses the output of 
"ps -p <pid> -o args="

Works on OS X. Give me a bit and I'll update the patch to use that instead for 
both OS X and Linux.

Original comment by john@nachtimwald.com on 29 Nov 2013 at 7:56

[GoogleCodeExporter]

I see that your patch also adds a try .. except around "cores = 
os.sysconf('SC_NPROCESSORS_ONLN')". Does this fail under OS-X? In that case 
there is no point in trying to detect the available memory, because S3QL will 
default to use 2 threads anyway.

Original comment by Nikolaus@rath.org on 29 Nov 2013 at 8:04

[GoogleCodeExporter]

> I see that your patch also adds a try .. except around "cores = 
os.sysconf('SC_NPROCESSORS_ONLN')". Does this fail under OS-X?

It succeeds under OS X. I added it because calling os.sysconf with a param that 
doesn't exist will cause an exception and mounting to fail completely. Since 
it's a known hard failure point and even though it works on OS X  and Linux I 
added it in case someone tries to use it on another OS. Specifically one of the 
BSD variants which may or may not include that param.

Original comment by john@nachtimwald.com on 29 Nov 2013 at 8:07

[GoogleCodeExporter]

New patch, error message and logs.

"The Finder can’t complete the operation because some data in 
“references.txt” can’t be read or written.
(Error code -36)"

Original comment by john@nachtimwald.com on 29 Nov 2013 at 8:09

Attachments:

• s3ql-osx.path
• mount.log
• mount.s3ql_crit.log

[GoogleCodeExporter]

Ok, I've integrated most of the patch and pushed the changes. I'm still not 
sure what do to about nonempty though. Does Fuse4x/OSxfuse allow mounting over 
a non-empty directory even without the nonempty option, or do they not allow it 
at all?

The finder problem is caused by FUSE4x / OSxfuse passing a weird flag to S3QL 
when setting extended attributes. 

Original comment by Nikolaus@rath.org on 29 Nov 2013 at 8:31

[GoogleCodeExporter]

> Does Fuse4x/OSxfuse allow mounting over a non-empty directory even without 
the nonempty option, or do they not allow it at all?

You can mount over a non-empty directly with Fuse4x. I tested and that's what 
happens. The documentation says that shouldn't be the case. The option is 
supported by Fuse4x itself but missing from the list of available options in 
the darwin c file. It is present in the BSD c file however. I believe it was 
omitted by mistake.

I don't know about OSxfuse because the Fuse4x merging is only in master (not a 
stable release) and master from what I've read doesn't currently build so I 
can't test with it.

Considering Fuse4x announced merging with OSxfuse over a year ago and OSxfuse 
hasn't made a release (and isn't ready to make a release) with fuse4x's changes 
I think it would be okay to omit the nonempty option on OS X for now. Once 
OSxfuse actually has a stable release for OS X it can be investigated whether 
the omission needs to be removed. At that time it would be worth moving to 
OSxfuse as a requirement for S3QL on OS X.

Original comment by john@nachtimwald.com on 29 Nov 2013 at 8:38

[GoogleCodeExporter]

According to 
https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/
man3/sysconf.3.html, SC_PHYS_PAGES and SC_PAGESIZE should also exist under OS 
X. 

Could you try to run python3 -c 'import os; print(os.sysconf_names.keys())' and 
post the output?

Original comment by Nikolaus@rath.org on 29 Nov 2013 at 8:38

[GoogleCodeExporter]

> According to ..., SC_PHYS_PAGES and SC_PAGESIZE should also exist under OS X. 

Indeed that's what it says but in practice SC_PHYS_PAGES does not exist. 
SC_PAGESIZE does exist. Not that SC_PAGESIZE helps without having SC_PHYS_PAGES.

> Could you try to run python3 -c 'import os; print(os.sysconf_names.keys())' 
and post the output?

$ python3 -c 'import os; print(os.sysconf_names.keys())'
dict_keys(['SC_SAVED_IDS', 'SC_MEMLOCK', 'SC_THREAD_THREADS_MAX', 
'SC_2_VERSION', 'SC_VERSION', 'SC_DELAYTIMER_MAX', 'SC_AIO_PRIO_DELTA_MAX', 
'SC_XOPEN_VERSION', 'SC_SHARED_MEMORY_OBJECTS', 'SC_XBS5_LP64_OFF64', 
'SC_XOPEN_CRYPT', 'SC_NPROCESSORS_ONLN', 'SC_THREAD_PRIO_INHERIT', 
'SC_SEM_VALUE_MAX', 'SC_NPROCESSORS_CONF', 'SC_2_FORT_DEV', 
'SC_XBS5_ILP32_OFFBIG', 'SC_REALTIME_SIGNALS', 'SC_XOPEN_SHM', 'SC_2_C_BIND', 
'SC_BC_SCALE_MAX', 'SC_XOPEN_XCU_VERSION', 'SC_ARG_MAX', 
'SC_THREAD_PROCESS_SHARED', 'SC_PAGESIZE', 'SC_2_FORT_RUN', 'SC_MQ_PRIO_MAX', 
'SC_AIO_MAX', 'SC_ATEXIT_MAX', 'SC_BC_DIM_MAX', 'SC_XOPEN_ENH_I18N', 
'SC_IOV_MAX', 'SC_STREAM_MAX', 'SC_OPEN_MAX', 'SC_PRIORITY_SCHEDULING', 
'SC_PRIORITIZED_IO', 'SC_2_UPE', 'SC_THREAD_SAFE_FUNCTIONS', 'SC_NGROUPS_MAX', 
'SC_LOGIN_NAME_MAX', 'SC_BC_BASE_MAX', 'SC_XBS5_LPBIG_OFFBIG', 'SC_2_C_DEV', 
'SC_THREAD_PRIORITY_SCHEDULING', 'SC_XOPEN_REALTIME', 
'SC_THREAD_ATTR_STACKADDR', 'SC_GETPW_R_SIZE_MAX', 'SC_XBS5_ILP32_OFF32', 
'SC_XOPEN_LEGACY', 'SC_MQ_OPEN_MAX', 'SC_MESSAGE_PASSING', 'SC_PASS_MAX', 
'SC_COLL_WEIGHTS_MAX', 'SC_ASYNCHRONOUS_IO', 'SC_THREAD_DESTRUCTOR_ITERATIONS', 
'SC_THREAD_PRIO_PROTECT', 'SC_THREAD_KEYS_MAX', 'SC_BC_STRING_MAX', 
'SC_XOPEN_UNIX', 'SC_AIO_LISTIO_MAX', 'SC_2_CHAR_TERM', 'SC_TTY_NAME_MAX', 
'SC_RTSIG_MAX', 'SC_XOPEN_REALTIME_THREADS', 'SC_MAPPED_FILES', 
'SC_SYNCHRONIZED_IO', 'SC_MEMLOCK_RANGE', 'SC_JOB_CONTROL', 'SC_THREADS', 
'SC_SEMAPHORES', 'SC_SIGQUEUE_MAX', 'SC_FSYNC', 'SC_TIMERS', 'SC_2_SW_DEV', 
'SC_CHILD_MAX', 'SC_LINE_MAX', 'SC_TZNAME_MAX', 'SC_THREAD_STACK_MIN', 
'SC_MEMORY_PROTECTION', 'SC_PAGE_SIZE', 'SC_CLK_TCK', 'SC_GETGR_R_SIZE_MAX', 
'SC_EXPR_NEST_MAX', 'SC_THREAD_ATTR_STACKSIZE', 'SC_RE_DUP_MAX', 
'SC_TIMER_MAX', 'SC_SEM_NSEMS_MAX', 'SC_2_LOCALEDEF'])

Original comment by john@nachtimwald.com on 29 Nov 2013 at 8:42

[GoogleCodeExporter]

> Ok, I've integrated most of the patch and pushed the changes.

in unmount.py I notice you didn't incorporate the change to use umount instead 
of fusermount for OS X. fusermount does not exist on OS X so umount must be 
used. It can be used by the user who mounted the mount point originally.

Keeping track of what's outstanding:

* not using fusermount on OS X.
* Determining memory for thread spawning.
* nonempty option.

Original comment by john@nachtimwald.com on 29 Nov 2013 at 8:49

[GoogleCodeExporter]

Fusermount and memory is integrated as well now. Please test if it's working, I 
used the interfaces from your patch but rewrote most of the code to make it 
more robust (e.g. no shell invocation for subprocesses, more detailed parsing 
of sysctl output).

The problem with just removing the nonempty option under OS X is that this 
means that once nonempty is implemented in OSXFuse/FUSE4X, S3QL will suddenly 
no longer be able to mount over non-empty directories. Is there a way to check 
if the system uses FUSE4X or OSXFuse? In that case I would drop nonempty only 
when running FUSE4X, but not when running OSXFuse (with the assumption that 
FUSE4X is not going to see a new release with fixed behavior, and OSXFuse will 
support nonempty when it's released).

With respect to the setxattr flag problem from finder: llfuse knows about the 
XATTR_CREATE and XATTR_REPLACE flags. If it receives any other flags, it bails 
out because it doesn't know what that means. According to 
https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/
man2/setxattr.2.html, OS-X does not know any other flags either (passing 
XATTR_NOFOLLOW to FUSE doesn't make sense), so I'm not sure what FUSE4X is 
doing here. My best guess is that maybe it passes XATTR_NOFOLLOW. I'll post a 
patch to check for that shortly. If that's not the case, you'll have to ask the 
FUSE4X guys.

Original comment by Nikolaus@rath.org on 29 Nov 2013 at 9:05

[GoogleCodeExporter]

> The problem with just removing the nonempty option under OS X is that this 
means that once nonempty is implemented in OSXFuse/FUSE4X, S3QL will suddenly 
no longer be able to mount over non-empty directories.

Right but leaving the option means that you can't mount anything. Fuse4x itself 
errors and won't mount anything if an invalid option is passed. I'd much rather 
lose the ability to mount over non-empty directories in the future and be 
required to upgrade s3ql in the future to get it back than not being able to 
use s3ql at all now.

> Is there a way to check if the system uses FUSE4X or OSXFuse?

The libraries are named differently... Otherwise I don't know enough about 
either project to say definitively if it's possible or not.

Original comment by john@nachtimwald.com on 29 Nov 2013 at 9:12

[GoogleCodeExporter]

Somehow these setxattr issues sounded familiar, and indeed, I talked to the 
fuse4x guys about this before:

https://groups.google.com/d/msg/fuse4x/bRnh7J_nsts/Z7raJ06DB4sJ

Original comment by Nikolaus@rath.org on 29 Nov 2013 at 9:12

[GoogleCodeExporter]

Here is a patch for llfuse to work around the FUSE4X mess. Please let me know 
if it helps.

Original comment by Nikolaus@rath.org on 29 Nov 2013 at 9:17

Attachments:

• llfuse.diff

[GoogleCodeExporter]

Original comment by Nikolaus@rath.org on 29 Nov 2013 at 9:18

• Changed title: Various problems when running under MacOS X

[GoogleCodeExporter]

Attached is a corrected version of the llfuse patch due to tab and space mixing 
causing indentation issues.

Due to the -Werror I'm getting the following build failure for s3ql:

clang -Wno-unused-result -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall 
-Wstrict-prototypes -I/usr/local/include -I/usr/local/opt/sqlite/include 
-isysroot 
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/S
DKs/MacOSX10.8.sdk 
-I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer
/SDKs/MacOSX10.8.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers

-I/usr/local/Cellar/python3/3.3.3/Frameworks/Python.framework/Versions/3.3/inclu
de/python3.3m -c src/s3ql/deltadump.c -o 
build/temp.macosx-10.8-x86_64-3.3/src/s3ql/deltadump.o -Wall -Werror -Wextra
src/s3ql/deltadump.c:6548:19: error: expression result unused 
[-Werror,-Wunused-value]
    PyObject_INIT(o, t);
                  ^
/usr/local/Cellar/python3/3.3.3/Frameworks/Python.framework/Versions/3.3/include
/python3.3m/objimpl.h:163:69: note: expanded from macro 'PyObject_INIT'
    ( Py_TYPE(op) = (typeobj), _Py_NewReference((PyObject *)(op)), (op) )
                                                                    ^
src/s3ql/deltadump.c:6635:19: error: expression result unused 
[-Werror,-Wunused-value]
    PyObject_INIT(o, t);
                  ^
/usr/local/Cellar/python3/3.3.3/Frameworks/Python.framework/Versions/3.3/include
/python3.3m/objimpl.h:163:69: note: expanded from macro 'PyObject_INIT'
    ( Py_TYPE(op) = (typeobj), _Py_NewReference((PyObject *)(op)), (op) )
                                                                    ^
2 errors generated.
error: command 'clang' failed with exit status 1

Original comment by john@nachtimwald.com on 29 Nov 2013 at 9:43

Attachments:

• llfuse-osx.patch

[GoogleCodeExporter]

It's working (Finder and all) with the following changes:

* Using the llfuse patch I attached.
* Removing nonempty from mount.py (again, fuse4x will fail mounting if an 
unsupported option is present).
* I removed -Werror from setup.py so `python3 setup.py build_cython build` 
would succeed.

Original comment by john@nachtimwald.com on 29 Nov 2013 at 9:47

[GoogleCodeExporter]

Great, thanks!

So the only thing that still needs to be resolved is the nonempty option. For 
that I'd need a test for FUSE4X vs OSXFUSE...

Original comment by Nikolaus@rath.org on 29 Nov 2013 at 9:53

[GoogleCodeExporter]

> For that I'd need a test for FUSE4X vs OSXFUSE...

Actually, looking at mount_darwin.c in OSXFuse nonempty isn't listed as a 
supported option... At this point it doesn't look like OSXFuse supports 
nonempty either.

Original comment by john@nachtimwald.com on 29 Nov 2013 at 9:59

[GoogleCodeExporter]

I'll consider this fixed on the S3QL side then. Thanks for your help and 
patches!

Original comment by Nikolaus@rath.org on 29 Nov 2013 at 10:41

• Changed state: Fixed