"File exists" error when writing out the QC in the final stage of the pipeline

[lbeltrame]

I wonder if it's due to the description having a comma:

[2013-12-03 14:53]  Storing directory in local filesystem: /mnt/data/projects/project_tumor_vs_normal/final/Sample_738,_normal/qc
Traceback (most recent call last):
  File "/usr/local/bin/bcbio_nextgen.py", line 5, in <module>
    pkg_resources.run_script('bcbio-nextgen==0.7.6a', 'bcbio_nextgen.py')
  File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 540, in run_script
  File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 1455, in run_script
  File "/usr/local/lib/python2.7/dist-packages/bcbio_nextgen-0.7.6a-py2.7.egg/EGG-INFO/scripts/bcbio_nextgen.py", line 54, in <module>
    main(**kwargs)
  File "/usr/local/lib/python2.7/dist-packages/bcbio_nextgen-0.7.6a-py2.7.egg/EGG-INFO/scripts/bcbio_nextgen.py", line 38, in main
    run_main(**kwargs)
  File "/usr/local/lib/python2.7/dist-packages/bcbio_nextgen-0.7.6a-py2.7.egg/bcbio/pipeline/main.py", line 50, in run_main
    fc_dir, run_info_yaml)
  File "/usr/local/lib/python2.7/dist-packages/bcbio_nextgen-0.7.6a-py2.7.egg/bcbio/pipeline/main.py", line 95, in _run_toplevel
    upload.from_sample(xs[0])
  File "/usr/local/lib/python2.7/dist-packages/bcbio_nextgen-0.7.6a-py2.7.egg/bcbio/upload/__init__.py", line 20, in from_sample
    approach.update_file(finfo, sample, upload_config)
  File "/usr/local/lib/python2.7/dist-packages/bcbio_nextgen-0.7.6a-py2.7.egg/bcbio/upload/filesystem.py", line 48, in update_file
    return copy_finfo_directory(finfo, storage_dir)
  File "/usr/local/lib/python2.7/dist-packages/bcbio_nextgen-0.7.6a-py2.7.egg/bcbio/upload/filesystem.py", line 32, in copy_finfo_directory
    shutil.copytree(finfo["path"], out_dir)
  File "/usr/lib/python2.7/shutil.py", line 177, in copytree
    os.makedirs(dst)
  File "/usr/lib/python2.7/os.py", line 157, in makedirs
    mkdir(name, mode)
OSError: [Errno 17] File exists: '/mnt/data/projects/project_tumor_vs_normal/final/Sample_738,_normal/qc'

[lbeltrame]

It looks like a bug in utils.file_exists. Investigating.

EDIT: looks like os.path.getsize() on that directory returns 0 even if there's a directory with contents there. EDIT2: It's a FUSE mountpoint with MooseFS, so perhaps that's the issue.

[chapmanb]

Luca; Thanks for helping diagnose the issue. I switched directory checking over to avoid using utils.file_exists and now use the standard exists without checking file sizes. Hopefully this will work cleanly on your system now. Thanks again.