Columbus XML read error

[dlogan]

This came from a Forum user who also works with Carolina at SciLife I believe (Jordi). This is the Forum issue: http://www.cellprofiler.org/forum/viewtopic.php?f=12&t=3378#p10082

When I load the provided MeasurementIndex.ColumbusIDX.xml file into CPA instead of the properties file, I get this error below, indicating to me that the path is not being parsed properly:

[dlogan@vm2c8-e3d:~/github]$ ./CPA_python.command
/Users/dlogan/github/CPhomebrew/Cellar/cellprofiler-dev-python/1/cpdev/lib/python2.7/site-packages/scikit_learn-0.10-py2.7-macosx-10.6-intel.egg/scikits/learn/__init__.py:2: UserWarning: scikits.learn namespace is deprecated, please use sklearn instead
  warnings.warn('scikits.learn namespace is deprecated, please use sklearn instead')
/Users/dlogan/github/CPhomebrew/Cellar/cellprofiler-dev-python/1/cpdev/lib/python2.7/site-packages/scikit_learn-0.10-py2.7-macosx-10.6-intel.egg/sklearn/hmm.py:24: UserWarning: sklearn.hmm is orphaned, undocumented and has known numerical stability issues. If nobody volunteers to write documentation and make it more stable, this module will be removed in version 0.11.
  warnings.warn('sklearn.hmm is orphaned, undocumented and has known numerical'
Traceback (most recent call last):
  File "./CellProfiler-Analyst/cpa/cpa.py", line 522, in <module>
    app = CPAnalyst(redirect=False)
  File "/usr/local/lib/wxPython-unicode-2.8.12.1/lib/python2.7/site-packages/wx-2.8-mac-unicode/wx/_core.py", line 7981, in __init__
    self._BootstrapApp()
  File "/usr/local/lib/wxPython-unicode-2.8.12.1/lib/python2.7/site-packages/wx-2.8-mac-unicode/wx/_core.py", line 7555, in _BootstrapApp
    return _core_.PyApp__BootstrapApp(*args, **kwargs)
  File "./CellProfiler-Analyst/cpa/cpa.py", line 440, in OnInit
    if not show_load_dialog():
  File "/Users/dlogan/github/CellProfiler-Analyst/cpa/guiutils.py", line 809, in show_load_dialog
    load_columbus(filename)
  File "/Users/dlogan/github/CellProfiler-Analyst/cpa/parseperkinelmer.py", line 104, in load_columbus
    plates, wells, images = get_plates_wells_and_images(image_index)
  File "/Users/dlogan/github/CellProfiler-Analyst/cpa/parseperkinelmer.py", line 425, in get_plates_wells_and_images
    doc = dom.parse(os.path.join(results_dir, image_index))
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/dom/minidom.py", line 1914, in parse
    return expatbuilder.parse(file)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/dom/expatbuilder.py", line 922, in parse
    fp = open(file, 'rb')
IOError: [Errno 2] No such file or directory: u'/Users/dlogan/Projects_local/Forum/Columbus index files/2013-10-16T091253Z[1157]/ImageIndex.ColumbusIDX.xml'

Attachments: http://cellprofiler.org/issues/uploaded/6JJvDo/MeasurementIndex.ColumbusIDX.xml

[dlogan]

See also the email between Carolina, Anne and Lee on November 8, 2013 regarding this very same issue.

[LeeKamentsky]

This one is pretty clear: The measurements XML file has an XML element, , which gives the location of the image index file. CPA correctly parses out the name, "2013-10-16T091253Z[1157]/ImageIndex.ColumbusIDX.xml". There should be a subfolder, "2013-10-16T091253Z[1157]", in the same folder as the measurements XML file and that folder should have the file, "ImageIndex.ColumbusIDX.xml". That file isn't there, or so says CPA.

[dlogan]

The user supplied the ImageIndex XML file, here in the proper subfolder: https://imageweb.broadinstitute.org/imaging/docs/People/David/Forum/Columbus%20index%20files/

When I run this, I get the error below. I told the user to remove the space character in "Alexa 488" and replace it with an underscore. Hopefully that fixes it.

Traceback (most recent call last):
  File "cpa.py", line 522, in <module>
    app = CPAnalyst(redirect=False)
  File "/usr/local/lib/wxPython-unicode-2.8.12.1/lib/python2.7/site-packages/wx-2.8-mac-unicode/wx/_core.py", line 7981, in __init__
    self._BootstrapApp()
  File "/usr/local/lib/wxPython-unicode-2.8.12.1/lib/python2.7/site-packages/wx-2.8-mac-unicode/wx/_core.py", line 7555, in _BootstrapApp
    return _core_.PyApp__BootstrapApp(*args, **kwargs)
  File "cpa.py", line 440, in OnInit
    if not show_load_dialog():
  File "/Users/dlogan/github/CellProfiler-Analyst/cpa/guiutils.py", line 809, in show_load_dialog
    load_columbus(filename)
  File "/Users/dlogan/github/CellProfiler-Analyst/cpa/parseperkinelmer.py", line 115, in load_columbus
    create_per_image_table(plates, channels, imagesets, image_dir)
  File "/Users/dlogan/github/CellProfiler-Analyst/cpa/parseperkinelmer.py", line 268, in create_per_image_table
    %(', '.join(col+' '+typ for col, typ in col_defs)))
  File "/Users/dlogan/github/CellProfiler-Analyst/cpa/dbconnect.py", line 64, in fn
    return f(db, *args, **kwargs)
  File "/Users/dlogan/github/CellProfiler-Analyst/cpa/dbconnect.py", line 512, in execute
    '\nSecond exception was: %s'%(connID, query, e, e2))
dbconnect.DBException: ERROR: Database query failed for connection "MainThread" and failed to reconnect
Query was: "CREATE TABLE per_image (ImageNumber INTEGER, Plate TEXT, Well VARCHAR(3), FieldID INTEGER, PlaneID INTEGER, TimepointID INTEGER, PlateColumns TEXT, PlateRows TEXT, Name TEXT, MeasurementStartTime TEXT, PlateTypeName TEXT, MeasurementID TEXT, Image_File_DAPI TEXT, Image_File_Alexa 488 TEXT, Image_Path_DAPI TEXT, Image_Path_Alexa 488 TEXT)"
First exception was: near "488": syntax error
Second exception was: ERROR: Database query failed for connection "MainThread"
Query was: "CREATE TABLE per_image (ImageNumber INTEGER, Plate TEXT, Well VARCHAR(3), FieldID INTEGER, PlaneID INTEGER, TimepointID INTEGER, PlateColumns TEXT, PlateRows TEXT, Name TEXT, MeasurementStartTime TEXT, PlateTypeName TEXT, MeasurementID TEXT, Image_File_DAPI TEXT, Image_File_Alexa 488 TEXT, Image_Path_DAPI TEXT, Image_Path_Alexa 488 TEXT)"
Exception was: near "488": syntax error

[braymp]

Fortunately, spaces in names are expressly forbidden in CP 2.1.

[dlogan]

That is a very good thing! ... though here CPA is reading in Columbus XML files not supplied by CP (afaik). So unfortunately we can't force an input standard in this odd case :/