search

VUIIS / dax

Distributed Automation for XNAT
MIT License
25 stars 24 forks source link

Xnatmirror fails to connect #119

Closed elijahrockers closed 8 years ago

elijahrockers commented 8 years ago

OK, I'm bringing this one back. I am currently attempting to use Xnatmirror to copy projects from my 1.6.3 production, over to a 1.6.5 VM, but I get a refused connection error telling me to check credentials. The credentials are definitely correct (I have tried several times - and in any case the credentials are written in plaintext) and I have verified that the user credentials are the owner of the test project.

One thing of note here is that the credentials are authenticated via LDAP, which I suspect might be causing the issue. Thoughts?

elijahrockers commented 8 years ago

Created a local account. I can log in locally via database, but I'm still getting the same access error through Xnatmirror... when I try to use that account...

elijahrockers commented 8 years ago

Nvm derp, I forgot to include http:// OK, different error now: With DEST_XNAT_HOST=http://myhost:8080/xnat

I think I have to specify port because my VM is connected via NAT

Traceback (most recent call last):
  File "/usr/local/bin/Xnatmirror", line 750, in <module>
    (%s) to be able to upload processed data (assessors).' % logs_info['desthost'])
KeyError: 'desthost'
elijahrockers commented 8 years ago

I commented out that dictionary call and now it works. I am copying files over from a 1.6.3 production server to a 1.6.5 VM, but I'm still getting some errors.

There are a lot of seemingly successful INFO:Processing resource:SNAPSHOTS... and INFO:Processing resource:DICOM... etc, and then this:

INFO:Processing scan:80168...
('PUT', '/data/projects/A4_study/subjects/A41774010/experiments/A41774010_FLORBETAPIRb/scans/80168?xsiType=xnat:scScanData')
Traceback (most recent call last):
  File "Xnatmirror_fix", line 779, in <module>
    copy_project(src_p, dst_p, p_cache_dir)
  File "Xnatmirror_fix", line 388, in copy_project
    copy_subject(src_subj, dst_subj, subj_cache_dir)
  File "Xnatmirror_fix", line 426, in copy_subject
    copy_session(src_sess, dst_sess, sess_cache_dir)
  File "Xnatmirror_fix", line 456, in copy_session
    copy_scan(src_scan, dst_scan, scan_cache_dir)
  File "Xnatmirror_fix", line 472, in copy_scan
    dst_scan.create(scans=scan_type)
  File "/usr/local/lib/python2.7/dist-packages/pyxnat/core/resources.py", line 381, in create
    output = self._intf._exec(create_uri, 'PUT')
  File "/usr/local/lib/python2.7/dist-packages/pyxnat/core/interfaces.py", line 362, in _exec
    '''.format(response=response))
  File "/usr/local/lib/python2.7/dist-packages/pyxnat/core/errors.py", line 72, in catch_error
    raise DatabaseError(full_response)
pyxnat.core.errors.DatabaseError: pyxnat._exec failure:
    URI: http://picasso:8080/xnat/data/projects/A4_study/subjects/A41774010/experiments/A41774010_FLORBETAPIRb/scans/80168?xsiType=xnat:scScanData
    status code: 500
    headers: {'Content-Length': '365', 'Set-Cookie': 'SESSION_EXPIRATION_TIME="1474648721389,900000"; Version=1; Path=/', 'Accept-Ranges': 'bytes', 'Server': 'Noelios-Restlet-Engine/1.1.6', 'Connection': 'close', 'Date': 'Fri, 23 Sep 2016 16:38:41 GMT', 'Content-Type': 'text/html;charset=ISO-8859-1'}
    content: <html>
<head>
   <title>Status page</title>
</head>
<body>
<h3>The server encountered an unexpected condition which prevented it from fulfilling the request</h3>
elijahrockers commented 8 years ago

It looks like this may be a problem with PET scans, as I was able to copy over a test project that had MRIs with no problems whatsoever, but these PET sessions are giving me huge trouble

elijahrockers commented 8 years ago

I've stepped through the program with the interpreter and narrowed it down to this line in the copy_attrs(src_obj, dest_obj, attr_list) function.

src_attrs = src_obj.attrs.mget(attr_list) is returning pyxnat error, and I'm not entirely sure why. Below is the error in full

>>> src_obj.attrs.mget(attr_list)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/pyxnat/core/attributes.py", line 172, in mget
    jdata = JsonTable(self._intf._get_json(get_uri)
  File "/usr/local/lib/python2.7/dist-packages/pyxnat/core/interfaces.py", line 388, in _get_json
    content = self._exec(uri, 'GET')
  File "/usr/local/lib/python2.7/dist-packages/pyxnat/core/interfaces.py", line 362, in _exec
    '''.format(response=response))
  File "/usr/local/lib/python2.7/dist-packages/pyxnat/core/errors.py", line 72, in catch_error
    raise DatabaseError(full_response)
pyxnat.core.errors.DatabaseError: pyxnat._exec failure:
    URI: http://hhvxnatp01/xnat/data/projects/testproject/subjects/testPETsub/experiments/hmri_E01567/scans?columns=ID,xnat:imageScanData/type,xnat:imageScanData/UID,xnat:imageScanData/note,xnat:imageScanData/quality,xnat:imageScanData/condition,xnat:imageScanData/series_description,xnat:imageScanData/documentation,xnat:imageScanData/frames,xnat:imageScanData/scanner/manufacturer,xnat:imageScanData/scanner/model,xnat:petScanData/parameters/orientation,xnat:petScanData/parameters/scanTime,xnat:petScanData/parameters/originalFileName,xnat:petScanData/parameters/systemType,xnat:petScanData/parameters/fileType,xnat:petScanData/parameters/transaxialFOV,xnat:petScanData/parameters/acqType,xnat:petScanData/parameters/facility,xnat:petScanData/parameters/numPlanes,xnat:petScanData/parameters/frames/numFrames,xnat:petScanData/parameters/numGates,xnat:petScanData/parameters/planeSeparation,xnat:petScanData/parameters/binSize,xnat:petScanData/parameters/dataType&format=csv
    status code: 500
    headers: {'Content-Length': '267', 'Content-Encoding': 'gzip', 'Set-Cookie': 'SESSION_EXPIRATION_TIME="1474914919485,900000"; Version=1; Path=/', 'Accept-Ranges': 'bytes', 'Vary': 'Accept-Charset,Accept-Encoding,Accept-Language,Accept', 'Server': 'Noelios-Restlet-Engine/1.1.6', 'Connection': 'close', 'Date': 'Mon, 26 Sep 2016 18:35:19 GMT', 'Content-Type': 'text/html;charset=ISO-8859-1'}
    content: <html>
<head>
   <title>Status page</title>
</head>
<body>
<h3>The server encountered an unexpected condition which prevented it from fulfilling the request</h3><p>You can get technical details <a href="http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.5.1">here</a>.<br>
Please continue your visit at our <a href="/">home page</a>.
</p>
</body>
</html>

>>>

PET_SCAN_ATTRS is defined earlier, and the src_obj exists and is a valid scan, so it looks like the problem might be with pyxnat

bud42 commented 8 years ago

Can you post an xml document from one of these PET sessions?

On Mon, Sep 26, 2016 at 1:41 PM Eli notifications@github.com wrote:

I've stepped through the program with the interpreter and narrowed it down to this line in the copy_attrs(src_obj, dest_obj, attr_list) function.

src_attrs = src_obj.attrs.mget(attr_list) is returning pyxnat error, and I'm not entirely sure why. Below is the error in full

src_obj.attrs.mget(attr_list) Traceback (most recent call last): File "", line 1, in File "/usr/local/lib/python2.7/dist-packages/pyxnat/core/attributes.py", line 172, in mget jdata = JsonTable(self._intf._get_json(get_uri) File "/usr/local/lib/python2.7/dist-packages/pyxnat/core/interfaces.py", line 388, in _get_json content = self._exec(uri, 'GET') File "/usr/local/lib/python2.7/dist-packages/pyxnat/core/interfaces.py", line 362, in _exec '''.format(response=response)) File "/usr/local/lib/python2.7/dist-packages/pyxnat/core/errors.py", line 72, in catch_error raise DatabaseError(full_response) pyxnat.core.errors.DatabaseError: pyxnat._exec failure: URI: http://hhvxnatp01/xnat/data/projects/testproject/subjects/testPETsub/experiments/hmri_E01567/scans?columns=ID,xnat:imageScanData/type,xnat:imageScanData/UID,xnat:imageScanData/note,xnat:imageScanData/quality,xnat:imageScanData/condition,xnat:imageScanData/series_description,xnat:imageScanData/documentation,xnat:imageScanData/frames,xnat:imageScanData/scanner/manufacturer,xnat:imageScanData/scanner/model,xnat:petScanData/parameters/orientation,xnat:petScanData/parameters/scanTime,xnat:petScanData/parameters/originalFileName,xnat:petScanData/parameters/systemType,xnat:petScanData/parameters/fileType,xnat:petScanData/parameters/transaxialFOV,xnat:petScanData/parameters/acqType,xnat:petScanData/parameters/facility,xnat:petScanData/parameters/numPlanes,xnat:petScanData/parameters/frames/numFrames,xnat:petScanData/parameters/numGates,xnat:petScanData/parameters/planeSeparation,xnat:petScanData/parameters/binSize,xnat:petScanData/parameters/dataType&format=csv status code: 500 headers: {'Content-Length': '267', 'Content-Encoding': 'gzip', 'Set-Cookie': 'SESSION_EXPIRATION_TIME="1474914919485,900000"; Version=1; Path=/', 'Accept-Ranges': 'bytes', 'Vary': 'Accept-Charset,Accept-Encoding,Accept-Language,Accept', 'Server': 'Noelios-Restlet-Engine/1.1.6', 'Connection': 'close', 'Date': 'Mon, 26 Sep 2016 18:35:19 GMT', 'Content-Type': 'text/html;charset=ISO-8859-1'} content:

Status page

The server encountered an unexpected condition which prevented it from fulfilling the request

You can get technical details here.
Please continue your visit at our home page.

PET_SCAN_ATTRS is defined earlier, and the src_obj exists and is a valid scan, so it looks like the problem might be with pyxnat

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/VUIIS/dax/issues/119#issuecomment-249659076, or mute the thread https://github.com/notifications/unsubscribe-auth/ACy88pi0L5e4PUbHqPIN2mY31pjx-HlTks5quBHhgaJpZM4KEY0p .

elijahrockers commented 8 years ago

petSession.xml.txt

I just downloaded a single scan from a PET session, and reuploaded it into a test project as a scan for a test subject/session, typically there would actually be about 5 scans for a PET session

bud42 commented 8 years ago

Looks like one of the PET attribute names was incorrect. This has now been fixed on the master branch. Will you update your dax and try again?

elijahrockers commented 8 years ago

Almost. Scan got copied over, but the 'type' 'series desc' and 'usability' fields weren't copied over, there's also some info missing from the "exploded" view. Screenshots attached

SOURCE SESSION: src

DEST SESSION: dest

elijahrockers commented 8 years ago

The XMLs are still a tad confusing for me, but I notice some scan fields are not in the destination session XML, as compared to the source XML

elijahrockers commented 8 years ago

I think this line in copy_attrs() is not working: dest_obj.attrs.mset(src_list)

If I do an mget(src_list) on the same object, the output is ['', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '']

bud42 commented 8 years ago

Ok, update and try again.

On Mon, Sep 26, 2016 at 3:14 PM Eli notifications@github.com wrote:

I think this line in copy_attrs() is not working: dest_obj.attrs.mset(src_list)

If I do an mget(src_list) on the same object, the output is ['', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '']

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/VUIIS/dax/issues/119#issuecomment-249683925, or mute the thread https://github.com/notifications/unsubscribe-auth/ACy88q3k0XTJyIUctBpEtMz91ocgMVCaks5quCeqgaJpZM4KEY0p .

elijahrockers commented 8 years ago

No change.

bud42 commented 8 years ago

Hmm, I tested a petScanData and it worked fine. Could you try deleting the session on the destination and try again?

On Mon, Sep 26, 2016 at 4:06 PM Eli notifications@github.com wrote:

No change.

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/VUIIS/dax/issues/119#issuecomment-249697477, or mute the thread https://github.com/notifications/unsubscribe-auth/ACy88v7fPYHL9aOXHOsHIfmeZNQ2SJHyks5quDO5gaJpZM4KEY0p .

elijahrockers commented 8 years ago

Yea I had actually deleted the entire project. I think there's something wrong with dest_obj.attrs.mset(src_list). As I mentioned, I have stepped through Xnatmirror with the interpreter, so I have all the interfaces up and everything. That line doesn't give me an error, but but when I try the mget(src_list), it returns all empty values. As if the dict is being created but all the values are empty, for the destination object. And if I try to get a value with a nonsense key, it kicks out a nasty error, whereas if I use a valid key, it just returns empty values.

elijahrockers commented 8 years ago

Maybe not necessarily with that function... it clearly works for MRI scans (relatively well) but not for PET... I suppose this is a symptom of the data type/xml wizardry

elijahrockers commented 8 years ago

I tried copying over a completely different PET scan, from a production project. It copied over the two CT scans just fine, but the third experiment, "Dose Info" which just conists of SNAPSHOT and secondary, failed fatally with status 417 "Need PUT contents"... I notice in the source XML, the Dose Info does not contain a definition for xsi:type anywhere, but the CT and PET scans do.

elijahrockers commented 8 years ago

I was able to set the type for the PET scan 'manually' by using pyxnat in the interpreter, with the attrs.set() function, (as opposed to attrs.mset() which Xnatmirror is using), so if all else fails I can bruteforce it that way...

bud42 commented 8 years ago

Could you try using the "-ca" flag? This should set each attribute individually rather than using mset.

elijahrockers commented 8 years ago

When I get back from lunch. also as a side note, I manually constructed the 'put' query URI, but excluded the entries with no values, and it worked. I suspect the fact that some of the fields have no value is interfering with the rest query

elijahrockers commented 8 years ago

-ca also didn't work. We're gonna try commenting out the missing paths from the definitions at the top of Xnatmirror

elijahrockers commented 8 years ago

We think the issue is with startTime. We can use mset() individually to set everything except for the time. For some reason startTime refuses to be set, or rather it's always empty

bud42 commented 8 years ago

Ok, looks like the startTime path was wrong. Would you update and try again?

elijahrockers commented 8 years ago

That did it. We are still having similar issues with otherDicomScanData though, some of our MR sessions have some misc. scans that don't get their attributes copied over, but the mrScanData is fine.

bud42 commented 8 years ago

Could you post an example XML from a session that doesn't copy properly?

On Tue, Sep 27, 2016 at 4:20 PM Eli notifications@github.com wrote:

That did it. We are still having similar issues with otherDicomScanData though, some of our MR sessions have some misc. scans that don't get their attributes copied over, but the mrScanData is fine.

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/VUIIS/dax/issues/119#issuecomment-250002411, or mute the thread https://github.com/notifications/unsubscribe-auth/ACy88llVADIDcKKKtDev7w-APIRdJmkUks5quYiQgaJpZM4KEY0p .

elijahrockers commented 8 years ago

Fixed by #120