search

4Quant / COPD-staging

Package of tools to quantify lung disease (COPD) using any number of popular medical quantifiers (PD15, LAA-x...)
1 stars 0 forks source link

LungQuantization Plugin returns Null/Zero values #78

Closed drtjre closed 8 years ago

drtjre commented 8 years ago

Using the lung quantization plugin (as currently installed on my system) directly in Fiji produces legitimate results. However, using the same routines on data retrieved from the PACS using RIQAE produces null or zero or infinite results. Use of another Fiji plugin, "Measure", produces expected results.

Example of Measure command producing legit results:

select patientname,seriesDescription,studydate,runmap(fetch_dicom_imagej(studyInstanceUID,seriesInstanceUID),"Measure","") from stolz3 where success=true;

+-----------------------+--------------------------+------------+--------------------------------------------------+--+
|      patientname      |    seriesDescription     | studydate  |                       _c3                        |
+-----------------------+--------------------------+------------+--------------------------------------------------+--+
| HIDDEN 01  | Thorax 1.0 I26f 3        | 20140621   | {"Area":[262144.0],"Mean":[513.5583724975586]}   |
| HIDDEN 02      | Thorax nativ 1.0 I26f 3  | 20130507   | {"Area":[262144.0],"Mean":[447.33084869384766]}  |
| HIDDEN 03      | Monitoring 10.0 B31f     | 20150529   | {"Area":[262144.0],"Mean":[625.2124786376953]}   |
| HIDDEN 04        | Thorax 3.0 I31f 3        | 20120316   | {"Area":[262144.0],"Mean":[474.97106170654297]}  |
+-----------------------+--------------------------+------------+--------------------------------------------------+--+

Example of LungQuant command producing erroneous results:

select patientname,seriesDescription,studydate,runmap(fetch_dicom_imagej(studyInstanceUID,seriesInstanceUID),"PD LAA Quantify SegmentedLung","") from stolz3 where success=true;

+-----------------------+--------------------------+------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--+
|      patientname      |    seriesDescription     | studydate  |                                                                                                                                    _c3                                                                                                                                    |
+-----------------------+--------------------------+------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--+
| HIDDEN01  | Thorax 1.0 I26f 3        | 20140621   | {"FOV voxels ":[8.0740352E7],"LAA-1024%":[0.0],"LAA-1024pixel^3":[0.0],"LAA-900%":[0.0],"LAA-900pixel^3":[0.0],"LAA-950%":[0.0],"LAA-950pixel^3":[0.0],"LungVol pixel^3":[0.0],"LungVol voxs":[0.0],"PD15":[0.0],"StudyID":[NaN],"sex":[NaN],"vox volume pixel^3":[1.0]}  |
| HIDDEN 02       | Thorax nativ 1.0 I26f 3  | 20130507   | {"FOV voxels ":[8.51968E7],"LAA-1024%":[0.0],"LAA-1024pixel^3":[0.0],"LAA-900%":[0.0],"LAA-900pixel^3":[0.0],"LAA-950%":[0.0],"LAA-950pixel^3":[0.0],"LungVol pixel^3":[0.0],"LungVol voxs":[0.0],"PD15":[0.0],"StudyID":[NaN],"sex":[NaN],"vox volume pixel^3":[1.0]}    |
| HIDDEN 03        | Thorax 3.0 I31f 3        | 20120316   | {"FOV voxels ":[5.4001664E7],"LAA-1024%":[0.0],"LAA-1024pixel^3":[0.0],"LAA-900%":[0.0],"LAA-900pixel^3":[0.0],"LAA-950%":[0.0],"LAA-950pixel^3":[0.0],"LungVol pixel^3":[0.0],"LungVol voxs":[0.0],"PD15":[0.0],"StudyID":[NaN],"sex":[NaN],"vox volume pixel^3":[1.0]}  |
| HIDDEN 04         | Monitoring 10.0 B31f     | 20150529   | {"FOV voxels ":[1310720.0],"LAA-1024%":[0.0],"LAA-1024pixel^3":[0.0],"LAA-900%":[0.0],"LAA-900pixel^3":[0.0],"LAA-950%":[0.0],"LAA-950pixel^3":[0.0],"LungVol pixel^3":[0.0],"LungVol voxs":[0.0],"PD15":[0.0],"StudyID":[NaN],"sex":[NaN],"vox volume pixel^3":[1.0]}    |
+-----------------------+--------------------------+------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--+
4 rows selected (28.809 seconds)
drtjre commented 8 years ago

To permit others to test this situation, I have made available on USBshare three cases in anonymous form. Below a description of how these can be tested to replicate this bug.

Anonymize 3 Stolz

Three Anonymized Stolz cases are privately (requires login) available on USBshare/StolzAnonymize.
Manually performing LungSegmentation/Quant using Fiji produces legitimate results:

PatientName (Anonymized) StudyID LungVol    PD15    LAA-950mm^3
Anonymous Sid 10007     SID10007    2.841   -858    49240.431
Anonymous Sid 10113     SID10113    3.68    -886    41568.794
Anonymous Sid 10201     SID10201    4.471   -937    390522.385

The 3 anonymized cases have been loaded into Osirix for testing with RIQAE - see below for results.

Create Table from csv-PACS:

CREATE TEMPORARY TABLE stolz3pacs USING fourquant.pacs OPTIONS (bind "YETI", username "OSIRIX", server "localhost", port "11112", path "StolzAnonymous3.csv");

where StolzAnonymous3.csv is:

Patient Name;Patient ID;Study Instance UID;Study Date;Study Description;Series Instance UID;Series Description;Result
Anonymous^Sid^10007;;;;;;;
Anonymous^Sid^10113;;;;;;;
Anonymous^Sid^10201;;;;;;;

Check table:


0: jdbc:hive2://127.0.0.1:8000/> select PatientName from stolz3pacs;
+----------------------+--+
|     PatientName      |
+----------------------+--+
| Anonymous Sid 10007  |
| Anonymous Sid 10201  |
| Anonymous Sid 10113  |
+----------------------+--+

Perform a simple Measure calculation:

This appears to produce non-zero reasonable results as expected:

select patientname,seriesDescription,runmap(fetch_dicom_imagej(studyInstanceUID,seriesInstanceUID),"Measure","") from stolz3pacs where success=true;
PACSToolCall:-L SERIES -c osirix@localhost:11112 -b yeti -m SeriesInstanceUID=1.3.12.2.1107.5.1.4.49001.30000008123113283818700000454 -m StudyInstanceUID=1.2.276.0.18.14.200.2.0.0.20081231.103551.64 --store-tcs /var/folders/cj/p00m_p1137l5wpxrjr5wr62c0000gp/T/store_tcs7121845210281147784.properties --directory emptyDir
PACSToolCall:-L SERIES -c osirix@localhost:11112 -b yeti -m SeriesInstanceUID=1.3.12.2.1107.5.1.4.54504.30000008112007153707800007751 -m StudyInstanceUID=1.2.276.0.18.14.200.2.0.0.20081120.125332.60 --store-tcs /var/folders/cj/p00m_p1137l5wpxrjr5wr62c0000gp/T/store_tcs3246684587818035570.properties --directory emptyDir
PACSToolCall:-L SERIES -c osirix@localhost:11112 -b yeti -m SeriesInstanceUID=1.3.12.2.1107.5.1.4.50021.30000008051505383531200008814 -m StudyInstanceUID=1.2.276.0.18.14.200.2.0.0.20080515.113315.4 --store-tcs /var/folders/cj/p00m_p1137l5wpxrjr5wr62c0000gp/T/store_tcs7126723019959534364.properties --directory emptyDir
+----------------------+------------------------+--------------------------------------------------+--+
|     patientname      |   seriesDescription    |                       _c2                        |
+----------------------+------------------------+--------------------------------------------------+--+
| Anonymous Sid 10007  | HWS+Thx art. 3.0 B30f  | {"Area":[262144.0],"Mean":[215.31087112426758]}  |
| Anonymous Sid 10201  | Thorax nativ 3.0 B30f  | {"Area":[262144.0],"Mean":[471.34604263305664]}  |
| Anonymous Sid 10113  | LE 3.0 B30f            | {"Area":[262144.0],"Mean":[762.6416549682617]}   |
+----------------------+------------------------+--------------------------------------------------+--+

Perform PD/LAA Quantification

Trial on "unsegmented" image data results in all values as Zero/null:

select patientname,seriesDescription,studydate,runmap(fetch_dicom_imagej(studyInstanceUID,seriesInstanceUID),"PD LAA Quantify SegmentedLung","") from stolz3pacs where success=true;
PACSToolCall:-L SERIES -c osirix@localhost:11112 -b yeti -m SeriesInstanceUID=1.3.12.2.1107.5.1.4.49001.30000008123113283818700000454 -m StudyInstanceUID=1.2.276.0.18.14.200.2.0.0.20081231.103551.64 --store-tcs /var/folders/cj/p00m_p1137l5wpxrjr5wr62c0000gp/T/store_tcs6227654999163457109.properties --directory emptyDir
PACSToolCall:-L SERIES -c osirix@localhost:11112 -b yeti -m SeriesInstanceUID=1.3.12.2.1107.5.1.4.54504.30000008112007153707800007751 -m StudyInstanceUID=1.2.276.0.18.14.200.2.0.0.20081120.125332.60 --store-tcs /var/folders/cj/p00m_p1137l5wpxrjr5wr62c0000gp/T/store_tcs4131042739232563902.properties --directory emptyDir
PACSToolCall:-L SERIES -c osirix@localhost:11112 -b yeti -m SeriesInstanceUID=1.3.12.2.1107.5.1.4.50021.30000008051505383531200008814 -m StudyInstanceUID=1.2.276.0.18.14.200.2.0.0.20080515.113315.4 --store-tcs /var/folders/cj/p00m_p1137l5wpxrjr5wr62c0000gp/T/store_tcs8837834746381452551.properties --directory emptyDir
+----------------------+------------------------+------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--+
|     patientname      |   seriesDescription    | studydate  |                                                                                                                                    _c3                                                                                                                                    |
+----------------------+------------------------+------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--+
| Anonymous Sid 10007  | HWS+Thx art. 3.0 B30f  | 20081231   | {"FOV voxels ":[8.2313216E7],"LAA-1024%":[0.0],"LAA-1024pixel^3":[0.0],"LAA-900%":[0.0],"LAA-900pixel^3":[0.0],"LAA-950%":[0.0],"LAA-950pixel^3":[0.0],"LungVol pixel^3":[0.0],"LungVol voxs":[0.0],"PD15":[0.0],"StudyID":[NaN],"sex":[NaN],"vox volume pixel^3":[1.0]}  |
| Anonymous Sid 10201  | Thorax nativ 3.0 B30f  | 20081120   | {"FOV voxels ":[4.6923776E7],"LAA-1024%":[0.0],"LAA-1024pixel^3":[0.0],"LAA-900%":[0.0],"LAA-900pixel^3":[0.0],"LAA-950%":[0.0],"LAA-950pixel^3":[0.0],"LungVol pixel^3":[0.0],"LungVol voxs":[0.0],"PD15":[0.0],"StudyID":[NaN],"sex":[NaN],"vox volume pixel^3":[1.0]}  |
| Anonymous Sid 10113  | LE 3.0 B30f            | 20080515   | {"FOV voxels ":[5.7933824E7],"LAA-1024%":[0.0],"LAA-1024pixel^3":[0.0],"LAA-900%":[0.0],"LAA-900pixel^3":[0.0],"LAA-950%":[0.0],"LAA-950pixel^3":[0.0],"LungVol pixel^3":[0.0],"LungVol voxs":[0.0],"PD15":[0.0],"StudyID":[NaN],"sex":[NaN],"vox volume pixel^3":[1.0]}  |
+----------------------+------------------------+------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--+
3 rows selected (17.768 seconds)

A trial which incorporates Image Segmentation also results in zero/null results:

select patientname,seriesDescription,studydate,runmap(run2(fetch_dicom_imagej(studyInstanceUID,seriesInstanceUID),"Segment Lung Tissue",""),"PD LAA Quantify SegmentedLung","") from stolz3pacs where success=true;

PACSToolCall:-L SERIES -c osirix@localhost:11112 -b yeti -m SeriesInstanceUID=1.3.12.2.1107.5.1.4.54504.30000008112007153707800007751 -m StudyInstanceUID=1.2.276.0.18.14.200.2.0.0.20081120.125332.60 --store-tcs /var/folders/cj/p00m_p1137l5wpxrjr5wr62c0000gp/T/store_tcs6194165006507264746.properties --directory emptyDir
PACSToolCall:-L SERIES -c osirix@localhost:11112 -b yeti -m SeriesInstanceUID=1.3.12.2.1107.5.1.4.49001.30000008123113283818700000454 -m StudyInstanceUID=1.2.276.0.18.14.200.2.0.0.20081231.103551.64 --store-tcs /var/folders/cj/p00m_p1137l5wpxrjr5wr62c0000gp/T/store_tcs7131212569564495806.properties --directory emptyDir
PACSToolCall:-L SERIES -c osirix@localhost:11112 -b yeti -m SeriesInstanceUID=1.3.12.2.1107.5.1.4.50021.30000008051505383531200008814 -m StudyInstanceUID=1.2.276.0.18.14.200.2.0.0.20080515.113315.4 --store-tcs /var/folders/cj/p00m_p1137l5wpxrjr5wr62c0000gp/T/store_tcs7668734660244417243.properties --directory emptyDir
+----------------------+------------------------+------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--+
|     patientname      |   seriesDescription    | studydate  |                                                                                                                                    _c3                                                                                                                                    |
+----------------------+------------------------+------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--+
| Anonymous Sid 10007  | HWS+Thx art. 3.0 B30f  | 20081231   | {"FOV voxels ":[8.2313216E7],"LAA-1024%":[0.0],"LAA-1024pixel^3":[0.0],"LAA-900%":[0.0],"LAA-900pixel^3":[0.0],"LAA-950%":[0.0],"LAA-950pixel^3":[0.0],"LungVol pixel^3":[0.0],"LungVol voxs":[0.0],"PD15":[0.0],"StudyID":[NaN],"sex":[NaN],"vox volume pixel^3":[1.0]}  |
| Anonymous Sid 10201  | Thorax nativ 3.0 B30f  | 20081120   | {"FOV voxels ":[4.6923776E7],"LAA-1024%":[0.0],"LAA-1024pixel^3":[0.0],"LAA-900%":[0.0],"LAA-900pixel^3":[0.0],"LAA-950%":[0.0],"LAA-950pixel^3":[0.0],"LungVol pixel^3":[0.0],"LungVol voxs":[0.0],"PD15":[0.0],"StudyID":[NaN],"sex":[NaN],"vox volume pixel^3":[1.0]}  |
| Anonymous Sid 10113  | LE 3.0 B30f            | 20080515   | {"FOV voxels ":[5.7933824E7],"LAA-1024%":[0.0],"LAA-1024pixel^3":[0.0],"LAA-900%":[0.0],"LAA-900pixel^3":[0.0],"LAA-950%":[0.0],"LAA-950pixel^3":[0.0],"LungVol pixel^3":[0.0],"LungVol voxs":[0.0],"PD15":[0.0],"StudyID":[NaN],"sex":[NaN],"vox volume pixel^3":[1.0]}  |
+----------------------+------------------------+------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--+

Note: All values are Null.

kmader commented 8 years ago

My main suspicious is that the offset is somehow being lost, so try running stats (stats(fetch_dicom_imagej(studyInstanceUID,seriesInstanceUID))) to get the information. The FOV voxels still comes out correct which I is strange.

Can you try using runtable and runrow instead of runmap since the result might for some reason be in another row?

drtjre commented 8 years ago

Runrow - gives zero. RunTABLE gives exception.

Runrow Results:


select patientname,seriesDescription,studydate,runrow(fetch_dicom_imagej(studyInstanceUID,seriesInstanceUID),"PD LAA Quantify SegmentedLung","") from stolz3pacs where success=true;
PACSToolCall:-L SERIES -c osirix@localhost:11112 -b yeti -m SeriesInstanceUID=1.3.12.2.1107.5.1.4.54504.30000008112007153707800007751 -m StudyInstanceUID=1.2.276.0.18.14.200.2.0.0.20081120.125332.60 --store-tcs /var/folders/cj/p00m_p1137l5wpxrjr5wr62c0000gp/T/store_tcs1399583431030434596.properties --directory emptyDir
PACSToolCall:-L SERIES -c osirix@localhost:11112 -b yeti -m SeriesInstanceUID=1.3.12.2.1107.5.1.4.49001.30000008123113283818700000454 -m StudyInstanceUID=1.2.276.0.18.14.200.2.0.0.20081231.103551.64 --store-tcs /var/folders/cj/p00m_p1137l5wpxrjr5wr62c0000gp/T/store_tcs5599936374050646508.properties --directory emptyDir
PACSToolCall:-L SERIES -c osirix@localhost:11112 -b yeti -m SeriesInstanceUID=1.3.12.2.1107.5.1.4.50021.30000008051505383531200008814 -m StudyInstanceUID=1.2.276.0.18.14.200.2.0.0.20080515.113315.4 --store-tcs /var/folders/cj/p00m_p1137l5wpxrjr5wr62c0000gp/T/store_tcs5668133840403478511.properties --directory emptyDir
+----------------------+------------------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--+
|     patientname      |   seriesDescription    | studydate  |                                                                                                                       _c3                                                                                                                       |
+----------------------+------------------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--+
| Anonymous Sid 10007  | HWS+Thx art. 3.0 B30f  | 20081231   | {"FOV voxels ":8.2313216E7,"LAA-1024%":0.0,"LAA-1024pixel^3":0.0,"LAA-900%":0.0,"LAA-900pixel^3":0.0,"LAA-950%":0.0,"LAA-950pixel^3":0.0,"LungVol pixel^3":0.0,"LungVol voxs":0.0,"PD15":0.0,"StudyID":NaN,"sex":NaN,"vox volume pixel^3":1.0}  |
| Anonymous Sid 10201  | Thorax nativ 3.0 B30f  | 20081120   | {"FOV voxels ":4.6923776E7,"LAA-1024%":0.0,"LAA-1024pixel^3":0.0,"LAA-900%":0.0,"LAA-900pixel^3":0.0,"LAA-950%":0.0,"LAA-950pixel^3":0.0,"LungVol pixel^3":0.0,"LungVol voxs":0.0,"PD15":0.0,"StudyID":NaN,"sex":NaN,"vox volume pixel^3":1.0}  |
| Anonymous Sid 10113  | LE 3.0 B30f            | 20080515   | {"FOV voxels ":5.7933824E7,"LAA-1024%":0.0,"LAA-1024pixel^3":0.0,"LAA-900%":0.0,"LAA-900pixel^3":0.0,"LAA-950%":0.0,"LAA-950pixel^3":0.0,"LungVol pixel^3":0.0,"LungVol voxs":0.0,"PD15":0.0,"StudyID":NaN,"sex":NaN,"vox volume pixel^3":1.0}  |
+----------------------+------------------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--+

RunTable Results:

select patientname,seriesDescription,studydate,runtable(fetch_dicom_imagej(studyInstanceUID,seriesInstanceUID),"PD LAA Quantify SegmentedLung","") from stolz3pacs where success=true;
PACSToolCall:-L SERIES -c osirix@localhost:11112 -b yeti -m SeriesInstanceUID=1.3.12.2.1107.5.1.4.54504.30000008112007153707800007751 -m StudyInstanceUID=1.2.276.0.18.14.200.2.0.0.20081120.125332.60 --store-tcs /var/folders/cj/p00m_p1137l5wpxrjr5wr62c0000gp/T/store_tcs4397479100887086077.properties --directory emptyDir
PACSToolCall:-L SERIES -c osirix@localhost:11112 -b yeti -m SeriesInstanceUID=1.3.12.2.1107.5.1.4.49001.30000008123113283818700000454 -m StudyInstanceUID=1.2.276.0.18.14.200.2.0.0.20081231.103551.64 --store-tcs /var/folders/cj/p00m_p1137l5wpxrjr5wr62c0000gp/T/store_tcs6777457007696729161.properties --directory emptyDir
Error: org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 12.0 failed 1 times, most recent failure: Lost task 1.0 in stage 12.0 (TID 226, localhost): java.lang.ClassCastException: java.lang.String cannot be cast to org.apache.spark.unsafe.types.UTF8String
    at org.apache.spark.sql.catalyst.util.GenericArrayData.getUTF8String(GenericArrayData.scala:55)
    at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
    at org.apache.spark.sql.execution.Project$$anonfun$1$$anonfun$apply$1.apply(basicOperators.scala:51)
    at org.apache.spark.sql.execution.Project$$anonfun$1$$anonfun$apply$1.apply(basicOperators.scala:49)
    at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
    at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
    at scala.collection.Iterator$class.foreach(Iterator.scala:727)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
    at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
    at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
    at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47)
    at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273)
    at scala.collection.AbstractIterator.to(Iterator.scala:1157)
    at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265)
    at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157)
    at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252)
    at scala.collection.AbstractIterator.toArray(Iterator.scala:1157)
    at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$12.apply(RDD.scala:927)
    at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$12.apply(RDD.scala:927)
    at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858)
    at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
    at org.apache.spark.scheduler.Task.run(Task.scala:89)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

Driver stacktrace: (state=,code=0)
drtjre commented 8 years ago

Stats also causes - Memory - exception:

select patientname,seriesDescription,studydate,stats(fetch_dicom_imagej(studyInstanceUID,seriesInstanceUID)) from stolz3pacs where success=true;
PACSToolCall:-L SERIES -c osirix@localhost:11112 -b yeti -m SeriesInstanceUID=1.3.12.2.1107.5.1.4.49001.30000008123113283818700000454 -m StudyInstanceUID=1.2.276.0.18.14.200.2.0.0.20081231.103551.64 --store-tcs /var/folders/cj/p00m_p1137l5wpxrjr5wr62c0000gp/T/store_tcs9146868133922682801.properties --directory emptyDir
PACSToolCall:-L SERIES -c osirix@localhost:11112 -b yeti -m SeriesInstanceUID=1.3.12.2.1107.5.1.4.54504.30000008112007153707800007751 -m StudyInstanceUID=1.2.276.0.18.14.200.2.0.0.20081120.125332.60 --store-tcs /var/folders/cj/p00m_p1137l5wpxrjr5wr62c0000gp/T/store_tcs8448026936204067464.properties --directory emptyDir
Error: org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 13.0 failed 1 times, most recent failure: Lost task 1.0 in stage 13.0 (TID 228, localhost): java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Arrays.java:3236)
    at java.io.ByteArrayOutputStream.toByteArray(ByteArrayOutputStream.java:191)
    at ch.fourquant.images.types.SerDeserHelper$$anonfun$bytify$1.apply(SerDeserHelper.scala:20)
    at ch.fourquant.images.types.SerDeserHelper$$anonfun$bytify$1.apply(SerDeserHelper.scala:15)
    at scala.util.Try$.apply(Try.scala:161)
    at ch.fourquant.images.types.SerDeserHelper$.bytify(SerDeserHelper.scala:15)
    at ch.fourquant.images.types.PipUDT.serialize(images.scala:44)
    at ch.fourquant.images.types.PipUDT.serialize(images.scala:16)
    at org.apache.spark.sql.catalyst.CatalystTypeConverters$UDTConverter.toCatalystImpl(CatalystTypeConverters.scala:142)
    at org.apache.spark.sql.catalyst.CatalystTypeConverters$CatalystTypeConverter.toCatalyst(CatalystTypeConverters.scala:102)
    at org.apache.spark.sql.catalyst.CatalystTypeConverters$$anonfun$createToCatalystConverter$2.apply(CatalystTypeConverters.scala:401)
    at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
    at org.apache.spark.sql.execution.Project$$anonfun$1$$anonfun$apply$1.apply(basicOperators.scala:51)
    at org.apache.spark.sql.execution.Project$$anonfun$1$$anonfun$apply$1.apply(basicOperators.scala:49)
    at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
    at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
    at scala.collection.Iterator$class.foreach(Iterator.scala:727)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
    at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
    at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
    at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47)
    at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273)
    at scala.collection.AbstractIterator.to(Iterator.scala:1157)
    at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265)
    at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157)
    at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252)
    at scala.collection.AbstractIterator.toArray(Iterator.scala:1157)
    at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$12.apply(RDD.scala:927)
    at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$12.apply(RDD.scala:927)
    at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858)
    at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
kmader commented 8 years ago

The memory exception means you need to start java with more memory using the -Xmx argument. For example java -Xmx8g will allocated 8gb of memory for java. I am not sure what the default is, but you should make it as high as possible for these tests.

kmader commented 8 years ago

You can also reduce memory usage by adding LIMIT 1 to the sql expression to only get one result for initial testing

drtjre commented 8 years ago

Where you able to run some LungQuantification using the anonymized data I posted on the shared drive?

drtjre commented 8 years ago

After increasing java memory allocation, the STAT command produces the following error:

select patientname,seriesDescription,studydate,stats(fetch_dicom_imagej(studyInstanceUID,seriesInstanceUID)) from stolz3pacs where success=true;
PACSToolCall:-L SERIES -c osirix@localhost:11112 -b yeti -m SeriesInstanceUID=1.3.12.2.1107.5.1.4.54504.30000008112007153707800007751 -m StudyInstanceUID=1.2.276.0.18.14.200.2.0.0.20081120.125332.60 --store-tcs /var/folders/cj/p00m_p1137l5wpxrjr5wr62c0000gp/T/store_tcs4259478624266578240.properties --directory emptyDir
PACSToolCall:-L SERIES -c osirix@localhost:11112 -b yeti -m SeriesInstanceUID=1.3.12.2.1107.5.1.4.49001.30000008123113283818700000454 -m StudyInstanceUID=1.2.276.0.18.14.200.2.0.0.20081231.103551.64 --store-tcs /var/folders/cj/p00m_p1137l5wpxrjr5wr62c0000gp/T/store_tcs3747596182260056837.properties --directory emptyDir
PACSToolCall:-L SERIES -c osirix@localhost:11112 -b yeti -m SeriesInstanceUID=1.3.12.2.1107.5.1.4.50021.30000008051505383531200008814 -m StudyInstanceUID=1.2.276.0.18.14.200.2.0.0.20080515.113315.4 --store-tcs /var/folders/cj/p00m_p1137l5wpxrjr5wr62c0000gp/T/store_tcs9156190453687388072.properties --directory emptyDir
Error: scala.MatchError: ch.fourquant.images.types.ImageStatisticsUDT@920bd5 (of class ch.fourquant.images.types.ImageStatisticsUDT) (state=,code=0)
drtjre commented 8 years ago

Even with increased memory. LungQuantification results are null/zero.

kmader commented 8 years ago

So I have now run the tests on my environment with your data and updated the stats function to work inside the command line tool. I have also added toarray to convert the images to an array so you can export them (if needed) and a function to get just a single image fetch_dicom_imagej_single. Finally there is a new repository which lets you run the tests automatically (https://github.com/4Quant/RIQAE-Java) using JUnit and the Java API for our tools. I have put the new jar and the java tools in the riqae folder and you can run the tests by just typing mvn test in the root directory

drtjre commented 8 years ago

Great! I will test this but likely not until Friday. TJ

kmader commented 8 years ago

As a check for your outputs here are the results from the `PACSCOPDTest suite integrated into the project

The updated stats returns the basic information about the image (so you can see if it is performing the offset correctly)

Map(mean -> -1988.5996551513672, min -> -4048.0, pts -> 262144.0, max -> 583.0, stdDev -> 1155.2388050224663)

And the results from the COPD plugin aren't null anymore since the calibration is paired to the entire stack and not just the individual slices

[1.3.6.1.4.1.14519.5.2.1.4320.5030.966354075876482042295761929295,
Map(LAA-950mm^3 -> 0.0, LAA-900mm^3 -> 0.0, LungVol mm^3 -> 0.0, PD15 -> -947.0, LAA-900% -> 46.7, StudyID -> NaN, LungVol voxs -> 2171109.0, sex -> NaN, LAA-1024% -> 1.1, LAA-1024mm^3 -> 0.0, LAA-950% -> 13.8, vox volume mm^3 -> 0.0, LungVol (Liters) -> 0.0, FOV voxels  -> 1.7825792E7)]

The rest of the results are here

drtjre commented 8 years ago

Principle aspect of bug - Zero/Null values - resolved!

CREATE TEMPORARY TABLE stolz3pacs USING fourquant.pacs OPTIONS (bind "YETI", username "OSIRIX", server "localhost", port "11112", path "StolzAnonymous3.csv");

select patientname,patientID AS StudyID,runmap(run2(fetch_dicom_imagej(studyInstanceUID,seriesInstanceUID),"Segment Lung Tissue",""),"PD LAA Quantify SegmentedLung","") AS Results from Stolz3pacs where success=true;
PACSToolCall:-L SERIES -c osirix@localhost:11112 -b yeti -m SeriesInstanceUID=1.3.12.2.1107.5.1.4.54504.30000008112007153707800007751 -m StudyInstanceUID=1.2.276.0.18.14.200.2.0.0.20081120.125332.60 --store-tcs /var/folders/cj/p00m_p1137l5wpxrjr5wr62c0000gp/T/store_tcs5403419203677837044.properties --directory emptyDir
PACSToolCall:-L SERIES -c osirix@localhost:11112 -b yeti -m SeriesInstanceUID=1.3.12.2.1107.5.1.4.49001.30000008123113283818700000454 -m StudyInstanceUID=1.2.276.0.18.14.200.2.0.0.20081231.103551.64 --store-tcs /var/folders/cj/p00m_p1137l5wpxrjr5wr62c0000gp/T/store_tcs8807823199842924643.properties --directory emptyDir
PACSToolCall:-L SERIES -c osirix@localhost:11112 -b yeti -m SeriesInstanceUID=1.3.12.2.1107.5.1.4.50021.30000008051505383531200008814 -m StudyInstanceUID=1.2.276.0.18.14.200.2.0.0.20080515.113315.4 --store-tcs /var/folders/cj/p00m_p1137l5wpxrjr5wr62c0000gp/T/store_tcs4370383041495005520.properties --directory emptyDir
+----------------------+-----------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--+
|     patientname      |  StudyID  |                                                                                                                                                                            Results                                                                                                                                                                             |
+----------------------+-----------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--+
| Anonymous Sid 10007  | SID10007  | {"FOV voxels ":[8.2313216E7],"LAA-1024%":[0.1],"LAA-1024mm^3":[4411.125],"LAA-900%":[4.1],"LAA-900mm^3":[233690.0625],"LAA-950%":[1.7],"LAA-950mm^3":[98480.8125],"LungVol (Liters)":[5.682],"LungVol mm^3":[5682457.0],"LungVol voxs":[3367382.0],"PD15":[-858.0],"StudyID":[NaN],"sex":[NaN],"vox volume mm^3":[1.6875]}                                     |
| Anonymous Sid 10201  | SID10201  | {"FOV voxels ":[4.6923776E7],"LAA-1024%":[0.1],"LAA-1024mm^3":[4544.734954833984],"LAA-900%":[45.3],"LAA-900mm^3":[3337090.7306671143],"LAA-950%":[8.7],"LAA-950mm^3":[643638.4320259094],"LungVol (Liters)":[7.37],"LungVol mm^3":[7369556.0],"LungVol voxs":[4579283.0],"PD15":[-937.0],"StudyID":[NaN],"sex":[NaN],"vox volume mm^3":[1.6093254089355469]}  |
| Anonymous Sid 10113  | SID10113  | {"FOV voxels ":[5.7933824E7],"LAA-1024%":[0.0],"LAA-1024mm^3":[779.0036201477051],"LAA-900%":[9.4],"LAA-900mm^3":[694324.988079071],"LAA-950%":[1.1],"LAA-950mm^3":[83137.51888275146],"LungVol (Liters)":[7.359],"LungVol mm^3":[7359262.0],"LungVol voxs":[3920513.0],"PD15":[-886.0],"StudyID":[NaN],"sex":[NaN],"vox volume mm^3":[1.8771171569824219]}    |
+----------------------+-----------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--+
Ground Truth:
PatientName (Anonymized) StudyID LungVol    PD15    LAA-950mm^3
Anonymous Sid 10007     SID10007    2.841   -858    49240.431
Anonymous Sid 10113     SID10113    3.68    -886    41568.794
Anonymous Sid 10201     SID10201    4.471   -937    390522.385