search

SystemsGenetics / KINC

Knowledge Independent Network Construction
MIT License
11 stars 4 forks source link

KINC run rmt failing #62

Closed mrbende closed 5 years ago

mrbende commented 5 years ago

KINC successfully generates a ccm, cmx, and emx file. When trying to run rmt, it fails. The following error is present in every output file.

[mrbende@node0021 EDS1014]$ kinc run rmt --input EDS1014.cmx --log EDS1014.log
"STD exception std::exception caught!\n"

The resulting log file that is generated is always empty. As a result, kinc run extract can not use an accurate threshold value to generate a network. I have updated my versions of ACE and KINC within the last week, so I do have all of the newest edits.

I have uploaded my correlation composition file to the following box link. Can a developer please attempt to run this command using my EDS1014.cmx file and review the resulting errors. Thanks

https://clemson.box.com/s/02bvxwfpjw0tz96oaq2w6gt337ocn1jf

bentsherman commented 5 years ago

Debugging this issue now, I think the problem has something to do with the fact that your CMX is pretty large and RMT hasn't been tested on such a large matrix, so there's probably some issues with integer overflow and memory allocation.

mrbende commented 5 years ago

We have tried to run a condensed version of my GEM, limited to the first 100 genes only. That job failed at rmt as well.

bentsherman commented 5 years ago

Was it the same error though? I think RMT would fail to find a threshold for a matrix that small. I might try to process your matrix using the first 5000 genes, that's around the size that I've tested before.

mrbende commented 5 years ago

I have uploaded the original GEM for you to work with and condense to whatever size you need to test. It can be downloaded at the box link here.

bentsherman commented 5 years ago

While you're waiting on this @mrbende I believe there are other methods that people in the lab are using to extract a network, something with KINC.R and sample enrichment, which I haven't worked with yet. It might be good to talk to Dr. Feltus about trying those other methods as well.

bentsherman commented 5 years ago

I just updated RMT to do pairwise reduction, so now it will select only one correlation per gene pair which uses much less memory. By default it just uses the first cluster like we discussed.

I also updated a different portion of RMT to use a faster library, so RMT should also be much faster overall. I was able to process the full EDS correlation matrix in under a minute. I updated the install script on Palmetto so just delete your KINC directory and run the install script again, let me know if you have any issues.

bentsherman commented 5 years ago

I was able to process your EDS1014 matrix from end to end, including RMT, without any errors. Closing this issue so I can move forward with the next versioned release, if you have any issues down the road we can discuss then.