search

ResearchIT / IGV-ScanFold

IGV + RNA folding tools
4 stars 2 forks source link

Potential multiprocessing error on Windows #14

Closed rjandr closed 3 years ago

rjandr commented 3 years ago

IGV-ScanFold on Windows 10 Home (Version 10.0.19042) does not complete scans.

windowsIGVerror

igv.log info:

ERROR [2021-03-07T15:29:32,523] [RuntimeUtils.java:88]  Traceback (most recent call last):
ERROR [2021-03-07T15:29:32,524] [RuntimeUtils.java:88]    File "D:\rjand\DDownloads\scanfoldigv-windows\scanfold\ScanFold-Scan_IGV.py", line 338, in <module>
ERROR [2021-03-07T15:29:32,525] [RuntimeUtils.java:88]      energy_list = energies(seqlist, temperature)
ERROR [2021-03-07T15:29:32,525] [RuntimeUtils.java:88]    File "D:\rjand\DDownloads\scanfoldigv-windows\scanfold\ScanFoldSharedIGV.py", line 234, in energies
ERROR [2021-03-07T15:29:32,535] [RuntimeUtils.java:88]      energy_list = multiprocessing(rna_folder, [(sequence, temperature) for sequence in seq_list], 12)
ERROR [2021-03-07T15:29:32,536] [RuntimeUtils.java:88]    File "D:\rjand\DDownloads\scanfoldigv-windows\scanfold\ScanFoldSharedIGV.py", line 255, in multiprocessing
ERROR [2021-03-07T15:29:32,536] [RuntimeUtils.java:88]      return list(res)
ERROR [2021-03-07T15:29:32,536] [RuntimeUtils.java:88]    File "concurrent\futures\process.py", line 484, in _chain_from_iterable_of_lists
ERROR [2021-03-07T15:29:32,536] [RuntimeUtils.java:88]    File "concurrent\futures\_base.py", line 611, in result_iterator
ERROR [2021-03-07T15:29:32,537] [RuntimeUtils.java:88]    File "concurrent\futures\_base.py", line 432, in result
ERROR [2021-03-07T15:29:32,537] [RuntimeUtils.java:88]    File "concurrent\futures\_base.py", line 388, in __get_result
ERROR [2021-03-07T15:29:32,538] [RuntimeUtils.java:88]  concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
ERROR [2021-03-07T15:29:32,872] [RuntimeUtils.java:88]  Traceback (most recent call last):
ERROR [2021-03-07T15:29:32,872] [RuntimeUtils.java:88]    File "D:\rjand\DDownloads\scanfoldigv-windows\scanfold\ScanFold-Fold_IGV.py", line 681, in <module>
ERROR [2021-03-07T15:29:32,873] [RuntimeUtils.java:88]      step_size, window_size = get_step_win(lines)
ERROR [2021-03-07T15:29:32,873] [RuntimeUtils.java:88]    File "D:\rjand\DDownloads\scanfoldigv-windows\scanfold\ScanFold-Fold_IGV.py", line 539, in get_step_win
ERROR [2021-03-07T15:29:32,873] [RuntimeUtils.java:88]      row2 = lines[1].split()
ERROR [2021-03-07T15:29:32,873] [RuntimeUtils.java:88]  IndexError: list index out of range
INFO [2021-03-07T15:29:32,873] [MessageUtils.java:76]  IndexError: list index out of range<br>See igv.log for more details
INFO [2021-03-07T15:29:32,907] [IGV.java:1316]  Loading 1 resources.
INFO [2021-03-07T15:29:32,908] [TrackLoader.java:123]  Loading resource, path C:\Users\rjand\AppData\Local\Temp\scanfold-results3741362279704692303\output8vtkmuoz.IGV.bp
ERROR [2021-03-07T15:29:32,909] [BasePairFileParser.java:97]  Error parsing base pair file
java.lang.NullPointerException: null
    at org.broad.igv.feature.basepair.BasePairFileParser.loadData(BasePairFileParser.java:37) [igv.jar:?]
    at org.broad.igv.feature.basepair.BasePairTrack.<init>(BasePairTrack.java:44) [igv.jar:?]
    at org.broad.igv.track.TrackLoader.loadBasePairFile(TrackLoader.java:1113) [igv.jar:?]
    at org.broad.igv.track.TrackLoader.load(TrackLoader.java:192) [igv.jar:?]
    at org.broad.igv.ui.IGV.load(IGV.java:1399) [igv.jar:?]
    at org.broad.igv.ui.IGV.loadResources(IGV.java:1331) [igv.jar:?]
    at org.broad.igv.ui.IGV$4.run(IGV.java:474) [igv.jar:?]
    at org.broad.igv.util.LongRunningTask.call(LongRunningTask.java:72) [igv.jar:?]
    at java.util.concurrent.FutureTask.run(Unknown Source) [?:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
    at java.lang.Thread.run(Unknown Source) [?:?]
ERROR [2021-03-07T15:29:32,910] [TrackLoader.java:252]  java.lang.NullPointerException
java.lang.RuntimeException: java.lang.NullPointerException
    at org.broad.igv.feature.basepair.BasePairFileParser.loadData(BasePairFileParser.java:101) ~[igv.jar:?]
    at org.broad.igv.feature.basepair.BasePairTrack.<init>(BasePairTrack.java:44) ~[igv.jar:?]
    at org.broad.igv.track.TrackLoader.loadBasePairFile(TrackLoader.java:1113) ~[igv.jar:?]
    at org.broad.igv.track.TrackLoader.load(TrackLoader.java:192) [igv.jar:?]
    at org.broad.igv.ui.IGV.load(IGV.java:1399) [igv.jar:?]
    at org.broad.igv.ui.IGV.loadResources(IGV.java:1331) [igv.jar:?]
    at org.broad.igv.ui.IGV$4.run(IGV.java:474) [igv.jar:?]
    at org.broad.igv.util.LongRunningTask.call(LongRunningTask.java:72) [igv.jar:?]
    at java.util.concurrent.FutureTask.run(Unknown Source) [?:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
    at java.lang.Thread.run(Unknown Source) [?:?]
Caused by: java.lang.NullPointerException
    at org.broad.igv.feature.basepair.BasePairFileParser.loadData(BasePairFileParser.java:37) ~[igv.jar:?]
    ... 11 more
ERROR [2021-03-07T15:29:32,910] [IGV.java:1334]  Error loading track
org.broad.igv.exceptions.DataLoadException: java.lang.NullPointerException
    at org.broad.igv.track.TrackLoader.load(TrackLoader.java:254) ~[igv.jar:?]
    at org.broad.igv.ui.IGV.load(IGV.java:1399) ~[igv.jar:?]
    at org.broad.igv.ui.IGV.loadResources(IGV.java:1331) [igv.jar:?]
    at org.broad.igv.ui.IGV$4.run(IGV.java:474) [igv.jar:?]
    at org.broad.igv.util.LongRunningTask.call(LongRunningTask.java:72) [igv.jar:?]
    at java.util.concurrent.FutureTask.run(Unknown Source) [?:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
    at java.lang.Thread.run(Unknown Source) [?:?]
INFO [2021-03-07T15:29:32,911] [MessageUtils.java:76]  Error loading C:\Users\rjand\AppData\Local\Temp\scanfold-results3741362279704692303\output8vtkmuoz.IGV.bp: java.lang.NullPointerException
INFO [2021-03-07T15:29:32,923] [IGV.java:1316]  Loading 1 resources.
INFO [2021-03-07T15:29:32,924] [TrackLoader.java:123]  Loading resource, path C:\Users\rjand\AppData\Local\Temp\scanfold-results3741362279704692303\outputaatu22bs.final_partners_zscore.wig
INFO [2021-03-07T15:29:32,946] [IGV.java:1316]  Loading 1 resources.
INFO [2021-03-07T15:29:32,947] [TrackLoader.java:123]  Loading resource, path C:\Users\rjand\AppData\Local\Temp\scanfold-results3741362279704692303\outputkznuual7.mfe.wig
INFO [2021-03-07T15:29:32,963] [IGV.java:1316]  Loading 1 resources.
INFO [2021-03-07T15:29:32,964] [TrackLoader.java:123]  Loading resource, path C:\Users\rjand\AppData\Local\Temp\scanfold-results3741362279704692303\output934ttvol.zscore.wig
INFO [2021-03-07T15:29:33,017] [IGV.java:1316]  Loading 1 resources.
INFO [2021-03-07T15:29:33,018] [TrackLoader.java:123]  Loading resource, path C:\Users\rjand\AppData\Local\Temp\scanfold-results3741362279704692303\output_bsqdh6r.pvalue.wig
INFO [2021-03-07T15:29:33,038] [IGV.java:1316]  Loading 1 resources.
INFO [2021-03-07T15:29:33,039] [TrackLoader.java:123]  Loading resource, path C:\Users\rjand\AppData\Local\Temp\scanfold-results3741362279704692303\outputh___iw6i.ed.wig
INFO [2021-03-07T15:29:33,064] [IGV.java:1316]  Loading 1 resources.
INFO [2021-03-07T15:29:33,065] [TrackLoader.java:123]  Loading resource, path C:\Users\rjand\AppData\Local\Temp\scanfold-results3741362279704692303\outputwgdlc5d_.-2filter.scanfoldvarna
ERROR [2021-03-07T15:29:33,066] [TrackLoader.java:252]  null
java.lang.NullPointerException: null
ERROR [2021-03-07T15:29:33,066] [IGV.java:1334]  Error loading track
java.lang.NullPointerException: null
    at org.broad.igv.exceptions.DataLoadException.<init>(DataLoadException.java:38) ~[igv.jar:?]
    at org.broad.igv.track.TrackLoader.load(TrackLoader.java:254) ~[igv.jar:?]
    at org.broad.igv.ui.IGV.load(IGV.java:1399) ~[igv.jar:?]
    at org.broad.igv.ui.IGV.loadResources(IGV.java:1331) [igv.jar:?]
    at org.broad.igv.ui.IGV$4.run(IGV.java:474) [igv.jar:?]
    at org.broad.igv.util.LongRunningTask.call(LongRunningTask.java:72) [igv.jar:?]
    at java.util.concurrent.FutureTask.run(Unknown Source) [?:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
    at java.lang.Thread.run(Unknown Source) [?:?]
INFO [2021-03-07T15:29:33,066] [MessageUtils.java:76]  Error loading C:\Users\rjand\AppData\Local\Temp\scanfold-results3741362279704692303\outputwgdlc5d_.-2filter.scanfoldvarna: null
rjandr commented 3 years ago

Even on scans that otherwise appear to work. There is an issue where only a portion of the scan completes. For example, only 150 of the >190nt are scanned in example below: see images and log file:

image

image

INFO [2021-03-07T16:28:14,105] [Main.java:205]  Startup  IGV Version user not_set
INFO [2021-03-07T16:28:14,115] [Main.java:206]  Java 11.0.9.1 (build 11.0.9.1+1) 2020-11-04
INFO [2021-03-07T16:28:14,116] [Main.java:209]  Java Vendor: AdoptOpenJDK https://adoptopenjdk.net/
INFO [2021-03-07T16:28:14,117] [Main.java:211]  JVM: OpenJDK 64-Bit Server VM AdoptOpenJDK   
INFO [2021-03-07T16:28:14,117] [Main.java:214]  Default User Directory: C:\Users\rjand\Documents
INFO [2021-03-07T16:28:14,117] [Main.java:215]  OS: Windows 10 10.0 amd64
INFO [2021-03-07T16:28:14,123] [Main.java:266]  Unknown version: user
INFO [2021-03-07T16:28:15,014] [GenomeManager.java:183]  Loading genome: https://s3.amazonaws.com/igv.org.genomes/hg38/hg38.genome
INFO [2021-03-07T16:28:21,970] [CommandListener.java:121]  Listening on port 60151
ERROR [2021-03-07T16:29:12,234] [RuntimeUtils.java:88]  Traceback (most recent call last):
ERROR [2021-03-07T16:29:12,235] [RuntimeUtils.java:88]    File "D:\rjand\DDownloads\scanfoldigv-windows\scanfold\ScanFold-Scan_IGV.py", line 338, in <module>
ERROR [2021-03-07T16:29:12,235] [RuntimeUtils.java:88]      energy_list = energies(seqlist, temperature)
ERROR [2021-03-07T16:29:12,235] [RuntimeUtils.java:88]    File "D:\rjand\DDownloads\scanfoldigv-windows\scanfold\ScanFoldSharedIGV.py", line 234, in energies
ERROR [2021-03-07T16:29:12,236] [RuntimeUtils.java:88]      energy_list = multiprocessing(rna_folder, [(sequence, temperature) for sequence in seq_list], 12)
ERROR [2021-03-07T16:29:12,236] [RuntimeUtils.java:88]    File "D:\rjand\DDownloads\scanfoldigv-windows\scanfold\ScanFoldSharedIGV.py", line 255, in multiprocessing
ERROR [2021-03-07T16:29:12,236] [RuntimeUtils.java:88]      return list(res)
ERROR [2021-03-07T16:29:12,237] [RuntimeUtils.java:88]    File "concurrent\futures\process.py", line 484, in _chain_from_iterable_of_lists
ERROR [2021-03-07T16:29:12,237] [RuntimeUtils.java:88]    File "concurrent\futures\_base.py", line 611, in result_iterator
ERROR [2021-03-07T16:29:12,237] [RuntimeUtils.java:88]    File "concurrent\futures\_base.py", line 432, in result
ERROR [2021-03-07T16:29:12,238] [RuntimeUtils.java:88]    File "concurrent\futures\_base.py", line 388, in __get_result
ERROR [2021-03-07T16:29:12,238] [RuntimeUtils.java:88]  concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
INFO [2021-03-07T16:29:21,908] [IGV.java:1316]  Loading 1 resources.
INFO [2021-03-07T16:29:21,919] [TrackLoader.java:123]  Loading resource, path C:\Users\rjand\AppData\Local\Temp\scanfold-results2916817773445814175\outputxywcisln.IGV.bp
INFO [2021-03-07T16:29:21,920] [IGV.java:1316]  Loading 1 resources.
INFO [2021-03-07T16:29:21,921] [TrackLoader.java:123]  Loading resource, path C:\Users\rjand\AppData\Local\Temp\scanfold-results2916817773445814175\output6b1xp2zd.final_partners_zscore.wig
INFO [2021-03-07T16:29:21,928] [IGV.java:1316]  Loading 1 resources.
INFO [2021-03-07T16:29:21,929] [TrackLoader.java:123]  Loading resource, path C:\Users\rjand\AppData\Local\Temp\scanfold-results2916817773445814175\output65xsbenv.mfe.wig
INFO [2021-03-07T16:29:21,936] [IGV.java:1316]  Loading 1 resources.
INFO [2021-03-07T16:29:21,937] [TrackLoader.java:123]  Loading resource, path C:\Users\rjand\AppData\Local\Temp\scanfold-results2916817773445814175\outputvd3c4w6f.zscore.wig
INFO [2021-03-07T16:29:21,945] [IGV.java:1316]  Loading 1 resources.
INFO [2021-03-07T16:29:21,946] [TrackLoader.java:123]  Loading resource, path C:\Users\rjand\AppData\Local\Temp\scanfold-results2916817773445814175\output1t7s4ieg.pvalue.wig
INFO [2021-03-07T16:29:21,993] [IGV.java:1316]  Loading 1 resources.
INFO [2021-03-07T16:29:21,993] [IGV.java:1316]  Loading 1 resources.
INFO [2021-03-07T16:29:21,994] [TrackLoader.java:123]  Loading resource, path C:\Users\rjand\AppData\Local\Temp\scanfold-results2916817773445814175\output91kp6hwl.-2filter.scanfoldvarna
INFO [2021-03-07T16:29:21,995] [TrackLoader.java:123]  Loading resource, path C:\Users\rjand\AppData\Local\Temp\scanfold-results2916817773445814175\outputi7w2f59c.ed.wig
rjandr commented 3 years ago

Fixed by running on less processors (went from 12 to 4).