The Human compendia failed at QN Verification step

[arielsvn]

Context

We were running the Human Compendia.

Problem or idea

It failed with this error:

2019-11-03 16:50:43,534 i-0119de425230c426b [volume: -1] data_refinery_workers.processors.create_compendia INFO: IterativeSVD algorithm: ARPACK
[IterativeSVD] Iter 1: observed MAE=0.613012
[IterativeSVD] Iter 2: observed MAE=0.463297
[IterativeSVD] Iter 3: observed MAE=0.378142
[IterativeSVD] Iter 4: observed MAE=0.331633
[IterativeSVD] Iter 5: observed MAE=0.319173
[IterativeSVD] Iter 6: observed MAE=0.319024
[IterativeSVD] Iter 7: observed MAE=0.318952
[IterativeSVD] Iter 8: observed MAE=0.318899
[IterativeSVD] Iter 9: observed MAE=0.318865
[IterativeSVD] Iter 10: observed MAE=0.318840
[IterativeSVD] Iter 11: observed MAE=0.318821
[IterativeSVD] Iter 12: observed MAE=0.318808
[IterativeSVD] Iter 13: observed MAE=0.318798
[IterativeSVD] Iter 14: observed MAE=0.318789
[IterativeSVD] Iter 15: observed MAE=0.318782
... DEBUG [total_cpu: 8.0] [job_id: 29323092] [process_ram: 73.87223434448242]: end SVD
... DEBUG [job_id: 29323092]: Duration: 5778.422582626343
... DEBUG [total_cpu: 0.7] [job_id: 29323092] [process_ram: 50.64521026611328]: start untranspose
... DEBUG [total_cpu: 0.8] [job_id: 29323092] [process_ram: 166.86474609375]: end untranspose
... DEBUG [job_id: 29323092]: Duration: 3010.8331418037415
... DEBUG [total_cpu: 2.1] [job_id: 29323092] [process_ram: 166.86474609375]: start quantile normalize
... ERROR [processor_job: 29323092] [no_retry: False]: Unhandled exception caught while running processor function _perform_imputation in pipeline:
Traceback (most recent call last):
File "/home/user/data_refinery_workers/processors/utils.py", line 383, in run_pipeline
last_result = processor(last_result)
File "/home/user/data_refinery_workers/processors/create_compendia.py", line 350, in _perform_imputation
job_context = smashing_utils.quantile_normalize(job_context, ks_check=False)
File "/home/user/data_refinery_workers/processors/smashing_utils.py", line 502, in quantile_normalize
combos = combn(ncol(reso), 2)
File "/usr/local/lib/python3.5/dist-packages/rpy2/robjects/functions.py", line 178, in __call__
return super(SignatureTranslatedFunction, self).__call__(*args, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/rpy2/robjects/functions.py", line 106, in __call__
res = super(Function, self).__call__(*new_args, **new_kwargs)
rpy2.rinterface.RRuntimeError: Error in matrix(r, nrow = len.r, ncol = count) :
invalid 'ncol' value (too large or NA)

Solution or next step

@davidsmejia

[davidsmejia]

@kurtwheeler

[kurtwheeler]

The human compendia passed while this was reenabled!