Mbio: Unable to fetch required data in some apps with some datasets

[asizemore]

From Dan: For example, both DIABIMMUNE and Bangladesh are throwing errors for multiple apps, but BONUS and HMP WGS working with the same apps. So, not a general app issue, but a dataset specific issue

Data service error

Caught Exception:
 java.lang.RuntimeException: Unable to fetch all required data
    at org.veupathdb.service.eda.common.client.StreamingDataClient.buildDataStreams(StreamingDataClient.java:80) ~[service.jar:3.0.0]
    at org.veupathdb.service.eda.common.client.StreamingDataClient.buildAndProcessStreams(StreamingDataClient.java:40) ~[service.jar:3.0.0]
    at org.veupathdb.service.eda.ds.plugin.AbstractPlugin.accept(AbstractPlugin.java:190) ~[service.jar:3.0.0]

...

Caused by: java.lang.RuntimeException: 500 Internal Server Error: {"status":"server-error","message":"Expected 14 columns but parsed 13 in line:\nentity_16SRRNAV4Assay_stable_id\tSample_stable_id\tParticipant_stable_id\tEUPATH_0009256_Bacteria_Actinobacteriota_Actinobacteria_Bifidobacteriales_Bifidobacteriaceae_Bifidobacterium\tEUPATH_0009256_Bacteria_Firmicutes_Clostridia_Lachnospirales_Lachnospiraceae_Blautia\tEUPATH_0009256_Bacteria_Firmicutes_Bacilli_Lactobacillales_Streptococcaceae_Streptococcus\tEUPATH_0009256_Bacteria_Firmicutes_Clostridia_Oscillospirales_Ruminococcaceae_Faecalibacterium\tEUPATH_0009256_Bacteria_Firmicutes_Bacilli_Lactobacillales_Lactobacillaceae_Lactobacillus\tEUPATH_0009256_Bacteria_Firmicutes_Clostridia_Lachnospirales_Lachnospiraceae_Dorea\tEUPATH_0009256_Bacteria_Firmicutes_Clostridia_Oscillospirales_Ruminococcaceae_Subdoligranulum\tEUPATH_0009256_Bacteria_Proteobacteria_Gammaproteobacteria_Enterobacterales_Enterobacteriaceae_Escherichia_Shigella\tEUPATH_0009256_Bacteria_Bacteroidota_Bacteroidia_Bacteroidales_Prevotellaceae_Prevotella\tEUPATH_0009256_Bacteria_Actinobacteriota_Coriobacteriia_Coriobacteriales_Coriobacteriaceae_Collinsella","requestId":"5juh71beyXD3mIRy7UDv36"}
    at org.gusdb.fgputil.client.ResponseFuture.lambda$getInputStream$0(ResponseFuture.java:49) ~[service.jar:3.0.0]
    at java.util.Optional.orElseThrow(Optional.java:403) ~[?:?]
    at org.gusdb.fgputil.functional.Either.leftOrElseThrow(Either.java:149) ~[service.jar:3.0.0]
    at org.gusdb.fgputil.client.ResponseFuture.getInputStream(ResponseFuture.java:48) ~[service.jar:3.0.0]

[d-callan]

I'll bet this is bc of the data being reloaded w a new repeated measures node. I haven't confirmed, but would make an initial guess, there is a cached result without that node.

[asizemore]

Just tried the compute below and got a failed compute. This is on my dev site and I did run this computation today.

In the compute service now we see

2023-03-02 15:26:35.962 [rid:     ][jid:0e0d7db60afdc39d8307f4dafda9c418][p:alphadiv] ERROR PluginExecutor:166 - Failed to fetch tabular data.
 java.lang.RuntimeException: 500 Internal Server Error: {"status":"server-error","message":"Unable to fetch all required data","requestId":"1Szkq4Svi3oDZKMgwVHuAA"}
    at org.gusdb.fgputil.client.ResponseFuture.lambda$getInputStream$0(ResponseFuture.java:49) ~[service.jar:1.0.0]
    at java.util.Optional.orElseThrow(Optional.java:403) ~[?:?]
    at org.gusdb.fgputil.functional.Either.leftOrElseThrow(Either.java:149) ~[service.jar:1.0.0]
    at org.gusdb.fgputil.client.ResponseFuture.getInputStream(ResponseFuture.java:48) ~[service.jar:1.0.0]
    at org.veupathdb.service.eda.compute.EDA.getMergeData(EDA.kt:119) ~[service.jar:1.0.0]
    at org.veupathdb.service.eda.compute.exec.PluginExecutor.execute(PluginExecutor.kt:157) ~[service.jar:1.0.0]
    at org.veupathdb.lib.compute.platform.intern.jobs.JobExecutionHandler.execute(JobExecutionHandler.kt:86) ~[service.jar:1.0.0]
    at org.veupathdb.lib.compute.platform.intern.queues.QueueWrapper.onJob(QueueWrapper.kt:98) ~[service.jar:1.0.0]
    at org.veupathdb.lib.compute.platform.intern.queues.QueueWrapper.access$onJob(QueueWrapper.kt:29) ~[service.jar:1.0.0]
    at org.veupathdb.lib.compute.platform.intern.queues.QueueWrapper$3.handle(QueueWrapper.kt:60) ~[service.jar:1.0.0]
    at org.veupathdb.lib.rabbit.jobs.pools.JobHandlers.execute(JobHandlers.kt:18) ~[service.jar:1.0.0]
    at org.veupathdb.lib.rabbit.jobs.QueueWorker.initCallbacks$lambda-5$lambda-3$lambda-2(QueueWorker.kt:82) ~[service.jar:1.0.0]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
    at java.lang.Thread.run(Thread.java:833) ~[?:?]

[d-callan]

that entity diagram says it has no repeated measures. We should check how the data are loaded

[asizemore]

Now on QA:

All are working!!! Closing ticket 🎉