search

broadinstitute / cromwell

Scientific workflow engine designed for simplicity & scalability. Trivially transition between one off use cases to massive scale production environments
http://cromwell.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
988 stars 357 forks source link

Workflow input processing failed: The label in the input is too long #6999

Open ibrahimkurt opened 1 year ago

ibrahimkurt commented 1 year ago

Hi,

Running a workflow on WSL/Ubuntu 20.04 using conda-installed Cromwell:

cromwell run ngs-ubuntu-20-04/iletisim/warp/pipelines/broad/dna_seq/germline/single_sample/exome/local_newGCP_ExomeGermlineSingleSample_deneme6_bcftools.wdl -i ngs-ubuntu-20-04/iletisim/json/S736Nr1.json -o ngs-ubuntu-20-04/iletisim/json/options2.json

Getting the error:

[2023-02-04 08:55:00,61] [info] Running with database db.url = jdbc:hsqldb:mem:bc9ad7e3-efc7-4f37-aecb-b283b104cbcd;shutdown=false;hsqldb.tx=mvcc
[2023-02-04 08:55:06,54] [info] Running migration RenameWorkflowOptionsInMetadata with a read batch size of 100000 and a write batch size of 100000
[2023-02-04 08:55:06,55] [info] [RenameWorkflowOptionsInMetadata] 100%
[2023-02-04 08:55:06,64] [info] Running with database db.url = jdbc:hsqldb:mem:a487ea75-b617-4523-a254-d0e694e68ff9;shutdown=false;hsqldb.tx=mvcc
[2023-02-04 08:55:06,92] [info] Slf4jLogger started
[2023-02-04 08:55:07,18] [info] Workflow heartbeat configuration:
{
  "cromwellId" : "cromid-b625dba",
  "heartbeatInterval" : "2 minutes",
  "ttl" : "10 minutes",
  "writeBatchSize" : 10000,
  "writeThreshold" : 10000
}
[2023-02-04 08:55:07,22] [info] Metadata summary refreshing every 2 seconds.
[2023-02-04 08:55:07,26] [info] KvWriteActor configured to flush with batch size 200 and process rate 5 seconds.
[2023-02-04 08:55:07,26] [info] WriteMetadataActor configured to flush with batch size 200 and process rate 5 seconds.
[2023-02-04 08:55:07,26] [info] CallCacheWriteActor configured to flush with batch size 100 and process rate 3 seconds.
[2023-02-04 08:55:07,63] [info] JobExecutionTokenDispenser - Distribution rate: 50 per 1 seconds.
[2023-02-04 08:55:07,64] [info] SingleWorkflowRunnerActor: Version 34-unknown-SNAP
[2023-02-04 08:55:07,65] [info] SingleWorkflowRunnerActor: Submitting workflow
[2023-02-04 08:55:07,68] [info] Unspecified type (Unspecified version) workflow 48f62f22-25fe-4f0f-b5fe-21191f035abd submitted
[2023-02-04 08:55:07,72] [info] SingleWorkflowRunnerActor: Workflow submitted 48f62f22-25fe-4f0f-b5fe-21191f035abd
[2023-02-04 08:55:07,75] [info] 1 new workflows fetched
[2023-02-04 08:55:07,75] [info] WorkflowManagerActor Starting workflow 48f62f22-25fe-4f0f-b5fe-21191f035abd
[2023-02-04 08:55:07,76] [warn] SingleWorkflowRunnerActor: received unexpected message: Done in state RunningSwraData
[2023-02-04 08:55:07,79] [warn] Couldn't find a suitable DSN, defaulting to a Noop one.
[2023-02-04 08:55:07,79] [info] Using noop to send events.
[2023-02-04 08:55:07,81] [info] WorkflowManagerActor Successfully started WorkflowActor-48f62f22-25fe-4f0f-b5fe-21191f035abd
[2023-02-04 08:55:07,81] [info] Retrieved 1 workflows from the WorkflowStoreActor
[2023-02-04 08:55:07,81] [info] WorkflowStoreHeartbeatWriteActor configured to flush with batch size 10000 and process rate 2 minutes.
[2023-02-04 08:55:07,81] [info] MaterializeWorkflowDescriptorActor [48f62f22]: Parsing workflow as WDL 1.0
[2023-02-04 08:55:08,24] [error] WorkflowManagerActor Workflow 48f62f22-25fe-4f0f-b5fe-21191f035abd failed (during MaterializingWorkflowDescriptorState): cromwell.engine.workflow.lifecycle.materialization.MaterializeWorkflowDescriptorActor$$anon$1: Workflow input processing failed:
The label in the input is too long
java.base/java.net.IDN.toASCIIInternal(IDN.java:340)
java.base/java.net.IDN.toASCII(IDN.java:122)
java.base/java.net.IDN.toASCII(IDN.java:151)
com.softwaremill.sttp.Uri.encodeHost(Uri.scala:171)
com.softwaremill.sttp.Uri.toString(Uri.scala:122)
com.softwaremill.sttp.asynchttpclient.AsyncHttpClientBackend.requestToAsync(AsyncHttpClientBackend.scala:152)
com.softwaremill.sttp.asynchttpclient.AsyncHttpClientBackend.send(AsyncHttpClientBackend.scala:40)
com.softwaremill.sttp.FollowRedirectsBackend.sendWithCounter(FollowRedirectsBackend.scala:15)
com.softwaremill.sttp.FollowRedirectsBackend.send(FollowRedirectsBackend.scala:10)
com.softwaremill.sttp.RequestT.send(RequestT.scala:242)
cromwell.languages.util.ImportResolver$.$anonfun$httpResolverWithHeaders$1(ImportResolver.scala:88)
common.transforms.package$CheckedAtoB$.$anonfun$firstSuccess$2(package.scala:32)
scala.collection.LinearSeqOptimized.foldLeft(LinearSeqOptimized.scala:122)
scala.collection.LinearSeqOptimized.foldLeft$(LinearSeqOptimized.scala:118)
scala.collection.immutable.List.foldLeft(List.scala:86)
common.transforms.package$CheckedAtoB$.$anonfun$firstSuccess$1(package.scala:29)
cats.data.Kleisli.$anonfun$andThen$1(Kleisli.scala:37)
wdl.transforms.base.wdlom2wom.FileElementToWomBundle$.wdl$transforms$base$wdlom2wom$FileElementToWomBundle$$importWomBundle(FileElementToWomBundle.scala:99)
wdl.transforms.base.wdlom2wom.FileElementToWomBundle$$anon$1.$anonfun$toWomBundle$11(FileElementToWomBundle.scala:72)
cats.instances.VectorInstances$$anon$1.$anonfun$traverse$2(vector.scala:77)
cats.instances.VectorInstances$$anon$1.loop$2(vector.scala:40)
cats.instances.VectorInstances$$anon$1.$anonfun$foldRight$2(vector.scala:41)
cats.Eval$.advance(Eval.scala:272)
cats.Eval$.loop$1(Eval.scala:354)
cats.Eval$.cats$Eval$$evaluate(Eval.scala:372)
cats.Eval$Defer.value(Eval.scala:258)
cats.instances.VectorInstances$$anon$1.traverse(vector.scala:76)
cats.instances.VectorInstances$$anon$1.traverse(vector.scala:12)
cats.Traverse$Ops.traverse(Traverse.scala:19)
cats.Traverse$Ops.traverse$(Traverse.scala:19)
cats.Traverse$ToTraverseOps$$anon$3.traverse(Traverse.scala:19)
wdl.transforms.base.wdlom2wom.FileElementToWomBundle$$anon$1.toWomBundle(FileElementToWomBundle.scala:72)
wdl.transforms.base.wdlom2wom.FileElementToWomBundle$$anon$1.toWomBundle(FileElementToWomBundle.scala:28)
wom.transforms.WomBundleMaker$Ops.toWomBundle(WomExecutableMaker.scala:16)
wom.transforms.WomBundleMaker$Ops.toWomBundle$(WomExecutableMaker.scala:16)
wom.transforms.WomBundleMaker$ops$$anon$2.toWomBundle(WomExecutableMaker.scala:16)
wdl.transforms.base.wdlom2wom.FileElementToWomBundle$.convert(FileElementToWomBundle.scala:81)
wdl.transforms.base.wdlom2wom.package$.$anonfun$fileElementToWomBundle$1(package.scala:12)
scala.util.Either$RightProjection.flatMap(Either.scala:702)
cats.instances.EitherInstances$$anon$1.flatMap(either.scala:36)
cats.instances.EitherInstances$$anon$1.flatMap(either.scala:32)
cats.data.Kleisli.$anonfun$andThen$1(Kleisli.scala:37)
languages.wdl.draft3.WdlDraft3LanguageFactory.getWomBundle(WdlDraft3LanguageFactory.scala:50)
wdl.transforms.base.wdlom2wom.FileElementToWomBundle$.$anonfun$importWomBundle$2(FileElementToWomBundle.scala:91)
common.transforms.package$CheckedAtoB$.$anonfun$firstSuccess$1(package.scala:28)
scala.util.Either$RightProjection.flatMap(Either.scala:702)
cats.instances.EitherInstances$$anon$1.flatMap(either.scala:36)
cats.instances.EitherInstances$$anon$1.flatMap(either.scala:32)
cats.data.Kleisli.$anonfun$andThen$1(Kleisli.scala:37)
wdl.transforms.base.wdlom2wom.FileElementToWomBundle$.wdl$transforms$base$wdlom2wom$FileElementToWomBundle$$importWomBundle(FileElementToWomBundle.scala:99)
wdl.transforms.base.wdlom2wom.FileElementToWomBundle$$anon$1.$anonfun$toWomBundle$11(FileElementToWomBundle.scala:72)
cats.instances.VectorInstances$$anon$1.$anonfun$traverse$2(vector.scala:77)
cats.instances.VectorInstances$$anon$1.loop$2(vector.scala:40)
cats.instances.VectorInstances$$anon$1.$anonfun$foldRight$2(vector.scala:41)
cats.Eval$.advance(Eval.scala:272)
cats.Eval$.loop$1(Eval.scala:354)
cats.Eval$.cats$Eval$$evaluate(Eval.scala:372)
cats.Eval$Defer.value(Eval.scala:258)
cats.instances.VectorInstances$$anon$1.traverse(vector.scala:76)
cats.instances.VectorInstances$$anon$1.traverse(vector.scala:12)
cats.Traverse$Ops.traverse(Traverse.scala:19)
cats.Traverse$Ops.traverse$(Traverse.scala:19)
cats.Traverse$ToTraverseOps$$anon$3.traverse(Traverse.scala:19)
wdl.transforms.base.wdlom2wom.FileElementToWomBundle$$anon$1.toWomBundle(FileElementToWomBundle.scala:72)
wdl.transforms.base.wdlom2wom.FileElementToWomBundle$$anon$1.toWomBundle(FileElementToWomBundle.scala:28)
wom.transforms.WomBundleMaker$Ops.toWomBundle(WomExecutableMaker.scala:16)
wom.transforms.WomBundleMaker$Ops.toWomBundle$(WomExecutableMaker.scala:16)
wom.transforms.WomBundleMaker$ops$$anon$2.toWomBundle(WomExecutableMaker.scala:16)
wdl.transforms.base.wdlom2wom.FileElementToWomBundle$.convert(FileElementToWomBundle.scala:81)
wdl.transforms.base.wdlom2wom.package$.$anonfun$fileElementToWomBundle$1(package.scala:12)
scala.util.Either$RightProjection.flatMap(Either.scala:702)
cats.instances.EitherInstances$$anon$1.flatMap(either.scala:36)
cats.instances.EitherInstances$$anon$1.flatMap(either.scala:32)
cats.data.Kleisli.$anonfun$andThen$1(Kleisli.scala:37)
languages.wdl.draft3.WdlDraft3LanguageFactory.getWomBundle(WdlDraft3LanguageFactory.scala:50)
wdl.transforms.base.wdlom2wom.FileElementToWomBundle$.$anonfun$importWomBundle$2(FileElementToWomBundle.scala:91)
common.transforms.package$CheckedAtoB$.$anonfun$firstSuccess$1(package.scala:28)
scala.util.Either$RightProjection.flatMap(Either.scala:702)
cats.instances.EitherInstances$$anon$1.flatMap(either.scala:36)
cats.instances.EitherInstances$$anon$1.flatMap(either.scala:32)
cats.data.Kleisli.$anonfun$andThen$1(Kleisli.scala:37)
wdl.transforms.base.wdlom2wom.FileElementToWomBundle$.wdl$transforms$base$wdlom2wom$FileElementToWomBundle$$importWomBundle(FileElementToWomBundle.scala:99)
wdl.transforms.base.wdlom2wom.FileElementToWomBundle$$anon$1.$anonfun$toWomBundle$11(FileElementToWomBundle.scala:72)
cats.instances.VectorInstances$$anon$1.$anonfun$traverse$2(vector.scala:77)
cats.instances.VectorInstances$$anon$1.loop$2(vector.scala:40)
cats.instances.VectorInstances$$anon$1.$anonfun$foldRight$2(vector.scala:41)
cats.Eval$.advance(Eval.scala:272)
cats.Eval$.loop$1(Eval.scala:354)
cats.Eval$.cats$Eval$$evaluate(Eval.scala:372)
cats.Eval$Defer.value(Eval.scala:258)
cats.instances.VectorInstances$$anon$1.traverse(vector.scala:76)
cats.instances.VectorInstances$$anon$1.traverse(vector.scala:12)
cats.Traverse$Ops.traverse(Traverse.scala:19)
cats.Traverse$Ops.traverse$(Traverse.scala:19)
cats.Traverse$ToTraverseOps$$anon$3.traverse(Traverse.scala:19)
wdl.transforms.base.wdlom2wom.FileElementToWomBundle$$anon$1.toWomBundle(FileElementToWomBundle.scala:72)
wdl.transforms.base.wdlom2wom.FileElementToWomBundle$$anon$1.toWomBundle(FileElementToWomBundle.scala:28)
wom.transforms.WomBundleMaker$Ops.toWomBundle(WomExecutableMaker.scala:16)
wom.transforms.WomBundleMaker$Ops.toWomBundle$(WomExecutableMaker.scala:16)
wom.transforms.WomBundleMaker$ops$$anon$2.toWomBundle(WomExecutableMaker.scala:16)
wdl.transforms.base.wdlom2wom.FileElementToWomBundle$.convert(FileElementToWomBundle.scala:81)
wdl.transforms.base.wdlom2wom.package$.$anonfun$fileElementToWomBundle$1(package.scala:12)
scala.util.Either$RightProjection.flatMap(Either.scala:702)
cats.instances.EitherInstances$$anon$1.flatMap(either.scala:36)
cats.instances.EitherInstances$$anon$1.flatMap(either.scala:32)
cats.data.Kleisli.$anonfun$andThen$1(Kleisli.scala:37)
languages.wdl.draft3.WdlDraft3LanguageFactory.getWomBundle(WdlDraft3LanguageFactory.scala:50)
languages.wdl.draft3.WdlDraft3LanguageFactory.$anonfun$validateNamespace$2(WdlDraft3LanguageFactory.scala:39)
scala.util.Either.flatMap(Either.scala:338)
languages.wdl.draft3.WdlDraft3LanguageFactory.validateNamespace(WdlDraft3LanguageFactory.scala:38)
cromwell.engine.workflow.lifecycle.materialization.MaterializeWorkflowDescriptorActor.$anonfun$buildWorkflowDescriptor$7(MaterializeWorkflowDescriptorActor.scala:242)
cats.data.EitherT.$anonfun$flatMap$1(EitherT.scala:80)
cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:138)
cats.effect.internals.IORunLoop$.start(IORunLoop.scala:35)
cats.effect.IO.unsafeRunAsync(IO.scala:269)
cats.effect.IO.unsafeToFuture(IO.scala:341)
cromwell.engine.workflow.lifecycle.materialization.MaterializeWorkflowDescriptorActor$$anonfun$1.$anonfun$applyOrElse$1(MaterializeWorkflowDescriptorActor.scala:146)
scala.concurrent.Future.$anonfun$flatMap$1(Future.scala:303)
scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:37)
scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:91)
scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:81)
akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:91)
akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44)
akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
    at cromwell.engine.workflow.lifecycle.materialization.MaterializeWorkflowDescriptorActor.cromwell$engine$workflow$lifecycle$materialization$MaterializeWorkflowDescriptorActor$$workflowInitializationFailed(MaterializeWorkflowDescriptorActor.scala:200)
    at cromwell.engine.workflow.lifecycle.materialization.MaterializeWorkflowDescriptorActor$$anonfun$2.applyOrElse(MaterializeWorkflowDescriptorActor.scala:173)
    at cromwell.engine.workflow.lifecycle.materialization.MaterializeWorkflowDescriptorActor$$anonfun$2.applyOrElse(MaterializeWorkflowDescriptorActor.scala:165)
    at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:34)
    at akka.actor.FSM.processEvent(FSM.scala:670)
    at akka.actor.FSM.processEvent$(FSM.scala:667)
    at cromwell.engine.workflow.lifecycle.materialization.MaterializeWorkflowDescriptorActor.akka$actor$LoggingFSM$$super$processEvent(MaterializeWorkflowDescriptorActor.scala:123)
    at akka.actor.LoggingFSM.processEvent(FSM.scala:806)
    at akka.actor.LoggingFSM.processEvent$(FSM.scala:788)
    at cromwell.engine.workflow.lifecycle.materialization.MaterializeWorkflowDescriptorActor.processEvent(MaterializeWorkflowDescriptorActor.scala:123)
    at akka.actor.FSM.akka$actor$FSM$$processMsg(FSM.scala:664)
    at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:658)
    at akka.actor.Actor.aroundReceive(Actor.scala:517)
    at akka.actor.Actor.aroundReceive$(Actor.scala:515)
    at cromwell.engine.workflow.lifecycle.materialization.MaterializeWorkflowDescriptorActor.aroundReceive(MaterializeWorkflowDescriptorActor.scala:123)
    at akka.actor.ActorCell.receiveMessage(ActorCell.scala:588)
    at akka.actor.ActorCell.invoke(ActorCell.scala:557)
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
    at akka.dispatch.Mailbox.run(Mailbox.scala:225)
    at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
    at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

[2023-02-04 08:55:08,24] [info] WorkflowManagerActor WorkflowActor-48f62f22-25fe-4f0f-b5fe-21191f035abd is in a terminal state: WorkflowFailedState
[2023-02-04 08:55:08,24] [info] $a [48f62f22]: Copying workflow logs from /mnt/g/ELM-WES-pipeline/cromwell-workflow-logs/workflow.48f62f22-25fe-4f0f-b5fe-21191f035abd.log to /mnt/g/ELM-WES-pipeline/cromwell_wf_logs/workflow.48f62f22-25fe-4f0f-b5fe-21191f035abd.log
[2023-02-04 08:55:15,88] [info] SingleWorkflowRunnerActor workflow finished with status 'Failed'.
[2023-02-04 08:55:17,27] [info] Workflow polling stopped
[2023-02-04 08:55:17,29] [info] Shutting down WorkflowStoreActor - Timeout = 5 seconds
[2023-02-04 08:55:17,30] [info] Shutting down WorkflowLogCopyRouter - Timeout = 5 seconds
[2023-02-04 08:55:17,31] [info] Shutting down JobExecutionTokenDispenser - Timeout = 5 seconds
[2023-02-04 08:55:17,31] [info] Aborting all running workflows.
[2023-02-04 08:55:17,31] [info] JobExecutionTokenDispenser stopped
[2023-02-04 08:55:17,31] [info] WorkflowStoreActor stopped
[2023-02-04 08:55:17,32] [info] Shutting down WorkflowManagerActor - Timeout = 3600 seconds
[2023-02-04 08:55:17,32] [info] WorkflowLogCopyRouter stopped
[2023-02-04 08:55:17,32] [info] WorkflowManagerActor All workflows finished
[2023-02-04 08:55:17,32] [info] WorkflowManagerActor stopped
[2023-02-04 08:55:17,32] [info] Connection pools shut down
[2023-02-04 08:55:17,33] [info] Shutting down SubWorkflowStoreActor - Timeout = 1800 seconds
[2023-02-04 08:55:17,33] [info] Shutting down JobStoreActor - Timeout = 1800 seconds
[2023-02-04 08:55:17,33] [info] SubWorkflowStoreActor stopped
[2023-02-04 08:55:17,33] [info] Shutting down CallCacheWriteActor - Timeout = 1800 seconds
[2023-02-04 08:55:17,33] [info] Shutting down ServiceRegistryActor - Timeout = 1800 seconds
[2023-02-04 08:55:17,33] [info] JobStoreActor stopped
[2023-02-04 08:55:17,33] [info] CallCacheWriteActor Shutting down: 0 queued messages to process
[2023-02-04 08:55:17,33] [info] Shutting down DockerHashActor - Timeout = 1800 seconds
[2023-02-04 08:55:17,33] [info] CallCacheWriteActor stopped
[2023-02-04 08:55:17,33] [info] WriteMetadataActor Shutting down: 0 queued messages to process
[2023-02-04 08:55:17,33] [info] Shutting down IoProxy - Timeout = 1800 seconds
[2023-02-04 08:55:17,33] [info] KvWriteActor Shutting down: 0 queued messages to process
[2023-02-04 08:55:17,33] [info] DockerHashActor stopped
[2023-02-04 08:55:17,34] [info] IoProxy stopped
[2023-02-04 08:55:17,34] [info] ServiceRegistryActor stopped
[2023-02-04 08:55:17,37] [info] Database closed
[2023-02-04 08:55:17,37] [info] Stream materializer shut down
[2023-02-04 08:55:17,40] [info] Automatic shutdown of the async connection
[2023-02-04 08:55:17,40] [info] Gracefully shutdown sentry threads.
[2023-02-04 08:55:17,40] [info] Shutdown finished.

Any feedback or ideas would be greatly appreciated.

aednichols commented 1 year ago

Can you post your WDL? Searching for The label in the input is too long on the 'net suggests that the HTTP resolver is being passed a domain that is too long. I suppose it's possible that you aren't even using HTTP inputs, but when we try all resolvers and the HTTP one fails, the error is not handled correctly.

ibrahimkurt commented 1 year ago

local_newGCP_ExomeGermlineSingleSample_deneme6_bcftools.wdl.zip

This is the main workflow I run, but of course it imports many other WDL tasks. Happy to share them as well if needed.