Closed drernie closed 8 months ago
ERROR ~ Unknown method invocation `multiply` on Integer type
-- Check 'nf-3VMrm7uIc5HIBT.log' file for details
ERROR ~ publish failed:
-- Check 'nf-3VMrm7uIc5HIBT.log' file for details
FAILED: QuiltPackage.vivos_production_tower_hlatyping
Get an error on Tower, and publish fails. But run Succeeds. And no LOG.
Jan-10 00:35:26.879 [main] ERROR nextflow.script.WorkflowMetadata - Failed to invoke workflow.onComplete
event handler
java.io.IOException: Cannot run program "mail": error=2, No such file or directory
THIS was due to providing an email address (Forge probably lacked the correct permission)
s3://qulit-demos WORKS https://demo.quiltdata.com/b/quilt-demos/tree/direct/hlatyping/
When I try it with vivo-pipes, I insanely get: `Error executing process > 'NFCORE_HLATYPING:HLATYPING:INPUT_CHECK:SAMPLESHEET_CHECK (samplesheet_full.csv)'
Which is probably due to:
at nextflow.proces [nf-4JZDgKLIZL7JQA.log](https://github.com/data-yaml/vivos/files/13882729/nf-4JZDgKLIZL7JQA.log) sor.PublishDir.createPublishDir(PublishDir.groovy:471)
Jan-10 03:39:45.881 [Task monitor] DEBUG nextflow.file.FileHelper - Creating a file system instance for provider: S3FileSystemProvider
Jan-10 03:39:45.882 [Task monitor] DEBUG nextflow.cloud.aws.config.AwsConfig - AWS S3 config properties: {upload_chunk_size=10485760, max_error_retry=10}
Jan-10 03:39:45.885 [Task monitor] DEBUG nextflow.cloud.aws.nio.S3Client - Setting S3 upload chunk size=10485760
Jan-10 03:39:45.885 [Task monitor] DEBUG nextflow.cloud.aws.nio.S3Client - Setting S3 glacierRetrievalTier=null
Jan-10 03:39:46.023 [Task monitor] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
task: name=NFCORE_HLATYPING:HLATYPING:INPUT_CHECK:SAMPLESHEET_CHECK (samplesheet_full.csv); work-dir=s3://quilt-demos/scratch/4JZDgKLIZL7JQA/73/def5d7fb0cdc50f01f9428cfcaccba
error [com.amazonaws.services.s3.model.AmazonS3Exception]: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: YN19P6RK3AQ5G8YA; S3 Extended Request ID: gxIWFIIh2ouRqoa53HJrSeukFul07DmsSL+rmpJQzN+oH7L0ZA2YpSz6j+AhPreU93aJ9v2MRn+fcTXfQTlj319hrsGqdZJmFMboOkislYw=; Proxy: null)
Jan-10 03:39:46.053 [Task monitor] ERROR nextflow.processor.TaskProcessor - Error executing process > 'NFCORE_HLATYPING:HLATYPING:INPUT_CHECK:SAMPLESHEET_CHECK (samplesheet_full.csv)'
Caused by:
Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: YN19P6RK3AQ5G8YA; S3 Extended Request ID: gxIWFIIh2ouRqoa53HJrSeukFul07DmsSL+rmpJQzN+oH7L0ZA2YpSz6j+AhPreU93aJ9v2MRn+fcTXfQTlj319hrsGqdZJmFMboOkislYw=; Proxy: null)
com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: YN19P6RK3AQ5G8YA; S3 Extended Request ID: gxIWFIIh2ouRqoa53HJrSeukFul07DmsSL+rmpJQzN+oH7L0ZA2YpSz6j+AhPreU93aJ9v2MRn+fcTXfQTlj319hrsGqdZJmFMboOkislYw=; Proxy: null)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1879)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1418)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1387)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403)
at com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:421)
at com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:6532)
at com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1861)
at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1821)
at nextflow.cloud.aws.nio.S3Client.putObject(S3Client.java:210)
at nextflow.cloud.aws.nio.S3FileSystemProvider.createDirectory(S3FileSystemProvider.java:488)
at java.base/java.nio.file.Files.createDirectory(Files.java:700)
at java.base/java.nio.file.Files.createAndCheckIsDirectory(Files.java:807)
at java.base/java.nio.file.Files.createDirectories(Files.java:753)
at nextflow.processor.PublishDir.makeDirs(PublishDir.groovy:480)
at nextflow.processor.PublishDir.createPublishDir(PublishDir.groovy:471)
at nextflow.processor.PublishDir.apply0(PublishDir.groovy:209)
at nextflow.processor.PublishDir.apply(PublishDir.groovy:286)
at nextflow.processor.PublishDir$apply.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:148)
at nextflow.processor.TaskProcessor.publishOutputs0(TaskProcessor.groovy:1362)
at nextflow.processor.TaskProcessor.publishOutputs(TaskProcessor.groovy:1337)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.codehaus.groovy.runtime.callsite.PlainObjectMetaMethodSite.doInvoke(PlainObjectMetaMethodSite.java:48)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:189)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:57)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:51)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:171)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:185)
at nextflow.processor.TaskProcessor.finalizeTask0(TaskProcessor.groovy:2337)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.codehaus.groovy.runtime.callsite.PlainObjectMetaMethodSite.doInvoke(PlainObjectMetaMethodSite.java:48)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:189)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:57)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:51)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:171)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:185)
at nextflow.processor.TaskProcessor.finalizeTask(TaskProcessor.groovy:2308)
at nextflow.processor.TaskPollingMonitor.checkTaskStatus(TaskPollingMonitor.groovy:631)
at nextflow.processor.TaskPollingMonitor.checkAllTasks(TaskPollingMonitor.groovy:537)
at nextflow.processor.TaskPollingMonitor.pollLoop(TaskPollingMonitor.groovy:412)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1254)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1030)
at org.codehaus.groovy.runtime.InvokerHelper.invokePogoMethod(InvokerHelper.java:1036)
at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:1019)
at org.codehaus.groovy.runtime.InvokerHelper.invokeMethodSafe(InvokerHelper.java:97)
at nextflow.processor.TaskPollingMonitor$_start_closure2.doCall(TaskPollingMonitor.groovy:293)
at nextflow.processor.TaskPollingMonitor$_start_closure2.call(TaskPollingMonitor.groovy)
at groovy.lang.Closure.run(Closure.java:498)
at java.base/java.lang.Thread.run(Thread.java:833)
Jan-10 03:39:46.068 [Task monitor] DEBUG nextflow.Session - Session aborted -- Cause: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: YN19P6RK3AQ5G8YA; S3 Extended Request ID: gxIWFIIh2ouRqoa53HJrSeukFul07DmsSL+rmpJQzN+oH7L0ZA2YpSz6j+AhPreU93aJ9v2MRn+fcTXfQTlj319hrsGqdZJmFMboOkislYw=; Proxy: null)
AH. CDK BUCKETS are created "Deny ALL" Principals.
Making them public, AND granting hackathon-shared principal access does NOT help. Trying DRIFT: manually deleting 'Deny' to see if that solves the problem.
NOPE: Even manually deleting the DENY does NOT allow direct S3 access. WTF?
Plan B: Enable sales-production bucket?!?
ALSO: quilt-demos has proper permissions, but STILL fails on nf-plugin:
"ERROR ~ Unknown method invocation multiply
on Integer type"
AH. That's a ~known bug with task-attempt scale checks:
cpus = { check_max( 1 * task.attempt, 'cpus' ) }
memory = { check_max( 6.GB * task.attempt, 'memory' ) }
time = { check_max( 4.h * task.attempt, 'time' ) }
versus
cpus = 20
memory = 72.GB
time = 96.h
DOH. Is this because I used test_full
(which uses base) versus [test](https://github.com/nf-core/hlatyping/blob/master/conf/test.config)
profile (which overrides)?
So.... getting rid of test_full removes the error. Now it runs without error: Then nothing.
Jan-10 04:49:23.196 [PublishDir-3] DEBUG nextflow.quilt.QuiltObserver - onFilePublish.Path[quilt-demos#package=direct%2fhlatyping&path=multiqc%2fmultiqc_plots]
Jan-10 04:49:23.196 [PublishDir-3] DEBUG nextflow.quilt.QuiltObserver - checkPath[quilt-demos#package=direct%2fhlatyping&path=multiqc%2fmultiqc_plots] published[true]
Jan-10 04:49:23.196 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'PublishDir' shutdown completed (hard=false)
Jan-10 04:49:23.198 [main] INFO nextflow.Nextflow - -[0;35m[nf-core/hlatyping][0;32m Pipeline completed successfully[0m-
Jan-10 04:49:23.204 [main] DEBUG n.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=17; failedCount=0; ignoredCount=0; cachedCount=0; pendingCount=0; submittedCount=0; runningCount=0; retriesCount=0; abortedCount=0; succeedDuration=11m 52s; failedDuration=0ms; cachedDuration=0ms;loadCpus=0; loadMemory=0; peakRunning=4; peakCpus=20; peakMemory=120 GB; ]
Jan-10 04:49:23.205 [main] DEBUG nextflow.trace.TraceFileObserver - Workflow completed -- saving trace file
Jan-10 04:49:23.207 [main] DEBUG nextflow.trace.ReportObserver - Workflow completed -- rendering execution report
Jan-10 04:49:24.385 [main] DEBUG nextflow.trace.TimelineObserver - Workflow completed -- rendering execution timeline
Jan-10 04:50:26.840 [tower-logs-checkpoint] DEBUG n.cloud.aws.nio.S3FileSystemProvider - S3 upload file from=nf-4xbsNXg1LhGfdf.txt to=s3://quilt-demos/scratch/4xbsNXg1LhGfdf/nf-4xbsNXg1LhGfdf.txt
Jan-10 04:50:27.078 [tower-logs-checkpoint] DEBUG n.cloud.aws.nio.S3FileSystemProvider - S3 upload file from=nf-4xbsNXg1LhGfdf.log to=s3://quilt-demos/scratch/4xbsNXg1LhGfdf/nf-4xbsNXg1LhGfdf.log
I hate my life...
End to End: Raw Data to Analysis
Sequence
NOTE: public website, do not mention any customer names