search

ncbi / egapx

Eukaryotic Genome Annotation Pipeline-External caller scripts and documentation
Other
70 stars 5 forks source link

Error in test run ( nextflow ) #15

Closed Artifice120 closed 3 months ago

Artifice120 commented 3 months ago

When running the test files on a HPC I get the following error.

The path to the virtual env and the git repository is /lustre/isaac/scratch/jtorre28/unstable/egapx

Seems to not find the specified bucket.

Noticed that the .nextflow file is in the home directory instead of the working directory. 


Jun-14 15:05:19.105 [main] DEBUG nextflow.cli.Launcher - $> nextflow -C /lustre/isaac/scratch/jtorre28/unstable/egapx/egapx_config/singularity.config,/lustre/isaac/scratch/jtorre28/unstable/egapx/ui/assets/config/default.config,/lustre/isaac/scratch/jtorre28/unstable/egapx/ui/assets/config/docker_image.config,/lustre/isaac/scratch/jtorre28/unstable/egapx/ui/assets/config/process_resources.config -log /lustre/isaac/scratch/jtorre28/unstable/egapx/example_out/nextflow.log run /lustre/isaac/scratch/jtorre28/unstable/egapx/ui/../nf/ui.nf --output /lustre/isaac/scratch/jtorre28/unstable/egapx/example_out -with-report /lustre/isaac/scratch/jtorre28/unstable/egapx/example_out/run.report.html -with-timeline /lustre/isaac/scratch/jtorre28/unstable/egapx/example_out/run.timeline.html -with-trace /lustre/isaac/scratch/jtorre28/unstable/egapx/example_out/run.trace.txt -params-file /lustre/isaac/scratch/jtorre28/unstable/egapx/example_out/run_params.yaml
Jun-14 15:05:19.214 [main] DEBUG nextflow.cli.CmdRun - N E X T F L O W  ~  version 24.04.2                                                                                                                       Jun-14 15:05:19.235 [main] DEBUG nextflow.plugin.PluginsFacade - Setting up plugin manager > mode=prod; embedded=false; plugins-dir=/nfs/home/jtorre28/.nextflow/plugins; core-plugins: nf-amazon@2.5.2,nf-azure@1.6.0,nf-cloudcache@0.4.1,nf-codecommit@0.2.0,nf-console@1.1.3,nf-ga4gh@1.3.0,nf-google@1.13.2,nf-tower@1.9.1,nf-wave@1.4.2
Jun-14 15:05:19.251 [main] INFO  o.pf4j.DefaultPluginStatusProvider - Enabled plugins: []
Jun-14 15:05:19.253 [main] INFO  o.pf4j.DefaultPluginStatusProvider - Disabled plugins: []
Jun-14 15:05:19.257 [main] INFO  org.pf4j.DefaultPluginManager - PF4J version 3.10.0 in 'deployment' mode
Jun-14 15:05:19.272 [main] INFO  org.pf4j.AbstractPluginManager - No plugins
Jun-14 15:05:19.297 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /lustre/isaac/scratch/jtorre28/unstable/egapx/egapx_config/singularity.config
Jun-14 15:05:19.298 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /lustre/isaac/scratch/jtorre28/unstable/egapx/ui/assets/config/default.config
Jun-14 15:05:19.298 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /lustre/isaac/scratch/jtorre28/unstable/egapx/ui/assets/config/docker_image.config
Jun-14 15:05:19.298 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /lustre/isaac/scratch/jtorre28/unstable/egapx/ui/assets/config/process_resources.config
Jun-14 15:05:19.326 [main] DEBUG n.secret.LocalSecretsProvider - Secrets store: /nfs/home/jtorre28/.nextflow/secrets/store.json
Jun-14 15:05:19.329 [main] DEBUG nextflow.secret.SecretsLoader - Discovered secrets providers: [nextflow.secret.LocalSecretsProvider@66e8997c] - activable => nextflow.secret.LocalSecretsProvider@66e8997c
Jun-14 15:05:19.384 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `standard`
Jun-14 15:05:19.824 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `standard`
Jun-14 15:05:19.848 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `standard`
Jun-14 15:05:19.864 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `standard`
Jun-14 15:05:19.997 [main] DEBUG nextflow.cli.CmdRun - Applied DSL=2 from script declaration
Jun-14 15:05:20.013 [main] DEBUG nextflow.cli.CmdRun - Launching `/lustre/isaac/scratch/jtorre28/unstable/egapx/ui/../nf/ui.nf` [extravagant_leakey] DSL2 - revision: 1a0ac50de2
Jun-14 15:05:20.015 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins default=[nf-amazon@2.5.2]
Jun-14 15:05:20.015 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins resolved requirement=[nf-amazon@2.5.2]                                                                                                  Jun-14 15:05:20.015 [main] DEBUG nextflow.plugin.PluginUpdater - Installing plugin nf-amazon version: 2.5.2                                                                                                      Jun-14 15:05:20.019 [main] INFO  nextflow.plugin.PluginUpdater - Downloading plugin nf-amazon@2.5.2                                                                                                              Jun-14 15:05:21.659 [main] INFO  org.pf4j.util.FileUtils - Expanded plugin zip 'nf-amazon-2.5.2.zip' in 'nf-amazon-2.5.2'                                                                                        Jun-14 15:05:21.665 [main] DEBUG nextflow.plugin.PluginUpdater - Failed atomic move for plugin /tmp/pf4j-update-downloader13127609559763656649/nf-amazon-2.5.2 -> /nfs/home/jtorre28/.nextflow/plugins/nf-amazon-2.5.2 - Reason: /tmp/pf4j-update-downloader13127609559763656649/nf-amazon-2.5.2 -> /nfs/home/jtorre28/.nextflow/plugins/nf-amazon-2.5.2: Invalid cross-device link - Fallback on safe move
Jun-14 15:05:27.196 [main] INFO  org.pf4j.AbstractPluginManager - Plugin 'nf-amazon@2.5.2' resolved
Jun-14 15:05:27.196 [main] INFO  org.pf4j.AbstractPluginManager - Start plugin 'nf-amazon@2.5.2'
Jun-14 15:05:27.324 [main] DEBUG nextflow.plugin.BasePlugin - Plugin started nf-amazon@2.5.2
Jun-14 15:05:27.355 [main] DEBUG nextflow.file.FileHelper - > Added 'S3FileSystemProvider' to list of installed providers [s3]
Jun-14 15:05:27.420 [main] DEBUG nextflow.Session - Session UUID: 5956cf49-b4b9-4368-8408-2c71c97c9eea
Jun-14 15:05:27.421 [main] DEBUG nextflow.Session - Run name: extravagant_leakey
Jun-14 15:05:27.421 [main] DEBUG nextflow.Session - Executor pool size: 4
Jun-14 15:05:27.444 [main] DEBUG nextflow.file.FileHelper - Creating a file system instance for provider: S3FileSystemProvider
Jun-14 15:05:27.468 [main] DEBUG nextflow.cloud.aws.config.AwsConfig - AWS S3 config properties: {max_error_retry=5}
Jun-14 15:05:28.063 [main] DEBUG nextflow.file.FilePorter - File porter settings maxRetries=3; maxTransfers=50; pollTimeout=null
Jun-14 15:05:28.069 [main] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'FileTransfer' minSize=10; maxSize=12; workQueue=LinkedBlockingQueue[10000]; allowCoreThreadTimeout=false
Jun-14 15:05:28.077 [main] DEBUG nextflow.cli.CmdRun -
  Version: 24.04.2 build 5914
  Created: 29-05-2024 06:19 UTC (02:19 EDT)
  System: Linux 4.18.0-425.19.2.el8_7.x86_64
  Runtime: Groovy 4.0.21 on OpenJDK 64-Bit Server VM 11.0.13+7-b1751.21
  Encoding: UTF-8 (UTF-8)
  Process: 199358@login1 [10.22.16.20]
  CPUs: 4 - Mem: 16 GB (11.8 GB) - Swap: 4 GB (0)
Jun-14 15:05:30.114 [main] DEBUG n.c.aws.util.S3CredentialsProvider - No AWS credentials available - falling back to anonymous access
Jun-14 15:05:30.394 [main] ERROR nextflow.cli.Launcher - @unknown
com.amazonaws.services.s3.model.AmazonS3Exception: The specified bucket does not exist (Service: Amazon S3; Status Code: 404; Error Code: NoSuchBucket; Request ID: Q5R163WSYZVZS2NS; S3 Extended Request ID: F1stO8PsZuqfnY2dgyoWSQqKSOBZXWomBZyEiXr0vYYq1l2cwOcWOU5yvOnrKj0z4A0CykdKJq4=; Proxy: null)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1879)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1418)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1387)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5520)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5467)
        at com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:422)
        at com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:6601)
        at com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1891)
        at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1851)
        at nextflow.cloud.aws.nio.S3Client.putObject(S3Client.java:203)
        at nextflow.cloud.aws.nio.S3FileSystemProvider.createDirectory(S3FileSystemProvider.java:488)
        at java.base/java.nio.file.Files.createDirectory(Files.java:690)
        at java.base/java.nio.file.Files.createAndCheckIsDirectory(Files.java:797)
        at java.base/java.nio.file.Files.createDirectories(Files.java:743)
        at org.codehaus.groovy.vmplugin.v8.IndyInterface.selectMethod(IndyInterface.java:355)
        at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
        at nextflow.extension.FilesEx.mkdirs(FilesEx.groovy:493)
        at nextflow.Session.init(Session.groovy:403)
        at nextflow.script.ScriptRunner.execute(ScriptRunner.groovy:129)
        at nextflow.cli.CmdRun.run(CmdRun.groovy:372)
        at nextflow.cli.Launcher.run(Launcher.groovy:503)
        at nextflow.cli.Launcher.main(Launcher.groovy:657)
pstrope commented 3 months ago

Hi, Please post the command you used to run egapx.

Artifice120 commented 3 months ago

Just copy pasted the example below adjusting for singularity ;

 python3 ui/egapx.py ./examples/input_D_farinae_small.yaml -e singularity -w s3://temp_datapath/D_farinae -o example_out
pstrope commented 3 months ago

That bucket in s3 is an example and can be used only with -e aws

For your run, use a unix path (for -w) where you can store a lot of temp files.

Artifice120 commented 3 months ago

That was it. Thanks for the quick response.