citiususc / BigSeqKit

BigSeqKit: a parallel Big Data toolkit to process FASTA and FASTQ files at scale
GNU General Public License v3.0
55 stars 4 forks source link

Failed to run locally. #5

Open shenwei356 opened 8 months ago

shenwei356 commented 8 months ago

Hi, I'm trying to install and run BigSeqKit locally.

Here are the commands. There's an error of Permission denied, but it remains after changing ownership of the ignis directory.

# no error
sudo docker pull ignishpc/full

cd ~/Downloads
# no error
docker run --rm -v $(pwd):/target ignishpc/submitter ignis-export /target

# set current directory as job directory
export IGNIS_DFS_ID=$(pwd)
# set docker as scheduler
export IGNIS_SCHEDULER_TYPE=docker
# set where docker is available
export IGNIS_SCHEDULER_URL=/var/run/docker.sock

./ignis/bin/ignis-submit ignishpc/full bigseqkit
Mar 10, 2024 09:03:37 <ERROR> org.ignis.submitter.Submit.call(Submit.java:271): java.io.IOException: com.sun.jna.LastErrorException: [13] Permission denied
java.lang.RuntimeException: java.io.IOException: com.sun.jna.LastErrorException: [13] Permission denied
        at com.github.dockerjava.httpclient5.ApacheDockerHttpClientImpl.execute(ApacheDockerHttpClientImpl.java:179) ~[docker-java-transport-httpclient5-3.2.11.jar:?]
        at com.github.dockerjava.httpclient5.ApacheDockerHttpClient.execute(ApacheDockerHttpClient.java:9) ~[docker-java-transport-httpclient5-3.2.11.jar:?]
        at com.github.dockerjava.core.DefaultInvocationBuilder.execute(DefaultInvocationBuilder.java:228) ~[docker-java-core-3.2.11.jar:?]
        at com.github.dockerjava.core.DefaultInvocationBuilder.post(DefaultInvocationBuilder.java:124) ~[docker-java-core-3.2.11.jar:?]
        at com.github.dockerjava.core.exec.CreateContainerCmdExec.execute(CreateContainerCmdExec.java:37) ~[docker-java-core-3.2.11.jar:?]
        at com.github.dockerjava.core.exec.CreateContainerCmdExec.execute(CreateContainerCmdExec.java:13) ~[docker-java-core-3.2.11.jar:?]
        at com.github.dockerjava.core.exec.AbstrSyncDockerCmdExec.exec(AbstrSyncDockerCmdExec.java:21) ~[docker-java-core-3.2.11.jar:?]
        at com.github.dockerjava.core.command.AbstrDockerCmd.exec(AbstrDockerCmd.java:35) ~[docker-java-core-3.2.11.jar:?]
        at com.github.dockerjava.core.command.CreateContainerCmdImpl.exec(CreateContainerCmdImpl.java:609) ~[docker-java-core-3.2.11.jar:?]
        at org.ignis.scheduler.Docker.createDriverContainer(Docker.java:318) ~[ignis-scheduler-docker-1.0.jar:?]
        at org.ignis.submitter.Submit.call(Submit.java:186) [ignis-submitter-1.0.jar:?]
        at org.ignis.submitter.Submit.call(Submit.java:46) [ignis-submitter-1.0.jar:?]
        at picocli.CommandLine.executeUserObject(CommandLine.java:1953) [picocli-4.6.1.jar:task ':jar' property 'archiveVersion']
        at picocli.CommandLine.access$1300(CommandLine.java:145) [picocli-4.6.1.jar:task ':jar' property 'archiveVersion']
        at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352) [picocli-4.6.1.jar:task ':jar' property 'archiveVersion']
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2346) [picocli-4.6.1.jar:task ':jar' property 'archiveVersion']
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2311) [picocli-4.6.1.jar:task ':jar' property 'archiveVersion']
        at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179) [picocli-4.6.1.jar:task ':jar' property 'archiveVersion']
        at picocli.CommandLine.execute(CommandLine.java:2078) [picocli-4.6.1.jar:task ':jar' property 'archiveVersion']
        at org.ignis.submitter.Submit.main(Submit.java:284) [ignis-submitter-1.0.jar:?]
Caused by: java.io.IOException: com.sun.jna.LastErrorException: [13] Permission denied
        at com.github.dockerjava.transport.DomainSocket.<init>(DomainSocket.java:63) ~[docker-java-transport-3.2.11.jar:?]
        at com.github.dockerjava.transport.LinuxDomainSocket.<init>(LinuxDomainSocket.java:41) ~[docker-java-transport-3.2.11.jar:?]
        at com.github.dockerjava.transport.DomainSocket.get(DomainSocket.java:138) ~[docker-java-transport-3.2.11.jar:?]
        at com.github.dockerjava.httpclient5.ApacheDockerHttpClientImpl$2.createSocket(ApacheDockerHttpClientImpl.java:137) ~[docker-java-transport-httpclient5-3.2.11.jar:?]
        at org.apache.hc.client5.http.impl.io.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:125) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:409) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.InternalExecRuntime.connectEndpoint(InternalExecRuntime.java:164) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.InternalExecRuntime.connectEndpoint(InternalExecRuntime.java:174) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.ConnectExec.execute(ConnectExec.java:135) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.ExecChainElement$1.proceed(ExecChainElement.java:57) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.ProtocolExec.execute(ProtocolExec.java:172) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.ExecChainElement$1.proceed(ExecChainElement.java:57) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.HttpRequestRetryExec.execute(HttpRequestRetryExec.java:93) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.ExecChainElement$1.proceed(ExecChainElement.java:57) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.ContentCompressionExec.execute(ContentCompressionExec.java:128) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.ExecChainElement$1.proceed(ExecChainElement.java:57) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.RedirectExec.execute(RedirectExec.java:116) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.InternalHttpClient.doExecute(InternalHttpClient.java:178) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:67) ~[httpclient5-5.0.3.jar:5.0.3]
        at com.github.dockerjava.httpclient5.ApacheDockerHttpClientImpl.execute(ApacheDockerHttpClientImpl.java:175) ~[docker-java-transport-httpclient5-3.2.11.jar:?]
        ... 19 more
Caused by: com.sun.jna.LastErrorException: [13] Permission denied
        at com.github.dockerjava.transport.LinuxDomainSocket.connect(Native Method) ~[docker-java-transport-3.2.11.jar:?]
        at com.github.dockerjava.transport.LinuxDomainSocket.connect(LinuxDomainSocket.java:49) ~[docker-java-transport-3.2.11.jar:?]
        at com.github.dockerjava.transport.DomainSocket.open(DomainSocket.java:69) ~[docker-java-transport-3.2.11.jar:?]
        at com.github.dockerjava.transport.DomainSocket.<init>(DomainSocket.java:59) ~[docker-java-transport-3.2.11.jar:?]
        at com.github.dockerjava.transport.LinuxDomainSocket.<init>(LinuxDomainSocket.java:41) ~[docker-java-transport-3.2.11.jar:?]
        at com.github.dockerjava.transport.DomainSocket.get(DomainSocket.java:138) ~[docker-java-transport-3.2.11.jar:?]
        at com.github.dockerjava.httpclient5.ApacheDockerHttpClientImpl$2.createSocket(ApacheDockerHttpClientImpl.java:137) ~[docker-java-transport-httpclient5-3.2.11.jar:?]
        at org.apache.hc.client5.http.impl.io.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:125) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:409) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.InternalExecRuntime.connectEndpoint(InternalExecRuntime.java:164) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.InternalExecRuntime.connectEndpoint(InternalExecRuntime.java:174) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.ConnectExec.execute(ConnectExec.java:135) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.ExecChainElement$1.proceed(ExecChainElement.java:57) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.ProtocolExec.execute(ProtocolExec.java:172) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.ExecChainElement$1.proceed(ExecChainElement.java:57) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.HttpRequestRetryExec.execute(HttpRequestRetryExec.java:93) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.ExecChainElement$1.proceed(ExecChainElement.java:57) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.ContentCompressionExec.execute(ContentCompressionExec.java:128) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.ExecChainElement$1.proceed(ExecChainElement.java:57) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.RedirectExec.execute(RedirectExec.java:116) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.InternalHttpClient.doExecute(InternalHttpClient.java:178) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:67) ~[httpclient5-5.0.3.jar:5.0.3]
        at com.github.dockerjava.httpclient5.ApacheDockerHttpClientImpl.execute(ApacheDockerHttpClientImpl.java:175) ~[docker-java-transport-httpclient5-3.2.11.jar:?]
        ... 19 more
cesarpomar commented 8 months ago

The Permission denied error is because your user doesn't have permission to access /var/run/docker.sock. Please check if you can run docker ps.

Another option is to run ignis-submit command with sudo.

Please let me know if you find any other errors

shenwei356 commented 8 months ago
$ sudo docker ps
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

$ sudo ./ignis/bin/ignis-submit ignishpc/full bigseqkit
Mar 12, 2024 09:59:28 <ERROR> org.ignis.submitter.Submit.call(Submit.java:271): Key 'ignis.scheduler.type' value error,  not found
org.ignis.properties.IPropertyException: Key 'ignis.scheduler.type' value error,  not found
        at org.ignis.properties.IProperties.getProperty(IProperties.java:69) ~[ignis-properties-1.0.jar:?]
        at org.ignis.submitter.Submit.call(Submit.java:113) [ignis-submitter-1.0.jar:?]
        at org.ignis.submitter.Submit.call(Submit.java:46) [ignis-submitter-1.0.jar:?]
        at picocli.CommandLine.executeUserObject(CommandLine.java:1953) [picocli-4.6.1.jar:task ':jar' property 'archiveVersion']
        at picocli.CommandLine.access$1300(CommandLine.java:145) [picocli-4.6.1.jar:task ':jar' property 'archiveVersion']
        at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352) [picocli-4.6.1.jar:task ':jar' property 'archiveVersion']
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2346) [picocli-4.6.1.jar:task ':jar' property 'archiveVersion']
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2311) [picocli-4.6.1.jar:task ':jar' property 'archiveVersion']
        at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179) [picocli-4.6.1.jar:task ':jar' property 'archiveVersion']
        at picocli.CommandLine.execute(CommandLine.java:2078) [picocli-4.6.1.jar:task ':jar' property 'archiveVersion']
        at org.ignis.submitter.Submit.main(Submit.java:284) [ignis-submitter-1.0.jar:?]
cesarpomar commented 8 months ago

You should use sudo -E to preserve the environment variables defined with the export command. The error indicates that ignis.scheduler.type created with export IGNIS_SCHEDULER_TYPE=docker does not exist.

shenwei356 commented 8 months ago

Oh thanks, it finally worked. However a new error comes out. It reported a file not exist error. But the input file does exist.

$ ll /home/shenwei/go/src/github.com/shenwei356/seqkit/tests/hairpin.fa
-rw-r--r-- 1 shenwei shenwei 4.6M  9月25日 08:16 /home/shenwei/go/src/github.com/shenwei356/seqkit/tests/hairpin.fa

$ sudo -E sh ./ignis/bin/ignis-submit ignishpc/full 'bigseqkit seq /home/shenwei/go/src/github.com/shenwei356/seqkit/tests/hairpin.fa | bigseqkit stats'

...
panic: stat /home/shenwei/go/src/github.com/shenwei356/seqkit/tests/hairpin.fa: no such file or directory
...
details. Mar 12, 2024 02:14:35 org.ignis.backend.Main.main(Main.java:46): Backend started Mar 12, 2024 02:14:35 org.ignis.backend.Main.main(Main.java:49): Loading environment variables Mar 12, 2024 02:14:35 org.ignis.backend.Main.main(Main.java:64): Loading configuration file Mar 12, 2024 02:14:35 org.ignis.backend.Main.main(Main.java:80): Loading scheduler Mar 12, 2024 02:14:35 org.ignis.backend.Main.main(Main.java:92): Checking scheduler docker Mar 12, 2024 02:14:35 org.ignis.backend.Main.main(Main.java:95): Scheduler docker /var/run/docker.sock ... OK Mar 12, 2024 02:14:35 org.ignis.backend.Main.main(Main.java:103): Getting Driver container info from scheduler Mar 12, 2024 02:14:35 org.ignis.backend.Main.main(Main.java:105): Driver container info found Mar 12, 2024 02:14:35 org.ignis.backend.Main.main(Main.java:125): Setting dynamic properties Mar 12, 2024 02:14:35 org.ignis.backend.services.IBackendServiceImpl.start(IBackendServiceImpl.java:48): Backend server started on port 4000 [INFO] 2024/03/12 14:14:35 ilog.go:37: ServerModule: go executor started Mar 12, 2024 02:14:35 org.ignis.backend.services.IBackendServiceImpl.startHealthServer(IBackendServiceImpl.java:117): Backend health server started Mar 12, 2024 02:14:36 org.ignis.backend.cluster.helpers.cluster.IClusterCreateHelper.create(IClusterCreateHelper.java:53): Cluster(0): Registering cluster with 1 containers Mar 12, 2024 02:14:36 org.ignis.backend.cluster.helpers.worker.IWorkerCreateHelper.create(IWorkerCreateHelper.java:57): Cluster(0), Worker(0): Registering worker with 1 executors with 1 cores panic: stat /home/shenwei/go/src/github.com/shenwei356/seqkit/tests/hairpin.fa: no such file or directory goroutine 1 [running]: main.checkError(...) /tmp/src/bigseqkit-cli/helper.go:34 main.getFileList({0xc000344720, 0x3, 0x3}, 0x1) /tmp/src/bigseqkit-cli/helper.go:193 +0x105 main.getFileListFromArgsAndFile(0xc000722390?, {0xc000344720, 0x3, 0x3}, 0x0?, {0x12b5059?, 0xd7e260?}, 0x40?) /tmp/src/bigseqkit-cli/helper.go:231 +0x79 main.readSeqs(0xc0007280e8?, {0xc000344720?, 0x2?, 0x0?}, 0x4?) /tmp/src/bigseqkit-cli/helper.go:56 +0x9a main.ignisDriver(0x108f840?, {0xc000344720, 0x3, 0x3}, 0x1305858) /tmp/src/bigseqkit-cli/helper.go:104 +0x315 main.init.17.func1.1(0xc000348000?, {0xc000344720?, 0x3?, 0x3?}) /tmp/src/bigseqkit-cli/seq.go:48 +0x25 github.com/spf13/cobra.(*Command).execute(0xc000348000, {0xc0003446f0, 0x3, 0x3}) /root/go/pkg/mod/github.com/spf13/cobra@v1.2.1/command.go:860 +0x663 github.com/spf13/cobra.(*Command).ExecuteC(0xc0000d4c80) /root/go/pkg/mod/github.com/spf13/cobra@v1.2.1/command.go:974 +0x3bd github.com/spf13/cobra.(*Command).Execute(...) /root/go/pkg/mod/github.com/spf13/cobra@v1.2.1/command.go:902 main.main() /tmp/src/bigseqkit-cli/main.go:4 +0x1e Mar 12, 2024 02:14:36 org.apache.thrift.transport.TIOStreamTransport.close(TIOStreamTransport.java:159): Error closing output stream. java.net.SocketException: Socket closed at sun.nio.ch.NioSocketImpl.ensureOpenAndConnected(Unknown Source) ~[?:?] at sun.nio.ch.NioSocketImpl.beginWrite(Unknown Source) ~[?:?] at sun.nio.ch.NioSocketImpl.implWrite(Unknown Source) ~[?:?] at sun.nio.ch.NioSocketImpl.write(Unknown Source) ~[?:?] at sun.nio.ch.NioSocketImpl$2.write(Unknown Source) ~[?:?] at java.net.Socket$SocketOutputStream.write(Unknown Source) ~[?:?] at java.io.BufferedOutputStream.flushBuffer(Unknown Source) ~[?:?] at java.io.BufferedOutputStream.flush(Unknown Source) ~[?:?] at java.io.FilterOutputStream.close(Unknown Source) ~[?:?] at org.apache.thrift.transport.TIOStreamTransport.close(TIOStreamTransport.java:157) [libthrift-0.15.0.jar:0.15.0] at org.apache.thrift.transport.TSocket.close(TSocket.java:252) [libthrift-0.15.0.jar:0.15.0] at org.apache.thrift.transport.TZlibTransport.close(TZlibTransport.java:83) [libthrift-0.15.0.jar:0.15.0] at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:267) [libthrift-0.15.0.jar:0.15.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?] at java.lang.Thread.run(Unknown Source) [?:?] Mar 12, 2024 02:14:36 org.ignis.backend.services.IBackendServiceImpl.stopAll(IBackendServiceImpl.java:86): Stopping Backend server Mar 12, 2024 02:14:36 org.apache.thrift.transport.TIOStreamTransport.close(TIOStreamTransport.java:159): Error closing output stream. java.io.IOException: org.apache.thrift.transport.TTransportException: Cannot write to null outputStream at org.apache.thrift.transport.TTransportOutputStream.write(TZlibTransport.java:139) ~[libthrift-0.15.0.jar:0.15.0] at java.util.zip.DeflaterOutputStream.deflate(Unknown Source) ~[?:?] at java.util.zip.DeflaterOutputStream.finish(Unknown Source) ~[?:?] at java.util.zip.DeflaterOutputStream.close(Unknown Source) ~[?:?] at org.apache.thrift.transport.TIOStreamTransport.close(TIOStreamTransport.java:157) [libthrift-0.15.0.jar:0.15.0] at org.apache.thrift.transport.TZlibTransport.close(TZlibTransport.java:81) [libthrift-0.15.0.jar:0.15.0] at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:270) [libthrift-0.15.0.jar:0.15.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?] at java.lang.Thread.run(Unknown Source) [?:?] Caused by: org.apache.thrift.transport.TTransportException: Cannot write to null outputStream at org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:194) ~[libthrift-0.15.0.jar:0.15.0] at org.apache.thrift.transport.TTransportOutputStream.write(TZlibTransport.java:137) ~[libthrift-0.15.0.jar:0.15.0] ... 9 more Mar 12, 2024 02:14:36 org.ignis.backend.services.IBackendServiceImpl.stopHealthServer(IBackendServiceImpl.java:127): Backend health server stopped Mar 12, 2024 02:14:36 org.ignis.backend.services.IBackendServiceImpl.start(IBackendServiceImpl.java:71): Backend server stopped Mar 12, 2024 02:14:36 org.ignis.backend.cluster.helpers.cluster.IClusterDestroyHelper.destroy(IClusterDestroyHelper.java:41): Cluster(0): Preparing cluster to destroy Mar 12, 2024 02:14:36 org.ignis.backend.cluster.helpers.worker.IWorkerDestroyHelper.destroy(IWorkerDestroyHelper.java:38): Cluster(0), Worker(0): Preparing worker to destroy Mar 12, 2024 02:14:36 org.ignis.backend.cluster.helpers.cluster.IClusterDestroyHelper.lambda$destroy$0(IClusterDestroyHelper.java:57): Cluster(0): Destroying cluster with 1 containers Mar 12, 2024 02:14:36 org.ignis.backend.cluster.helpers.cluster.IClusterDestroyHelper.lambda$destroy$0(IClusterDestroyHelper.java:59): Cluster(0): Cluster destroyed Mar 12, 2024 02:14:36 org.ignis.backend.Main.main(Main.java:176): Backend stopped
cesarpomar commented 8 months ago

Using Docker containers, only the IGNIS_DFS_ID is mounted and set as the working directory. Therefore, you can reference files within that directory using relative paths.

You can use IgnisHPC properties to manually mount other directories using -p just after the script.

...ignis-submit -p ignis.driver.bind.{containerpath}={hostpath} -p ignis.executor.bind.{containerpath}={hostpath}...

We're working on a simpler implementation, but it's not ready yet.