search

rabix / bunny

[Legacy] Executor for CWL workflows. Executes sbg:draft-2 and CWL 1.0
http://rabix.io
Apache License 2.0
74 stars 28 forks source link

Provider "s3" not found #417

Closed adamstruck closed 6 years ago

adamstruck commented 6 years ago

I am still getting this error frequently when I build bunny on a new VM. My build command is mvn package -P tes -P cwl. Did the build process change since you merged S3 support into master? 

Feb 08, 2018 11:29:27 PM com.google.inject.internal.MessageProcessor visit
INFO: An exception was caught and reported. Message: java.nio.file.ProviderNotFoundException: Provider "s3" not found
java.nio.file.ProviderNotFoundException: Provider "s3" not found
        at java.nio.file.FileSystems.newFileSystem(FileSystems.java:341)
        at org.rabix.backend.tes.TESModule.configure(TESModule.java:41)
        at com.google.inject.AbstractModule.configure(AbstractModule.java:62)
        at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:340)
        at com.google.inject.AbstractModule.install(AbstractModule.java:122)
        at org.rabix.cli.BackendCommandLine$1.configure(BackendCommandLine.java:251)
        at com.google.inject.AbstractModule.configure(AbstractModule.java:62)
        at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:340)
        at com.google.inject.spi.Elements.getElements(Elements.java:110)
        at com.google.inject.internal.InjectorShell$Builder.build(InjectorShell.java:138)
        at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:104)
        at com.google.inject.Guice.createInjector(Guice.java:99)
        at com.google.inject.Guice.createInjector(Guice.java:73)
        at com.google.inject.Guice.createInjector(Guice.java:62)
        at org.rabix.cli.BackendCommandLine.main(BackendCommandLine.java:226)

Exception in thread "main" com.google.inject.CreationException: Unable to create injector, see the following errors:

1) An exception was caught and reported. Message: Provider "s3" not found
  at org.rabix.cli.BackendCommandLine$1.configure(BackendCommandLine.java:251) (via modules: org.rabix.cli.BackendCommandLine$1 -> org.rabix.backend.tes.TESModule)

1 error
        at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:470)
        at com.google.inject.internal.InternalInjectorCreator.initializeStatically(InternalInjectorCreator.java:155)
        at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:107)
        at com.google.inject.Guice.createInjector(Guice.java:99)
        at com.google.inject.Guice.createInjector(Guice.java:73)
        at com.google.inject.Guice.createInjector(Guice.java:62)
        at org.rabix.cli.BackendCommandLine.main(BackendCommandLine.java:226)
Caused by: java.nio.file.ProviderNotFoundException: Provider "s3" not found
        at java.nio.file.FileSystems.newFileSystem(FileSystems.java:341)
        at org.rabix.backend.tes.TESModule.configure(TESModule.java:41)
        at com.google.inject.AbstractModule.configure(AbstractModule.java:62)
        at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:340)
        at com.google.inject.AbstractModule.install(AbstractModule.java:122)
        at org.rabix.cli.BackendCommandLine$1.configure(BackendCommandLine.java:251)
        at com.google.inject.AbstractModule.configure(AbstractModule.java:62)
        at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:340)
        at com.google.inject.spi.Elements.getElements(Elements.java:110)
        at com.google.inject.internal.InjectorShell$Builder.build(InjectorShell.java:138)
        at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:104)
        ... 4 more
adamstruck commented 6 years ago

I was able to address the above issue by adding the file rabix-backend-tes/src/main/resources/META-INF/services/java.nio.file.spi.FileSystemProvider

The contents of the file:

com.upplication.s3fs.S3FileSystemProvider
adamstruck commented 6 years ago

With the above fix(?) I now encounter the following:

In my config: rabix.tes.storage.base=s3://s3.amazonaws.com/dev-kf-data

[2018-02-09 00:50:35.402] [ERROR] Failed to use Bindings
org.rabix.bindings.BindingException: org.rabix.bindings.cwl.processor.CWLPortProcessorException: org.rabix.bindings.cwl.processor.CWLPortProcessorException: Error: /dev-kf-data/sequence_grouping.txt (No such file or directory) while processing value: {metadata=null, format=null, dirname=null, nameroot=sequence_grouping, path=/dev-kf-data/sequence_grouping.txt, basename=sequence_grouping.txt, nameext=.txt, size=89451, contents=null, checksum=null, location=s3://s3.amazonaws.com/dev-kf-data/sequence_grouping.txt, secondaryFiles=[], class=File}
    at org.rabix.bindings.cwl.CWLProcessor.preprocess(CWLProcessor.java:107) ~[rabix-cli-1.0.4.jar:na]
    at org.rabix.bindings.cwl.CWLBindings.preprocess(CWLBindings.java:79) ~[rabix-cli-1.0.4.jar:na]
    at org.rabix.backend.tes.service.impl.LocalTESWorkerServiceImpl$TaskRunCallable.call(LocalTESWorkerServiceImpl.java:236) [rabix-cli-1.0.4.jar:na]
    at org.rabix.backend.tes.service.impl.LocalTESWorkerServiceImpl$TaskRunCallable.call(LocalTESWorkerServiceImpl.java:220) [rabix-cli-1.0.4.jar:na]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_151]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_151]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_151]
    at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151]
Caused by: org.rabix.bindings.cwl.processor.CWLPortProcessorException: org.rabix.bindings.cwl.processor.CWLPortProcessorException: Error: /dev-kf-data/sequence_grouping.txt (No such file or directory) while processing value: {metadata=null, format=null, dirname=null, nameroot=sequence_grouping, path=/dev-kf-data/sequence_grouping.txt, basename=sequence_grouping.txt, nameext=.txt, size=89451, contents=null, checksum=null, location=s3://s3.amazonaws.com/dev-kf-data/sequence_grouping.txt, secondaryFiles=[], class=File}
    at org.rabix.bindings.cwl.processor.callback.CWLPortProcessorHelper.loadInputContents(CWLPortProcessorHelper.java:86) ~[rabix-cli-1.0.4.jar:na]
    at org.rabix.bindings.cwl.CWLProcessor.preprocess(CWLProcessor.java:98) ~[rabix-cli-1.0.4.jar:na]
    ... 7 common frames omitted
Caused by: org.rabix.bindings.cwl.processor.CWLPortProcessorException: Error: /dev-kf-data/sequence_grouping.txt (No such file or directory) while processing value: {metadata=null, format=null, dirname=null, nameroot=sequence_grouping, path=/dev-kf-data/sequence_grouping.txt, basename=sequence_grouping.txt, nameext=.txt, size=89451, contents=null, checksum=null, location=s3://s3.amazonaws.com/dev-kf-data/sequence_grouping.txt, secondaryFiles=[], class=File}
    at org.rabix.bindings.cwl.processor.CWLPortProcessor.processValues(CWLPortProcessor.java:56) ~[rabix-cli-1.0.4.jar:na]
    at org.rabix.bindings.cwl.processor.CWLPortProcessor.processInputs(CWLPortProcessor.java:31) ~[rabix-cli-1.0.4.jar:na]
    at org.rabix.bindings.cwl.processor.callback.CWLPortProcessorHelper.loadInputContents(CWLPortProcessorHelper.java:84) ~[rabix-cli-1.0.4.jar:na]
    ... 8 common frames omitted
Caused by: java.io.FileNotFoundException: /dev-kf-data/sequence_grouping.txt (No such file or directory)
    at java.io.FileInputStream.open0(Native Method) ~[na:1.8.0_151]
    at java.io.FileInputStream.open(FileInputStream.java:195) ~[na:1.8.0_151]
    at java.io.FileInputStream.<init>(FileInputStream.java:138) ~[na:1.8.0_151]
    at org.rabix.bindings.cwl.helper.CWLFileValueHelper.loadContents(CWLFileValueHelper.java:235) ~[rabix-cli-1.0.4.jar:na]
    at org.rabix.bindings.cwl.helper.CWLFileValueHelper.setContents(CWLFileValueHelper.java:138) ~[rabix-cli-1.0.4.jar:na]
    at org.rabix.bindings.cwl.processor.callback.CWLLoadContentsPortProcessorCallback.process(CWLLoadContentsPortProcessorCallback.java:25) ~[rabix-cli-1.0.4.jar:na]
    at org.rabix.bindings.cwl.processor.CWLPortProcessor.processValue(CWLPortProcessor.java:70) ~[rabix-cli-1.0.4.jar:na]
    at org.rabix.bindings.cwl.processor.CWLPortProcessor.processValues(CWLPortProcessor.java:54) ~[rabix-cli-1.0.4.jar:na]
    ... 10 common frames omitted
[2018-02-09 00:50:35.698] [ERROR] Failed to retrieve TESTask
java.util.concurrent.ExecutionException: org.rabix.backend.tes.service.TESServiceException: Failed to use Bindings
    at java.util.concurrent.FutureTask.report(FutureTask.java:122) [na:1.8.0_151]
    at java.util.concurrent.FutureTask.get(FutureTask.java:192) [na:1.8.0_151]
    at org.rabix.backend.tes.service.impl.LocalTESWorkerServiceImpl$1.run(LocalTESWorkerServiceImpl.java:169) ~[rabix-cli-1.0.4.jar:na]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_151]
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_151]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_151]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_151]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_151]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_151]
    at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151]
Caused by: org.rabix.backend.tes.service.TESServiceException: Failed to use Bindings
    at org.rabix.backend.tes.service.impl.LocalTESWorkerServiceImpl$TaskRunCallable.call(LocalTESWorkerServiceImpl.java:302) ~[rabix-cli-1.0.4.jar:na]
    at org.rabix.backend.tes.service.impl.LocalTESWorkerServiceImpl$TaskRunCallable.call(LocalTESWorkerServiceImpl.java:220) ~[rabix-cli-1.0.4.jar:na]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_151]
    ... 3 common frames omitted
Caused by: org.rabix.bindings.BindingException: org.rabix.bindings.cwl.processor.CWLPortProcessorException: org.rabix.bindings.cwl.processor.CWLPortProcessorException: Error: /dev-kf-data/sequence_grouping.txt (No such file or directory) while processing value: {metadata=null, format=null, dirname=null, nameroot=sequence_grouping, path=/dev-kf-data/sequence_grouping.txt, basename=sequence_grouping.txt, nameext=.txt, size=89451, contents=null, checksum=null, location=s3://s3.amazonaws.com/dev-kf-data/sequence_grouping.txt, secondaryFiles=[], class=File}
    at org.rabix.bindings.cwl.CWLProcessor.preprocess(CWLProcessor.java:107) ~[rabix-cli-1.0.4.jar:na]
    at org.rabix.bindings.cwl.CWLBindings.preprocess(CWLBindings.java:79) ~[rabix-cli-1.0.4.jar:na]
    at org.rabix.backend.tes.service.impl.LocalTESWorkerServiceImpl$TaskRunCallable.call(LocalTESWorkerServiceImpl.java:236) ~[rabix-cli-1.0.4.jar:na]
    ... 5 common frames omitted
Caused by: org.rabix.bindings.cwl.processor.CWLPortProcessorException: org.rabix.bindings.cwl.processor.CWLPortProcessorException: Error: /dev-kf-data/sequence_grouping.txt (No such file or directory) while processing value: {metadata=null, format=null, dirname=null, nameroot=sequence_grouping, path=/dev-kf-data/sequence_grouping.txt, basename=sequence_grouping.txt, nameext=.txt, size=89451, contents=null, checksum=null, location=s3://s3.amazonaws.com/dev-kf-data/sequence_grouping.txt, secondaryFiles=[], class=File}
    at org.rabix.bindings.cwl.processor.callback.CWLPortProcessorHelper.loadInputContents(CWLPortProcessorHelper.java:86) ~[rabix-cli-1.0.4.jar:na]
    at org.rabix.bindings.cwl.CWLProcessor.preprocess(CWLProcessor.java:98) ~[rabix-cli-1.0.4.jar:na]
    ... 7 common frames omitted
Caused by: org.rabix.bindings.cwl.processor.CWLPortProcessorException: Error: /dev-kf-data/sequence_grouping.txt (No such file or directory) while processing value: {metadata=null, format=null, dirname=null, nameroot=sequence_grouping, path=/dev-kf-data/sequence_grouping.txt, basename=sequence_grouping.txt, nameext=.txt, size=89451, contents=null, checksum=null, location=s3://s3.amazonaws.com/dev-kf-data/sequence_grouping.txt, secondaryFiles=[], class=File}
    at org.rabix.bindings.cwl.processor.CWLPortProcessor.processValues(CWLPortProcessor.java:56) ~[rabix-cli-1.0.4.jar:na]
    at org.rabix.bindings.cwl.processor.CWLPortProcessor.processInputs(CWLPortProcessor.java:31) ~[rabix-cli-1.0.4.jar:na]
    at org.rabix.bindings.cwl.processor.callback.CWLPortProcessorHelper.loadInputContents(CWLPortProcessorHelper.java:84) ~[rabix-cli-1.0.4.jar:na]
    ... 8 common frames omitted
Caused by: java.io.FileNotFoundException: /dev-kf-data/sequence_grouping.txt (No such file or directory)
    at java.io.FileInputStream.open0(Native Method) ~[na:1.8.0_151]
    at java.io.FileInputStream.open(FileInputStream.java:195) ~[na:1.8.0_151]
    at java.io.FileInputStream.<init>(FileInputStream.java:138) ~[na:1.8.0_151]
    at org.rabix.bindings.cwl.helper.CWLFileValueHelper.loadContents(CWLFileValueHelper.java:235) ~[rabix-cli-1.0.4.jar:na]
    at org.rabix.bindings.cwl.helper.CWLFileValueHelper.setContents(CWLFileValueHelper.java:138) ~[rabix-cli-1.0.4.jar:na]
    at org.rabix.bindings.cwl.processor.callback.CWLLoadContentsPortProcessorCallback.process(CWLLoadContentsPortProcessorCallback.java:25) ~[rabix-cli-1.0.4.jar:na]
    at org.rabix.bindings.cwl.processor.CWLPortProcessor.processValue(CWLPortProcessor.java:70) ~[rabix-cli-1.0.4.jar:na]
    at org.rabix.bindings.cwl.processor.CWLPortProcessor.processValues(CWLPortProcessor.java:54) ~[rabix-cli-1.0.4.jar:na]
    ... 10 common frames omitted
adamstruck commented 6 years ago

So I tracked the above error down. Looks like its failing on an ExpressionTool becasue it can't load the contents of the remote file. Here is the tool in question:

cwlVersion: v1.0
class: ExpressionTool
id: expression_createsequencegrouping
requirements:
  - class: InlineJavascriptRequirement

inputs:
  sequence_grouping_tsv:
    type: File
    inputBinding:
      loadContents: true

outputs:
  sequence_grouping_array:
    type:
      type: array
      items:
        type: array
        items: string

expression:
  "${
      var lines = inputs.sequence_grouping_tsv.contents.split('\\n');
      var nline = lines.length;
      var eachgroup = [];
      var grouparray = [];
      for (var i = 0; i < nline; i++) {
        eachgroup.push(lines[i].split('\\t'));
        grouparray[i] = lines[i].split('\\t');
      }
      return {'sequence_grouping_array': grouparray};
  }"