Error in Nextflow workflow: parameterTemplate in wrong format

[AleSR13]

Hello,

I was following part of the tutorial to be able to test a Nextflow workflow with AWS genomics (notebooks/200-omics_workflows.ipynb) and it failed with the error:

botocore.exceptions.ParamValidationError: Parameter validation failed:
Invalid type for parameter parameterTemplate.greeting, value: (string) greeting to use, type: <class 'str'>, valid types: <class 'dict'>
Invalid type for parameter parameterTemplate.addressee, value: (string) who to greet, type: <class 'str'>, valid types: <class 'dict'>

I realized that the way you pass parameters is not right. Instead of:

{
    "greeting": "(string) greeting to use",
    "addressee": "(string) who to greet"
}

It should be:

{
    "greeting": {
        "description": "(string) greeting to use"
    }
    "addressee": {
        "description": "(string) who to greet"
    }
}

I modified it in my fork but notebooks are not great for version control so I don't know if a pull request is valuable.

After that, the workflow ran fine. However, a major issue that I see is that I cannot find my output anywhere!!! It ran properly but it is not in my output_uri (s3 bucket). Any help finding it? As it is, the notebook shows how to create workflow and run it but without output (I did not change anything in your workflow example, so the publishDir is '/mnt/workflow/pubdir'), it is not very useful. I had the same problem with a custom nextflow workflow using containers where I could also not find my output. I was wondering whether I needed to use containerOptions and do a mount but first I decided to try your tutorial. I cannot find an answer in the official AWS documentation on how they "sync" the /mnt/workflow/pubdir with your output_uri. Any help would be very much appreciated!

[AleSR13]

I found the problem with the publishing! My policy was allowing write access to only a specific folder, so it ran successfully but it didn't give any error when failing to publish in S3. Still the other problem stands I think 😄

[wleepang]

Thanks for pointing this out and submitting a pull request! The original parameter format was what was used during our beta and was updated just prior to our launch release.

Regarding your outputs, /mnt/workflow/pubdir is a path recognized by the engine used by Omics Workflows for Nextflow. Any process with a publishDir directive set to this location will have its outputs written to output_uri. The IAM role used by the run needs to have write permissions to that location. I'll double check if that currently needs more than what is specified in the notebook.

[AleSR13]

Hi @wleepang , I ended up finding my output. It was a silent error in my workflow. However, I have to say it is VERY difficult to debug the workflows. I cannot find all the output and error messages of Nextflow and since the working directory is ephemeral, I cannot find the complete log files of my command either. Sometimes something breaks (probably due to a core dump or god knows what) without giving a proper error message and I have no way to debug.

Also, if you get an error while making a new workflow or a task within your run, the information about the error is not easy to find (is it even stored somewhere? Definitely not in CloudWatch /aws/omics/WorkflowLog since I have read every single file that is in there). Anyway, your examples work very nicely! It is more like the more complicated real-life workflows what is difficult to debug! Thanks for the answer and for putting together these examples!

[wleepang]

Closing this issue. Feel free to re-open if needed.