Singularity testing

[rssk]

Task Description

Test

• [ ] vbm preprocesser
• [ ] vbm regression
• [ ] decentralized mancova Data can be found in each computation repo

[dylanmartin]

VBM preprocessor

How should the computation step be configured?

Is this the right mapping?

[dylanmartin]

There are 2 issues here:

1. The download failed for some reason
2. It appears that the file exists and therefor passes the isSingularityImageLatest check but the image is bad

[dylanmartin]

rm -rf the image gets permission denied sudo works

[dylanmartin]

It looks like permission is denied to add the files when downloading

[dylanmartin]

Adding the --fix-perms option so singularity build was a suggested solution in the error text. That seemed to potentially fix the problem but now there's an error saying 'not enough space on device' I'll clear some space to see if it works. There is a warning in the error text that says: WARNING: The --fix-perms option modifies the filesystem permissions on the resulting container. Let's revisit to make sure it isn't a security issue.

[dylanmartin]

We may need more space on this volume

[dylanmartin]

Increased size of storage on the EC2 instance from 30GB to 60GB. No longer running int device space limits.

[dylanmartin]

Running VBM pre

[dylanmartin]

It looks like the docker image for the remote isn't pulled

[dylanmartin]

[dylanmartin]

Computation is running

[dylanmartin]

Is this the expected result from VBM pre?

[dylanmartin]

@rssk which of these computations is "VBM regression"? I'm guessing it's Regression - VBM

[dylanmartin]

Configuring the VBM regression pipeline step this way:

[dylanmartin]

Files are showing up in the wrong place

[dylanmartin]

VBM preprocessor is taking a while. Moving on to VBM regression to see if that completes. Using this configuration:

[dylanmartin]

[dylanmartin]

[dylanmartin]

It isn't necessarily true that these runs are hanging because we're using Singularity. It's likely there is some other issue.

[dylanmartin]

https://github.com/trendscenter/coinstac/issues/1693

[dylanmartin]

[dylanmartin]

Line 79 in the startscript is exec $@. To see the contents of $@, the values can be echoed and sent to the logs. The container must be rebuilt from this container recipe to run the modified startscript.

[dylanmartin]

Modify runscript and not startscript

[dylanmartin]

Echo successfully logged. It looks like a json object is being passed into the exec command. Are these the expected arguments?

[dylanmartin]

Are these the correct args?

[dylanmartin]

[dylanmartin]

ServiceFunctionGenerator is not getting a compspecVersion argument so it defaults to 1. the opts parameter to createService does contain a key called version that appears to reflect the compspec version.

[dylanmartin]

After feeding the correct compspecVersion to serviceFunctionGenerator, the code goes down compspecVersion 2 switch case. it then encounters this error

The error executing the exec function in the startscript happens as well.

[dylanmartin]

This is what gets passed to spawn when starting the container

[dylanmartin]

Are these the commands that need to be executed by the exec line in the runscript?

[rssk]

• [ ] ssr regression, compspec type 2 https://github.com/trendscenter/coinstac-regression-fsl
• [ ] vbm preprocessor, compspec type 1 https://github.com/trendscenter/Coinstac_VBM_computation
• [ ] vbm regression, compspec type 2 https://github.com/trendscenter/coinstac-regression-vbm
• [ ] decentralized mancova, compspec type 1 https://github.com/trendscenter/coinstac-mancova/tree/master/coinstac_mancova

[dylanmartin]

We must not be running from the .sif

[dylanmartin]

ws error not bubbling up

[dylanmartin]

The python server inside the singularity container for compspecVersion 2 needs to know what port to listen on for ws connections.

[rssk]

Investigate if a non zero exit code process triggers singularity instance to propagate or other show the error

• what happens when a container cannot be immediately started? (errors on startup) -- needs more testing -- immediately crashing containers leave logs, but no other events? -- fixed this? for whatever reason singularity has its stderr and stdout streams swapped leading to the error output being missed, reversed these and seems fine for now. Asked the singularity devs whats up
• what happens when a container exits randomly? -- needs more testing
• what happens when a pipeline computation internally fails and throws? -- exits to the UI as expected 👍

[dylanmartin]

The above comment is referring to the entry command script. For example:

    "command": [
      "python",
      "\/computation\/scripts\/local.py"
    ],

[rssk]

Got vbm working, had to fix the passed in env to singularity's start, they need to be comma separated --env B=1,C=1