molgenis / molgenis-compute

MOLGENIS Compute is a framework for bioinformatics which enables large scale data and computational workflow management in a distributed execution environment.
http://wiki.gcc.rug.nl/wiki/ComputeStart
GNU Lesser General Public License v3.0
4 stars 16 forks source link

removing .env files #277

Open RoanKanninga opened 6 years ago

RoanKanninga commented 6 years ago

Removing out of generators/impl/TaskGenerator.java

// source its environment
                        scriptBuilder.append(Parameters.SOURCE_COMMAND).append(" ")
                                .append(Parameters.ENVIRONMENT_DIR_VARIABLE).append(File.separator).append(prevJobName)
                                .append(Parameters.ENVIRONMENT_EXTENSION).append("\n");
scriptBuilder.append("\n\n# Connect parameters to environment\n");

// append footer that appends the task's parameters to
            // environment of this task
            String myEnvironmentFile = Parameters.ENVIRONMENT_DIR_VARIABLE + File.separator + task.getName()
                    + Parameters.ENVIRONMENT_EXTENSION;
            scriptBuilder.append("\n#\n## End of your protocol template\n#\n");
            scriptBuilder.append("\n# Save output in environment file: '" + myEnvironmentFile
                    + "' with the output vars of this step\n");

            Iterator<Output> itOutput = step.getProtocol().getOutputs().iterator();
            while (itOutput.hasNext())
            {
                String parameterName = itOutput.next().getName();
                if (dataEntityValues.containsKey(parameterName))
                {
                    // If parameter not set at runtime then ERROR
                    String line = "if [[ -z \"$" + parameterName + "\" ]]; then echo \"In step '" + step.getName()
                            + "', parameter '" + parameterName + "' has no value! Please assign a value to parameter '"
                            + parameterName + "'." + "\" >&2; exit 1; fi\n";

                    // Else set parameters at right indexes.
                    // Explanation: if param file is collapsed in this
                    // template, then we should not output a single
                    // value but a list of values because next step may
                    // be run in uncollapsed fashion

                    List<String> rowIndex = dataEntity.getList(Parameters.ID_COLUMN);
                    for (int i = 0; i < rowIndex.size(); i++)
                    {
                        Object rowIndexObject = rowIndex.get(i);
                        String rowIndexString = rowIndexObject.toString();
                        line += "echo \"" + step.getName() + Parameters.STEP_PARAM_SEP_SCRIPT + parameterName + "["
                                + rowIndexString + "]=\\\"${" + parameterName + "[" + i + "]}\\\"\" >> "
                                + myEnvironmentFile + "\n";
                    }

                    scriptBuilder.append(line);
                }

            }
            scriptBuilder.append("\necho \"\" >> " + myEnvironmentFile + "\nchmod 755 " + myEnvironmentFile + "\n");```
pneerincx commented 6 years ago

Why disabling the creation of .env files. This would kill the feature to pass dynamic values computed at runtime of the script from one analysis step to the next. I would prefer a fix to make sure the .env files are created only when necessarry (and actually used) and with the correct permissions.

freerkvandijk commented 6 years ago

You only want to have/create the .env files when needed in an analysis step. So I would assume you only check if the .env files are present in your generated script when you actually need parameters values from them, instead of completely removing the dynamic values feature.

On 17 October 2017 at 10:06, Pieter Neerincx notifications@github.com wrote:

Why disabling the creation of .env files. This would kill the feature to pass dynamic values computed at runtime of the script from one analysis step to the next. I would prefer a fix to make sure the .env files are created only when necessarry (and actually used) and with the correct permissions.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/molgenis/molgenis-compute/issues/277#issuecomment-337151764, or mute the thread https://github.com/notifications/unsubscribe-auth/ABxneW9781Tbz17_GTqAX4aXCx7ipc6pks5stGAegaJpZM4P7ox6 .

RoanKanninga commented 6 years ago

Fine by me, but you can already pass dynamic values from the generatedscript as a parameter via the -o argument in compute. Where do you want to set other dynamic values?