[ANT-525] Xpansion Sensitivity not finding study output

JasonMarechal25 commented 1 year ago

Description

Provide a clear and concise description of the issue.

Steps to Reproduce

Steps to reproduce the behavior:

Start a new run of a study
Select Xpansion mode
Select sensitivity analysis
Select a simulation output
Run

Actual Behavior

Simulation fail. Xpansion report the following error:

[22743] Failed to execute script 'launch' due to unhandled exception!
Traceback (most recent call last):
  File "launch.py", line 39, in <module>
  File "antares_xpansion/driver.py", line 91, in launch
  File "antares_xpansion/driver.py", line 149, in launch_sensitivity_step
  File "antares_xpansion/config_loader.py", line 408, in _create_sensitivity_dir
  File "antares_xpansion/config_loader.py", line 420, in _sensitivity_dir
  File "antares_xpansion/config_loader.py", line 339, in xpansion_simulation_output
  File "antares_xpansion/config_loader.py", line 489, in _set_last_simulation_name
  File "antares_xpansion/config_loader.py", line 534, in get_last_modified_dir
IndexError: list index out of range

This means that id didn't find the output zip

Expected Behavior

Xpansion sensitivity analysis performed properly

Possible Solution

The path to the simulation given in command line may not be good. Here is a command line example: ./antares-xpansion-launcher --step sensitivity -i ../simulation/ --simulationName 20230302-1618eco.zip

The -i parameter is the path to an Antares Study. The --simulationName is the name of an output zip archive in the output directory of the study given in the previous parameter E.g. /path/to/study/simulation/output/20230302-1618eco.zip

Environment

~Production~
Staging

Additional Information

Add any other context about the problem here.

If applicable, provide the study ID that this issue relates to: 0b95e9d0-571b-4906-b2b6-b7de4fb9f2ca

laurent-laporte-pro commented 1 year ago

Le script antares-xpansion-launcher est actuellement lancé par Antares Launcher en utilisant le script installé sur la machine SLURM. Ce script est généré à la volée lors de la construction de l'image Docker (dans antares-cd).

Voici un extrait de ce script :

if [ "$JOB_TYPE" = "ANTARES_XPANSION" ]; then
  if (( $ANTARES_VERSION >= 800 )); then
    antares-xpansion-launcher -i ${STUDY_PATH} ${XPANSION_SENSITIVITY_OPT} -m mpibenders -n ${SLURM_CPUS_PER_TASK} --antares-n-cpu ${SLURM_CPUS_PER_TASK} --oversubscribe
  else
    srun Rscript $XPANSION_RSCRIPT --study="$STUDY_PATH" --path_antares=$ANTARES_SOLVER --n-cpu=${SLURM_CPUS_PER_TASK}
    # just to have code 0
    ls
  fi
elif [ "$JOB_TYPE" = "ANTARES_XPANSION_CPP" ]; then
    antares-xpansion-launcher -i ${STUDY_PATH} ${XPANSION_SENSITIVITY_OPT} -m mpibenders -n ${SLURM_CPUS_PER_TASK} --antares-n-cpu ${SLURM_CPUS_PER_TASK} --oversubscribe
elif [ "$JOB_TYPE" = "ANTARES_XPANSION_R" ]; then
    srun Rscript $XPANSION_RSCRIPT --study="$STUDY_PATH" --path_antares=$ANTARES_SOLVER --n-cpu=${SLURM_CPUS_PER_TASK}
  # just to have code 0
    ls
else
  if has_opt "xpress" ; then
    module load ortools/8.13.0
    srun antares-solver --force-parallel=${SLURM_CPUS_PER_TASK} --use-ortools --ortools-solver xpress -i "$STUDY_PATH"
  elif has_opt "zip" ; then
    srun antares-solver --force-parallel=${SLURM_CPUS_PER_TASK} -z -i "$STUDY_PATH"
  else
    srun antares-solver --force-parallel=${SLURM_CPUS_PER_TASK} -i "$STUDY_PATH"
  fi
fi

laurent-laporte-pro commented 1 year ago

According to the documentation, the --simulationName parameter is only used when the step is problem_generation, benders, study_update.

Again, the documentation states that if the parameter is not provided, Xpansion should use the "last" one: in that case, the most recent run will be used. So, I think that this is an Xpansion issue.

laurent-laporte-pro commented 1 year ago

Parameters used by Antares Launcher compared to the expected parameters:	Actual parameters	Expected parameters
antares-xpansion-launcher	antares-xpansion-launcher	Script name
-i ${STUDY_PATH}	-i ../simulation/	Specifies the Antares study folder path. Use quotes “antares study path” in case of a space in the path.
${XPANSION_SENSITIVITY_OPT}	--step sensitivity	Launch sensitivity analysis, see Sensitivity analysis.
-m mpibenders		Launch the MPI version of Benders decomposition if the user has MPI. Use the option -n or --np to set the number of processes to use.
-n ${SLURM_CPUS_PER_TASK}		Sets the number of MPI processes to use for the MPI version of Benders decomposition. This option only has an effect when -m is set to mpibenders.
--antares-n-cpu ${SLURM_CPUS_PER_TASK}		Sets the number of threads to use for Antares-Simulator in the initial antares step.
--oversubscribe		Enable mpi oversubscribe option (linux only), default = False
	--simulationName 20230302-1618eco.zip	Defines the name of the Antares-Simulator output directory that Antares-Xpansion uses to generate the expansion problem. If the value is last, the most recent run will be used. This option only has an effect when --step is among {problem_generation, benders, study_update}.

JasonMarechal25 commented 1 year ago

According to the documentation, the --simulationName parameter is only used when the step is problem_generation, benders, study_update.

Again, the documentation states that if the parameter is not provided, Xpansion should use the "last" one: in that case, the most recent run will be used. So, I think that this is an Xpansion issue.

This is a mistake in documentation. The value is used for sensitivity step too

JasonMarechal25 commented 1 year ago

Parameters used by Antares Launcher compared to the expected parameters: Actual parameters Expected parameters Descriptions (documentation) antares-xpansion-launcher antares-xpansion-launcher Script name -i ${STUDY_PATH} -i ../simulation/ Specifies the Antares study folder path. Use quotes “antares study path” in case of a space in the path. ${XPANSION_SENSITIVITY_OPT} --step sensitivity Launch sensitivity analysis, see Sensitivity analysis. -m mpibenders Launch the MPI version of Benders decomposition if the user has MPI. Use the option -n or --np to set the number of processes to use. -n ${SLURM_CPUS_PER_TASK} Sets the number of MPI processes to use for the MPI version of Benders decomposition. This option only has an effect when -m is set to mpibenders. --antares-n-cpu ${SLURM_CPUS_PER_TASK} Sets the number of threads to use for Antares-Simulator in the initial antares step. --oversubscribe Enable mpi oversubscribe option (linux only), default = False --simulationName 20230302-1618eco.zip Defines the name of the Antares-Simulator output directory that Antares-Xpansion uses to generate the expansion problem. If the value is last, the most recent run will be used. This option only has an effect when --step is among {problem_generation, benders, study_update}.

This is probably an issue. In Antares web it is possible to select a simulation to run sensitivity on it. That means this option is lost at some point

laurent-laporte-pro commented 1 year ago

L'analyse montre que le répertoire output (et donc le sous-dossier de la sortie sélectionné par l'utilisateur) ne se trouve pas dans le ZIP de l'étude :

$ ls -1 -F b69e6647-2625-45c7-8ef2-053550f955e4
Desktop.ini
input/
layers/
logs/
patch.json
settings/
study.antares
user/

laurent-laporte-pro commented 1 year ago

New release of Antares Launcher v1.3.0

laurent-laporte-pro commented 1 year ago

Xpansion trouve bien la sortie, mais il s'attend à avoir un ZIP et non pas un répertoire.

Nous avons ouvert un nouveau ticket : #1645

AntaresSimulatorTeam / AntaREST