Create mso4sc local directory to store temporary files and logs

[gdolle]

I think it would be nice to store all files generated by the plugin on the cluster in a specific local user directory to be more clean, and avoid to potentially erase data in ${HOME}.

For example:

~{HOME}/.mso4sc_orchestrator/logs
~{HOME}/.mso4sc_orchestrator/scripts

And later use a MSO4SC_ORCHESTRATOR_DIR variable maybe to let users change this root path.

[emepetres]

This is being addressed by creating a new work directory for each instance.

[victorsndvg]

Hi @emepetres ,

as @Trophime said, we need to avoid using always $HOME as root directory. Some HPC's can have some limitations in some particular directories (like disk space or number of files per user).

I think the Infrastructure admin role should define a default base directory per HPC as starting point for the storage. Then apps providers (developers) can overwrite it if they want.

In order to avoid creating the workdir always in the $HOME directory we could prepend to it a base directory per HPC. e.g we can have a default $BASE_DIR variable per HPC/Cloud

What do you think?

[Trophime]

I fully agree with @victorsndvg

[Trophime]

And right now it breaks any blueprint that is using or setting workdir to any sub directory of $LUSTRE for instance

[gdolle]

I agree too. It would be better to have the absolute path MSO4SC_ORCHESTRATOR_DIR with ${HOME} by default for example. And let the system admins manage this path system wide when they configure the orchestrator for their cluster.

I think it's not a good idea to let blueprint devs and end-users manage this path since system admins are the best suited to choose the best file system for their machine in my opinion.

[emepetres]

So where this base directory should be defined? When adding the infrastructure in the portal? or in the blueprint?

[gdolle]

Personally, I would let this option in the plugin directly here https://github.com/MSO4SC/cloudify-hpc-plugin/blob/c8187685da595d90412c451f706a554fc98e9726/hpc_plugin/workload_managers/workload_manager.py#L257 Check if the variable MSO4SC_ORCHESTRATOR_DIR is set, go in, otherwise you go in the default orchestrator config directory ${HOME}/.mso4sc_orchestrator for example.

PS: And if user want to change, he can change the environment variable in the userland in the cluster directly. Most end-user won't touch that part.

[emepetres]

@gdolle the problem with that approach is that if you define the variable in the bluerprint, it becomes specific for a concrete HPC. Either way if you set this as input, the user must know the folder of the HPC in which it should run...

[gdolle]

I mean you hide it in the plugin (no blueprint). You retrieve the environment variable via ssh remote command for example, sth like

wrkdir=sh_client.send_command( "echo ${MSO4SC_ORCHESTRATOR_DIR}", wait_result=wait_result)
if workdir
   return sh_client.send_command( ...)
else
   cd ${HOME}/.mso4sc_orchestrator
   return sh_client.send_command( ...)

It's not specific to a special HPC, since you set home as default. It's just that you have the possibility to let admins/users to better configure the orchestrator by setting it in the cluster directly.