search

cnr-ibf-pa / hbp-bsp-issues

Ticketing system for developers/testers and power users of the Brain Simulation Platform of the Human Brain Project
4 stars 0 forks source link

Unicore access denied while sbatch works #439

Closed antonelepfl closed 5 years ago

antonelepfl commented 5 years ago

Hi @BerndSchuller We are updating our module stack to some new version on Jureca and for instance if I enter into Jureca and run sbatch everything works fine but when I try to do the same thing with Unicore API, the result of Unicore is an error 

PSI: handleAnswer: spawn to node 3235 failed: "testExecutable: stat(): S_ISREG error
PSI: ter/Stages/2019a/softw/2019a/software/gperf/3.1-GCCcor.33.1-GCCcore-8.3.0/bin:/usr/loc/local/software/jurecabooster/Stoftware/jurecabooster/Stages/201r/Stages/2019a/software/gperf/3.-linux/2.33.1-GCCcore-8.3.0/bin:bin:/usr/local/software/jurecaboabooster/Stages/2019a/software/ure/fontconfig/2.13.1-GCCcore-8.3-8.3.0/bin:/usr/local/software/joftware/jurecabooster/Stages/201"

And then ....

Could not spawn '/p/project/cvsk25/software-deployment/HBP/jureca-booster/28-05-2019/install/linux-centos7-x86_64/intel-19.0.3/neurodamus-hippocampus-0.2-zepe22/lib/hoc' process 253: Permission denied

Which is not totally true because I'm able to run ls to that directory and it has executable access too.

You could find the job with the ls and the full log in this folder: /p/scratch/cvsk25/unicore-jobs/364cfef9-2d4a-40bd-b42c-af8213a1b1e4

So we are wondering if Unicore requires some extra permissions to run a job that is different of what we usually do manually on sbatch.

CC: @pramodk

BerndSchuller commented 5 years ago

There are no extra permissions apart from the user id and group (antonel1 / cvsk25). Server logs look fine (I can't access files on the file system)

Does the same error happen when you login (as antonel1) and run the sbatch on the UNICORE-generated slurm file?

cd /p/scratch/cvsk25/unicore-jobs/364cfef9-2d4a-40bd-b42c-af8213a1b1e4 sbatch bsssubmit*

NOTE the stdout/stderr in /p/scratch/cvsk25/unicore-jobs/364cfef9-2d4a-40bd-b42c-af8213a1b1e4 will be overwritten when you do this...

antonelepfl commented 5 years ago

Hi @BerndSchuller When I run sbatch to the bsssubmit I get the same access denied. Do you have any other idea of what could it be?

BerndSchuller commented 5 years ago

Not sure I understand what the real error is. What do you do in the case when things are working? Possible differences are in the "input.sh" script that you run, or in the sbatch parameters (nodes, selected queue...) Could you attach stdout/stderr or send them to me in a zip file?

antonelepfl commented 5 years ago

The new deployment changed some paths so I fixed that. Thanks for the help @BerndSchuller