search

lcls-psana / psocake

3 stars 12 forks source link

Added path fixes to run psocake at CFEL #2

Closed kartikayyer closed 6 years ago

kartikayyer commented 6 years ago

If the PSOCAKE_FACILITY environment variable contains the string CFEL, then certain directory attributes are modified. Specifically, self.dir is set to the appropriate value and args.outDir is overwridden and set to the user scratch folder.

This seems to work here and will not interfere with operation at either LCLS or PAL (the other two options for PSOCAKE_FACILITY). The only change will be that if a user sets args.outDir from the command line, the psocakeDir attribute will have the experimentName rather than the username in the value.

chuckie82 commented 6 years ago

"The only change will be that if a user sets args.outDir from the command line, the psocakeDir attribute will have the experimentName rather than the username in the value." Is this needed at CFEL?

kartikayyer commented 6 years ago

The outDir is the user scratch folder at CFEL, regardless of the experiment, unlike at LCLS where the default folder is the experiment scratch folder for all users.

What are other situations where people use a custom outDir?

chuckie82 commented 6 years ago

At LCLS, outDir is used by data analysis group. We have read access to all the experiments, but no write access to experiment folders so we use outDir to redirect to our own scratch space. So I would like to keep this feature unchanged.

Can you not set outDir to be your user scratch folder + '/experimentName'? I guess I don't fully understand the usage at CFEL.

kartikayyer commented 6 years ago

But if you write to your own scratch space, why would you want to have your username in the folder structure? Currently the psocakeDir attribute does not have the experiment name anywhere in it, so I don't understand how it could be used with multiple experiments writing to the same outDir, which seems to be what you described.

The following setup would work at CFEL without affecting the workflow for you guys, but it would not solve the problem of different experiments clobbering each other's output: In gui.py,

if 'CFEL' in os.environ['PSOCAKE_FACILITY']:
    self.facility = self.facilityLCLS
    args.outDir = '/gpfs/cfel/cxi/scratch/user'
    self.dir = '/gpfs/cfel/cxi/common/slac/reg/d/psdm'
chuckie82 commented 6 years ago

It would be up to the user to make outDir unique. How about changing Line 790 to:

self.parent.psocakeDir = self.parent.rootDir + '/' + self.parent.experimentName + '/' + self.parent.username + '/psocake'

kartikayyer commented 6 years ago

That would work if we don't do the changes in my previous comment. It would add extra levels to the folder hierarchy, but I guess we have to live with that.

I think the fundamental problem is that here, the folders are organized as scratch/$USER/$EXPERIMENT/, while at LCLS it is $EXPERIMENT first and then $USER.

To summarize, the path at CFEL would be: /gpfs/cfel/cxi/scratch/user/$USER/$EXPERIMENT/$USER/psocake while the path at LCLS if someone specifies a custom outDir would be: $OUTDIR/$EXPERIMENT/$USER/psocake