search

CoBrALab / minc-bpipe-library

Library of bpipe implemented commands commonly applied to minc datafiles, suitable for importing into other pipelines
Other
13 stars 8 forks source link

Job loss due to file lock on scinet #29

Closed cfhammill closed 5 years ago

cfhammill commented 7 years ago

I periodically get job losses when running the bpipe preprocessing pipeline:

----------------------------------------
Begin PBS Prologue Thu Sep 22 11:01:51 EDT 2016 1474556511
Job ID:         37929662.gpc-sched-ib0
Username:       chammill
Group:          jlerch
Nodes:          gpc-f107n074-ib0
End PBS Prologue Thu Sep 22 11:01:52 EDT 2016 1474556512
----------------------------------------
java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.codehaus.groovy.tools.GroovyStarter.rootLoader(GroovyStarter.java:106)
        at org.codehaus.groovy.tools.GroovyStarter.main(GroovyStarter.java:128)
Caused by: java.io.IOException: Couldn't get lock for .bpipe/bpipe.log
        at java.util.logging.FileHandler.openFiles(FileHandler.java:438)
        at java.util.logging.FileHandler.<init>(FileHandler.java:292)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
        at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:77)
        at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:102)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:57)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:182)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:190)
        at bpipe.Runner.initializeLogging(Runner.groovy:497)
        at bpipe.Runner.main(Runner.groovy:157)
        ... 6 more

This problem may be upstream in bpipe itself.

gdevenyi commented 7 years ago

Looks like bpipe and/or filesystem issues.

I've just updated bpipe to the latest upstream, let me know if that helps.

cfhammill commented 7 years ago

I'll keep you posted, thanks!

cfhammill commented 7 years ago

Hi Gabe,

Looks like something may be broken with the file permissions for the bpipe update

$ module load scinet
utility.c(2245):ERROR:50: Cannot open file '/project/m/mchakrav/quarantine/modules/bpipe/0.9.9.2' for 'reading'
utility.c(2245):ERROR:50: Cannot open file '/project/m/mchakrav/quarantine/modules/bpipe/0.9.9.2' for 'reading'
gdevenyi commented 7 years ago

You're an external group user... I forgot to re-run my "fix all permissions" after install. Running now.

cfhammill commented 7 years ago

I imagined it was something like that, thanks for the fix.

gdevenyi commented 7 years ago

Script done.

gdevenyi commented 5 years ago

Still an issue? I've had no reports of anything like this recently, with the Niagara upgrade.

cfhammill commented 5 years ago

Haven't used it in quite a while, so not sure

gdevenyi commented 5 years ago

Okay, closing for now, reopen if you catch anything.