fragpipe diaumpire is only running check centroid

mremachine1 commented 9 months ago

Describe the issue or question: I am trying to run diaumpire via fragpipe only but with it being the only tab checked in fragpipe, but it only runs check centroid. We have found that the diaumpire portion of fragpipe runs much faster splitting the total file set into multiple separate jobs (1 day instead of 5 days) we then issue a second fragpipe run starting from diaumpire mzmls to run in msfragger (all together for the mass calibration)

I have attached a zip of the manifest, workflow and log for one of the diaumpire fragpipe jobs below

Upload your log file diaumpire_runfiles.zip

fcyu commented 9 months ago

The error was cause by the SDRF writer. @dpolasky , can you take a look when you have time?

2024-01-04 10:14:53,763 ERROR - java.nio.file.FileAlreadyExistsException: /fastscratch/myscratch/smburke/fenna_out/sdrf.tsv
    at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:94)
    at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
    at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
    at java.base/sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:248)
    at java.base/sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:110)
    at java.base/java.nio.file.Files.deleteIfExists(Files.java:1191)
    at com.dmtavt.fragpipe.tabs.TabWorkflow.getSaveFilePath(TabWorkflow.java:1666)
    at com.dmtavt.fragpipe.tabs.TabWorkflow.on(TabWorkflow.java:1372)
    at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
    at java.base/java.lang.reflect.Method.invoke(Method.java:578)
    at org.greenrobot.eventbus.EventBus.invokeSubscriber(EventBus.java:517)
    at org.greenrobot.eventbus.EventBus.postToSubscription(EventBus.java:440)
    at org.greenrobot.eventbus.EventBus.postSingleEventForEventType(EventBus.java:421)
    at org.greenrobot.eventbus.EventBus.postSingleEvent(EventBus.java:394)
    at org.greenrobot.eventbus.EventBus.post(EventBus.java:275)
    at com.dmtavt.fragpipe.api.Bus.post(Bus.java:76)
    at com.dmtavt.fragpipe.FragpipeRun.lambda$run$13(FragpipeRun.java:523)
    at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
    at java.base/java.lang.Thread.run(Thread.java:1589)

Thanks,

Fengchao

jossmi commented 9 months ago

Hi FragPipe team, thanks for the quick reply. This seems to have appeared out of nowhere on our system - we run in headless mode on a Linux computing cluster and we recently were running FragPipe v20.0 and it was working fine, but then within the past day or so we started getting this error. Upgrading to v21.0 did not resolve it. Is this a headless mode or Linux issue? We can run it fine using FragPipe v20.0 on our Windows desktop, but as Sean mentioned, much, much slower than on the cluster, which is a problem for the number of files we are processing. Any thoughts on a potential workaround for the time being until the SDRF writer issue can be resolved?

Thanks, Josh

dpolasky commented 9 months ago

Hi Josh, I am traveling this week and won't be able to make a fix until next week, but in the meantime, I think a workaround would be to disable the SDRF writing (either by unchecking the 'write technical SDRF' box in the GUI or setting 'workflow.misc.save-sdrf' to false in the workflow file for headless mode).

Best, Dan

jossmi commented 9 months ago

Thanks for the suggestion!

Josh

Joshua W. Smith, Ph.D. Research Associate

Department of Environmental Health and Engineering Johns Hopkins Bloomberg School of Public Health —— Mass Spectrometry and Proteomics Facility Department of Biological Chemistry Johns Hopkins University School of Medicine

On Jan 4, 2024, at 6:19 PM, Daniel Polasky @.***> wrote:

  External Email - Use Caution

Hi Josh, I am traveling this week and won't be able to make a fix until next week, but in the meantime, I think a workaround would be to disable the SDRF writing (either by unchecking the 'write technical SDRF' box in the GUI or setting 'workflow.misc.save-sdrf' to false in the workflow file for headless mode).

Best, Dan

— Reply to this email directly, view it on GitHubhttps://github.com/Nesvilab/FragPipe/issues/1395#issuecomment-1877893487, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ASOSZHHLR4VPRZFPTTEN2BDYM42FFAVCNFSM6AAAAABBNDXDD6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZXHA4TGNBYG4. You are receiving this because you commented.Message ID: @.***>

fcyu commented 5 months ago

@dpolasky Is this bug fixed?

Thanks,

Fengchao

dpolasky commented 5 months ago

Yes, should be. I couldn't reproduce it, which makes me think it might be a permissions issue. But I have added a check to the method to delete a previous table if one exists prior to writing, which should prevent it in any case.

fcyu commented 5 months ago

Sounds great.

Thanks,

Fengchao

Nesvilab / FragPipe

fragpipe diaumpire is only running check centroid #1395