search

wfondrie / depthcharge

A deep learning toolkit for mass spectrometry
https://wfondrie.github.io/depthcharge/
Apache License 2.0
59 stars 18 forks source link

Problem with AnnotatedSpectrumDataset and n_workers>0 #45

Closed andradesalazar closed 4 months ago

andradesalazar commented 7 months ago

Hi there, I started to use depthcharge and I really like it :) However, I have a bit of a problem with the dataset AnnotatedSpectrumDataset from the latest depthcharge version. I am testing it with the function from https://github.com/wfondrie/depthcharge/blob/main/tests/unit_tests/test_data/test_loaders.py#L47 That works well. However, whenever I increase the number of workers it gets stuck and the code never terminates. Have you encountered problems like this? Do you have any idea why this may happen? I am using Python 3.10.12 on a Linux machine. Here are some of my package versions:

torch==2.1.0
pytorch-lightning==1.9.5
pylance==0.8.16
pyteomics==4.6.3

Thanks a lot in advance!

wfondrie commented 7 months ago

Hi @andradesalazar - thanks for reaching out.

Can you try updating pylance and see if the issue still exists?

I'll try and reproduce this issue as well.

andradesalazar commented 6 months ago

Hi @wfondrie Thanks for your answer! I upgraded pylance to 0.9.14 but unfortunately, the issue persists. Did you mange to reproduce the issue?

Any suggestions/help would be highly appreciated :)

wfondrie commented 6 months ago

Thanks for checking - I'm working on a fix for this now. Lance relatively recently added full PyTorch integration, which I'll update depthcharge to use. It should solve these issues and make it more efficient! https://lancedb.github.io/lance/api/python/lance.torch.html

wfondrie commented 5 months ago

Hi @andradesalazar - quick update here: I haven't been able to reproduce the original issue locally. However, I am still looking into migration to the official PyTorch api for Lance.

andradesalazar commented 5 months ago

Hi @wfondrie, thanks a lot for the update. Migration to the latest Lance using full PyTorch integration would be great :)