search

limebit / medmodels

MedModels is a high-speed RWE framework to apply the latest methods from scientific research to medical data.
https://www.medmodels.de
BSD 3-Clause "New" or "Revised" License
4 stars 0 forks source link

Add support for pickling MedRecords #258

Open JabobKrauskopf opened 3 days ago

JabobKrauskopf commented 3 days ago

When using 

train_loader = DataLoader(
            train_dataset, batch_size=self.batch_size, shuffle=True, num_workers=2
        )

and a MedRecord is inside the train_dataset, this error pops up:

====================================================================== ERROR: test_train_real_gru (main.TestRealGRU) Test the train_real_gru method of RealGRU.

Traceback (most recent call last): File "/Users/martin.iniguez/Desktop/projects/medmodels-plus/medmodels/data_synthesis/mtgan/model/real_gru/tests/test_real_gru.py", line 284, in test_train_real_gru loss = self.real_gru.train_real_gru(self.train_dataset) File "/Users/martin.iniguez/Desktop/projects/medmodels-plus/medmodels/data_synthesis/mtgan/model/real_gru/real_gru.py", line 139, in train_real_gru for ( File "/Users/martin.iniguez/Desktop/projects/medmodels-plus/.venv/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 440, in iter return self._get_iterator() File "/Users/martin.iniguez/Desktop/projects/medmodels-plus/.venv/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 388, in _get_iterator return _MultiProcessingDataLoaderIter(self) File "/Users/martin.iniguez/Desktop/projects/medmodels-plus/.venv/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1038, in init w.start() File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/multiprocessing/process.py", line 121, in start self._popen = self._Popen(self) File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/multiprocessing/context.py", line 224, in _Popen return _default_context.get_context().Process._Popen(process_obj) File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/multiprocessing/context.py", line 284, in _Popen return Popen(process_obj) File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 32, in init super().init(process_obj) File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_fork.py", line 19, in init self._launch(process_obj) File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 47, in _launch reduction.dump(process_obj, fp) File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/multiprocessing/reduction.py", line 60, in dump ForkingPickler(file, protocol).dump(obj) TypeError: cannot pickle 'builtins.PyMedRecord' object

It is solved the moment the num_workerks is equal to default value (0).