search

deepmodeling / deepmd-kit

A deep learning package for many-body potential energy representation and molecular dynamics
https://docs.deepmodeling.com/projects/deepmd/
GNU Lesser General Public License v3.0
1.41k stars 486 forks source link

[BUG] `source/tests/pt/test_training.py::TestFparam::test_dp_train` throws a warning (instead of an error) #3901

Open njzjz opened 1 week ago

njzjz commented 1 week ago

While the test seems wrong, it's unclear why this doesn't raise the error to fail the test.

2024-06-24T21:21:39.8587714Z source/tests/pt/test_training.py::TestFparam::test_dp_train
2024-06-24T21:21:39.8589430Z   /__w/_tool/Python/3.11.8/x64/lib/python3.11/site-packages/_pytest/threadexception.py:77: PytestUnhandledThreadExceptionWarning: Exception in thread Thread-2421
2024-06-24T21:21:39.8590638Z   
2024-06-24T21:21:39.8590985Z   Traceback (most recent call last):
2024-06-24T21:21:39.8603736Z     File "/__w/_tool/Python/3.11.8/x64/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
2024-06-24T21:21:39.8604688Z       self.run()
2024-06-24T21:21:39.8605695Z     File "/__w/deepmd-kit/deepmd-kit/deepmd/pt/utils/dataloader.py", line 217, in run
2024-06-24T21:21:39.8606482Z       for item in self._source:
2024-06-24T21:21:39.8607649Z     File "/__w/_tool/Python/3.11.8/x64/lib/python3.11/site-packages/torch/utils/data/dataloader.py", line 631, in __next__
2024-06-24T21:21:39.8608649Z       data = self._next_data()
2024-06-24T21:21:39.8609282Z              ^^^^^^^^^^^^^^^^^
2024-06-24T21:21:39.8610422Z     File "/__w/_tool/Python/3.11.8/x64/lib/python3.11/site-packages/torch/utils/data/dataloader.py", line 675, in _next_data
2024-06-24T21:21:39.8611636Z       data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
2024-06-24T21:21:39.8612326Z              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-24T21:21:39.8613453Z     File "/__w/_tool/Python/3.11.8/x64/lib/python3.11/site-packages/torch/utils/data/_utils/fetch.py", line 53, in fetch
2024-06-24T21:21:39.8614490Z       data = self.dataset[possibly_batched_index]
2024-06-24T21:21:39.8615051Z              ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-24T21:21:39.8616353Z     File "/__w/deepmd-kit/deepmd-kit/deepmd/pt/utils/dataloader.py", line 173, in __getitem__
2024-06-24T21:21:39.8617184Z       batch = next(self.iters[idx])
2024-06-24T21:21:39.8617652Z               ^^^^^^^^^^^^^^^^^^^^^
2024-06-24T21:21:39.8618936Z     File "/__w/_tool/Python/3.11.8/x64/lib/python3.11/site-packages/torch/utils/data/dataloader.py", line 631, in __next__
2024-06-24T21:21:39.8620396Z       data = self._next_data()
2024-06-24T21:21:39.8620829Z              ^^^^^^^^^^^^^^^^^
2024-06-24T21:21:39.8621978Z     File "/__w/_tool/Python/3.11.8/x64/lib/python3.11/site-packages/torch/utils/data/dataloader.py", line 675, in _next_data
2024-06-24T21:21:39.8623171Z       data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
2024-06-24T21:21:39.8623835Z              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-24T21:21:39.8625007Z     File "/__w/_tool/Python/3.11.8/x64/lib/python3.11/site-packages/torch/utils/data/_utils/fetch.py", line 51, in fetch
2024-06-24T21:21:39.8626132Z       data = [self.dataset[idx] for idx in possibly_batched_index]
2024-06-24T21:21:39.8626812Z              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-24T21:21:39.8628016Z     File "/__w/_tool/Python/3.11.8/x64/lib/python3.11/site-packages/torch/utils/data/_utils/fetch.py", line 51, in <listcomp>
2024-06-24T21:21:39.8629305Z       data = [self.dataset[idx] for idx in possibly_batched_index]
2024-06-24T21:21:39.8630007Z               ~~~~~~~~~~~~^^^^^
2024-06-24T21:21:39.8630897Z     File "/__w/deepmd-kit/deepmd-kit/deepmd/pt/utils/dataset.py", line 40, in __getitem__
2024-06-24T21:21:39.8631747Z       b_data = self._data_system.get_item_torch(index)
2024-06-24T21:21:39.8632332Z                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-24T21:21:39.8633259Z     File "/__w/deepmd-kit/deepmd-kit/deepmd/utils/data.py", line 245, in get_item_torch
2024-06-24T21:21:39.8634049Z       frames = self._load_set(self.dirs[i])
2024-06-24T21:21:39.8634575Z                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-24T21:21:39.8635432Z     File "/__w/deepmd-kit/deepmd-kit/deepmd/utils/data.py", line 505, in _load_set
2024-06-24T21:21:39.8636244Z       data["find_" + kk], data[kk] = self._load_data(
2024-06-24T21:21:39.8636796Z                                      ^^^^^^^^^^^^^^^^
2024-06-24T21:21:39.8637681Z     File "/__w/deepmd-kit/deepmd-kit/deepmd/utils/data.py", line 660, in _load_data
2024-06-24T21:21:39.8638528Z       raise RuntimeError(f"{path} not found!")
2024-06-24T21:21:39.8639731Z   RuntimeError: /__w/deepmd-kit/deepmd-kit/source/tests/pt/water/data/data_0/set.000/fparam.npy not found!
2024-06-24T21:21:39.8640600Z   
2024-06-24T21:21:39.8641156Z     warnings.warn(pytest.PytestUnhandledThreadExceptionWarning(msg))
2024-06-24T21:21:39.8641712Z 
2024-06-24T21:21:39.8642047Z source/tests/pt/test_training.py::TestFparam::test_trainable
2024-06-24T21:21:39.8643630Z   /__w/_tool/Python/3.11.8/x64/lib/python3.11/site-packages/_pytest/threadexception.py:77: PytestUnhandledThreadExceptionWarning: Exception in thread Thread-2453
2024-06-24T21:21:39.8644862Z   
2024-06-24T21:21:39.8645203Z   Traceback (most recent call last):
2024-06-24T21:21:39.8646079Z     File "/__w/_tool/Python/3.11.8/x64/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
2024-06-24T21:21:39.8646892Z       self.run()
2024-06-24T21:21:39.8647673Z     File "/__w/deepmd-kit/deepmd-kit/deepmd/pt/utils/dataloader.py", line 217, in run
2024-06-24T21:21:39.8648441Z       for item in self._source:
2024-06-24T21:21:39.8649682Z     File "/__w/_tool/Python/3.11.8/x64/lib/python3.11/site-packages/torch/utils/data/dataloader.py", line 631, in __next__
2024-06-24T21:21:39.8650685Z       data = self._next_data()
2024-06-24T21:21:39.8651121Z              ^^^^^^^^^^^^^^^^^
2024-06-24T21:21:39.8652235Z     File "/__w/_tool/Python/3.11.8/x64/lib/python3.11/site-packages/torch/utils/data/dataloader.py", line 675, in _next_data
2024-06-24T21:21:39.8653409Z       data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
2024-06-24T21:21:39.8654308Z              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-24T21:21:39.8655466Z     File "/__w/_tool/Python/3.11.8/x64/lib/python3.11/site-packages/torch/utils/data/_utils/fetch.py", line 53, in fetch
2024-06-24T21:21:39.8656499Z       data = self.dataset[possibly_batched_index]
2024-06-24T21:21:39.8657039Z              ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-24T21:21:39.8658160Z     File "/__w/deepmd-kit/deepmd-kit/deepmd/pt/utils/dataloader.py", line 173, in __getitem__
2024-06-24T21:21:39.8659188Z       batch = next(self.iters[idx])
2024-06-24T21:21:39.8659818Z               ^^^^^^^^^^^^^^^^^^^^^
2024-06-24T21:21:39.8660977Z     File "/__w/_tool/Python/3.11.8/x64/lib/python3.11/site-packages/torch/utils/data/dataloader.py", line 631, in __next__
2024-06-24T21:21:39.8661982Z       data = self._next_data()
2024-06-24T21:21:39.8662412Z              ^^^^^^^^^^^^^^^^^
2024-06-24T21:21:39.8663518Z     File "/__w/_tool/Python/3.11.8/x64/lib/python3.11/site-packages/torch/utils/data/dataloader.py", line 675, in _next_data
2024-06-24T21:21:39.8664709Z       data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
2024-06-24T21:21:39.8665394Z              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-24T21:21:39.8666531Z     File "/__w/_tool/Python/3.11.8/x64/lib/python3.11/site-packages/torch/utils/data/_utils/fetch.py", line 51, in fetch
2024-06-24T21:21:39.8667634Z       data = [self.dataset[idx] for idx in possibly_batched_index]
2024-06-24T21:21:39.8668306Z              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-24T21:21:39.8669656Z     File "/__w/_tool/Python/3.11.8/x64/lib/python3.11/site-packages/torch/utils/data/_utils/fetch.py", line 51, in <listcomp>
2024-06-24T21:21:39.8670802Z       data = [self.dataset[idx] for idx in possibly_batched_index]
2024-06-24T21:21:39.8671402Z               ~~~~~~~~~~~~^^^^^
2024-06-24T21:21:39.8672267Z     File "/__w/deepmd-kit/deepmd-kit/deepmd/pt/utils/dataset.py", line 40, in __getitem__
2024-06-24T21:21:39.8673152Z       b_data = self._data_system.get_item_torch(index)
2024-06-24T21:21:39.8673738Z                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-24T21:21:39.8674644Z     File "/__w/deepmd-kit/deepmd-kit/deepmd/utils/data.py", line 245, in get_item_torch
2024-06-24T21:21:39.8675449Z       frames = self._load_set(self.dirs[i])
2024-06-24T21:21:39.8675974Z                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-24T21:21:39.8676811Z     File "/__w/deepmd-kit/deepmd-kit/deepmd/utils/data.py", line 505, in _load_set
2024-06-24T21:21:39.8677619Z       data["find_" + kk], data[kk] = self._load_data(
2024-06-24T21:21:39.8678198Z                                      ^^^^^^^^^^^^^^^^
2024-06-24T21:21:39.8679183Z     File "/__w/deepmd-kit/deepmd-kit/deepmd/utils/data.py", line 660, in _load_data
2024-06-24T21:21:39.8679990Z       raise RuntimeError(f"{path} not found!")
2024-06-24T21:21:39.8681073Z   RuntimeError: /__w/deepmd-kit/deepmd-kit/source/tests/pt/water/data/data_0/set.000/fparam.npy not found!
2024-06-24T21:21:39.8681958Z   
2024-06-24T21:21:39.8682489Z     warnings.warn(pytest.PytestUnhandledThreadExceptionWarning(msg))
njzjz commented 1 week ago

Reopen. The warning still exists in the latest runs: https://github.com/deepmodeling/deepmd-kit/actions/runs/9684066404/job/26720945997