jacobbieker
commented
11 months ago Hi,
Sorry for the delayed response, I was on annual leave. I just pushed a minor update that should also fix some incompatibilities from using newer dependencies.
For 2. you can try running the test_train_dgmr in tests/test_model.py and set it to use GPUs, that should give a quicker and faster feedback loop on using the model. When I ran it on my CPU, there was no errors, so I think you might be right that its an issue with moving the batches to the GPU for some reason. That also gets rid of a lot of the extra code that might be masking the issue.
-
You should be able to run that from the folder it is in, I believe, but moving it to the root should also work.
-
It was tensorflow>2.0, newer ones should work though as well.
CUDA version is 11.8, the model currently doesn't run on my local machine, so I can try taking a look later on our ML machine.
rutkovskii
Javelin1991
Describe the bug
I am trying to run the
./train/run.py. I have several issues:To
Due to having the newer version of Pytorch than it is originally was developed on.
I keep encountering this issue, which I feel like comes from miss match from the versions of the dependencies on which the DGMR was developed: I can assume that there's an issue with transferring a batch to the GPU device (I am not sure). Let me know if you have any suggestions on how I can verify it.
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "run.py", line 239, in
trainer.fit(model, datamodule)
File "/home/arutkovskii_umass_edu/skillful_nowcasting/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 532, in fit
call._call_and_handle_interrupt(
File "/home/arutkovskii_umass_edu/skillful_nowcasting/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 67, in _call_and_handle_interrupt
trainer._teardown()
File "/home/arutkovskii_umass_edu/skillful_nowcasting/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1003, in _teardown
self.strategy.teardown()
File "/home/arutkovskii_umass_edu/skillful_nowcasting/venv/lib/python3.8/site-packages/pytorch_lightning/strategies/strategy.py", line 498, in teardown
self.lightning_module.cpu()
File "/home/arutkovskii_umass_edu/skillful_nowcasting/venv/lib/python3.8/site-packages/lightning_fabric/utilities/device_dtype_mixin.py", line 79, in cpu
return super().cpu()
File "/home/arutkovskii_umass_edu/skillful_nowcasting/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 967, in cpu
return self._apply(lambda t: t.cpu())
File "/home/arutkovskii_umass_edu/skillful_nowcasting/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 810, in _apply
module._apply(fn)
File "/home/arutkovskii_umass_edu/skillful_nowcasting/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 810, in _apply
module._apply(fn)
File "/home/arutkovskii_umass_edu/skillful_nowcasting/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 810, in _apply
module._apply(fn)
[Previous line repeated 3 more times]
File "/home/arutkovskii_umass_edu/skillful_nowcasting/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 857, in _apply
self._buffers[key] = fn(buf)
File "/home/arutkovskii_umass_edu/skillful_nowcasting/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 967, in
return self._apply(lambda t: t.cpu())
RuntimeError: CUDA error: unspecified launch failure
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with
TORCH_USE_CUDA_DSAto enable device-side assertions.wandb: Waiting for W&B process to finish... (failed 1). Press Control-C to abort syncing. wandb: 🚀 View run glowing-salad-28 at: https://wandb.ai/nowcasting-research/dgmr/runs/edi3updl wandb: ️⚡ View job at https://wandb.ai/nowcasting-research/dgmr/jobs/QXJ0aWZhY3RDb2xsZWN0aW9uOjEwMzkwMzkwNA==/version_details/v6 wandb: Synced 5 W&B file(s), 0 media file(s), 2 artifact file(s) and 0 other file(s) wandb: Find logs at: ./wandb/run-20231008_213620-edi3updl/logs (venv) ➜ skillful_nowcasting git:(main) ✗
torch==2.1.0 antialiased-cnns==0.3 pytorch-msssim==1.0.0 numpy==1.24.3 torchvision==0.16.0 pytorch-lightning==2.0.9.post0 einops==0.7.0 huggingface-hub==0.17.3 tensorflow==2.13.1