search

dreamquark-ai / tabnet

PyTorch implementation of TabNet paper : https://arxiv.org/pdf/1908.07442.pdf
https://dreamquark-ai.github.io/tabnet/
MIT License
2.61k stars 485 forks source link

RuntimeError: CUDA error: device-side assert triggered #418

Closed omkarmutreja12 closed 2 years ago

omkarmutreja12 commented 2 years ago

Describe the bug I get this CUDA error when trying to fit the trained Tabnet classifier on unseen inference dataset (with GPU).

What is the current behavior?

If the current behavior is a bug, please provide the steps to reproduce.

Expected behavior

Screenshots

Other relevant information: poetry version:
python version: Operating System: Additional tools:

Additional context

image

Optimox commented 2 years ago

can you try to make the inference on CPU and see if you still have an error ?

omkarmutreja12 commented 2 years ago

@Optimox - Tried on CPU, the model is predicting on inference data, for GPU, it raises the CUDA error: device-side assert triggered.

Optimox commented 2 years ago

Have you trained the model with the same machine on GPU?

What does your inference computer says when trying torch.cuda.is_available?

Optimox commented 2 years ago

@omkarmutreja12 do you have any new information about your issue ?

omkarmutreja12 commented 2 years ago

@Optimox - Hey apologies for the delayed response, the problem is solved, there was an issue with the categorical column encoding, had one extra level which was not present in train, fixed it and it's running fine now, thanks!