Computation on GPU is much slower than on CPU

[mc-karsa-tech]

Hi,

our dataset has 1300 rows and 5 columns. On GPU causal impact takes around 17 minutes while on CPU in takes only 2-3 minutes using this code:

casual_impact = CausalImpact(input_data_frame, pre_period, post_period, model_args = { 
    'nseasons' : 1,
    'niter': 1000,
    'prior_level_sd': 0.1,
    'fit_method': 'vi'
})

I also tried different niter and fit_method but GPU was always much slower than CPU. Is there any issue with library in utilization of GPU or could I do something to fix it?

GPU is GeForce GTX 1080 and on the same computer we use Tensorflow in another python program, where it is much faster on GPU than on CPU as expected. We have an issue only when using this causal impact library.

[WillianFuks]

Hi @mc-karsa-tech ,

Just tested this issue here and confirmed the same behavior; that was quite surprising indeed.

Not sure what's going on, I'll run some tests here and confirm where the time consumption is coming from. As soon as I have some results I'll let you know.

Thanks for opening this issue :)!

Best,

Will

[WillianFuks]

Hi @mc-karsa-tech ,

For what I've tested this issue seems to be related to Tensorflow. I've opened this issue on their repository asking for some help, hopefully they can shed some light into this issue.

I do remember that when I was implementing this package running on GPUs was considerably faster. But as I've tested on previous versions of TP and TFP the issue remained; not sure what's happening.

For now I recommend sticking with using CPUs and let's see what the team has to tell us in the meantime.

Best,

Will

[WillianFuks]

Hi @mc-karsa-tech ,

As an update, Dave Moore from TFP team replied back to the issue and for what he said it's not quite expected indeed that GPUs will cause any improvements in processing time.

As it seems it's indeed better to use CPUs for now. Funny thing is that when I was first developing this package I did have the impression that GPUs were helping and a lot but maybe that was just a coincidence back at the time.

I'll update the documentation accordingly.

Best,

Will

[mc-karsa-tech]

Hi, thank you for your effort and feedback. We stayed to use CPU instead of GPU since it is faster for our datasets although still slow. I apologize for my late response.