[BUG] Q28 fails fitting Naive Bayes due to cuML CumlArray refactor

[beckernick]

The cuML CumlArray cleanup refactor appears to have caused the Dask Multinomial Naive Bayes implementation to fail trying a binaryop between attributes that are now CumlArrays during _merge_counts_to_model.

This does not occur in the small example code in the docstring for Dask MNB in cuML.

Note that this needs to be run as a stand-alone query for now due to https://github.com/rapidsai/tpcx-bb/issues/59

[beckernick]

@VibhuJawa are we now able to refactor this code consistent with https://github.com/rapidsai/tpcx-bb/issues/50 now that https://github.com/rapidsai/cuml/pull/2554 has landed?

[beckernick]

python tpcx_bb_query_28.py --config_file=../../benchmark_runner/benchmark_config.yaml
Using default arguments
Connected to ucx://172.22.1.26:8786
[I] [12:52:20.604337] [<Future: finished, type: cuml.MultinomialNB, key: _fit-8335bdf0-8835-466c-9c42-4ae7c0fa7db1>]
[I] [12:52:20.605043] [Delayed('_merge_counts_to_model-a30656f7-9ec8-4de4-b737-f0586861e450')]
[I] [12:52:20.608186] [Delayed('_merge_counts_to_model-b001de66-5977-4340-9fca-5c45eec0aacd')]
[I] [12:52:20.608563] [Delayed('_merge_counts_to_model-5eb8dd94-b371-4ad6-aa2a-6feec7dc575a')]
[I] [12:52:20.608922] [Delayed('_merge_counts_to_model-4fc97c15-01dc-4c10-8664-c064e321606b')]
[I] [12:52:20.609299] [Delayed('_merge_counts_to_model-1b8f7650-f51f-4063-b7a3-e2e2d175c8a2')]
[I] [12:52:20.609637] [Delayed('_merge_counts_to_model-8fedd205-1d62-42d4-9e95-5af19504b509')]
[I] [12:52:20.609682] [<Future: finished, type: cuml.MultinomialNB, key: _fit-5b521a3f-8f6d-4905-af81-0d286b3408b8>]
[I] [12:52:20.610252] [Delayed('_merge_counts_to_model-ffce0918-0422-4c77-8184-68000636b8d9')]
[I] [12:52:20.610286] [<Future: finished, type: cuml.MultinomialNB, key: _fit-05dfcca3-2b96-4b60-86b5-fdbab7adbdbb>]
[I] [12:52:20.610690] [Delayed('_merge_counts_to_model-34814cd8-60af-4031-80cd-da294720ec51')]
[I] [12:52:20.610725] [<Future: finished, type: cuml.MultinomialNB, key: _fit-a7069349-3c0c-4441-bfba-e0b6901c16dc>]
[I] [12:52:20.610747] [<Future: finished, type: cuml.MultinomialNB, key: _fit-8562ae07-a7e2-467d-9fb5-9d6b5fce5147>]
[I] [12:52:20.610768] [<Future: finished, type: cuml.MultinomialNB, key: _fit-eb5dd4a9-15db-4880-bf4e-8bc900add5ff>]
[I] [12:52:20.610788] [<Future: finished, type: cuml.MultinomialNB, key: _fit-6601b505-1d73-43db-9d14-392a09886ff9>]
[I] [12:52:20.610807] [<Future: finished, type: cuml.MultinomialNB, key: _fit-6fc5dad4-e948-444c-9dec-763aac6ef64c>]
distributed.worker - WARNING -  Compute Failed
Function:  _merge_counts_to_model
args:      ([MultinomialNB(), MultinomialNB()])
kwargs:    {}
Exception: TypeError("unsupported operand type(s) for +=: 'CumlArray' and 'CumlArray'")

distributed.worker - WARNING -  Compute Failed
Function:  _merge_counts_to_model
args:      ([MultinomialNB(), MultinomialNB()])
kwargs:    {}
Exception: TypeError("unsupported operand type(s) for +=: 'CumlArray' and 'CumlArray'")

distributed.worker - WARNING -  Compute Failed
Function:  _merge_counts_to_model
args:      ([MultinomialNB(), MultinomialNB()])
kwargs:    {}
Exception: TypeError("unsupported operand type(s) for +=: 'CumlArray' and 'CumlArray'")

distributed.worker - WARNING -  Compute Failed
Function:  _merge_counts_to_model
args:      ([MultinomialNB(), MultinomialNB()])
kwargs:    {}
Exception: TypeError("unsupported operand type(s) for +=: 'CumlArray' and 'CumlArray'")

distributed.worker - WARNING -  Compute Failed
Function:  _merge_counts_to_model
args:      ([MultinomialNB(), MultinomialNB()])
kwargs:    {}
Exception: TypeError("unsupported operand type(s) for +=: 'CumlArray' and 'CumlArray'")

distributed.worker - WARNING -  Compute Failed
Function:  _merge_counts_to_model
args:      ([MultinomialNB(), MultinomialNB()])
kwargs:    {}
Exception: TypeError("unsupported operand type(s) for +=: 'CumlArray' and 'CumlArray'")

distributed.worker - WARNING -  Compute Failed
Function:  _merge_counts_to_model
args:      ([MultinomialNB(), MultinomialNB()])
kwargs:    {}
Exception: TypeError("unsupported operand type(s) for +=: 'CumlArray' and 'CumlArray'")

distributed.worker - WARNING -  Compute Failed
Function:  _merge_counts_to_model
args:      ([MultinomialNB(), MultinomialNB()])
kwargs:    {}
Exception: TypeError("unsupported operand type(s) for +=: 'CumlArray' and 'CumlArray'")

Encountered Exception while running query
Traceback (most recent call last):
  File "/raid/nicholasb/miniconda3/envs/rapids-tpcxbb-20200817/lib/python3.7/site-packages/xbb_tools/utils.py", line 280, in run_dask_cudf_query
    config=config,
  File "/raid/nicholasb/miniconda3/envs/rapids-tpcxbb-20200817/lib/python3.7/site-packages/xbb_tools/utils.py", line 61, in benchmark
    result = func(*args, **kwargs)
  File "tpcx_bb_query_28.py", line 346, in main
    client=client, train_data=train_data, test_data=test_data
  File "tpcx_bb_query_28.py", line 290, in post_etl_processing
    model.fit(X_train, y_train)
  File "/raid/nicholasb/miniconda3/envs/rapids-tpcxbb-20200817/lib/python3.7/site-packages/cuml/common/memory_utils.py", line 56, in cupy_rmm_wrapper
    return func(*args, **kwargs)
  File "/raid/nicholasb/miniconda3/envs/rapids-tpcxbb-20200817/lib/python3.7/site-packages/cuml/dask/naive_bayes/naive_bayes.py", line 190, in fit
    client=self.client)
  File "/raid/nicholasb/miniconda3/envs/rapids-tpcxbb-20200817/lib/python3.7/site-packages/cuml/dask/common/func.py", line 63, in reduce
    workers = [(first(who_has[m.key]), m) for m in futures]
  File "/raid/nicholasb/miniconda3/envs/rapids-tpcxbb-20200817/lib/python3.7/site-packages/cuml/dask/common/func.py", line 63, in <listcomp>
    workers = [(first(who_has[m.key]), m) for m in futures]
  File "/raid/nicholasb/miniconda3/envs/rapids-tpcxbb-20200817/lib/python3.7/site-packages/toolz/itertoolz.py", line 376, in first
    return next(iter(seq))
StopIteration

@VibhuJawa

[VibhuJawa]

This should be resolved now as it ran successfully based @beckernick 's test and https://github.com/rapidsai/cuml/issues/2704 is closed . Working cuml library version.

0.15.0a200825