1954

References

Checklist:

[ ] I have followed the contribution guidelines and code style for this project.
[ ] I have added tests covering my contributions.
[ ] I have updated the documentation accordingly.
[ ] I have signed the commits, e.g. git commit -s -m "your commit message".
[ ] This PR is being made to staging branch AND NOT TO main branch.

miguelgfierro commented 6 months ago

with scipy 1.11.1, and python 3.9 it works:

$ pytest tests/unit/examples/test_notebooks_python.py::test_sar_deep_dive_runs --disable-warnings
================================================= test session starts ==================================================
platform linux -- Python 3.9.16, pytest-7.4.0, pluggy-1.2.0
rootdir: /home/miguel/MS/recommenders
configfile: pyproject.toml
plugins: hypothesis-6.80.0, cov-4.1.0, mock-3.11.1, typeguard-4.0.0, anyio-3.7.0
collected 1 item

tests/unit/examples/test_notebooks_python.py .                                                                   [100%]

============================================ 1 passed, 3 warnings in 7.23s =============================================

With scipy 1.13.0 and python 3.9:

$ pytest tests/unit/examples/test_notebooks_python.py::test_sar_deep_dive_runs --disable-warnings
================================================= test session starts ==================================================
platform linux -- Python 3.9.16, pytest-7.4.0, pluggy-1.2.0
rootdir: /home/miguel/MS/recommenders
configfile: pyproject.toml
plugins: hypothesis-6.80.0, cov-4.1.0, mock-3.11.1, typeguard-4.0.0, anyio-3.7.0
collected 1 item

tests/unit/examples/test_notebooks_python.py .                                                                   [100%]

============================================ 1 passed, 3 warnings in 6.62s =============================================

For benchmarking purpose, using the previous version of scipy 1.10.1, it is slower:

$ pytest tests/unit/examples/test_notebooks_python.py::test_sar_deep_dive_runs --disable-warnings
================================================= test session starts ==================================================
platform linux -- Python 3.9.16, pytest-7.4.0, pluggy-1.2.0
rootdir: /home/miguel/MS/recommenders
configfile: pyproject.toml
plugins: hypothesis-6.80.0, cov-4.1.0, mock-3.11.1, typeguard-4.0.0, anyio-3.7.0
collected 1 item

tests/unit/examples/test_notebooks_python.py .                                                                   [100%]

============================================ 1 passed, 3 warnings in 8.39s =============================================

miguelgfierro commented 6 months ago

Error in python 3.11:

============================= test session starts ==============================
platform linux -- Python 3.11.5, pytest-8.1.1, pluggy-1.4.0
rootdir: /mnt/azureml/cr/j/e62514dcd77647bcb13baecdeaa9a748/exe/wd
configfile: pyproject.toml
plugins: anyio-4.3.0, hypothesis-6.100.0, cov-5.0.0, typeguard-4.2.1, mock-3.14.0
collected 19 items

tests/unit/examples/test_notebooks_python.py s...                        [ 21%]
tests/unit/recommenders/utils/test_notebook_utils.py ..........          [ 73%]
tests/unit/examples/test_notebooks_python.py 

=================================== FAILURES ===================================
__________________________ test_sar_single_node_runs ___________________________

notebooks = ***'als_deep_dive': '/mnt/azureml/cr/j/e62514dcd77647bcb13baecdeaa9a748/exe/wd/examples/02_model_collaborative_filtering...rk_movielens': '/mnt/azureml/cr/j/e62514dcd77647bcb13baecdeaa9a748/exe/wd/examples/06_benchmarks/movielens.ipynb', ...***
output_notebook = 'output.ipynb', kernel_name = 'python3'

    @pytest.mark.notebooks
    def test_sar_single_node_runs(notebooks, output_notebook, kernel_name):
        notebook_path = notebooks["sar_single_node"]
>       execute_notebook(notebook_path, output_notebook, kernel_name=kernel_name)

tests/unit/examples/test_notebooks_python.py:34: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
recommenders/utils/notebook_utils.py:102: in execute_notebook
    executed_notebook, _ = execute_preprocessor.preprocess(
/azureml-envs/azureml_e95b5901212d39f15b2a44eca8fce43f/lib/python3.11/site-packages/nbconvert/preprocessors/execute.py:102: in preprocess
    self.preprocess_cell(cell, resources, index)
/azureml-envs/azureml_e95b5901212d39f15b2a44eca8fce43f/lib/python3.11/site-packages/nbconvert/preprocessors/execute.py:123: in preprocess_cell
    cell = self.execute_cell(cell, index, store_history=True)
/azureml-envs/azureml_e95b5901212d39f15b2a44eca8fce43f/lib/python3.11/site-packages/jupyter_core/utils/__init__.py:165: in wrapped
    return loop.run_until_complete(inner)
/azureml-envs/azureml_e95b5901212d39f15b2a44eca8fce43f/lib/python3.11/asyncio/base_events.py:653: in run_until_complete
    return future.result()
/azureml-envs/azureml_e95b5901212d39f15b2a44eca8fce43f/lib/python3.11/site-packages/nbclient/client.py:1062: in async_execute_cell
    await self._check_raise_for_error(cell, cell_index, exec_reply)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <nbconvert.preprocessors.execute.ExecutePreprocessor object at 0x149eec95f690>
cell = ***'cell_type': 'code', 'execution_count': 8, 'metadata': ***'execution': ***'iopub.status.busy': '2024-04-08T15:15:06.80546...d_k_items(test, top_k=TOP_K, remove_seen=True)\n\nprint("Took *** seconds for prediction.".format(test_time.interval))'***
cell_index = 16
exec_reply = ***'buffers': [], 'content': ***'ename': 'TypeError', 'engine_info': ***'engine_id': -1, 'engine_uuid': 'bd67981e-ed10-462d-...e, 'engine': 'bd67981e-ed10-462d-bdbd-359a88e1244a', 'started': '2024-04-08T15:15:06.805810Z', 'status': 'error'***, ...***

    async def _check_raise_for_error(
        self, cell: NotebookNode, cell_index: int, exec_reply: dict[str, t.Any] | None
    ) -> None:
        if exec_reply is None:
            return None

        exec_reply_content = exec_reply["content"]
        if exec_reply_content["status"] != "error":
            return None

        cell_allows_errors = (not self.force_raise_errors) and (
            self.allow_errors
            or exec_reply_content.get("ename") in self.allow_error_names
            or "raises-exception" in cell.metadata.get("tags", [])
        )
        await run_hook(
            self.on_cell_error, cell=cell, cell_index=cell_index, execute_reply=exec_reply
        )
        if not cell_allows_errors:
>           raise CellExecutionError.from_cell_and_msg(cell, exec_reply_content)
E           nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:
E           ------------------
E           with Timer() as test_time:
E               top_k = model.recommend_k_items(test, top_k=TOP_K, remove_seen=True)
E           
E           print("Took *** seconds for prediction.".format(test_time.interval))
E           ------------------
E           
E           ----- stderr -----
E           2024-04-08 15:15:06,821 INFO     Calculating recommendation scores
E           ------------------
E           
E           ---------------------------------------------------------------------------
E           TypeError                                 Traceback (most recent call last)
E           Cell In[8], line 2
E                 1 with Timer() as test_time:
E           ----> 2     top_k = model.recommend_k_items(test,top_k=TOP_K,remove_seen=True)
E                 4 print("Took *** seconds for prediction.".format(test_time.interval))
E           
E           File /mnt/azureml/cr/j/e62514dcd77647bcb13baecdeaa9a748/exe/wd/recommenders/models/sar/sar_singlenode.py:535, in SARSingleNode.recommend_k_items(self, test, top_k, sort_top_k, remove_seen)
E               522 def recommend_k_items(self, test, top_k=10, sort_top_k=True, remove_seen=False):
E               523     """Recommend top K items for all users which are in the test set
E               524 
E               525     Args:
E              (...)
E               532         pandas.DataFrame: top k recommendation items for each user
E               533     """
E           --> 535     test_scores = self.score(test,remove_seen=remove_seen)
E               537     top_items, top_scores = get_top_k_scored_items(
E               538         scores=test_scores, top_k=top_k, sort_top_k=sort_top_k
E               539     )
E               541     df = pd.DataFrame(
E               542         ***
E               543             self.col_user: np.repeat(
E              (...)
E               548         ***
E               549     )
E           
E           File /mnt/azureml/cr/j/e62514dcd77647bcb13baecdeaa9a748/exe/wd/recommenders/models/sar/sar_singlenode.py:357, in SARSingleNode.score(self, test, remove_seen)
E               354 if self.normalize:
E               355     counts = self.unity_user_affinity[user_ids, :].dot(self.item_similarity)
E               356     user_min_scores = (
E           --> 357         np.tile(counts.min(axis=1)[:,np.newaxis], test_scores.shape[1])
E               358         * self.rating_min
E               359     )
E               360     user_max_scores = (
E               361         np.tile(counts.max(axis=1)[:, np.newaxis], test_scores.shape[1])
E               362         * self.rating_max
E               363     )
E               364     test_scores = rescale(
E               365         test_scores,
E               366         self.rating_min,
E              (...)
E               369         user_max_scores,
E               370     )
E           
E           TypeError: 'coo_matrix' object is not subscriptable

/azureml-envs/azureml_e95b5901212d39f15b2a44eca8fce43f/lib/python3.11/site-packages/nbclient/client.py:918: CellExecutionError

miguelgfierro commented 5 months ago

@gramhagen it seems we still have the same error

gramhagen commented 5 months ago

oh, i see. I didn't realize that only occurred on scipy 1.13. counts should be a csr matrix, but if you get the min it looks like it's converting it back to coo which you can't index with [:, np.newaxis] at this point the scores are dense arrays, so I think the right solution is to convert the min and max results to arrays as well. I will add a suggestion to fix this

miguelgfierro commented 5 months ago

New error:

$ pytest tests/unit/recommenders/models/test_sar_singlenode.py --disabl
e-warnings
================================================= test session starts ==================================================
platform linux -- Python 3.10.14, pytest-8.1.1, pluggy-1.4.0
rootdir: /home/miguel/MS/recommenders
configfile: pyproject.toml
plugins: hypothesis-6.100.1, anyio-4.3.0, cov-5.0.0, typeguard-4.2.1, mock-3.14.0
collected 36 items                                               FF.FFFFFF.....F.F........                                                                      [100%]

============================================================================== FAILURES ===============================================================================
_______________________________________________________________ test_sar_item_similarity[1-cosine-cos] ________________________________________________________________
threshold = 1, similarity_type = 'cosine', file = 'cos'
demo_usage_data =                  UserId    MovieId     Timestamp  Rating
0      0003000098E85347  DQF-00358  1.433879e+09       1
1   ...4C72  DQF-00248  1.416293e+09       1
11675  00037FFE818E4C72  DAF-00375  1.416293e+09       1

[7189 rows x 4 columns]
sar_settings = {'ATOL': 1e-08, 'FILE_DIR': 'https://recodatasets.z20.web.core.windows.net/sarunittest/', 'TEST_USER_ID': '0003000098E85347'}
header = {'col_item': 'MovieId', 'col_rating': 'Rating', 'col_timestamp': 'Timestamp', 'col_user': 'UserId'}

    @pytest.mark.parametrize(
        "threshold,similarity_type,file",
        [
            (1, "cooccurrence", "count"),
            (1, "cosine", "cos"),
            (1, "inclusion index", "incl"),
            (1, "jaccard", "jac"),
            (1, "lexicographers mutual information", "lex"),
            (1, "lift", "lift"),
            (1, "mutual information", "mi"),
            (3, "cooccurrence", "count"),
            (3, "cosine", "cos"),
            (3, "inclusion index", "incl"),
            (3, "jaccard", "jac"),
            (3, "lexicographers mutual information", "lex"),
            (3, "lift", "lift"),
            (3, "mutual information", "mi"),
        ],
    )
    def test_sar_item_similarity(
        threshold, similarity_type, file, demo_usage_data, sar_settings, header
    ):

        model = SAR(
            similarity_type=similarity_type,
            timedecay_formula=False,
            time_decay_coefficient=30,
            threshold=threshold,
            **header
        )

        # Remove duplicates
        demo_usage_data = demo_usage_data.sort_values(
            header["col_timestamp"], ascending=False
        )
        demo_usage_data = demo_usage_data.drop_duplicates(
            [header["col_user"], header["col_item"]], keep="first"
        )

        model.fit(demo_usage_data)

        true_item_similarity = pd.read_csv(
            sar_settings["FILE_DIR"] + "sim_" + file + str(threshold) + ".csv", index_col=0
        )
        item2index = pd.Series(model.item2index)
        index = item2index[true_item_similarity.index]
        columns = item2index[true_item_similarity.columns]

        if similarity_type == "cooccurrence":
            test_item_similarity = pd.DataFrame(model.item_similarity.todense())
            test_item_similarity = test_item_similarity.reindex(
                index=index, columns=columns
            )
            assert np.array_equal(
                true_item_similarity.astype("float64"),
                test_item_similarity.astype("float64"),
            )
        else:
>           test_item_similarity = pd.DataFrame(model.item_similarity)

tests/unit/recommenders/models/test_sar_singlenode.py:138:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../anaconda/envs/recommenders310/lib/python3.10/site-packages/pandas/core/frame.py:843: in __init__
    data = list(data)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <101x101 sparse matrix of type '<class 'numpy.float64'>'
        with 3581 stored elements in COOrdinate format>

    def __iter__(self):
        for r in range(self.shape[0]):
>           yield self[r]
E           TypeError: 'coo_matrix' object is not subscriptable

../../anaconda/envs/recommenders310/lib/python3.10/site-packages/scipy/sparse/_base.py:260: TypeError

miguelgfierro commented 5 months ago

Trying with @anargyri and @SimonYansenZhao to roll back and change the array casting pytest tests/unit/examples/test_notebooks_python.py::test_sar_single_node_runs

    def score(self, test, remove_seen=False):
        """Score all items for test users.

        Args:
            test (pandas.DataFrame): user to test
            remove_seen (bool): flag to remove items seen in training from recommendation

        Returns:
            numpy.ndarray: Value of interest of all items for the users.
        """

        # get user / item indices from test set
        user_ids = list(
            map(
                lambda user: self.user2index.get(user, np.NaN),
                test[self.col_user].unique(),
            )
        )
        if any(np.isnan(user_ids)):
            raise ValueError("SAR cannot score users that are not in the training set")

        # calculate raw scores with a matrix multiplication
        logger.info("Calculating recommendation scores")
        test_scores = self.user_affinity.toarray()[user_ids, :].dot(
            self.item_similarity
        )

Small example:

user_affinity = csr_array([[1, 2, 0], [0, 0, 3], [4, 0, 5]])
item_similarity = np.array([[1, 1, -1],[1,2,3],[4,5,6]])
user_affinity.dot(item_similarity)

array([[ 3,  5,  5],
       [12, 15, 18],
       [24, 29, 26]])

user_ids = [0, 1]
user_affinity[user_ids, :].dot(item_similarity)

array([[ 3,  5,  5],
       [12, 15, 18]])

SimonYansenZhao commented 5 months ago

@miguelgfierro @anargyri I think numpy or scipy has different behaviors with Python 3.8 and other Python versions, because the same tests in group_notebooks_cpu_001 with Python 3.9+ passed but failed with Python 3.8 after I changed item_similarity from np.array(result) to result.toarray().

And I found that if testing with Python 3.8, the latest supported scipy version is 1.10 which worked with np.array(result), but when testing with Python 3.9, scipy 1.11+ will be installed but works with result.toarray().

miguelgfierro commented 5 months ago

@SimonYansenZhao the world is falling if one can't trust numpy or scipy anymore.

miguelgfierro commented 5 months ago

Awesome! @SimonYansenZhao @anargyri @loomlike can you guys accept the PR? since I started it, I can't accept it

recommenders-team / recommenders

SAR sparse multiplcation modification due to a breaking change in scipy #2083

Description

Related Issues

1954

References

Checklist: