PreferredAI / cornac

A Comparative Framework for Multimodal Recommender Systems
https://cornac.preferred.ai
Apache License 2.0
868 stars 142 forks source link

Enhance serving evaluation endpoints #595

Closed darrylong closed 7 months ago

darrylong commented 7 months ago

Description

This PR involves 3 changes to the serving functions:

  1. Add metric_user_results to evaluation results as user_result
  2. Modified /evaluate endpoint to accept evaluation data in the form of a json if data is included. Else, it will follow previous behavior of using feedback data added through the /feedback endpoint.
  3. query is removed from reponse

Sample request for /evaluate:

{
    "metrics": ["RMSE()", "NDCG(k=10)"],
    "data": [
        ["123", "1539", 1],
        ["123", "2", 1],
        ["124", "1", 1]
    ]
}

Response:

{
    "result": {
        "NDCG@10": 0.3175294778309396,
        "RMSE": 2.781925109617526
    },
    "user_result": {
        "NDCG@10": {
            "123": 0.20438239758848611,
            "124": 0.43067655807339306
        },
        "RMSE": {
            "123": 2.244862849697699,
            "124": 3.3189873695373535
        }
    }
}

Related Issues

Checklist:

tqtg commented 7 months ago

I think we can remove "query" from the response to minimize the bandwidth because "data" could be quite significant.

tqtg commented 7 months ago

Also, the returned user_result contains mapped user indices. Shall we try to map them back to the original user IDs, using the mapping in train_set, for it to be consistent with data in the request?

darrylong commented 7 months ago

Also, the returned user_result contains mapped user indices. Shall we try to map them back to the original user IDs, using the mapping in train_set, for it to be consistent with data in the request?

Now using mapped user indices, new response is updated in main post.