How use customized dataset?

[chiyuzhang94]

Hi @andreeaiana ,

Thanks for this great work!

I wonder if I can run models with a customized dataset or not. Any guidance?

Best, Chiyu

[andreeaiana]

Hi Chiyu,

You can run the implemented models with a customized dataset. For this, you would have to do the following:

• Implement your own data preprocessing class (see MIND preprocessing for an example).

• The output of the data preprocessing should be in the following format: _ news should be a pd.DataFrame containing news features as specified as a list of _datasetattributes (see MIND config file for an example of supported features) _ behaviors should be a pd.DataFrame containing the click behavior of users in the format: user (ID), history (list of clicked news IDs), candidates (list of candidate news IDs), labels (binary, for the candidates).

• Create a dedicated config file for your dataset (see MIND config file for an example). Most importantly, you need to define which attributes your dataset has, by modifying the _datasetattributes variable (e.g., if you don't have any sentiment scores, remove the value from the list of used attributes).

• If you're planning to use categorical variables (e.g., topical category), this should be encoded during the dataset preprocessing.

Let me know if you have any more questions.

Best, Andreea

[chiyuzhang94]

Hi @andreeaiana ,

Thanks for the explanation. I wonder if you can provide a sample of processed dataset?

[andreeaiana]

I sent you an email with samples of the processed dataset.

[chiyuzhang94]

Hi @andreeaiana ,

I wonder how you split the development and testing set. Where I can find the code for this? Particularly, I want to use my split of MIND dataset instead of downloading the original file and split. I can prepare a dataset to match the original format of MIND dataset. I would like to know how to pass my prepared data and splits to your tool.

[andreeaiana]

I split the MIND dataset in the mind_dataframe.py script.

• If you want to re-use that script, you should set download = False, and modified the _load_behaviors function where the train/val splits are created.
• If you have a custom dataset with your own preprocessing script, pass the different splits directly to the mind_rec_datamodule.py

[chiyuzhang94]

Thanks.

Should I change these lines (1, 2) to be data_dir=to_my_file_path

Should I give the root path to the path to each tsv file?

[andreeaiana]

Yes, you can change the data_dir=my_file_path either in the configuration of your datamodule or that of your experiment.