Closed chiyuzhang94 closed 5 months ago
Hi Chiyu,
You can run the implemented models with a customized dataset. For this, you would have to do the following:
Implement your own data preprocessing class (see MIND preprocessing for an example).
The output of the data preprocessing should be in the following format: _ news should be a pd.DataFrame containing news features as specified as a list of _datasetattributes (see MIND config file for an example of supported features) _ behaviors should be a pd.DataFrame containing the click behavior of users in the format: user (ID), history (list of clicked news IDs), candidates (list of candidate news IDs), labels (binary, for the candidates).
Create a dedicated config file for your dataset (see MIND config file for an example). Most importantly, you need to define which attributes your dataset has, by modifying the _datasetattributes variable (e.g., if you don't have any sentiment scores, remove the value from the list of used attributes).
If you're planning to use categorical variables (e.g., topical category), this should be encoded during the dataset preprocessing.
Let me know if you have any more questions.
Best, Andreea
Hi @andreeaiana ,
Thanks for the explanation. I wonder if you can provide a sample of processed dataset?
I sent you an email with samples of the processed dataset.
Hi @andreeaiana ,
I wonder how you split the development and testing set. Where I can find the code for this? Particularly, I want to use my split of MIND dataset instead of downloading the original file and split. I can prepare a dataset to match the original format of MIND dataset. I would like to know how to pass my prepared data and splits to your tool.
I split the MIND dataset in the mind_dataframe.py script.
download = False
, and modified the _load_behaviors
function where the train/val splits are created.Yes, you can change the data_dir=my_file_path
either in the configuration of your datamodule or that of your experiment.
Hi @andreeaiana ,
Thanks for this great work!
I wonder if I can run models with a customized dataset or not. Any guidance?
Best, Chiyu