search

dzhv / Spatio-Temporal-mobile-traffic-forecasting

Mobile Traffic Prediction using Deep Learning models
24 stars 18 forks source link

Missing Files in Data Folder #1

Closed saloni103 closed 2 years ago

saloni103 commented 4 years ago

Hi, I am not able to find mini_train.npy and similar .npy files under the data folder.

AssertionError: Data file does not exist at expected path: data\mini_train.npy

dzhv commented 4 years ago

Hi. I've just added mini_train.npy, mini_val.npy and mini_test.npy files. However, these include only a small number of examples from the whole dataset.

I cannot include the whole dataset as it is too large for a git repository. However, if you need the whole dataset let me know and I will try to help you acquire and preprocess it.

saloni103 commented 4 years ago

Hello, I am unable to get the whole dataset and the website does not seem to have it. Also for the notebooks I am unable to locate predictions.npy file in the mentioned path. If you could help me with that. Thanks.

dzhv commented 4 years ago

It looks as if the Dandelion API website is not providing the dataset anymore: https://dandelion.eu/datamine/open-big-data/

However, maybe it's possible to get the data from here: https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/EGZHFV

If you manage to get the data files, try looking into this script for preprocessing: https://github.com/dzhv/Spatio-Temporal-mobile-traffic-forecasting/blob/master/utilities/data_grid_mapping.py

Sorry, I know the script might not be very clear as it was written in a "do once -> use once -> forget" manner. But try looking into map_november or map_decemberfunctions. They expect November or December data files (named something like sms-call-internet-mi-2013-11-06.txt) to be in the data/november_input or data/december_input directories. The functions map the raw data files to numpy tensors and saves them.

dzhv commented 4 years ago

Regarding the predictions.npy, I have a feeling that, at least at the start, you do not need it (together with the notebook analyzing predictions).

predictions.npy should be a file containing a trained model's predictions. It is generated by https://github.com/dzhv/Spatio-Temporal-mobile-traffic-forecasting/blob/master/experiments/model_evaluator.py evaluate()function. And I have only used it for analysis of the trained models' predictions visualizing them and trying to get a sense of the prediction quality.