search

benedekrozemberczki / pytorch_geometric_temporal

PyTorch Geometric Temporal: Spatiotemporal Signal Processing with Neural Machine Learning Models (CIKM 2021)
MIT License
2.62k stars 368 forks source link

Question about example code #87

Closed smatovski closed 3 years ago

smatovski commented 3 years ago

Hi, I've been trying to reuse the example code, MPNNLSTM, GCLSTM, and GConvLSTM with my own dataset. My MSE errors are around .2 but the models do not return correct target values. When I run the examples the MSE is around 1. What does MSE represent? How should I structure data my data for these models? I have a graph with 20 nodes, 2 features per node, and 32 edges. The dataset has 360 separate graphs. My dataset looks similar to the one used in MPNNLSTM.

benedekrozemberczki commented 3 years ago

Hi there,

What are we talking about? Which dataset?

MSE is mean squared error.

https://en.wikipedia.org/wiki/Mean_squared_error

The documentation described the data types needed for loading datasets. If you are willing to contribute the dataset we can create an integrated loader. Bests,

Benedek

smatovski commented 3 years ago

If I run the mpnnlstm_example.py. I get an MSE of 1.0663 using the Chickenpox dataset. gconvlstm_example.py gives an MSE of 1.0904 with the Chickenpox dataset. gclstm_example.py gives an MSE of 1.0784 with the Chickenpox dataset. These MSE all look to be really high not sure if there is something wrong in the code or my environment.

If I replace the Chickenpox dataset with one I built. The MSE drops to .2 for each model but when I look at the target data, y_hat, I am not seeing the values I am expecting. Not sure what I am doing wrong.

benedekrozemberczki commented 3 years ago

Why would the performance be the same on two different datasets?

On Mon, 2 Aug 2021 at 22:16, smatovski @.***> wrote:

If I run the mpnnlstm_example.py. I get an MSE of 1.0663 using the Chickenpox dataset. gconvlstm_example.py gives an MSE of 1.0904 with the Chickenpox dataset. gclstm_example.py gives an MSE of 1.0784 with the Chickenpox dataset. These MSE all look to be really high not sure if there is something wrong in the code or my environment.

If I replace the Chickenpox dataset with one I built. The MSE drops to .2 for each model but when I look at the target data, y_hat, I am not seeing the values I am expecting. Not sure what I am doing wrong.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/benedekrozemberczki/pytorch_geometric_temporal/issues/87#issuecomment-891339104, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEETMF574HW5ZVL5MSCXQR3T24DLZANCNFSM5BNKQUEA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

smatovski commented 3 years ago

I'm not sure why the code examples from the repo have a similar MSE. I did not modify them only ran the code on my local environment/machine. Using Acondana for my python environment.

smatovski commented 3 years ago

I copied the code from the Epidemiological Forecasting, https://pytorch-geometric-temporal.readthedocs.io/en/latest/notes/introduction.html#epidemiological-forecasting, and ran it. I receive an MSE of 1.0232, not 0.6866. Can you please explain why?

benedekrozemberczki commented 3 years ago

Different datasets have a different loss value.

On Tue, 3 Aug 2021 at 02:21, smatovski @.***> wrote:

I copied the code from the Epidemiological Forecasting, https://pytorch-geometric-temporal.readthedocs.io/en/latest/notes/introduction.html#epidemiological-forecasting, and ran it. I receive an MSE of 1.0232, not 0.6866. Can you please explain why?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/benedekrozemberczki/pytorch_geometric_temporal/issues/87#issuecomment-891437602, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEETMFYSDY4MSLUZVWSI7R3T25ABPANCNFSM5BNKQUEA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

smatovski commented 3 years ago

But if I copy the code from GitHub shouldn't I get similar results? Your example on Epidemiological Forecasting shows an MSE of .6866. I run the exact same code and got an MSE of 1.0232.

benedekrozemberczki commented 3 years ago

That is fine, the read the docs should be updated.

On Tue, 3 Aug 2021 at 13:38, smatovski @.***> wrote:

But if I copy the code from GitHub shouldn't I get similar results? Your example on Epidemiological Forecasting shows an MSE of .6866. I run the exact same code and got an MSE of 1.0232.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/benedekrozemberczki/pytorch_geometric_temporal/issues/87#issuecomment-891813333, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEETMF5ROFFOSECALMGPXPDT27PNRANCNFSM5BNKQUEA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

smatovski commented 3 years ago

Doesn't an MSE of 1.0232 mean the model is not predicting values correctly?

benedekrozemberczki commented 3 years ago

It means that out of sample it is roughly random - it can be distribution shift, seasonality, and trends.

On Tue, 3 Aug 2021 at 13:55, smatovski @.***> wrote:

Doesn't an MSE of 1.0232 mean the model is not predicting values correctly?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/benedekrozemberczki/pytorch_geometric_temporal/issues/87#issuecomment-891823625, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEETMF57HTSCNX5BFXAK6Y3T27RM5ANCNFSM5BNKQUEA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

smatovski commented 3 years ago

Ok, so the example code evaluate the dataset and comes back with the conclusion that the dataset is very random. Meaning my dataset, SC_Graph_synthetic, is not very random. I want to understand how I can get one of the models to predict the target values. Do I need more data, a different model, ...

benedekrozemberczki commented 3 years ago

Yes, that is something that you have to experiment with.

On Tue, 3 Aug 2021 at 14:29, smatovski @.***> wrote:

Ok, so the example code evaluate the dataset and comes back with the conclusion that the dataset is very random. Meaning my dataset, SC_Graph_synthetic, is not very random. I want to understand how I can get one of the models to predict the target values. Do I need more data, a different model, ...

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/benedekrozemberczki/pytorch_geometric_temporal/issues/87#issuecomment-891847490, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEETMF3EYZUGRPAAYJIXX7TT27VKHANCNFSM5BNKQUEA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

benedekrozemberczki commented 3 years ago

@smatovski Could you star the repo?

smatovski commented 3 years ago

Started a repo of my code and dataset. https://github.com/smatovski/GNN_Supply_Chain (currently private)