Cannot achieve results as reported in the paper

[thasorn]

Dear @LiDan456 and all,

I have some problem with testing part and would like to ask for help.

After trying to run the model follows your instructions and code, I managed to train RGAN.py model with a SWAT data set (Pure normal physical data), which working great so far.

However, I have some problem with test in AD.py part. I am trying to test the model (GAN-AD with 5 PCA components) with the SWAT data set (Abnormal physical data) with your original code and default setting but I could not get the results close to reported performance in your paper. (The paper reported that GAN-AD with 5 PCA components can outperform other methods. The Accuracy is 94.80, Precision is 93.33, Recall is 63.64, F-1 is 0.75 and FPR is 0.46.) The result that I get is pretty poor and not as good as it should. (Please see the attached images)

I also try training and testing again more than 10 times, but the best result I could achieve is Accuracy around 60-70% (which is far from the reported result)

As mention above, I would like to ask you questions as follows:

1. Why I could not achieve the same result as reported in your paper? Do you have any suggestion about my mistake I might make in code or implementation processes or some setting might be missing?

2. Did you do the cross-validation when you test? I am trying to read your paper and understand your code, but I still could not find any cross-validation method that you might implement. Please let me know if I misunderstood.

I would appreciate any comments, helps or suggestions. Thank you very much for your answer in advance.

[Kbishou]

Hay @thasorn , I get some some trouble with obtain the Swat data set , Thanks a lot if you can send me that? and what prepare work should I to on the dataset before doing the experiment

[LiDan456]

Hi Kbishou,

I cannot distribute the Swat dataset by myself. You can apply the dataset via the online application form here https://itrust.sutd.edu.sg/itrust-labs_datasets/

For the preparation, it is based on your purpose of using this dataset. Usually, you can use it directly.

Best, Dan

From: Kbishoumailto:notifications@github.com Sent: Tuesday, 1 September 2020 11:28 AM To: LiDan456/GAN-ADmailto:GAN-AD@noreply.github.com Cc: Li Danmailto:dli006@hotmail.com; Mentionmailto:mention@noreply.github.com Subject: Re: [LiDan456/GAN-AD] Cannot achieve results as reported in the paper (#3)

Hay @thasornhttps://github.com/thasorn , I get some some trouble with obtain the Swat data set , Thanks a lot if you can send me that? and what prepare work should I to on the dataset before doing the experiment

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/LiDan456/GAN-AD/issues/3#issuecomment-684174124, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AJAVURJEFGKXZWRGY25ESYLSDRS5XANCNFSM4II2J7AQ.

[Kbishou]

@LiDan456 Thank you for your reply , but I don't find any .csv and .npy file in dataset from https://itrust.sutd.edu.sg/itrust-labs_datasets/ . How can I get files in these type? Thanks again

[LiDan456]

I think you should fill in the application form first (https://docs.google.com/forms/d/1GOLYXa7TX0KlayqugUOOPMvbcwSQiGNMOjHuNqKcieA/viewform?edit_requested=true).

Base on my experience, after submitting the application form, you will receive an email which will tell you where to download the dataset.

Best, Dan

From: Kbishoumailto:notifications@github.com Sent: Tuesday, 1 September 2020 4:41 PM To: LiDan456/GAN-ADmailto:GAN-AD@noreply.github.com Cc: Li Danmailto:dli006@hotmail.com; Mentionmailto:mention@noreply.github.com Subject: Re: [LiDan456/GAN-AD] Cannot achieve results as reported in the paper (#3)

@LiDan456https://github.com/LiDan456 Thank you for your reply , but I don't find any .csv and .npy file in dataset from https://itrust.sutd.edu.sg/itrust-labs_datasets/ . How can I get files in these type? Thanks again

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/LiDan456/GAN-AD/issues/3#issuecomment-684584432, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AJAVURKEMNLVKG4KODTORQLSDSXSHANCNFSM4II2J7AQ.

[Kbishou]

Dear@LiDan456, Actually i have got the Swat dataset. I mean there are several folders (see attached images) and which one should be used. Besides, i cannot find any .csv and .npy file in the folders

[LiDan456]

Sorry for misunderstanding your problem. Maybe they have changed the format. Can you ask iTrust directly? The one who share you the dataset.

Best, Dan

发件人: Kbishoumailto:notifications@github.com 发送时间: Tuesday, 1 September 2020 6:04 PM 收件人: LiDan456/GAN-ADmailto:GAN-AD@noreply.github.com 抄送: Li Danmailto:dli006@hotmail.com; Mentionmailto:mention@noreply.github.com 主题: Re: [LiDan456/GAN-AD] Cannot achieve results as reported in the paper (#3)

Dear@LiDan456, Actually i have got the Swat dataset. I mean there are several folders (see attached images) and which one should be used. Besides, i cannot find any .csv and .npy file in the folders [image]https://user-images.githubusercontent.com/54734032/91836516-78349300-ec7d-11ea-84e3-227606d10308.png

[image]https://user-images.githubusercontent.com/54734032/91836436-589d6a80-ec7d-11ea-88c9-e7225b372290.png

― You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/LiDan456/GAN-AD/issues/3#issuecomment-684707548, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AJAVURP2WK7UKB4RAO6QOSTSDTBJFANCNFSM4II2J7AQ.

[Kbishou]

Dear @thasorn Did you get this issue (see attached image) when running "python RGAN.py --settings_file swat_gen"? It puzzles me for days.
Do you know how to fix it. Best wishes KB

[thasorn]

Dear @Kbishou ,

I cannot remember it clearly, since I have not touched this code over a year, but it seems like you face with the data shape problem. I would suggest you to investigate your code that related to data shape in line 70, or you might want to take a look in GAN-AD -> experiments -> setting, then choose a swat_gen.txt file or file with the model name that you currently work on. Try to tune the parameter that might related to. If I were you, I might change 'seq_length' number to 7200.

Hope this help,

C. Thasorn

[Kbishou]

@thasorn Thanks for help and it is work by changing 'seq_length' number to 7200. However, there arise another issue...... I extremely want to know wether you get this problem, and can you tell me how to solve it ? soooory iam a beginner on GAN ....

[jyoti922]

Dear @LiDan456 and all, can anyone help me understand this code .I m new to this field. as I am running the code its showing error.Can anyone suggest me some reference from which I can understand the proper way of using this code and files.

[shengbinzheng]

Dear @LiDan456 and all,

I have some problem with testing part and would like to ask for help.

After trying to run the model follows your instructions and code, I managed to train RGAN.py model with a SWAT data set (Pure normal physical data), which working great so far.

However, I have some problem with test in AD.py part. I am trying to test the model (GAN-AD with 5 PCA components) with the SWAT data set (Abnormal physical data) with your original code and default setting but I could not get the results close to reported performance in your paper. (The paper reported that GAN-AD with 5 PCA components can outperform other methods. The Accuracy is 94.80, Precision is 93.33, Recall is 63.64, F-1 is 0.75 and FPR is 0.46.) The result that I get is pretty poor and not as good as it should. (Please see the attached images)

I also try training and testing again more than 10 times, but the best result I could achieve is Accuracy around 60-70% (which is far from the reported result)

As mention above, I would like to ask you questions as follows:

1. Why I could not achieve the same result as reported in your paper? Do you have any suggestion about my mistake I might make in code or implementation processes or some setting might be missing?
2. Did you do the cross-validation when you test? I am trying to read your paper and understand your code, but I still could not find any cross-validation method that you might implement. Please let me know if I misunderstood.

I would appreciate any comments, helps or suggestions. Thank you very much for your answer in advance.

hey @thasorn , I have some issue with you , I got a very poor performence in anomaly dectection(AD.py) . Do you find any solution about it?

zheng

[shengbinzheng]

@thasorn Thanks for help and it is work by changing 'seq_length' number to 7200. However, there arise another issue...... I extremely want to know wether you get this problem, and can you tell me how to solve it ? soooory iam a beginner on GAN ....

@Kbishou same issue, did you figure it out?

[shengbinzheng]

hey@thasorn.I have a question .What is """python RGAN.py --settings_file swat_gen""" going to do ? looking forward to your replying