Help reproducing result

[tejas-kale]

Hi! Many thanks for making the code available. It was helpful to a beginner like me in understanding your work better.

I have a quick question about matching the result I get with the one published (CIC-IDS2017 autoencoder). I cloned the repository and ran main.py with the default arguments. If I understand the code correctly, the results I get should match the results you have shown in section 6.1 (CICIDS2017 Autoencoder results) of your paper. Is that correct? If yes, the results I get (shown below) seem different. Can you please help me understand what I am doing wrong and how I can reproduce the results? Thanks... :)

BEGIN ARGUMENTS
normal_path, DataFiles/CIC/biflow_Monday-WorkingHours_Fixed.csv
attack_paths, DataFiles/CIC/
output, Results.csv
epochs, 50
archi, U15,U9,U15
regu, l2
l1_value, 0.01
l2_value, 0.0001
correlation_value, 0.9
dropout, 0.05
model, autoencoder
nu, 0.01
kern, rbf
loss, mse
END ARGUMENTS
Model: "model"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_2 (InputLayer)         [(None, 18)]              0         
_________________________________________________________________
dense (Dense)                (None, 15)                285       
_________________________________________________________________
dense_1 (Dense)              (None, 9)                 144       
_________________________________________________________________
dense_2 (Dense)              (None, 15)                150       
_________________________________________________________________
dense_3 (Dense)              (None, 18)                288       
=================================================================
Total params: 867
Trainable params: 867
Non-trainable params: 0
_________________________________________________________________
Training accuracy, 0.274017870426178 
 Validation accuracy,0.2714956998825073
File, accuracy, detection accuracy (1- AE acc]
Result using threshold = 0.05 
training, 0.274017870426178, 0.725982129573822, 186455 
0.05, , 0.0, 1.0, 0
0.1, , 0.004344211740098147, 0.9956557882599019, 810
0.15, , 0.07076774556863587, 0.9292322544313641, 13195
0.2, , 0.2616663538119117, 0.7383336461880883, 48789
0.25, , 0.4389262824810276, 0.5610737175189724, 81840
0.3, , 0.5335818293958328, 0.46641817060416724, 99489
0.35, , 0.6005095063151967, 0.39949049368480327, 111968
0.4, , 0.6476951543267813, 0.35230484567321874, 120766
0.45, , 0.6816819071625861, 0.31831809283741386, 127103
0.5, , 0.7110455605910273, 0.2889544394089727, 132578
0.55, , 0.7395779142420423, 0.26042208575795767, 137898
0.6, , 0.7708562387707489, 0.22914376122925106, 143730
0.65, , 0.7992706014856131, 0.20072939851438687, 149028
0.7, , 0.819618674747258, 0.18038132525274198, 152822
0.75, , 0.8330320988978573, 0.16696790110214266, 155323
0.8, , 0.8431632297337159, 0.15683677026628406, 157212
0.85, , 0.85106325923145, 0.14893674076855, 158685
0.9, , 0.8613552867984232, 0.13864471320157679, 160604
0.95, , 0.8762489608752783, 0.12375103912472174, 163381
Result using threshold = 0.05 
validation, 0.2714956998825073, 0.7285043001174927, 62152 
0.05, , 0.0, 1.0, 0
0.1, , 0.004167202986227313, 0.9958327970137727, 259
0.15, , 0.07074591324494788, 0.9292540867550522, 4397
0.2, , 0.26171321920453083, 0.7382867807954692, 16266
0.25, , 0.4385860471103102, 0.5614139528896898, 27259
0.3, , 0.5313264255373922, 0.4686735744626078, 33023
0.35, , 0.599401467370318, 0.40059853262968204, 37254
0.4, , 0.6447258334405972, 0.3552741665594028, 40071
0.45, , 0.6801068348564809, 0.31989316514351906, 42270
0.5, , 0.7087302098082121, 0.2912697901917879, 44049
0.55, , 0.7375949285622345, 0.2624050714377655, 45843
0.6, , 0.7689213540996267, 0.2310786459003733, 47790
0.65, , 0.7969333247522203, 0.20306667524777966, 49531
0.7, , 0.8164821727378041, 0.1835178272621959, 50746
0.75, , 0.8312202342643841, 0.16877976573561593, 51662
0.8, , 0.8418554511520144, 0.15814454884798557, 52323
0.85, , 0.8494336465439567, 0.1505663534560433, 52794
0.9, , 0.8598275196292959, 0.14017248037070407, 53440
0.95, , 0.8743403269404042, 0.1256596730595958, 54342
Result using threshold = 0.05 
new_biflow_Tuesday-WorkingHours_SSH.csv, 0.18941274285316467, 0.8105872571468353, 3627 
0.05, , 0.0, 1.0, 0
0.1, , 0.0, 1.0, 0
0.15, , 0.0016542597187758478, 0.9983457402812241, 6
0.2, , 0.586435070306038, 0.41356492969396197, 2127
0.25, , 0.7821891370278468, 0.21781086297215324, 2837
0.3, , 0.8133443617314585, 0.18665563826854148, 2950
0.35, , 0.8472566859663634, 0.1527433140336366, 3073
0.4, , 0.8753791011855528, 0.12462089881444716, 3175
0.45, , 0.8886131789357595, 0.11138682106424047, 3223
0.5, , 0.8974358974358975, 0.10256410256410253, 3255
0.55, , 0.9076371657016818, 0.09236283429831815, 3292
0.6, , 0.9183898538737248, 0.08161014612627515, 3331
0.65, , 0.926661152467604, 0.07333884753239595, 3361
0.7, , 0.9321753515301903, 0.06782464846980973, 3381
0.75, , 0.9363110008271298, 0.06368899917287019, 3396
0.8, , 0.9363110008271298, 0.06368899917287019, 3396
0.85, , 0.9365867107802591, 0.06341328921974088, 3397
0.9, , 0.9365867107802591, 0.06341328921974088, 3397
0.95, , 0.9365867107802591, 0.06341328921974088, 3397
Result using threshold = 0.05 
new_biflow_Wednesday-WorkingHours_slowloris.csv, 0.15809859335422516, 0.8419014066457748, 2840 
0.05, , 0.0, 1.0, 0
0.1, , 0.00035211267605633805, 0.9996478873239436, 1
0.15, , 0.04894366197183098, 0.951056338028169, 139
0.2, , 0.09612676056338028, 0.9038732394366197, 273
0.25, , 0.20035211267605635, 0.7996478873239437, 569
0.3, , 0.22464788732394367, 0.7753521126760563, 638
0.35, , 0.2545774647887324, 0.7454225352112676, 723
0.4, , 0.2591549295774648, 0.7408450704225352, 736
0.45, , 0.2640845070422535, 0.7359154929577465, 750
0.5, , 0.2679577464788732, 0.7320422535211268, 761
0.55, , 0.2721830985915493, 0.7278169014084507, 773
0.6, , 0.2764084507042254, 0.7235915492957746, 785
0.65, , 0.2813380281690141, 0.7186619718309859, 799
0.7, , 0.2862676056338028, 0.7137323943661972, 813
0.75, , 0.2880281690140845, 0.7119718309859155, 818
0.8, , 0.2880281690140845, 0.7119718309859155, 818
0.85, , 0.28838028169014085, 0.7116197183098592, 819
0.9, , 0.2897887323943662, 0.7102112676056338, 823
0.95, , 0.2897887323943662, 0.7102112676056338, 823
Result using threshold = 0.05 
new_biflow_Friday-WorkingHours_DDoS.csv, 0.004986490122973919, 0.9950135098770261, 45523 
0.05, , 0.0, 1.0, 0
0.1, , 0.0, 1.0, 0
0.15, , 0.0, 1.0, 0
0.2, , 0.0005272060277222503, 0.9994727939722777, 24
0.25, , 0.0021747248643542824, 0.9978252751356457, 99
0.3, , 0.002657997056433012, 0.997342002943567, 121
0.35, , 0.0032950376732640645, 0.9967049623267359, 150
0.4, , 0.0041078136326692, 0.9958921863673308, 187
0.45, , 0.004283548975243284, 0.9957164510247567, 195
0.5, , 0.004415350482173846, 0.9955846495178261, 201
0.55, , 0.004525185071282648, 0.9954748149287174, 206
0.6, , 0.004722887331678492, 0.9952771126683215, 215
0.65, , 0.004832721920787294, 0.9951672780792127, 220
0.7, , 0.005315994112866024, 0.994684005887134, 242
0.75, , 0.005425828701974826, 0.9945741712980252, 247
0.8, , 0.005425828701974826, 0.9945741712980252, 247
0.85, , 0.0054477956197965865, 0.9945522043802034, 248
0.9, , 0.0054477956197965865, 0.9945522043802034, 248
0.95, , 0.0054477956197965865, 0.9945522043802034, 248
Result using threshold = 0.05 
new_biflow_Wednesday-WorkingHours_Slowhttptest.csv, 0.2744464874267578, 0.7255535125732422, 2168 
0.05, , 0.0, 1.0, 0
0.1, , 0.0, 1.0, 0
0.15, , 0.0, 1.0, 0
0.2, , 0.11346863468634687, 0.8865313653136532, 246
0.25, , 0.1881918819188192, 0.8118081180811808, 408
0.3, , 0.38330258302583026, 0.6166974169741697, 831
0.35, , 0.5488929889298892, 0.45110701107011075, 1190
0.4, , 0.5618081180811808, 0.4381918819188192, 1218
0.45, , 0.5710332103321033, 0.4289667896678967, 1238
0.5, , 0.5811808118081181, 0.4188191881918819, 1260
0.55, , 0.584409594095941, 0.41559040590405905, 1267
0.6, , 0.5950184501845018, 0.40498154981549817, 1290
0.65, , 0.6023985239852399, 0.39760147601476015, 1306
0.7, , 0.6070110701107011, 0.3929889298892989, 1316
0.75, , 0.6116236162361623, 0.38837638376383765, 1326
0.8, , 0.6120848708487084, 0.38791512915129156, 1327
0.85, , 0.6130073800738007, 0.38699261992619927, 1329
0.9, , 0.6134686346863468, 0.3865313653136532, 1330
0.95, , 0.613929889298893, 0.38607011070110697, 1331
Result using threshold = 0.05 
new_biflow_Tuesday-WorkingHours_FTP.csv, 0.14747513830661774, 0.8525248616933823, 5228 
0.05, , 0.0, 1.0, 0
0.1, , 0.0, 1.0, 0
0.15, , 0.004781943381790359, 0.9952180566182096, 25
0.2, , 0.041698546289211935, 0.9583014537107881, 218
0.25, , 0.09506503442999235, 0.9049349655700076, 497
0.3, , 0.11610558530986993, 0.8838944146901301, 607
0.35, , 0.1478576893649579, 0.8521423106350421, 773
0.4, , 0.16545524100994644, 0.8345447589900535, 865
0.45, , 0.17253251721499618, 0.8274674827850038, 902
0.5, , 0.1796097934200459, 0.8203902065799541, 939
0.55, , 0.18592195868400918, 0.8140780413159908, 972
0.6, , 0.19376434583014537, 0.8062356541698547, 1013
0.65, , 0.2008416220351951, 0.7991583779648049, 1050
0.7, , 0.2090665646518745, 0.7909334353481254, 1093
0.75, , 0.2113618974751339, 0.7886381025248661, 1105
0.8, , 0.21174445294567712, 0.7882555470543229, 1107
0.85, , 0.21365723029839326, 0.7863427697016068, 1117
0.9, , 0.21423106350420812, 0.7857689364957918, 1120
0.95, , 0.21442234123947973, 0.7855776587605203, 1121
Result using threshold = 0.05 
new_biflow_Wednesday-WorkingHours_GoldenEye.csv, 0.0445588044822216, 0.9554411955177784, 7967 
0.05, , 0.0, 1.0, 0
0.1, , 0.0, 1.0, 0
0.15, , 0.0, 1.0, 0
0.2, , 0.0038910505836575876, 0.9961089494163424, 31
0.25, , 0.010041420861051838, 0.9899585791389481, 80
0.3, , 0.012175222794025355, 0.9878247772059746, 97
0.35, , 0.014560060248525167, 0.9854399397514748, 116
0.4, , 0.016568344420735535, 0.9834316555792645, 132
0.45, , 0.05422367264967993, 0.94577632735032, 432
0.5, , 0.07568720974017823, 0.9243127902598218, 603
0.55, , 0.0776954939123886, 0.9223045060876114, 619
0.6, , 0.07957826032383582, 0.9204217396761641, 634
0.65, , 0.08271620434291452, 0.9172837956570855, 659
0.7, , 0.08648173716580897, 0.913518262834191, 689
0.75, , 0.08811346805572988, 0.9118865319442702, 702
0.8, , 0.08886657462030878, 0.9111334253796912, 708
0.85, , 0.08936864566336136, 0.9106313543366387, 712
0.9, , 0.0902472699887034, 0.9097527300112966, 719
0.95, , 0.09037278774946655, 0.9096272122505334, 720
Result using threshold = 0.05 
new_biflow_Thursday-WorkingHours_Wev_XSS.csv, 0.004026845563203096, 0.9959731544367969, 745 
0.05, , 0.0, 1.0, 0
0.1, , 0.0, 1.0, 0
0.15, , 0.0, 1.0, 0
0.2, , 0.009395973154362415, 0.9906040268456375, 7
0.25, , 0.43087248322147653, 0.5691275167785235, 321
0.3, , 0.87248322147651, 0.12751677852348997, 650
0.35, , 0.9154362416107382, 0.08456375838926178, 682
0.4, , 0.9422818791946309, 0.05771812080536909, 702
0.45, , 0.9422818791946309, 0.05771812080536909, 702
0.5, , 0.9422818791946309, 0.05771812080536909, 702
0.55, , 0.9422818791946309, 0.05771812080536909, 702
0.6, , 0.9422818791946309, 0.05771812080536909, 702
0.65, , 0.9422818791946309, 0.05771812080536909, 702
0.7, , 0.9449664429530201, 0.055033557046979875, 704
0.75, , 0.9476510067114094, 0.05234899328859055, 706
0.8, , 0.9476510067114094, 0.05234899328859055, 706
0.85, , 0.9503355704697987, 0.04966442953020134, 708
0.9, , 0.9503355704697987, 0.04966442953020134, 708
0.95, , 0.9503355704697987, 0.04966442953020134, 708
Result using threshold = 0.05 
biflow_Monday-WorkingHours_Fixed.csv, 0.2733873128890991, 0.7266126871109009, 248607 
0.05, , 0.0, 1.0, 0
0.1, , 0.004299959373629866, 0.9957000406263702, 1069
0.15, , 0.07076228746575922, 0.9292377125342408, 17592
0.2, , 0.26167807020719447, 0.7383219297928055, 65055
0.25, , 0.43884122329620645, 0.5611587767037935, 109099
0.3, , 0.5330179761631813, 0.46698202383681875, 132512
0.35, , 0.6002324954647295, 0.39976750453527055, 149222
0.4, , 0.6469528211192767, 0.35304717888072334, 160837
0.45, , 0.681288137502162, 0.318711862497838, 169373
0.5, , 0.7104667205669993, 0.28953327943300067, 176627
0.55, , 0.7390821658279936, 0.2609178341720064, 183741
0.6, , 0.7703725156572422, 0.22962748434275781, 191520
0.65, , 0.7986862799518919, 0.2013137200481081, 198559
0.7, , 0.8188345460908181, 0.18116545390918193, 203568
0.75, , 0.8325791309174722, 0.16742086908252785, 206985
0.8, , 0.8428362837731842, 0.1571637162268158, 209535
0.85, , 0.8506558544208329, 0.1493441455791671, 211479
0.9, , 0.8609733434698138, 0.1390266565301862, 214044
0.95, , 0.8757718004722312, 0.12422819952776876, 217723
Result using threshold = 0.05 
new_biflow_Thursday-WorkingHours_Dropbox1.csv, 0.0, 1.0, 17 
0.05, , 0.0, 1.0, 0
0.1, , 0.0, 1.0, 0
0.15, , 0.0, 1.0, 0
0.2, , 0.058823529411764705, 0.9411764705882353, 1
0.25, , 0.29411764705882354, 0.7058823529411764, 5
0.3, , 0.35294117647058826, 0.6470588235294117, 6
0.35, , 0.47058823529411764, 0.5294117647058824, 8
0.4, , 0.7058823529411765, 0.2941176470588235, 12
0.45, , 0.7058823529411765, 0.2941176470588235, 12
0.5, , 0.7058823529411765, 0.2941176470588235, 12
0.55, , 0.7058823529411765, 0.2941176470588235, 12
0.6, , 0.7058823529411765, 0.2941176470588235, 12
0.65, , 0.7058823529411765, 0.2941176470588235, 12
0.7, , 0.7058823529411765, 0.2941176470588235, 12
0.75, , 0.7058823529411765, 0.2941176470588235, 12
0.8, , 0.7058823529411765, 0.2941176470588235, 12
0.85, , 0.7058823529411765, 0.2941176470588235, 12
0.9, , 0.7058823529411765, 0.2941176470588235, 12
0.95, , 0.7058823529411765, 0.2941176470588235, 12
Result using threshold = 0.05 
new_biflow_Thursday-WorkingHours_Dropbox2.csv, 0.0, 1.0, 7 
0.05, , 0.0, 1.0, 0
0.1, , 0.0, 1.0, 0
0.15, , 0.0, 1.0, 0
0.2, , 0.0, 1.0, 0
0.25, , 0.14285714285714285, 0.8571428571428572, 1
0.3, , 0.14285714285714285, 0.8571428571428572, 1
0.35, , 0.2857142857142857, 0.7142857142857143, 2
0.4, , 0.42857142857142855, 0.5714285714285714, 3
0.45, , 0.42857142857142855, 0.5714285714285714, 3
0.5, , 0.42857142857142855, 0.5714285714285714, 3
0.55, , 0.42857142857142855, 0.5714285714285714, 3
0.6, , 0.42857142857142855, 0.5714285714285714, 3
0.65, , 0.42857142857142855, 0.5714285714285714, 3
0.7, , 0.42857142857142855, 0.5714285714285714, 3
0.75, , 0.42857142857142855, 0.5714285714285714, 3
0.8, , 0.42857142857142855, 0.5714285714285714, 3
0.85, , 0.42857142857142855, 0.5714285714285714, 3
0.9, , 0.42857142857142855, 0.5714285714285714, 3
0.95, , 0.42857142857142855, 0.5714285714285714, 3
Result using threshold = 0.05 
new_biflow_Thursday-WorkingHours_CoolDisk.csv, 0.03846153989434242, 0.9615384601056576, 52 
0.05, , 0.0, 1.0, 0
0.1, , 0.0, 1.0, 0
0.15, , 0.0, 1.0, 0
0.2, , 0.0, 1.0, 0
0.25, , 0.21153846153846154, 0.7884615384615384, 11
0.3, , 0.23076923076923078, 0.7692307692307692, 12
0.35, , 0.3076923076923077, 0.6923076923076923, 16
0.4, , 0.5384615384615384, 0.46153846153846156, 28
0.45, , 0.5384615384615384, 0.46153846153846156, 28
0.5, , 0.5384615384615384, 0.46153846153846156, 28
0.55, , 0.5384615384615384, 0.46153846153846156, 28
0.6, , 0.5384615384615384, 0.46153846153846156, 28
0.65, , 0.5576923076923077, 0.4423076923076923, 29
0.7, , 0.5576923076923077, 0.4423076923076923, 29
0.75, , 0.6538461538461539, 0.34615384615384615, 34
0.8, , 0.6538461538461539, 0.34615384615384615, 34
0.85, , 0.7115384615384616, 0.28846153846153844, 37
0.9, , 0.7115384615384616, 0.28846153846153844, 37
0.95, , 0.7115384615384616, 0.28846153846153844, 37
Result using threshold = 0.05 
new_biflow_Wednesday-WorkingHours_Heartbleed.csv, 0.5289017558097839, 0.47109824419021606, 346 
0.05, , 0.0, 1.0, 0
0.1, , 0.0, 1.0, 0
0.15, , 0.0, 1.0, 0
0.2, , 0.08092485549132948, 0.9190751445086706, 28
0.25, , 0.3208092485549133, 0.6791907514450868, 111
0.3, , 0.42196531791907516, 0.5780346820809248, 146
0.35, , 0.5404624277456648, 0.4595375722543352, 187
0.4, , 0.661849710982659, 0.338150289017341, 229
0.45, , 0.6936416184971098, 0.3063583815028902, 240
0.5, , 0.7196531791907514, 0.28034682080924855, 249
0.55, , 0.7601156069364162, 0.23988439306358378, 263
0.6, , 0.7687861271676301, 0.23121387283236994, 266
0.65, , 0.7976878612716763, 0.20231213872832365, 276
0.7, , 0.815028901734104, 0.18497109826589597, 282
0.75, , 0.8294797687861272, 0.17052023121387283, 287
0.8, , 0.8294797687861272, 0.17052023121387283, 287
0.85, , 0.8294797687861272, 0.17052023121387283, 287
0.9, , 0.8294797687861272, 0.17052023121387283, 287
0.95, , 0.8294797687861272, 0.17052023121387283, 287
Result using threshold = 0.05 
new_biflow_Thursday-WorkingHours_Web_BF.csv, 0.09825196117162704, 0.901748038828373, 1659 
0.05, , 0.0, 1.0, 0
0.1, , 0.0006027727546714888, 0.9993972272453285, 1
0.15, , 0.02109704641350211, 0.9789029535864979, 35
0.2, , 0.08257986738999397, 0.917420132610006, 137
0.25, , 0.6172393007836046, 0.38276069921639544, 1024
0.3, , 0.8390596745027125, 0.16094032549728754, 1392
0.35, , 0.8758288125376733, 0.1241711874623267, 1453
0.4, , 0.8824593128390597, 0.11754068716094035, 1464
0.45, , 0.890295358649789, 0.10970464135021096, 1477
0.5, , 0.8945147679324894, 0.10548523206751059, 1484
0.55, , 0.8993369499698614, 0.10066305003013865, 1492
0.6, , 0.9005424954792043, 0.09945750452079571, 1494
0.65, , 0.9017480409885473, 0.09825195901145267, 1496
0.7, , 0.9065702230259193, 0.09342977697408072, 1504
0.75, , 0.917420132610006, 0.08257986738999401, 1522
0.8, , 0.917420132610006, 0.08257986738999401, 1522
0.85, , 0.9222423146473779, 0.07775768535262206, 1530
0.9, , 0.9228450874020494, 0.0771549125979506, 1531
0.95, , 0.9228450874020494, 0.0771549125979506, 1531
Result using threshold = 0.05 
new_biflow_Friday-WorkingHours_PortScan.csv, 0.021700190380215645, 0.9782998096197844, 165252 
0.05, , 0.0, 1.0, 0
0.1, , 1.8154091932321547e-05, 0.9999818459080677, 3
0.15, , 0.0002844141069397042, 0.9997155858930603, 47
0.2, , 0.0032858906397501995, 0.9967141093602498, 543
0.25, , 0.01363372304117348, 0.9863662769588265, 2253
0.3, , 0.017597366446397016, 0.982402633553603, 2908
0.35, , 0.019576162467020066, 0.9804238375329799, 3235
0.4, , 0.02324328903734902, 0.9767567109626509, 3841
0.45, , 0.023921041802822358, 0.9760789581971776, 3953
0.5, , 0.024604845932273134, 0.9753951540677268, 4066
0.55, , 0.025294701425701352, 0.9747052985742987, 4180
0.6, , 0.026111635562655822, 0.9738883644373442, 4315
0.65, , 0.026928569699610293, 0.9730714303003897, 4450
0.7, , 0.02763052792099339, 0.9723694720790066, 4566
0.75, , 0.028090431583278872, 0.9719095684167212, 4642
0.8, , 0.028332486142376493, 0.9716675138576235, 4682
0.85, , 0.02833853750635393, 0.971661462493646, 4683
0.9, , 0.028350640234308812, 0.9716493597656912, 4685
0.95, , 0.22725897417277854, 0.7727410258272215, 37555
Result using threshold = 0.05 
new_biflow_Friday-WorkingHours_Botnet.csv, 0.2979539632797241, 0.7020460367202759, 1564 
0.05, , 0.0, 1.0, 0
0.1, , 0.0, 1.0, 0
0.15, , 0.0, 1.0, 0
0.2, , 0.052429667519181586, 0.9475703324808185, 82
0.25, , 0.17519181585677748, 0.8248081841432225, 274
0.3, , 0.21547314578005114, 0.7845268542199488, 337
0.35, , 0.27301790281329924, 0.7269820971867007, 427
0.4, , 0.3446291560102302, 0.6553708439897699, 539
0.45, , 0.3599744245524297, 0.6400255754475703, 563
0.5, , 0.37531969309462915, 0.6246803069053708, 587
0.55, , 0.391304347826087, 0.6086956521739131, 612
0.6, , 0.40281329923273657, 0.5971867007672634, 630
0.65, , 0.41368286445012786, 0.5863171355498722, 647
0.7, , 0.4264705882352941, 0.5735294117647058, 667
0.75, , 0.43350383631713557, 0.5664961636828645, 678
0.8, , 0.710997442455243, 0.289002557544757, 1112
0.85, , 0.7218670076726342, 0.27813299232736577, 1129
0.9, , 0.7218670076726342, 0.27813299232736577, 1129
0.95, , 0.7244245524296675, 0.2755754475703325, 1133
Result using threshold = 0.05 
new_biflow_Wednesday-WorkingHours_Hulk.csv, 0.02925071120262146, 0.9707492887973785, 14427 
0.05, , 0.0, 1.0, 0
0.1, , 0.0, 1.0, 0
0.15, , 0.0, 1.0, 0
0.2, , 0.0015942330352810702, 0.9984057669647189, 23
0.25, , 0.0056144728633811605, 0.9943855271366189, 81
0.3, , 0.007000762459277743, 0.9929992375407223, 101
0.35, , 0.009842656130865738, 0.9901573438691342, 142
0.4, , 0.011159631246967491, 0.9888403687530325, 161
0.45, , 0.011714147085326124, 0.9882858529146739, 169
0.5, , 0.012337977403479587, 0.9876620225965204, 178
0.55, , 0.01289249324183822, 0.9871075067581618, 186
0.6, , 0.013654952519581341, 0.9863450474804186, 197
0.65, , 0.014140153878145144, 0.9858598461218548, 204
0.7, , 0.01448672627711929, 0.9855132737228807, 209
0.75, , 0.014833298676093436, 0.9851667013239066, 214
0.8, , 0.014833298676093436, 0.9851667013239066, 214
0.85, , 0.014902613155888264, 0.9850973868441117, 215
0.9, , 0.014902613155888264, 0.9850973868441117, 215
0.95, , 0.014902613155888264, 0.9850973868441117, 215

[HananHindy]

Hi,

Yes, you should get results that match the ones in the paper (or at least comparable based on the training process and randomisation weights). I cannot figure out the problem based on the results, but it looks like the AE did not train appropriately.

I couldn't verify the issue, I cloned the repo now and run the main.py script and I attached the results to this reply.

Could you check python and keras versions you are using? Also, check that you didn't change any of the learning parameters.

04_04_2021__15_51_Results.txt

[tejas-kale]

Hi, thanks for your response. I reran main.py and I agree that there is a problem with training at my end. In the logs that you attached, training and validation accuracy is around 98% whereas, in my runs, it hovers around 30%. I am attaching the epoch-wise loss and accuracy values for reference.

I am using Python version 3.6.8, TensorFlow (TF) version 2.4.1, and Keras version 2.4.0. I am using the Keras that comes bundled with TF. What environment are you using?

I did not change any of the learning parameters. Again, I have attached the results for reference. But, I had to make one change in autoencoder.py. I changed the line:

 self.model.compile(loss='mean_squared_error', optimizer = 'adadelta',  metrics=[self.accuracy])

to

self.model.compile(loss='mean_squared_error', optimizer = 'adadelta',  metrics=["accuracy"])

as self.accuracy is not defined.

04_04_2021__21_32_Results.txt 04_04_2021__21_32_epoch_wise_results.txt

[HananHindy]

Hi again,

Here are the versions I use: python: 3.7.3 keras: 2.2.5 TF: 1.14.0 But I doubt this is the problem.

Changing this line in autoencoder.py will affect the results as the model will use the generic accuracy function not the one defined in the script, that is responsible to deal with the reconstruction error and threshold. If self. accuracy is not defined, then you might want to check the indentation of this file, the accuracy function is defined in line 53.

I hope this helps

[tejas-kale]

Hi. Thanks again for the quick response. I will try my luck with TF 1.14.0 just in case.

Yes, the accuracy function is defined after all. Not sure how I missed it. When I ran it with TF 2.4.1, I encountered the following issue which I could not find a solution to. Replacing it with a generic accuracy function was not sensible (and it explains the difference in results) but at least I was able to train the autoencoder.

2021-04-01 20:37:15.877150: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-04-01 20:37:15.877575: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
Model: "model"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_2 (InputLayer)         [(None, 18)]              0         
_________________________________________________________________
dense (Dense)                (None, 15)                285       
_________________________________________________________________
dense_1 (Dense)              (None, 9)                 144       
_________________________________________________________________
dense_2 (Dense)              (None, 15)                150       
_________________________________________________________________
dense_3 (Dense)              (None, 18)                288       
=================================================================
Total params: 867
Trainable params: 867
Non-trainable params: 0
_________________________________________________________________
2021-04-01 20:37:16.696389: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
Epoch 1/50
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/contextlib.py", line 99, in __exit__
    self.gen.throw(type, value, traceback)
  File "/Users/tejas/pyenvs/bolt_dl_368/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 2825, in variable_creator_scope
    yield
  File "/Users/tejas/pyenvs/bolt_dl_368/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py", line 1100, in fit
    tmp_logs = self.train_function(iterator)
  File "/Users/tejas/pyenvs/bolt_dl_368/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 828, in __call__
    result = self._call(*args, **kwds)
  File "/Users/tejas/pyenvs/bolt_dl_368/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 871, in _call
    self._initialize(args, kwds, add_initializers_to=initializers)
  File "/Users/tejas/pyenvs/bolt_dl_368/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 726, in _initialize
    *args, **kwds))
  File "/Users/tejas/pyenvs/bolt_dl_368/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 2969, in _get_concrete_function_internal_garbage_collected
    graph_function, _ = self._maybe_define_function(args, kwargs)
  File "/Users/tejas/pyenvs/bolt_dl_368/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 3361, in _maybe_define_function
    graph_function = self._create_graph_function(args, kwargs)
  File "/Users/tejas/pyenvs/bolt_dl_368/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 3206, in _create_graph_function
    capture_by_value=self._capture_by_value),
  File "/Users/tejas/pyenvs/bolt_dl_368/lib/python3.6/site-packages/tensorflow/python/framework/func_graph.py", line 990, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/Users/tejas/pyenvs/bolt_dl_368/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 634, in wrapped_fn
    out = weak_wrapped_fn().__wrapped__(*args, **kwds)
  File "/Users/tejas/pyenvs/bolt_dl_368/lib/python3.6/site-packages/tensorflow/python/framework/func_graph.py", line 977, in wrapper
    raise e.ag_error_metadata.to_exception(e)
TypeError: in user code:

    /Users/tejas/pyenvs/bolt_dl_368/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py:805 train_function  *
        return step_function(self, iterator)
    /Users/tejas/tejas/code/NNModels_release/mini_projects/zero_day_detection/autoencoder.py:58 accuracy  *
        temp = K.ones(K.shape(mse))
    /Users/tejas/pyenvs/bolt_dl_368/lib/python3.6/site-packages/tensorflow/python/util/dispatch.py:201 wrapper  **
        return target(*args, **kwargs)
    /Users/tejas/pyenvs/bolt_dl_368/lib/python3.6/site-packages/tensorflow/python/keras/backend.py:1519 ones
        v = array_ops.ones(shape=shape, dtype=tf_dtype, name=name)
    /Users/tejas/pyenvs/bolt_dl_368/lib/python3.6/site-packages/tensorflow/python/util/dispatch.py:201 wrapper
        return target(*args, **kwargs)
    /Users/tejas/pyenvs/bolt_dl_368/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py:3132 ones
        output = fill(shape, constant(one, dtype=dtype), name=name)
    /Users/tejas/pyenvs/bolt_dl_368/lib/python3.6/site-packages/tensorflow/python/util/dispatch.py:201 wrapper
        return target(*args, **kwargs)
    /Users/tejas/pyenvs/bolt_dl_368/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py:239 fill
        result = gen_array_ops.fill(dims, value, name=name)
    /Users/tejas/pyenvs/bolt_dl_368/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py:3353 fill
        dims, value, name=name, ctx=_ctx)
    /Users/tejas/pyenvs/bolt_dl_368/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py:3378 fill_eager_fallback
        ctx=ctx, name=name)
    /Users/tejas/pyenvs/bolt_dl_368/lib/python3.6/site-packages/tensorflow/python/eager/execute.py:75 quick_execute
        raise e
    /Users/tejas/pyenvs/bolt_dl_368/lib/python3.6/site-packages/tensorflow/python/eager/execute.py:60 quick_execute
        inputs, attrs, num_outputs)

    TypeError: An op outside of the function building code is being passed
    a "Graph" tensor. It is possible to have Graph tensors
    leak out of the function building context by including a
    tf.init_scope in your function building code.
    For example, the following function will fail:
      @tf.function
      def has_init_scope():
        my_constant = tf.constant(1.)
        with tf.init_scope():
          added = my_constant * 2
    The graph tensor has name: Shape_1:0

Process finished with exit code 1