search

zcakhaa / DeepLOB-Deep-Convolutional-Neural-Networks-for-Limit-Order-Books

This jupyter notebook is used to demonstrate our recent work, "DeepLOB: Deep Convolutional Neural Networks for Limit Order Books", published in IEEE Transactions on Singal Processing. We use FI-2010 dataset and present how model architecture is constructed here. The FI-2010 is publicly avilable and interested readers can check out their paper.
374 stars 205 forks source link

the accuracy is very low when applying to 1 second ES futures orderbook data #21

Closed Ray-0403 closed 2 years ago

Ray-0403 commented 2 years ago

Hi, this work is really promising. I tried to apply the deeplob to 1 second ES orderbook data(10 levels each side) from 2022.1.2 to 2022.1.6 with alpha=0.00004 and k=20 and using DecPre normalization (labels are balanced). The best accuracy i can get is around 35%. I can't figure out what is wrong in this experiment. Any suggestion? Thank you very much.

A screenshot of my dataset:

Screenshot 2022-02-27 at 08 04 31 Screenshot 2022-02-27 at 07 57 32

And the best accuracy is around 40% when using Z-score normalization

Screenshot 2022-02-27 at 09 25 12
zcakhaa commented 2 years ago

Hi,

You need to check your label balance (the percentage each class accounts for when alpha=0.00004). However, it is a different dataset so it is hard for me to tell which is wrong.

Best, Zihao

From: Ray @.> Sent: Sunday, February 27, 2022 7:58 AM To: zcakhaa/DeepLOB-Deep-Convolutional-Neural-Networks-for-Limit-Order-Books @.> Cc: Subscribed @.***> Subject: [zcakhaa/DeepLOB-Deep-Convolutional-Neural-Networks-for-Limit-Order-Books] the accuracy is very low when applying to 1 second ES futures orderbook data (Issue #21)

Hi, this work is really promising. I tried to apply the deeplob to 1 second ES orderbook data with alpha=0.00004 and k=20 and DecPre normalization. The best accuracy i can get is around 35%. I can't figure out what is wrong in this experiment. [Screenshot 2022-02-27 at 07 57 32]https://user-images.githubusercontent.com/49976973/155873837-39dfffef-fdb5-4c62-8463-47261aacb023.png

— Reply to this email directly, view it on GitHubhttps://github.com/zcakhaa/DeepLOB-Deep-Convolutional-Neural-Networks-for-Limit-Order-Books/issues/21, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AFN66TCD4ASEEDJJCHD2JNDU5HKTBANCNFSM5POQ232Q. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you are subscribed to this thread.Message ID: @.***>

Ray-0403 commented 2 years ago

Thank you for replying to me. I have checked that the labels are balanced. I added 3 dropout layers and changed the learning rate to 0.001, the Val accuracy increased to around 55%. I am going to collect more data to train the network. I wonder if the percentage of labels is like up: 45% down: 45% no movement: 10%, is it still meaningful to train deeplob on this dataset and be able to build a profitable strategy on it?

zcakhaa commented 2 years ago

Hi,

For experiment purposes, you can have a balanced dataset like 33% for each label. However, whether you can turn it into a profitable strategy depends on many other factors, for example, trading costs, turnover rate and etc.

From: Ray @.> Sent: Monday, February 28, 2022 8:45 AM To: zcakhaa/DeepLOB-Deep-Convolutional-Neural-Networks-for-Limit-Order-Books @.> Cc: Zihao Zhang @.>; Comment @.> Subject: Re: [zcakhaa/DeepLOB-Deep-Convolutional-Neural-Networks-for-Limit-Order-Books] the accuracy is very low when applying to 1 second ES futures orderbook data (Issue #21)

Thank you for replying to me. I have checked that the labels are balanced. I added 3 dropout layers and changed the learning rate to 0.001, the Val accuracy increased to around 55%. I am going to collect more data to train the network. I wonder if the percentage of labels is like up: 45% down: 45% no movement: 10%, is it still meaningful to train deeplob on this dataset and be able to build a profitable strategy on it?

— Reply to this email directly, view it on GitHubhttps://github.com/zcakhaa/DeepLOB-Deep-Convolutional-Neural-Networks-for-Limit-Order-Books/issues/21#issuecomment-1054020621, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AFN66TECCHFTYJPE2VEFCBDU5MYZ7ANCNFSM5POQ232Q. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you commented.Message ID: @.***>

AvivAlloni commented 2 years ago

Thank you for replying to me. I have checked that the labels are balanced. I added 3 dropout layers and changed the learning rate to 0.001, the Val accuracy increased to around 55%. I am going to collect more data to train the network. I wonder if the percentage of labels is like up: 45% down: 45% no movement: 10%, is it still meaningful to train deeplob on this dataset and be able to build a profitable strategy on it?

Hey @Ray-0403, I'm having a similar issue. Can you elaborate on where did you add the dropout layers?

Ray-0403 commented 2 years ago

I added another 3 dropout layers in the inception module, but i don't think i have fully solved this problem. I am thinking about other methods to deal with it, maybe we can try to figure it out together lol. It's my email: rayzzz0403@gmail.com