search

Albert-Z-Guo / Deep-Reinforcement-Stock-Trading

A light-weight deep reinforcement learning framework for portfolio management. This project explores the possibility of applying deep reinforcement learning algorithms to stock trading in a highly modular and scalable framework.
GNU General Public License v3.0
593 stars 135 forks source link

Saved models are only working on ^DJI_2016 stock #6

Closed RaghavML closed 3 years ago

RaghavML commented 3 years ago

I get error or session ends abruptly in google collab hence I am not able to train DQN on other stocks and hence I am using your saved models. Only able to train till 2 episodes with 1 year of stock data

Your DQN saved models are only producing profits on stock ^DJI_2016 and not on any other stocks ( not even on ^DJI_2010-2015 ) DQN_DJI_2016 DQN_GSPC2018 DQN_NVDIA2018

I have tried evaluating DQN on other stocks such as Alphabet Inc , Shanghai Composite etc still model produces only a straight line of portfolio value DDPG_PetroChina_2018 DQN_FB2018

I have tried changing initial portfolio value as well from 50k to 500k and 5k still to no avail .

What should I do ?

Please help !!

Thanks Raghav

RaghavML commented 3 years ago

I have added print(e, loss, action_dict[action], reward, agent.balance, len(agent.inventory),len(agent.memory),agent.buffer_size) line to train.py file above line containing logging.info('Episode: {}\tLoss: {:.2f}\tAction: {}\tReward: {:.2f}\tBalance: {:.2f}\tNumber of Stocks: {}'.format(e, loss, action_dict[action], reward, agent.balance, len(agent.inventory)))

and the training print results are as follows ( losses are increasing and portfolio balance is decreasing ) :

print(e, loss, action_dict[action], reward, agent.balance, len(agent.inventory),len(agent.memory),agent.buffer_size) 1 3.2852203845977783 Sell 3.188052999992763 50003.42217599999 0 61 60 1 3.7918121814727783 Buy 3.42217599999276 49911.123133999994 1 62 60 1 8.354686737060547 Buy 5.056048999998893 49817.190219 2 63 60 1 1.2688887119293213 Hold 2.0008249475438267 49817.190219 2 64 60 1 69.08196258544922 Sell 14.445857999997386 49913.708446 1 65 60 1 10.50949478149414 Hold 5.664885952514078 49913.708446 1 66 60 1 32.754364013671875 Buy 9.96266499999183 49817.454226999995 2 67 60 1 37.736663818359375 Buy 10.689928999992844 49720.83637599999 3 68 60 1 9.73847484588623 Hold 5.455053724254936 49720.83637599999 3 69 60 1 3.3992910385131836 Hold 3.2433457242596675 49720.83637599999 3 70 60 1 6.83476448059082 Sell -4.478223000009763 49812.990954999994 2 71 60 1 5.08433198928833 Sell -3.855558000002546 49906.126856999996 1 72 60 1 0.006941396277397871 Sell 0.19426199999416838 50001.469484999994 0 73 60 1 1.7587047815322876 Hold -2.247018139317391 50001.469484999994 0 74 60 1 0.6716785430908203 Buy 1.4694849999941653 49899.740785999995 1 75 60 1 4.0459160804748535 Hold 3.5338961390468735 49899.740785999995 1 76 60 1 40.61582946777344 Buy 11.088426999995136 49788.393144999995 2 77 60 1 60.87764358520508 Sell 13.564150000001462 49897.359713 1 78 60 1 3.163397789001465 Hold 3.130593119150215 49897.359713 1 79 60 1 14.630191802978516 Buy 6.674975000001723 49788.044451 2 80 60 1 9.772372245788574 Sell 5.46450899999914 49897.633684 1 81 60 1 7.0682902336120605 Hold 4.654856755473512 49897.633684 1 82 60 1 14.290316581726074 Hold 6.597574755471934 49897.633684 1 83 60 1 84.43083953857422 Sell 15.965150000003845 50011.457048000004 0 84 60 1 43.37368392944336 Buy 11.457048000003851 49898.390848 1 85 60 1 51.61090087890625 Sell 12.493166000005672 50011.975107000006 0 86 60 1 22.456186294555664 Hold 8.257823000100498 50011.975107000006 0 87 60 1 22.4561824798584 Sell 8.257823000100498 50011.975107000006 0 88 60 1 22.4561824798584 Hold 8.257823000100498 50011.975107000006 0 89 60 1 22.4561767578125 Hold 8.257823000100498 50011.975107000006 0 90 60 1 22.4561767578125 Hold 8.257823000100498 50011.975107000006 0 91 60 1 22.4561710357666 Hold 8.257823000100498 50011.975107000006 0 92 60 1 47.40279769897461 Buy 11.975107000005664 49895.845392 1 93 60 1 74.2903060913086 Buy 14.978853000007803 49776.711931000005 2 94 60 1 74.48859405517578 Buy 14.998765000003914 49657.568514000006 3 95 60 1 115.07421112060547 Buy 18.630160000000615 49537.214632 4 96 60 1 710.14990234375 Buy 46.206788000003144 49409.966593000005 5 97 60 1 1157.1492919921875 Sell 58.968986000006595 49537.48861100001 4 98 60 1 1087.2120361328125 Buy 57.160731000010855 49407.57058100001 5 99 60 1 814.3901977539062 Hold 49.47843610616369 49407.57058100001 5 100 60 1 2318.918701171875 Sell 83.45717200001074 49540.07392300001 4 100 60 1 4141.63525390625 Sell 111.51705600000938 49678.19123300001 3 100 60 1 5776.04833984375 Sell 131.68641100001082 49821.653498000014 2 100 60 1 3655.795166015625 Hold 104.7752802015021 49821.653498000014 2 100 60 1 3388.018798828125 Buy 100.86693800001376 49682.04677800001 3 100 60 1 4545.630859375 Buy 116.82709700000851 49537.12000500001 4 100 60 1 6522.65380859375 Sell 139.93552100001645 49683.132716000015 3 100 60 1 3390.697998046875 Buy 100.90679400000954 49543.87469000001 4 100 60 1 4355.31640625 Buy 114.35637800001132 49401.25426800001 5 100 60 1 4456.10595703125 Sell 115.67144200001377 49541.97680200001 4 100 60 1 3697.0908203125 Buy 105.36510600001202 49401.129726000014 5 100 60 1 2967.695556640625 Buy 94.40617100000964 49262.47443700001 6 100 60 1 2188.41162109375 Sell 81.07613600001784 49399.361354000015 5 100 60 1 2329.456298828125 Sell 83.6464670000141 49537.56333500001 4 100 60 1 3073.275390625 Buy 96.06992700001865 49397.936687000016 5 100 60 1 4448.04736328125 Sell 115.5668430000166 49540.75138400002 4 100 60 1 4349.6240234375 Buy 114.28165600001375 49397.36881600002 5 100 60 1 4748.736328125 Sell 119.40748200001681 49541.47866400002 4 100 60 1 4973.69970703125 Sell 122.20194800002074 49685.79273600002 3 100 60 1 5313.8349609375 Hold 126.30967244368999 49685.79273600002 3 100 60 1 6210.41552734375 Hold 136.54632044368822 49685.79273600002 3 100 60 1 5928.12548828125 Hold 133.40807744369425 49685.79273600002 3 100 60 1 4759.29248046875 Hold 119.54007944369273 49685.79273600002 3 100 60 1 5235.04150390625 Buy 125.3700970000209 49539.26694900002 4 100 60 1 4763.41259765625 Buy 119.59177700001601 49394.185742000016 5 100 60 1 5374.533203125 Hold 127.02873697285618 49394.185742000016 5 100 60 1 4817.15380859375 Buy 120.26423200001591 49248.970044000016 6 100 60 1 5828.181640625 Buy 132.27915400001802 49101.75185900002 7 100 60 1 6042.21142578125 Buy 134.68519400001242 48954.189954000016 8 100 60 1 5055.9765625 Buy 123.20814600001904 48808.06268000002 9 100 60 1 4494.58837890625 Buy 116.16961500001344 48662.717465000016 10 100 60 1 7639.38232421875 Buy 151.43740500001877 48513.845471000015 11 100 60 1 8416.8974609375 Hold 158.95467379738864 48513.845471000015 11 100 60 1 8982.2333984375 Buy 164.20450000001438 48363.81283200002 12 100 60 1 7935.291015625 Buy 154.3415200000236 48214.60210800002 13 100 60 1 16336.3828125 Buy 221.4301000000196 48060.23072400002 14 100 60 1 18011.435546875 Hold 232.5028090191107 48060.23072400002 14 100 60 1 20169.080078125 Hold 246.03219901911405 48060.23072400002 14 100 60 1 9528.896484375 Buy 169.1259880000216 47909.595348000024 15 100 60 1 4810.75830078125 Buy 120.18440800003009 47762.22274400003 16 100 60 1 4965.177734375 Buy 122.09725600003003 47614.73058700003 17 100 60 1 5560.83447265625 Buy 129.21077000002697 47466.819988000025 18 100 60 1 3227.907958984375 Buy 98.4559160000208 47320.61799200002 19 100 60 1 548.4328002929688 Hold 40.61227886425725 47320.61799200002 19 100 60 1 2161.0517578125 Buy 80.56804300002113 47175.35746300002 20 100 60 1 8115.640625 Buy 156.08500300002197 47026.321086000025 21 100 60 1 7076.53369140625 Sell 145.75381700002845 47174.415996000025 20 100 60 1 6584.57373046875 Buy 140.5979360000274 47026.10689900003 21 100 60 1 2716.317626953125 Sell 90.32155100002703 47171.73604900003 20 100 60 1 1791.4613037109375 Buy 73.36018900002819 47026.654842000025 21 100 60 1 1958.7889404296875 Buy 76.70746300002793 46881.41424100003 22 100 60 1 24.795351028442383 Hold 8.674734947216166 46881.41424100003 22 100 60 1 28.44803810119629 Hold -9.188187052776811 46881.41424100003 22 100 60 1 1059.7027587890625 Buy 56.433583000027284 46737.095180000026 23 100 60 1 53.12704086303711 Hold -12.574623123181407 46737.095180000026 23 100 60 1 13.143616676330566 Hold 6.329398876821035 46737.095180000026 23 100 60 1 0.00016036813030950725 Hold 0.028065876817966018 46737.095180000026 23 100 60 1 1160.486572265625 Sell -58.95389199997405 46876.567409000025 22 100 60 1 1007.7514038085938 Buy -54.93412499997794 46737.09020600002 23 100 60 1 5383.58544921875 Hold -127.03562375347785 46737.09020600002 23 100 60 1 1191.8548583984375 Buy -59.746023999978206 46597.82221600002 24 100 60 1 192.8033905029297 Buy -24.000159999981406 46457.06481500002 25 100 60 1 188.83018493652344 Buy -23.751059999980498 46316.29745000002 26 100 60 1 40.17798614501953 Buy -10.92879599997832 46175.03692100002 27 100 60 1 990.8313598632812 Buy 54.57058500001585 46031.35048900002 28 100 60 1 399.9950256347656 Hold 34.69080038232292 46031.35048900002 28 100 60 1 88.44153594970703 Sell 16.338789000020398 46173.81649200002 27 100 60 1 137.07656860351562 Hold 20.328800186592304 46173.81649200002 27 100 60 1 898.05615234375 Sell 51.95538100002224 46317.50292400002 26 100 60 1 352.1391906738281 Hold 32.55257827896577 46317.50292400002 26 100 60 1 5896.84912109375 Hold 133.05581227896192 46317.50292400002 26 100 60 1 6833.78759765625 Buy 143.2329620000237 46170.35946100002 27 100 60 1 12409.4501953125 Buy 192.99650000002293 46021.372904000025 28 100 60 1 40965.4375 Sell 350.6156940000225 46175.664576000025 27 100 60 1 45038.87109375 Buy 367.63211500002944 46020.406519000026 28 100 60 1 20872.021484375 Sell 250.28201900002918 46171.34078400003 27 100 60 1 18877.052734375 Hold 238.02302120072366 46171.34078400003 27 100 60 1 14817.0751953125 Sell 210.8845990000331 46320.88027400003 26 100 60 1 21735.013671875 Buy 255.40278800002852 46169.55248500003 27 100 60 1 33711.79296875 Sell 318.06789900002843 46322.93258100003 26 100 60 1 42959.91796875 Hold 359.04826570408915 46322.93258100003 26 100 60 1 42145.04296875 Buy 355.6271690000285 46167.82894300003 27 100 60 1 59076.67578125 Sell 421.03696400002383 46324.92013000003 26 100 60 1 54118.796875 Sell 402.98472500002765 46481.473337000025 25 100 60 1 50965.34375 Buy 391.069212000024 46325.089502000024 26 100 60 1 27025.916015625 Hold 284.79154638471897 46325.089502000024 26 100 60 1 38367.6484375 Sell 339.3182440000211 46479.321405000024 25 100 60 1 61254.96484375 Sell 428.72806100002435 46636.96053500003 24 100 60 1 66540.453125 Buy 446.8400550000224 46478.215555000024 25 100 60 1 41230.1875 Sell 351.74667300002557 46633.13488200003 24 100 60 1 40294.21484375 Buy 347.7318100000266 46478.360010000026 25 100 60 1 47689.05859375 Sell 378.29222000002693 46634.15107100003 24 100 60 1 49278.49609375 Sell 384.54381700003046 46790.06168500003 23 100 60 1 40634.359375 Buy 349.1962250000288 46635.31670500003 24 100 60 1 29947.283203125 Sell 299.78629800002693 46787.79517500003 23 100 60 1 18561.451171875 Buy 236.02532500003144 46637.87212500003 24 100 60 1 12548.8994140625 Sell 194.0775760000316 46786.036767000034 23 100 60 1 10823.3603515625 Sell 180.24457100003931 46933.55384200004 22 100 60 1 24749.716796875 Hold 272.53696452131334 46933.55384200004 22 100 60 1 21444.041015625 Hold 253.68778252130792 46933.55384200004 22 100 60 1 33395.70703125 Buy 316.57350400003634 46779.78022100004 23 100 60 1 21422.388671875 Buy 253.55969100003858 46628.74633100004 24 100 60 1 158148.203125 Hold 688.8493712168962 46628.74633100004 24 100 60 1 211668.8125 Buy 796.9229550000382 46455.07230500004 25 100 60 1 200313.921875 Buy 775.2543300000398 46282.26502400004 26 100 60 1 261476.453125 Buy 885.7301500000394 46105.20867300004 27 100 60 1 260650.5 Sell 884.3302360000367 46281.07946200004 26 100 60 1 279936.90625 Hold 916.4618575576314 46281.07946200004 26 100 60 1 401387.875 Hold 1097.3938555576315 46281.07946200004 26 100 60 1 472916.09375 Hold 1191.1622315576315 46281.07946200004 26 100 60 1 478751.0 Buy 1198.4877500000366 46091.948374000036 27 100 60 1 545570.875 Hold 1279.3906902701228 46091.948374000036 27 100 60 1 642616.8125 Sell 1388.520641000031 46286.434397000034 26 100 60 1 655760.0625 Buy 1402.6477070000328 46089.65696200003 27 100 60 1 592830.8125 Buy 1333.6513670000277 45895.43494700003 28 100 60 1 464331.53125 Hold 1180.301904696207 45895.43494700003 28 100 60 1 607804.4375 Buy 1350.3883110000315 45700.61518400003 29 100 60 1 598735.4375 Buy 1340.2761850000315 45506.14411500003 30 100 60 1 737892.25 Sell 1487.8929730000302 45703.88792000003 29 100 60 1 633116.8125 Buy 1378.2191750000275 45508.22132500003 30 100 60 1 708704.0625 Buy 1458.1694450000286 45309.88972100003 31 100 60 1 790495.0625 Buy 1540.0127310000316 45108.91801100003 32 100 60 1 738361.75 Buy 1488.366267000034 44909.56025300003 33 100 60 1 895142.75 Buy 1638.7776930000327 44705.644573000034 34 100 60 1 1027638.4375 Hold 1755.8731952903452 44705.644573000034 34 100 60 1 1245406.0 Sell 1932.9798860000337 44916.28504500003 33 100 60 1 1283221.0 Buy 1962.1058050000283 44702.77532500003 34 100 60 1 1171877.25 Buy 1875.0518650000304 44491.82601500003 35 100 60 1 777604.4375 Buy 1527.4049500000328 44290.80947400003 36 100 60 1 802821.75 Buy 1551.9731140000367 44089.11048400003 37 100 60 1 987416.1875 Buy 1721.1683790000316 43882.838649000034 38 100 60 1 1063492.375 Hold 1786.2406927089414 43882.838649000034 38 100 60 1 816557.5625 Buy 1565.1930470000298 43680.67142800003 39 100 60 1 790219.1875 Buy 1539.7439090000335 43479.15674900003 40 100 60 1 842098.3125 Sell 1589.4822180000315 43680.51201900003 39 100 60 1 916580.1875 Buy 1658.2840810000343 43475.95376100003 40 100 60 1 950748.6875 Sell 1688.9093880000316 43679.78972900003 39 100 60 1 958341.0625 Buy 1695.6394020000298 43474.25512200003 40 100 60 1 1071407.25 Buy 1792.875122000034 43266.28962200003 41 100 60 1 1193307.25 Sell 1892.118087000033 43474.987376000034 40 100 60 1 1196224.625 Buy 1894.4294960000334 43264.50132300003 41 100 60 1 1504643.125 Sell 2124.6510750000334 43478.77317400003 40 100 60 1 1253255.625 Buy 1939.0618540000287 43267.26595700003 41 100 60 1 1392629.875 Buy 2044.038664000036 43053.198330000036 42 100 60 1 1292905.625 Hold 1969.4957731136167 43053.198330000036 42 100 60 1 1458972.5 Buy 2092.1583160000373 42837.984997000036 43 100 60 1 1421612.375 Hold 2065.198140464039 42837.984997000036 43 100 60 1 1339169.0 Sell 2004.4219760000367 43049.512142000036 42 100 60 1 1302173.375 Buy 1976.5417920000327 42836.96381700003 43 100 60 1 1209585.625 Sell 1904.9796400000291 43046.25434400003 42 100 60 1 992716.5 Buy 1725.7816060000332 42839.59893300003 43 100 60 1 1558154.5 Buy 2162.1007570000365 42622.79656500003 44 100 60 1 1885406.0 Hold 2378.330466963625 42622.79656500003 44 100 60 1 2104982.25 Buy 2513.005729000033 42398.019084000036 45 100 60 1 2681332.25 Buy 2836.2441040000413 42166.05852800004 46 100 60 1 2729344.25 Hold 2861.523776332865 42166.05852800004 46 100 60 1 2861910.25 Buy 2930.191882000043 41932.05562900004 47 100 60 1 3136716.0 Sell 3067.6463960000406 42166.99003100004 46 100 60 1 2595834.0 Buy 2790.6597170000387 41936.04069000004 47 100 60 1 2711206.75 Buy 2852.000122000034 41703.786234000036 48 100 60 1 2750443.5 Buy 2872.5628420000285 41471.10338800003 49 100 60 1 1810633.875 Buy 2330.693990000029 41249.47909000003 50 100 60 1 1498039.875 Buy 2119.98384000003 41032.06899500003 51 100 60 1 464299.46875 Buy 1180.261206000032 40833.08483400003 52 100 60 1 1211616.75 Buy 1906.57827800003 40620.13303700003 53 100 60 1 1788780.625 Buy 2316.5867020000296 40399.44523200003 54 100 60 1 1396378.5 Buy 2046.787744000023 40183.753704000024 55 100 60 1 1444259.125 Buy 2081.5823940000264 39967.42954600002 56 100 60 1 1414223.5 Buy 2059.8244340000238 39751.493923000024 57 100 60 1 1166118.125 Buy 1870.438799000025 39538.880855000025 58 100 60 1 1385999.125 Buy 2039.1667730000263 39323.35868400003 59 100 60 1 426808.375 Buy 1131.6087720000287 39123.21885200003 60 100 60 1 342458.84375 Sell 1013.645848000029 39320.49940000003 59 100 60 1 95102.6796875 Buy 534.1923280000265 39130.43680800003 60 100 60 1 135399.109375 Hold 637.3861056546584 39130.43680800003 60 100 60 1 5113.83154296875 Hold 123.91078565466069 39130.43680800003 60 100 60 1 8687.5556640625 Buy 161.4893680000314 38946.58593200003 61 100 60 1 4821.56201171875 Sell -120.21922299996882 39125.30106600003 60 100 60 1 9.82805061340332 Sell -5.379930999975471 39305.92902600003 59 100 60 1 95071.1484375 Buy -534.0037629999715 39133.724497000025 60 100 60 1 122083.6015625 Buy -605.1365829999777 38962.70551500002 61 100 60 1 133879.53125 Buy -633.6996499999732 38792.154780000026 62 100 60 1 4029.146484375 Buy 109.99289200002386 38609.60900400003 63 100 60 1 11018.0576171875 Buy 181.85806300002878 38425.922511000026 64 100 60

RaghavML commented 3 years ago

1 36112128.0 Buy 10408.526678618644 194.2391160000196 250 100 60 1 39281000.0 Buy 10855.601678618641 194.2391160000196 250 100 60 1 50318844.0 Hold 12286.48692861864 194.2391160000196 250 100 60 1 49617488.0 Buy 12200.56092861864 194.2391160000196 250 100 60 1 52171668.0 Buy 12510.646178618637 194.2391160000196 250 100 60 1 59329520.0 Buy 13341.28517861864 194.2391160000196 250 100 60 1 73082320.0 Buy 14807.040428618635 194.2391160000196 250 100 60 1 72358792.0 Buy 14733.561928618645 194.2391160000196 250 100 60 1 68509416.0 Buy 14336.30692861864 194.2391160000196 250 100 60 1 69022112.0 Buy 14389.849928618645 194.2391160000196 250 100 60 1 64153308.0 Buy 13873.042428618643 194.2391160000196 250 100 60 1 56809344.0 Buy 13054.858428618641 194.2391160000196 250 100 60 1 57319876.0 Buy 13113.387428618636 194.2391160000196 250 100 60 1 54736796.0 Buy 12814.509678618644 194.2391160000196 250 100 60 1 58029724.0 Buy 13194.33542861864 194.2391160000196 250 100 60 1 68676144.0 Buy 14353.740178618635 194.2391160000196 250 100 60 1 67619672.0 Buy 14242.907928618635 194.2391160000196 250 100 60 1 62585072.0 Buy 13702.430178618644 194.2391160000196 250 100 60 1 67383352.0 Buy 14217.997928618646 194.2391160000196 250 100 60 1 80825160.0 Buy 15571.673678618641 194.2391160000196 250 100 60 1 84010368.0 Buy 15875.537178618648 194.2391160000196 250 100 60 1 87795240.0 Buy 16229.21317861864 194.2391160000196 250 100 60

Albert-Z-Guo commented 3 years ago

Hi there - Thanks for reaching out!

To avoid issues due to incompatibility, I'd suggest to install the dependencies specified in requirements.txt in a Python virtual environment.

As for model's performance, this repository is really a toy framework I built when I was learning reinforcement learning. So unless a lot of effort is put to designing the features extracted from price movement time-series; and incorporating some state-of-the-art deep reinforcement learning models, the model's performance won't be very well I'm afraid.

For much more mature implementations, here are some good resources: https://github.com/AI4Finance-LLC

Keep implementing and learning!