search

quickfix-j / quickfixj

QuickFIX/J is a full featured messaging engine for the FIX protocol. - This is the official project repository.
http://www.quickfixj.org
Other
942 stars 610 forks source link

if config ResendRequestChunkSize arg. is didn't work，probably oom #778

Open wajncn opened 4 months ago

wajncn commented 4 months ago

Describe the bug If the amount of data is particularly large 。 

ResetOnLogon=N
ResetOnLogout=N
ResetOnDisconnect=N
ResetOnError=N

in quickfixj-initiator delete local quickfixj file。 restart quickfixj-acceptor OOM （If there is no excessive memory）

need config

ResendRequestChunkSize=1000

Can solve this problem

system information:

https://github.com/quickfix-j/quickfixj/pull/777

wajncn commented 4 months ago

image

ResetOnLogon=N
ResetOnLogout=N
ResetOnDisconnect=N
ResetOnError=N
DisconnectOnError=N
ContinueInitializationOnError=N
ResendRequestChunkSize=3
INFO quickfix.Session - load beginSeqNo1,endSeqNo:3  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo4,endSeqNo:6  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo7,endSeqNo:9  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo10,endSeqNo:12  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo12,endSeqNo:14  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo15,endSeqNo:17  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo18,endSeqNo:20  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo21,endSeqNo:23  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo24,endSeqNo:26  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo27,endSeqNo:29  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo30,endSeqNo:32  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo33,endSeqNo:35  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo36,endSeqNo:38  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo39,endSeqNo:41  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo42,endSeqNo:44  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo45,endSeqNo:47  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo48,endSeqNo:50  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo51,endSeqNo:53  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo54,endSeqNo:56  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo57,endSeqNo:59  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo60,endSeqNo:62  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo63,endSeqNo:65  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo66,endSeqNo:68  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo69,endSeqNo:71  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo72,endSeqNo:74  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo75,endSeqNo:77  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo78,endSeqNo:80  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo81,endSeqNo:83  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo84,endSeqNo:86  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo87,endSeqNo:89  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo90,endSeqNo:92  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo93,endSeqNo:95  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo95,endSeqNo:97  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo98,endSeqNo:100  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo101,endSeqNo:103  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo104,endSeqNo:106  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo107,endSeqNo:109  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo110,endSeqNo:112  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo113,endSeqNo:115  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo116,endSeqNo:118  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo119,endSeqNo:121  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo122,endSeqNo:124  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo125,endSeqNo:127  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo128,endSeqNo:130  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo131,endSeqNo:133  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo134,endSeqNo:136  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo137,endSeqNo:139  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo140,endSeqNo:142  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo142,endSeqNo:144  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo145,endSeqNo:147  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo148,endSeqNo:150  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo151,endSeqNo:153  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo154,endSeqNo:156  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo157,endSeqNo:159  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo160,endSeqNo:162  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo163,endSeqNo:165  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo166,endSeqNo:168  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo169,endSeqNo:171  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo172,endSeqNo:174  beginString：FIXT.1.1
chrjohn commented 4 months ago

Hi @wajncn thanks but I don't fully understand the issue and the related PR. You can set chunk size in the config, but why do we need to change the code for this?

wajncn commented 4 months ago

Hi @wajncn thanks but I don't fully understand the issue and the related PR. You can set chunk size in the config, but why do we need to change the code for this?

If you don't change the code， endSeqNo is Max SeqNo。 FIleStore all the data is loaded。
Although the parameters are configured 【ResendRequestChunkSize=3】 。

So, change the code。

wajncn commented 4 months ago

Hi @wajncn thanks but I don't fully understand the issue and the related PR. You can set chunk size in the config, but why do we need to change the code for this?

Suppose the store file is 3GB，100000rows。 java max heap is 2G。
acceptor and initiator config【ResendRequestChunkSize=1000】。 acceptor online。 initiator offline。 then start initiator after。 acceptor need load 3GBdata to memory，bug memory limit 2G。 then OOM exception～ 。

if merge this issues，there wouldn't be this problem。

chrjohn commented 4 months ago

OK, starting to understand. Are you able to supply a unit test?

wajncn commented 4 months ago

OK, starting to understand. Are you able to supply a unit test?

I wrote it locally acceptor and initiator demo。 It meets my expectations

wajncn commented 4 months ago

OK, starting to understand. Are you able to supply a unit test? I submit a unit test as you requested 。 you can debug 【Session#2293 line https://github.com/wajncn/quickfixj/blob/e0c1d9ef730b524fe11953b459bebb629823680a/quickfixj-core/src/main/java/quickfix/Session.java#L2293】 and 【quickfix.Session#resendMessages】 【https://github.com/wajncn/quickfixj/blob/e0c1d9ef730b524fe11953b459bebb629823680a/quickfixj-core/src/main/java/quickfix/Session.java#L2335】

wajncn commented 4 months ago

OK, starting to understand. Are you able to supply a unit test?

hi ，Please help me look at this pr。

chrjohn commented 4 months ago

Thanks again for the PR. I'm looking at the issues as time permits, please be patient. :)

wajncn commented 4 months ago

Thanks again for the PR. I'm looking at the issues as time permits, please be patient. :)

how test passed ？

wajncn commented 4 months ago

??

chrjohn commented 4 months ago

https://github.com/quickfix-j/quickfixj/issues/778#issuecomment-2010365727

:)

wajncn commented 2 months ago

oh....

wajncn commented 2 months ago

merge ？

chrjohn commented 2 months ago

Did you see my comment?
https://github.com/quickfix-j/quickfixj/pull/777#discussion_r1548045908