Closed mrut2pac closed 3 years ago
@mrut2pac I think we need some kind of benchmark to support this improvement no?
@mrut2pac I was not expecting to change the default non dynamic interface (IUpdateSummaryMessage) at least for now. Is there a way that we can merge your PR without impacting this one and just when we use DynamicFields ? Just wondering,
@mrut2pac I was not expecting to change the default non dynamic interface (IUpdateSummaryMessage) at least for now. Is there a way that we can merge your PR without impacting this one and just when we use DynamicFields ? Just wondering,
My changes were 100% backwards compatible, except for the case if someone has a logic of listing the IUpdateSummaryMessage fields using reflection. Using DynamicFields won't work cause it is a class and not interface. I added a new interface that adds the dynamic fields on top of IUpdateSummaryMessage and for now the caller has to cast the received message to access the new fields. Hope this will suffice.
@mrut2pac I think we need some kind of benchmark to support this improvement no?
For memory it is obvious that the new solution is better since it creates objects with less number of fields (except when you select all fields, then it will be the same). In terms of CPU usage, I ensured that the generated IL code for each concrete class is practically the same set of instructions (only negligible differences that only affect the compiled IL code size), since there are no loops and branching compared to the current solution where we have to make a check for every field to see if it is included or not (or if the value is different than the default), it "SHOULD" be faster or at least achieve same speed. I agree that benchmarking would have been nice, but unfortunately I don't have more time to spend on this. Already went above my initial estimate. :)
@mrut2pac I think we need some kind of benchmark to support this improvement no?
For memory it is obvious that the new solution is better since it creates objects with less number of fields (except when you select all fields, then it will be the same). In terms of CPU usage, I ensured that the generated IL code for each concrete class is practically the same set of instructions (only negligible differences that only affect the compiled IL code size), since there are no loops and branching compared to the current solution where we have to make a check for every field to see if it is included or not (or if the value is different than the default), it "SHOULD" be faster or at least achieve same speed. I agree that benchmarking would have been nice, but unfortunately I don't have more time to spend on this. Already went above my initial estimate. :)
thats very good and interesting work I have to say. few questions though. (just doing my devil advocate here)
there's a slight performance hit carrying out unused properties for UpdateSummaryMessage, observed it using Level1MessageHandlerPerformanceTests after and before your changes. Just wondering if it would make sense to whether create a new Level1Client like Level1ClientDynamic able to deal with UpdateSummaryMessage with this new set of properties OR make Level1Client generic like we've done float, double, decimal before. Personally, from a user perspective, I think its more convenient the first option but what are your thoughts on this.
checking ILevel1Event, do you think having Summary and Update using IUpdateSummaryMessage instead of the class still relevant?
do you think there's some duplication between StreamingLevel1DynamicExample and StreamingLevel1UniversalExample can we combine both? Like keeping both seems to be a little bit overkill. Not sure I like the term Universal...
@mrut2pac I was not expecting to change the default non dynamic interface (IUpdateSummaryMessage) at least for now. Is there a way that we can merge your PR without impacting this one and just when we use DynamicFields ? Just wondering,
My changes were 100% backwards compatible, except for the case if someone has a logic of listing the IUpdateSummaryMessage fields using reflection. Using DynamicFields won't work cause it is a class and not interface. I added a new interface that adds the dynamic fields on top of IUpdateSummaryMessage and for now the caller has to cast the received message to access the new fields. Hope this will suffice.
Yes sorry, I was checking the code on my phone :)
@BunkerCoder what do you think?
@mrut2pac I think we need some kind of benchmark to support this improvement no?
For memory it is obvious that the new solution is better since it creates objects with less number of fields (except when you select all fields, then it will be the same). In terms of CPU usage, I ensured that the generated IL code for each concrete class is practically the same set of instructions (only negligible differences that only affect the compiled IL code size), since there are no loops and branching compared to the current solution where we have to make a check for every field to see if it is included or not (or if the value is different than the default), it "SHOULD" be faster or at least achieve same speed. I agree that benchmarking would have been nice, but unfortunately I don't have more time to spend on this. Already went above my initial estimate. :)
thats very good and interesting work I have to say. few questions though. (just doing my devil advocate here)
- there's a slight performance hit carrying out unused properties for UpdateSummaryMessage, observed it using Level1MessageHandlerPerformanceTests after and before your changes. Just wondering if it would make sense to whether create a new Level1Client like Level1ClientDynamic able to deal with UpdateSummaryMessage with this new set of properties OR make Level1Client generic like we've done float, double, decimal before. Personally, from a user perspective, I think its more convenient the first option but what are your thoughts on this.
- checking ILevel1Event, do you think having Summary and Update using IUpdateSummaryMessage instead of the class still relevant?
- do you think there's some duplication between StreamingLevel1DynamicExample and StreamingLevel1UniversalExample can we combine both? Like keeping both seems to be a little bit overkill. Not sure I like the term Universal...
Thanks!
IUpdateSummaryMessage
and remove the new interface. With new solution I don't see why anyone wouldn't want to use dynamic fields. I'd deprecate the Level1DynamicFields
property and remove it in the next major version. ILevel1Event I
think we can add few extra events there - allowing client to hook into any exception that could be thrown while parsing the messages received from IQ feed. Currently the application crashes when you throw exception from any of these handlers. An unhandled exception hits the async socket receiver thread and kills it. Why not protect it and feed the exception to the client? This way the use can at least log the exception details, or in some cases even handle it and continue.@mrut2pac check Level1MessageHandlerPerformanceTests class and run it on master and your branch. It takes slightly more time on yours probably because of the additional properties on UpdateSummaryMessage.
I'm not talking about the Dynamic Fields at all. Seems that even Level1MessageHandler is impacted because the set of new fields.
I'd personally prefer setting the needed fields when creating a client via factory and live with it. If you need to change it just create a new client with the new fields set. Currently, both mine and your existing dynamic solutions are suffering from a possible Race condition - if you change the fields set while there is something already on watch there is a possibility of crashing the application or, what is worse, getting wrong data. I don't see much benefit of supporting on the fly changes, because most applications would just select the desired set of fields on startup and always use them. So yes, I'd vote for creating a new client which gets the list of fields during initialization and doesn't allow changing it later. Makes sense.
You mentioned a performance hit - is the new solution slower or the old one? check Level1MessageHandlerPerformanceTests class and run it on master and your branch. It takes slightly more time on yours probably because of the additional properties on UpdateSummaryMessage. I'm not talking about the Dynamic Fields at all. Seems that even Level1MessageHandler is impacted because the set of new fields.
If you ask me, I'd move back all the dynamic fields into IUpdateSummaryMessage and remove the new interface. With new solution I don't see why anyone wouldn't want to use dynamic fields. I'd deprecate the Level1DynamicFields property and remove it in the next major version. SOrry which interface are you talking about?
As far as ILevel1Event I think we can add few extra events there - allowing client to hook into any exception that could be thrown while parsing the messages received from IQ feed. Currently the application crashes when you throw exception from any of these handlers. An unhandled exception hits the async socket receiver thread and kills it. Why not protect it and feed the exception to the client? This way the use can at least log the exception details, or in some cases even handle it and continue. Not sure for exceptions. The idea here is to avoid all of them to avoid Try Catch blocks. The way it has been design from the start, all handlers should be resilient to incorrect data.
There certainly is a lot of duplication in the example code, we can consolidate it when we decide how to finalize the solution, I wanted to have it there in order to demonstrate (read show off) the new capabilities. :) ok we can live with that for now
The name Universal should definitely go away and we should stick with Dynamic, I just didn't want to have DynamicV2 or anything like that. An initially I was thinking it can support both versions, until I thought about the race condition. Cool
@mrut2pac I think we are very close to merge this. Can we get on the same page for the next small refactor about passing dynamic fields in the constructor ?
I was thinking to have a Level1DynamicClient that supports your new Handler and instead of creating all these new properties in existing UpdateSummaryMessage we could have a new UpdateSummaryDynamicMessage and this one will be available in ILevel1Event as a generic for your new Handler. What do you think? That way, there's a clear separation. Personally, I dont use dynamic fields so, I'd prefer to have the leanest possible path.
Any thoughts ?
Level1MessageHandlerPerformanceTests
The current version doesn't have any new fields and I'm getting my branch run slower than master for Debug but faster with Release. Since there is nothing changed for this test at the moment I am inclined to ignore the slight variations as statistically insignificant. :) I will look to write similar test for the current dynamic and new solutions, would be interesting to compare.
*** Update:
Actually I just realized the parsing code is changed. Interestingly the Release build runs faster...
Are you sure you want to keep the code duplication across the board and not use FieldParser
?
I think we should use the FieldParser
...
@mrut2pac I think we are very close to merge this. Can we get on the same page for the next small refactor about passing dynamic fields in the constructor ?
I was thinking to have a Level1DynamicClient that supports your new Handler and instead of creating all these new properties in existing UpdateSummaryMessage we could have a new UpdateSummaryDynamicMessage and this one will be available in ILevel1Event as a generic for your new Handler. What do you think? That way, there's a clear separation. Personally, I dont use dynamic fields so, I'd prefer to have the leanest possible path.
Any thoughts ?
Have you looked at my latest version? There is IUpdateSummaryDynamicMessage interface and generated object now implement it.
Personally, I dont use dynamic fields so, I'd prefer to have the leanest possible path.
Are you using every single field from IUpdateSummaryMessage? If you don't you can still benefit by omitting the unneeded ones...
I will find time and create a new client to work with the new handler.
Level1MessageHandlerPerformanceTests
The current version doesn't have any new fields and I'm getting my branch run slower than master for Debug but faster with Release. Since there is nothing changed for this test at the moment I am inclined to ignore the slight variations as statistically insignificant. :) I will look to write similar test for the current dynamic and new solutions, would be interesting to compare. *** Update: Actually I just realized the parsing code is changed. Interestingly the Release build runs faster... Are you sure you want to keep the code duplication across the board and not use
FieldParser
? I think we should use theFieldParser
...
okay lets try that.
@mrut2pac do you want to wait for the new client or you want to merge this ASAP?
@mrut2pac do you want to wait for the new client or you want to merge this ASAP?
@mrut2pac are we ready to merge this one?
I am still working on it. I wrote the new client and separated the common code. Still got to fo few things. Please wait.
Regards, Vardan
On Apr 3, 2021, at 7:26 AM, Mathieu Paquette @.***> wrote:
@mrut2pac do you want to wait for the new client or you want to merge this ASAP?
@mrut2pac are we ready to merge this one?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.
I am still working on it. I wrote the new client and separated the common code. Still got to fo few things. Please wait. Regards, Vardan … On Apr 3, 2021, at 7:26 AM, Mathieu Paquette @.***> wrote: @mrut2pac do you want to wait for the new client or you want to merge this ASAP? @mrut2pac are we ready to merge this one? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.
@mrut2pac Ill merge protocol 6.1 PR first. @NichUK proposed that for a while and now almost ready. We might have some minor conflicts afterward.
@mathpaquette finalized the solution.
UpdateSummaryMessage
parse code when the standard set of fields are selected. The difference is much more drastic in Debug build. For Release the compiler helps to optimize it, but still my code beats it.
Here are some stats:
https://docs.google.com/spreadsheets/d/1DkrHW9Doyc_gs2xz0Q90CsulqmUEeshAgrXZlv23_Dw/edit?usp=sharingPlease let me know what you think.
P.S. a kind ask - next time please let me do the reverse merge from master
to my branch as it will reduce the thrash and iterations if I do it at the right time ;)
I am still working on it. I wrote the new client and separated the common code. Still got to fo few things. Please wait. Regards, Vardan … On Apr 3, 2021, at 7:26 AM, Mathieu Paquette @.***> wrote: @mrut2pac do you want to wait for the new client or you want to merge this ASAP? @mrut2pac are we ready to merge this one? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.
@mrut2pac Ill merge protocol 6.1 PR first. @NichUK proposed that for a while and now almost ready. We might have some minor conflicts afterward.
No worries, if you are ready to merge please go ahead. I will fix the conflicts in my branch, if any.
@mathpaquette finalized the solution.
- Refactored to completely separate this new Dynamic flow. Refactored some of the existing classes to minimize the code duplication. When we remove the old dynamic code we can clean it up even more.
- Added performance tests to cover both old and new dynamic handler.
- The new code generated is much faster than the old dynamic code and is even slightly faster than
UpdateSummaryMessage
parse code when the standard set of fields are selected. The difference is much more drastic in Debug build. For Release the compiler helps to optimize it, but still my code beats it. Here are some stats: https://docs.google.com/spreadsheets/d/1DkrHW9Doyc_gs2xz0Q90CsulqmUEeshAgrXZlv23_Dw/edit?usp=sharingPlease let me know what you think.
P.S. a kind ask - next time please let me do the reverse merge from
master
to my branch as it will reduce the thrash and iterations if I do it at the right time ;)
sorry about it... i wont to it again, or at least without notifying you.
I am still working on it. I wrote the new client and separated the common code. Still got to fo few things. Please wait. Regards, Vardan … On Apr 3, 2021, at 7:26 AM, Mathieu Paquette @.***> wrote: @mrut2pac do you want to wait for the new client or you want to merge this ASAP? @mrut2pac are we ready to merge this one? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.
@mrut2pac Ill merge protocol 6.1 PR first. @NichUK proposed that for a while and now almost ready. We might have some minor conflicts afterward.
No worries, if you are ready to merge please go ahead. I will fix the conflicts in my branch, if any.
Okay just merged, please rebase yours, Ill merge it as well.
I am still working on it. I wrote the new client and separated the common code. Still got to fo few things. Please wait. Regards, Vardan … On Apr 3, 2021, at 7:26 AM, Mathieu Paquette @.***> wrote: @mrut2pac do you want to wait for the new client or you want to merge this ASAP? @mrut2pac are we ready to merge this one? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.
@mrut2pac Ill merge protocol 6.1 PR first. @NichUK proposed that for a while and now almost ready. We might have some minor conflicts afterward.
No worries, if you are ready to merge please go ahead. I will fix the conflicts in my branch, if any.
Okay just merged, please rebase yours, Ill merge it as well.
done. thanks!
@mrut2pac damn. I wanted to squash these commits... one second.
@mrut2pac damn. I wanted to squash these commits... one second.
please re-create the PR from your branch.,
Nice work guys. Thanks! Quick heads-up. I've been working on getting my test environment up-to-date so that I can do some testing on my live code. Just updated to VS 16.9.3, and started getting missing NumberStyles and CultureInfo errors in Level1DynamicFields.cs. Adding "using System.Globalization;" fixed it. I haven't tracked down the root cause (probably something I goofed), but wanted to pass it along.
@mrut2pac damn. I wanted to squash these commits... one second.
Jumped the gun, sorry :) created a new PR
Nice work guys. Thanks! Quick heads-up. I've been working on getting my test environment up-to-date so that I can do some testing on my live code. Just updated to VS 16.9.3, and started getting missing NumberStyles and CultureInfo errors in Level1DynamicFields.cs. Adding "using System.Globalization;" fixed it. I haven't tracked down the root cause (probably something I goofed), but wanted to pass it along.
@BunkerCoder could you please check if you're on the latest? Might have been an intermediate state
@mrut2pac damn. I wanted to squash these commits... one second.
Jumped the gun, sorry :) created a new PR
HAHAHAH i'll call you lucky luke... pretty quick on the trigger!!! Merged now!! Thank you so much
Nice work guys. Thanks! Quick heads-up. I've been working on getting my test environment up-to-date so that I can do some testing on my live code. Just updated to VS 16.9.3, and started getting missing NumberStyles and CultureInfo errors in Level1DynamicFields.cs. Adding "using System.Globalization;" fixed it. I haven't tracked down the root cause (probably something I goofed), but wanted to pass it along.
@BunkerCoder could you please check if you're on the latest? Might have been an intermediate state
@mathpaquette - Per https://docs.microsoft.com/en-us/visualstudio/releases/2019/release-notes, it's the latest. Released 3/30/2021.
@mrut2pac @NichUK quite a productive weekend.. Protocol 6.1 and now this live Dynamic client!! Congrats.
Nice work guys. Thanks! Quick heads-up. I've been working on getting my test environment up-to-date so that I can do some testing on my live code. Just updated to VS 16.9.3, and started getting missing NumberStyles and CultureInfo errors in Level1DynamicFields.cs. Adding "using System.Globalization;" fixed it. I haven't tracked down the root cause (probably something I goofed), but wanted to pass it along.
@BunkerCoder could you please check if you're on the latest? Might have been an intermediate state
@mathpaquette - Per https://docs.microsoft.com/en-us/visualstudio/releases/2019/release-notes, it's the latest. Released 3/30/2021.
Probably a bad intermediate state .... Make sure you Clean solution and Build.. Usually it helps. We dont merge anything without a successfully CI build in order to avoid such frustrating situation that the build doesnt pass :)
Thanks guys and Happy Easter, if you're celebrating!
On Sun, Apr 4, 2021 at 12:30 PM Mathieu Paquette @.***> wrote:
@mrut2pac https://github.com/mrut2pac @NichUK https://github.com/NichUK quite a productive weekend.. Protocol 6.1 and now this live Dynamic client!! Congrats.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mathpaquette/IQFeed.CSharpApiClient/pull/107#issuecomment-813087559, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABTDKSVOGAYC53YU634OHGLTHC463ANCNFSM42F4YCWQ .
Nice work guys. Thanks! Quick heads-up. I've been working on getting my test environment up-to-date so that I can do some testing on my live code. Just updated to VS 16.9.3, and started getting missing NumberStyles and CultureInfo errors in Level1DynamicFields.cs. Adding "using System.Globalization;" fixed it. I haven't tracked down the root cause (probably something I goofed), but wanted to pass it along.
@BunkerCoder could you please check if you're on the latest? Might have been an intermediate state
@mathpaquette - Per https://docs.microsoft.com/en-us/visualstudio/releases/2019/release-notes, it's the latest. Released 3/30/2021.
Probably a bad intermediate state .... Make sure you Clean solution and Build.. Usually it helps. We dont merge anything without a successfully CI build in order to avoid such frustrating situation that the build doesnt pass :)
@mathpaquette - I just pulled latest master (73d15c0), cleaned solution and rebuilt. Now I'm seeing "Merge conflict marker encountered" in Level1Client.cs, Level1DynamicFields.cs, and iUpdateSummaryMessage.cs. Looks legit, but probably my issue again, Checking.
@mathpaquette - my mistake - those are on my side...I'll resolve. Sorry...
Nice work guys. Thanks! Quick heads-up. I've been working on getting my test environment up-to-date so that I can do some testing on my live code. Just updated to VS 16.9.3, and started getting missing NumberStyles and CultureInfo errors in Level1DynamicFields.cs. Adding "using System.Globalization;" fixed it. I haven't tracked down the root cause (probably something I goofed), but wanted to pass it along.
@BunkerCoder could you please check if you're on the latest? Might have been an intermediate state
@mathpaquette - Per https://docs.microsoft.com/en-us/visualstudio/releases/2019/release-notes, it's the latest. Released 3/30/2021.
Probably a bad intermediate state .... Make sure you Clean solution and Build.. Usually it helps. We dont merge anything without a successfully CI build in order to avoid such frustrating situation that the build doesnt pass :)
@mathpaquette - I just pulled latest master (73d15c0), cleaned solution and rebuilt. Now I'm seeing "Merge conflict marker encountered" in Level1Client.cs, Level1DynamicFields.cs, and iUpdateSummaryMessage.cs. Looks legit, but probably my issue again, Checking.
Sorry earlier I force pushed on master to come back in a previous state. If you dont have any changes on master (technically should be the case) do this:
git fetch origin
git checkout master
git reset origin/master --hard
you will come back to HEAD
Nice work guys. Thanks! Quick heads-up. I've been working on getting my test environment up-to-date so that I can do some testing on my live code. Just updated to VS 16.9.3, and started getting missing NumberStyles and CultureInfo errors in Level1DynamicFields.cs. Adding "using System.Globalization;" fixed it. I haven't tracked down the root cause (probably something I goofed), but wanted to pass it along.
@BunkerCoder could you please check if you're on the latest? Might have been an intermediate state
@mathpaquette - Per https://docs.microsoft.com/en-us/visualstudio/releases/2019/release-notes, it's the latest. Released 3/30/2021.
Probably a bad intermediate state .... Make sure you Clean solution and Build.. Usually it helps. We dont merge anything without a successfully CI build in order to avoid such frustrating situation that the build doesnt pass :)
@mathpaquette - I just pulled latest master (73d15c0), cleaned solution and rebuilt. Now I'm seeing "Merge conflict marker encountered" in Level1Client.cs, Level1DynamicFields.cs, and iUpdateSummaryMessage.cs. Looks legit, but probably my issue again, Checking.
Sorry earlier I force pushed on master to come back in a previous state. If you dont have any changes on master (technically should be the case) do this:
git fetch origin git checkout master git reset origin/master --hard
you will come back to HEAD
I should have seen that. Back to HEAD now. Clean/Rebuild All worked. Now I'm frozen trying to open Test Explorer...probably an issue in 16.9.3. I'll figure it out and let you know what happened.
Update - It's Windows... Reboot fixed it... Ran tests locally without failures (but we knew that because of CI...). Thanks again.