fair-acc / chart-fx

A scientific charting library focused on performance optimised real-time data visualisation at 25 Hz update rates for data sets with a few 10 thousand up to 5 million data points.
GNU Lesser General Public License v3.0
500 stars 92 forks source link

Financial charts #294

Closed raven2cz closed 3 years ago

raven2cz commented 3 years ago

We are planning to implement new financial platform. There are requirements for realtime processing. Almost all reqs are based on time series. The type of charts are candlesticks, HiLo renderers. About UI control is necessary to move dynamically and fast in windowed time range, to see detailed structure of prices. The last important req is possibility to add lines to the chart for visualization entries and exits of trades. There can be a thousands these objects which can be added to the chart for visualization. Is it possible to used your library for it? Or do you recommend to start from javafx directly? Thanks for your advice.

RalphSteinhagen commented 3 years ago

@raven2cz thanks for your interest in chart-fx. Of course, this is possible. There is an existing prototype implementation for the candlestick charts by @petrov9 that builds on top of chart-fx see #64 for details.

We could merge this into the master provided @petrov9 agrees to this, as this is his brainchild.

Otherwise, the Zoomer and other features are available. Notably dealing with large sample sizes is not an issue... this library was designed to handle 10k to millions of data points. :smirk:

Since chart-fx is a library we won't be able to write your application though and you may need to apply some of your own customizations to adjust the specifics to your taste. :smile:

raven2cz commented 3 years ago

The candlesticks and HiLo renderers are most important part of whole visualization. Most important is support for fast rendering and moving of sliding window for visualization of specific time range. Yes, zoom is important too, but the selected "fixed" window which is moved by sliders are more important. So, the correct control for sliders is necessary. I know problematic algorithms for candlestick visualization. For example, correct calculation of candles width is not trivial and several platforms have problems with it.

We have several UI visualizations in JFreeChart after several years. I post here the examples which are planned for the realization. I know that there is possible to use bridge between jfreechart and javafx, but this is another league. We plan to create new fast platform with perfect fast visualization, no bridges :-)

How about CPU and GPU. Do you have some tests if the scene can be process by GPU without software rendering?

alexis-finwin alexis-analysis-T30min alexis-intraday-volume-profiles-3-dark-theme alexis-intraday-volume-profiles-2 alexis-orderflow-3 alexis-orderflow-1

RalphSteinhagen commented 3 years ago

How about CPU and GPU. Do you have some tests if the scene can be process by GPU without software rendering?

@raven2cz for compatibility reasons, the library uses plain Java (i.e. w/o custom JNI interfaces) and JavaFX as its UI backend. JavaFX supports natively OpenGL- and/or SW-based rendering depending on the hardware you are running it on. Internally, our rendering routines use Canvas and GraphicsContext, which -- based on various tests -- seems to be the fastest access to the lower JavaFX rendering pipe-line. For more details see this link.

Most important is support for fast rendering [...]

There are the following ingredients for developing a fast rendering in JavaFX:

Again, the library is designed to be highly customizable and it's possible to draw financially related chart as demonstrated [here]](https://github.com/petrov9/chart-fx/blob/new_charts/chartfx-chart/src/main/java/de/gsi/chart/renderer/spi/CandleStickRenderer.java) but you may need to adjust some of the plotting choices/renderers to your liking.

Hope this brief information is helpful and please feel invited to make a PR contribution in this particular area.

raven2cz commented 3 years ago

Thanks for your all details. I checked and tested samples and speed. I already try to implement simple renderer and dataset. It seems as good library! I expect that we will start usage and extend chart-fx with financial package in next years. I have just few questions before closing this issue. It will very help with my tests and next work:

chart-fx-financial-test1

ennerf commented 3 years ago

Regarding (1):

I assume the end result would also have to know about Holidays etc. and be more complex to implement than just filtering every 6th & 7th value. Wouldn't it be simpler to just use trading day indices with a custom label mapper? e.g. trading_day_index[278] = 2017-Nov-30

wirew0rm commented 3 years ago

Thanks for your all details. I checked and tested samples and speed. I already try to implement simple renderer and dataset. It seems as good library! I expect that we will start usage and extend chart-fx with financial package in next years. I have just few questions before closing this issue. It will very help with my tests and next work:

Nice to hear that chart-fx works well for you. Please don't hesitate to share your progress, especially if you at some point want to merge your efforts back into chart-fx (which we definitely encurage).

  • timeseries need more improvement for financial toolbox. How about time filters, do you have possibility to filter weekends? It is more complicated because there hasn't to be present gap in the chart, it means time filter has to completely remove filtered parts.

There is an open feature request for this #77. I agree with @ennerf that there is a lot of hidden complexity in this filtering aproach (e.g. what happens if there is data for an area where there shouldn't be data, due to timezone mixup etc, how should Zoom and other tools handle these gaps ...).

  • sliding the zoom window by xaxis slider works very well. Is it possible to configure chart data "alignment" vertically to see all vertical data (min, max in window range) that the window is sliding? This is important feature for financial toolbox where the all ohlcv data has to be visible all time in the selected time range.

The auto ranging feature only supports auto ranging on the complete DataSet. The axes do not know of each other, so this is not completely straightforward to implement in chartfx. You can use a custom JavaFx Binding which computes the shown range whenever the xRange changes from the dataset. Beware that the Binding will effectively disable the Zoomer for this axis. Alternatively you could probably wrap your Dataset in a custom DataSet, which updates its range depending on your xAxis range.

  • last point is common, which seems as the bug. The mouse wheel works very good for zoom out, but doesn't work for zoom in, just usage of rectangle can be used for zooming. This is very frustrating for every library user. I expect same functionality for mouse wheel.

This is interesting. I actually observed the same behaviour on my development machine but attributed it to my machine setup as others do not seem to have the problem (actually it even works on my machine when I access it remotely via X2Go). Can you share some information about the versions of java, javafx and your operating system?

raven2cz commented 3 years ago

Regarding (1):

I assume the end result would also have to know about Holidays etc. and be more complex to implement than just filtering every 6th & 7th value. Wouldn't it be simpler to just use trading day indices with a custom label mapper? e.g. trading_day_index[278] = 2017-Nov-30

Yes. This seems as good trick ( at first glance ). So, which component can be used for extension? I'm chart-fx newbie. I expect AbstractAxisTransform? Like LogarithmicAxisTransform?

raven2cz commented 3 years ago

Thanks for your all details. I checked and tested samples and speed. I already try to implement simple renderer and dataset. It seems as good library! I expect that we will start usage and extend chart-fx with financial package in next years. I have just few questions before closing this issue. It will very help with my tests and next work:

Nice to hear that chart-fx works well for you. Please don't hesitate to share your progress, especially if you at some point want to merge your efforts back into chart-fx (which we definitely encurage).

Yes, of course. Financial package/toolbox can be part of the chart-fx "family". There is one potential problem to solve. The API between common chart-fx and financial framework. In the principle, the based API is mainly domain objects which are defined in the separate financial framework. These filled DOs are placed under the datasets and other your interfaces. This is important to keep just one ohlcv instances in the memory. This cannot be stored to directly the datasets defined by chart-fx, because financial framework cannot be depended on the datasets in chart-fx. This has to be discussed which part can be placed to common chart-fx. Maybe defined some datasets as examples for usage of financial toolbox.

  • timeseries need more improvement for financial toolbox. How about time filters, do you have possibility to filter weekends? It is more complicated because there hasn't to be present gap in the chart, it means time filter has to completely remove filtered parts.

There is an open feature request for this #77. I agree with @ennerf that there is a lot of hidden complexity in this filtering aproach (e.g. what happens if there is data for an area where there shouldn't be data, due to timezone mixup etc, how should Zoom and other tools handle these gaps ...).

separate topic before this post.

  • sliding the zoom window by xaxis slider works very well. Is it possible to configure chart data "alignment" vertically to see all vertical data (min, max in window range) that the window is sliding? This is important feature for financial toolbox where the all ohlcv data has to be visible all time in the selected time range.

The auto ranging feature only supports auto ranging on the complete DataSet. The axes do not know of each other, so this is not completely straightforward to implement in chartfx. You can use a custom JavaFx Binding which computes the shown range whenever the xRange changes from the dataset. Beware that the Binding will effectively disable the Zoomer for this axis. Alternatively you could probably wrap your Dataset in a custom DataSet, which updates its range depending on your xAxis range.

Thanks for this idea! The wrapping of Dataset is not good idea, but the javafx binding seems OK. Can you show me some example of this binding when xrange is changed? About Zoomer. It has to be different way, a zoomer has to work, but if the zoomer is used, this feature has to be disabled. I already implement this in jfreechart. So if a slider is used, the feature is activated and works. If the zoomer is used, this feature is disabled temporary. We can add some button too for activate this feature. Technically, there is not necessary to activate or deactivate the feature. If the xaxis slider is used the correct zoom of dataset is necessary to see min-max of yRange.

  • last point is common, which seems as the bug. The mouse wheel works very good for zoom out, but doesn't work for zoom in, just usage of rectangle can be used for zooming. This is very frustrating for every library user. I expect same functionality for mouse wheel.

This is interesting. I actually observed the same behaviour on my development machine but attributed it to my machine setup as others do not seem to have the problem (actually it even works on my machine when I access it remotely via X2Go). Can you share some information about the versions of java, javafx and your operating system?

Testing station is:
openjdk 11.0.8 2020-07-14 OpenJDK Runtime Environment (build 11.0.8+10) OpenJDK 64-Bit Server VM (build 11.0.8+10, mixed mode)

javafx is not changed, I'm using your chart-fx master branch with

13.0.1 11.0.0 kernel is 5.9.3-arch1-1, Arch Linux x86_64
RalphSteinhagen commented 3 years ago

@raven2cz I'd suggest to store the trading dates as data labels in the DataSet and to use then CategoryAxis. This should then be also compatible with the Zoomer and other plugins.

raven2cz commented 3 years ago

@raven2cz I'd suggest to store the trading dates as data labels in the DataSet and to use then CategoryAxis. This should then be also compatible with the Zoomer and other plugins.

Yes, I tested it, and it works. Dataset sends labels to category axis, no gaps, and labels are shown. BUT, there will be a lot of work how to correctly organize minor and major labels, and mainly changes in the zooming time. In actual test case, it is disaster because there is no more automatic logic for labels. See picture example. Which parameters of a category axis are important for some better formatting / organize category minor and major labels, minimal gaps between them, for example to show just few labels, possibility to read labels, no huge overlapping of labels? I tried to sample with category axis, but there is no correct config which will work for it...

obrazek

RalphSteinhagen commented 3 years ago

@raven2cz hat seems to be related to the chosen/default AxisLabelOverlapPolicy maybe you want to set this to a different value, e.g.:

xAxis.setOverlapPolicy(AxisLabelOverlapPolicy.SKIP_ALT);

You may try the other values as well.

raven2cz commented 3 years ago

@raven2cz hat seems to be related to the chosen/default AxisLabelOverlapPolicy maybe you want to set this to a different value, e.g.:

xAxis.setOverlapPolicy(AxisLabelOverlapPolicy.SKIP_ALT);

You may try the other values as well.

Thanks, works. But it is terrible usage after several testing. There are many problems with visualization, mainly minor and major grid lines. Not possibility to use correctly unit scaling, to see years if zoom is out, the x-axis still needs good quality processing like TimeAxis. So, this is not about some quick solution. It is missing and it will be part of our future implementation. Maybe it has to be combination TimeAxis which has internally indexed vector (like category axis) for internal processing, but labels, grid, and time scaling and showing is driven by defined time values in the vector, not just some formatting of dataset label values only.

raven2cz commented 3 years ago

So, we will start with prototypes for basic candlesticks first. Which type of co-op do you prefer? Can I create fork of chart-fx, and we will plan merge to repo after feature stabilization ok? Can I ask for your help about an evaluation and code review after finished steps?

RalphSteinhagen commented 3 years ago

@raven2cz sure, a PR request would be highly appreciated.

We follow the usual GH process outlined here and that would start with forking and then performing PR against our repository. Please feel free to make use of GitHub's 'draft PR' option as early as possible to check that your build/modifications pass our automatic build pipeline, static code-analysis, code-coverage, etc. Once you feel that you are finished, you can change the 'draft PR' and request a review.

Looking forward to your PR! :+1: :tada:

gregwis commented 3 years ago

Hi @RalphSteinhagen, I'm developing similar candlestick chart. Could you give me a hint how to calculate width of a candle that would be proportional to visible subset of dataset.

RalphSteinhagen commented 3 years ago

@gregwis maybe have a look at the HistogramRenderer and corresponding sample for getting inspiration. The idea is to base the width on the physical pixel distance between the neighbouring bin centres...

Maybe you and @raven2cz could join forces and/or merge your approaches? In any case, we are looking forward to your PRs.

raven2cz commented 3 years ago

Hi, yes @gregwis. From my point of view, it is wasting time to make same thing independently... In addition, I have implementation already, but it is just small piece of whole toolbox. The financial toolbox needs much more changes, not just a candlestick renderer. What do you expect from your work? In any cases, I will create fork in the next week and push my actual work.

gregwis commented 3 years ago

My requirements are very similar. I also have already developed a draft version. We can try to develope on one version. Currently I'm trying @RalphSteinhagen sugestions. I think HistogramRenderer options of calculation bar dimentions are overcomplicated for our purposes. I simplified them and I'm testing it on my candlestick renderer.

raven2cz commented 3 years ago

Candlestick renderer supports themes (not just color schemes, this is called SAND theme) with shadows, without effects or blur performance problems. Full perf still ensured ;-)

obrazek

milo-gsi commented 3 years ago

Great, I really like that theme. Simple and slick!

gregwis commented 3 years ago

I prefer simple red and green candles without shadows: shadows make it difficult to draw accurate trend lines two distinctable colors allow you to see price changes at a glance

I tried to implement custom Tooltip. I couldn't inherit from DataPointTooltip and I had to write new one inherited from AbstractDataFormattingPlugin (but only value formatters properties were reused). These two classes are hard to customize, especially for data set with data points with more than one y value.

obraz

raven2cz commented 3 years ago

Super, @gregwis! Tooltips or "top" information chart lines are important too. It is next topic for impl.

Of course that a candlestick renderer supports more color schemes. Actually I tested 5. And your example is called OLDSCHOOL in actual implementation.

obrazek

For prototype is important to try more schemes to see where are the implementation gaps in the design. For example shadows are present in the SAND scheme only and it is applied to all chart line components, for example trend lines too. If the shadow is correctly implemented, it causes "plastic" visualization which is necessary for meetings or projectors. Other schemes are in dark modes, eyecare problems etc.

In any way, good work @gregwis!

RalphSteinhagen commented 3 years ago

I couldn't inherit from DataPointTooltip and I had to write new one inherited from AbstractDataFormattingPlugin (but only value formatters properties were reused).

@gregwis and @raven2cz we have been recently pushed and released updates regarding the tool tip plugin. You may want to rebase your repositories against the master branch and see whether you still need to provide a parallel development.

Keep up the good work and do not be afraid to make a draft PR. It's very helpful to already see already intermediate results. This will make the review process and merge request much easier!

Again, looking forward!

raven2cz commented 3 years ago

Thanks for info. I expect that raw draft will be ready this or next week. But it is just a prototype, It needs more iteration to final PR to your library. I expect more cycles in the fork.

obrazek

gregwis commented 3 years ago

@raven2cz, charts look great!

@RalphSteinhagen, I tried with current version and I had problems with customizing your Tooltip: method formatData in AbstractDataFormattingPlugin cannot be simply overriden because it uses private methods getXValueFormatter(final Axis xAxis) and getYValueFormatter(final Axis yAxis). Method formatData takes Tuple with two Numbers - I need 4 values. Therefore methods invoking formatData also have to be overriden. Extended DataPoint subclass added even more compexity. Mouse proximity calculation needs to be customized too (candles can be long and Tooltip should be displayed on any point on candle) - it is on my TODO list. Therefore it was easier to rewrite the entire DataTooltip class. Maybe it is not preffered way...

I am looking for a function that allows me to double-click on the chart and automatically scale the data range of Y values to min/max visible X range. Do I have to write a new plugin?

raven2cz commented 3 years ago

@RalphSteinhagen, and @gregwis. Thanks for all your work and questions. Mainly this one: "I am looking for a function that allows me to double-click on the chart and automatically scale the data range of Y values to min/max visible X range. Do I have to write a new plugin?"

In my opinion, this completely missing in actual chart-fx. I wrote before that keeping of min-max Y values are alpha-omega for all financial charts. It is a feature for sliding with the slider, for "double-click", button-zooming etc. Top ideal is addition of arrow keys: Up, Down keys for zoom-in/out, and Left/Right key for sliding in time (see video for zooming and movement in the chart).

We will have a lot of work with renderers now. I cannot waste time to study "best" solution correct keeping of Y ranges. Is it possible to create new issue for "core" developers to ensure correctly this functionality? In my opinion, this is not just financial charts that Y-values have to be visible all time (in the restricted X-axis range, not for all data of course). Note: Standard zooming with mouse scroll button has to be kept, no change.

What I mean is presented in this amibroker youtube video from 1:24 for sliding, zooming etc. please look to this short video for understanding. Required chart features - example video from Amibroker >1:24

RalphSteinhagen commented 3 years ago

[..] because it uses private methods [..]

@gregwis feel free to change the private to protected or even improve that class as long as the other user-cases will not break with these changes (changing 'private -> protected' is certainly not an issue). That class is very old and hasn't been revisited/refactored for a while.

[..] Mouse proximity calculation needs to be customized too (candles can be long and Tooltip should be displayed on any point on candle) [..]

True, we haven't considered the candle-stick use-case, since our applications mostly deal with (x,y, + x/y errors)-type data. If you find an elegant way to support both use-cases, please feel free to update that class.

I am looking for a function that allows me to double-click on the chart and automatically scale the data range of Y values to min/max visible X range. Do I have to write a new plugin?

Valid point, this is not implemented as such, and would require either an extension of the Zoomer and/or new second Zoomer plugin implementation. The auto-range feature takes the whole/combined (X or Y) range of the DataSets into account. The corresponding code can be found here. The automatic range computation is automatically disabled if you zoom in for a given axis. For example, if you zoom in only for X then the auto-range is disabled for X but preserves the auto-range for Y and vice versa. We need this for our use-case when scrolling along the x-axis such that the y-axis range does not change and that the zoomed data is related to the y-range of the full (un-zoomed) dataset. At the same time, I understand that for a financial context you may be only interested in the relative y-range of the actually visible data points/tuples.

May I suggest that you open another dedicated issue/feature request, detailing your/this use-case? This issue/track is already getting quite long and mixes-up different features requests and extensions. Thanks in advance.

raven2cz commented 3 years ago

High Low (OHLC) Renderer preview. obrazek

wirew0rm commented 3 years ago

Hey everyone,

thanks for the updates, this all looks really nice. We will have a small presentation of chart-fx at JFX Days online. Would it be okay for you if we show some of your images there as an example of ongoing contributions? cc/ @petrov9 @gregwis @raven2cz

Keep up the good work,

Alex

raven2cz commented 3 years ago

Approved required for merging #318. Reviewer needs write access rights.

raven2cz commented 3 years ago

Hi @wirew0rm, sure from me. There is new screenshot from AdvancedCandlestickSample. The PR is waiting.

chart-fx-advanced-candlestick-sample1

raven2cz commented 3 years ago

Hello @RalphSteinhagen or others, I opened draft PR and made several junits and samples according to your CI. But restyle branch needs some required review and write access rights. In addition, the travis crashed in the microservices (it is not my work, I tested restyle branch too, and all pass. It seems to another problem. Can you fix it? Which next steps are important from my side?

RalphSteinhagen commented 3 years ago

@raven2cz very good. Had a brief look at your PR. At a first glance, there seem to be only very few and minor issues. I especially appreciate that you minimised the PMD notifications and provided some unit tests. That makes a big difference. :+1:

Regarding our restyled-io bot: the fix-PR seems to erroneously try to merge the fixes directly into the chart-fx master and not your repository's feature branch. Seems to be a bot setup problem that we need to follow-up later. In the mean-time, could you run either the formatter script on your branch and/or merge the proposed restyled-io bot commits to your feature branch? Thanks in advance.

We are presently occupied with unrelated internal matters, but will have a more detailed look over the weekend and we will provide feedback early next week. However, since your PR passes all formal CI checks, we are quite confident that we can merge this then very soon afterwards. :+1:

raven2cz commented 3 years ago

Thanks for info. I checked the state of branch restyled/pull-316 (in GSI-CS-CO / chart-fx), and it seems that it is not up to date by my last commits of PMDs. In addition, there is some your new class DefaultTimeFormatter2 which doesn't fit to the result.

PR Restyle Financial Chart: CandleStickRenderer, HighLowRenderer, Basic Financial API #318 is not relevant and has to be deleted.

In actual time, there is no chance to merge restyle results which are corrupted. Is there some possibility to launch script or restyled-io process in my fork? Can you give me some link, script or settings for it? I do not work with this restyled service before.

RalphSteinhagen commented 3 years ago

Info courtesy by @wirew0rm: You do not have to use the restyled bot, it is only a convenience tool for people submitting pull requests who do not have formatting tools available.

If you have clang format > 9.0.0 (which should be available on archlinux) you can just run git rebase master --exec ./restyleLastCommit.sh and all your commits will be formatted according to our guidelines and restyle will automatically close its restyle commit.

RalphSteinhagen commented 3 years ago

@raven2cz got merged in #316 ! Thanks also to all others who contributed or provided ideas. :+1:

If you agree, I propose to close this issue.

raven2cz commented 3 years ago

Yes, close it. For next financial components we will create new issues. Step by step.

RalphSteinhagen commented 3 years ago

For next financial components we will create new issues.

:open_mouth: is it Christmas already? Please keep these good PRs coming! :smile: :+1:

lacgit commented 1 year ago

Regarding (1):

I assume the end result would also have to know about Holidays etc. and be more complex to implement than just filtering every 6th & 7th value. Wouldn't it be simpler to just use trading day indices with a custom label mapper? e.g. trading_day_index[278] = 2017-Nov-30

Hi all, I am new to chart-fx, but I can see its potential and I am trying to work with the financial samples which is quite easy to follow. I am trying to build candle stick chart without weekend/holiday gap. I managed to modify the finance advance sample to use the dataset index as the axis to simulate this effect, but to work on the tick label is still far from away. I also saw another thought from the team to supply date time into the category axis. Before going further, I am seeking your thoughts which is better path to invest more time on this feature. Any thought are welcome. Thanks in advance. Regards,

yezhengli-Mr9 commented 9 months ago

The candlesticks and HiLo renderers are most important part of whole visualization. Most important is support for fast rendering and moving of sliding window for visualization of specific time range. Yes, zoom is important too, but the selected "fixed" window which is moved by sliders are more important. So, the correct control for sliders is necessary. I know problematic algorithms for candlestick visualization. For example, correct calculation of candles width is not trivial and several platforms have problems with it.

Dear all friends, Is there simple examples https://github.com/fair-acc/chart-fx/tree/main/chartfx-samples/src/main/java/io/fair_acc/sample/financial showing charts with time-series scrollable (with adaptive min, max, volume distribution, etc.) as well as better with adaptive "selected window" scrollable? If chart-fx does not have it, such functionality in other libraries (javafx, etc.) are also welcome.

Can also work as developer/contributor to chart-fx if necessary~

Best and thanks,

sliding the zoom window by xaxis slider works very well. Is it possible to configure chart data "alignment" vertically to see all vertical data (min, max in window range) that the window is sliding? This is important feature for financial toolbox where the all ohlcv data has to be visible all time in the selected time range.

Mainly this one: "I am looking for a function that allows me to double-click on the chart and automatically scale the data range of Y values to min/max visible X range. Do I have to write a new plugin?"

image https://github.com/fair-acc/chart-fx/issues/77 https://github.com/fair-acc/chart-fx/discussions/394

raven2cz commented 9 months ago

@yezhengli-Mr9 I'm not sure if I understand your requirements. Perhaps try to describe it in your own words and let chat-gpt translate the precise description into English.

So far, I have understood it as you wanting to automate the Y-axis so that the min-max is always visible in the given window. However, this has already been implemented long ago. In the examples mentioned in the toolbox, there is a boolean auto-y. If you check this box, this feature will be activated for the current width of the sliding window.

The library should be intended primarily for users with more advanced skills in Java and JavaFX. In most cases, it is necessary to write some custom parts or connect it to your own trading platform. The examples show a simulation of a trading platform and simple data loading, but it is important to realize that these are just examples, and for full functionality, you need to integrate this library with more advanced parts of your frameworks.