highcharts / node-export-server

Highcharts Node.js export server
Other
356 stars 260 forks source link

chart rendering breaks around 5000 x-Axis data points #523

Open bvogel opened 5 months ago

bvogel commented 5 months ago

node v20 highcharts version 10.4.0 export server version: 3.1.1

Expected behaviour

Graph rendered correctly (4999 data points)

image

Actual behaviour

Graph rendered with increasing errors (5000 data points)

image

graph don't start at the same point any more

Graph broken (5001 data points)

image

graph is missing 3/4 of data

Reproduction steps

infiles for the cases:

working json

partially broken

broken

bvogel commented 5 months ago

In the meantime I found out that the error in the second graph may result from unbalanced series lengths, so that error may be expected, however the series have the same length for the third graph and it's still broken

bvogel commented 5 months ago

I can add an additional example for failing data around the 5000 data points mark:

working (4999 data points) image

broken (5000 data points) image

Although the graph is still being rendered it looses the requested gradient.

jszuminski commented 5 months ago

Thanks for reporting @bvogel!

It seems to be a duplicate of this issue: https://github.com/highcharts/node-export-server/issues/490

It should work in the newest version.

bvogel commented 5 months ago

Thank you, any news on the release date?

I'd be happy to test a beta or rc release as well

bvogel commented 4 months ago

Another month has passed, I can't really take this issue as resolved as we can't use the proposed solution - it still hasn't been released yet. Please advise how to proceed. (Due to internal guidelines we aren't in a position to use branches as reference for our packages)

bvogel commented 4 months ago

bump!

jszuminski commented 4 months ago

@bvogel it has been fixed (on the master branch) and on the official deployed Highcharts Export Server. Please try it out :)

bvogel commented 4 months ago

@jszuminski unfortunately we can't use the external server due to company policies, so I'll have to wait for the official release (my test data was deleted already)

jszuminski commented 4 months ago

Absolutely, I understand. If you'd like, you can always try to download the package from GitHub.

bvogel commented 3 months ago

Hi, I just tested with:

INFO 2024-08-08T18:05:17.669282664Z [resource.labels.containerName: highcharts-export-server] Thu Aug 08 2024 18:05:17 GMT+0000 [verbose] - [cache] Fetching script - https://code.highcharts.com/11.4/highcharts-more.js
INFO 2024-08-08T18:05:17.608273297Z [resource.labels.containerName: highcharts-export-server] Thu Aug 08 2024 18:05:17 GMT+0000 [verbose] - [cache] Fetching script - https://code.highcharts.com/11.4/highcharts.js
INFO 2024-08-08T18:05:17.602820134Z [resource.labels.containerName: highcharts-export-server] Thu Aug 08 2024 18:05:17 GMT+0000 [notice] - [cache] Updating cache version to Highcharts: 11.4/.
INFO 2024-08-08T18:05:17.602413336Z [resource.labels.containerName: highcharts-export-server] Thu Aug 08 2024 18:05:17 GMT+0000 [notice] - [cache] Fetching and caching Highcharts dependencies.
INFO 2024-08-08T18:05:17.600292924Z [resource.labels.containerName: highcharts-export-server] Thu Aug 08 2024 18:05:17 GMT+0000 [notice] - [process] Attaching exit listeners to the process.
INFO 2024-08-08T18:05:17.598449415Z [resource.labels.containerName: highcharts-export-server] Starting Highcharts Export Server v4.0.1...

but still seeing the same error. 5000 x-axis points will break rendering.

In contrast to #490 I don't get an 500 error but just a broken graph.

bvogel commented 2 months ago

i.e. can we have this re-opened please?

jszuminski commented 2 months ago

I've reopened this issue and added it to our backlog. If the problem indeed persists (we will have to investigate that), we will try to make it a part of one of the upcoming releases.

bvogel commented 2 months ago

Thank you!

bvogel commented 2 months ago

@jszuminski were you able to reproduce? should I sent test data from our end?

jszuminski commented 2 months ago

@bvogel if possible, you can share the test data, that would be helpful. You can also try to create a minimal reproducible example in JSFiddle, starting from there: https://jsfiddle.net/ftpyxewh/

bvogel commented 2 months ago

Hi @jszuminski,

this fiddle will work as fiddle but is broken if exported through the server https://jsfiddle.net/5ejsu9go/

this one will work: https://jsfiddle.net/j6ns0d3v/

Some additional data: the export server receives a post with:

json: { infile: <options from the fiddle>, scale: 2, type: "jpeg" }
bvogel commented 1 month ago

@jszuminski any news on this, were you able to reproduce the error we are seeing?

jszuminski commented 1 month ago

@bvogel I seem to have different errors when exporting this, but do not encounter what you encounter. The good news is that this https://github.com/highcharts/node-export-server/pull/561 pull request will introduce many beneficial changes and will allow to detect errors in request/CLI options if they are the ones causing an error so a part of the problem will get solved there; you will get an error or a warning message if the configuration is not optimal or not supported.

However, for now, let's get back to the main problem which is the partial rendering of the line chart. Could you please tell me:

  1. Are you sending an HTTP request or are you doing this via CLI? currently, There are separate exporting processes happening; we're currently working on unifying them which will also get rid of this problem.

  2. Could you share the full CLI command and a broken .json file so that I can 100% get the error that you get? I will then just run the command and share the result with you to compare. If this causes an error on my side as well, I will debug it and get to the root cause.

bvogel commented 1 month ago

we use a HTTP POST request, so I have no CLI or json file. The jsfiddles are very close to the requests we send. as mentioned in the previous post we use this code:

    def request_chart(for_pdf: false)
      scaling = for_pdf ? 2 : 1
      result = HTTP.post(HIGHCHARTS_EXPORT_SERVER_URL, json: { infile: @chart_options, scale: scaling, type: "jpeg" })

      raise HighchartsError, "status: #{result.status.code} reason: #{result.body}" unless result.status.success?

      StringIO.new(result.body)
    end

^^^ we see the error for for_pdf = true i.e. scale: 2

with the @chart_options being 100% identical to the ones in the fiddles. We use Highcharts Export Server v4.0.1 with Highcharts library 11.4.0

edit: this is the log from a export server for a failing request

2024-09-13 14:42:26 Fri Sep 13 2024 12:42:26 GMT+0000 [verbose] - [export] Got an incoming HTTP request with ID d510f5bd30cc4ad98ad04fd4ba82ae58.
2024-09-13 14:42:26 Fri Sep 13 2024 12:42:26 GMT+0000 [verbose] - [chart] Starting the exporting process.
2024-09-13 14:42:26 Fri Sep 13 2024 12:42:26 GMT+0000 [verbose] - [chart] Attempting to export from a raw input.
2024-09-13 14:42:26 Fri Sep 13 2024 12:42:26 GMT+0000 [verbose] - [pool] Work received, starting to process.
2024-09-13 14:42:26 Fri Sep 13 2024 12:42:26 GMT+0000 [verbose] - [pool] Acquiring a worker handle.
2024-09-13 14:42:26 Fri Sep 13 2024 12:42:26 GMT+0000 [verbose] - [pool] Acquired a worker handle.
2024-09-13 14:42:26 Fri Sep 13 2024 12:42:26 GMT+0000 [verbose] - [pool] Starting work on pool entry with ID c7d29e24-f160-4ac3-befe-28c17e54e271.
2024-09-13 14:42:26 Fri Sep 13 2024 12:42:26 GMT+0000 [verbose] - [export] Determining export path.
2024-09-13 14:42:26 Fri Sep 13 2024 12:42:26 GMT+0000 [verbose] - [export] Treating as config.
2024-09-13 14:42:26 Fri Sep 13 2024 12:42:26 GMT+0000 [verbose] - [pool] Work completed in 129 ms.

edit 2: you can use this link to get a broken sample: https://capinside.com/products/pictet-digital-hi-eur-lu0386392772/fact_sheet?performance_start_date=2011-01-05&performance_end_date=2024-09-13&peergroup_id=13&add_personalization=false&risk_return_range=1_year and this one (one day less) for a working one https://capinside.com/products/pictet-digital-hi-eur-lu0386392772/fact_sheet?performance_start_date=2011-01-06&performance_end_date=2024-09-13&peergroup_id=13&add_personalization=false&risk_return_range=1_year - and for good measure, this is the page we start from, probably some data can be deduced from the graph in use there: https://capinside.com/p/pictet-digital-hi-eur-lu0386392772?locale=en

jszuminski commented 1 month ago

Okay, thank you very much @bvogel! I have successfully managed to reproduce it locally and get the same result as you do:

Screenshot 2024-09-18 at 09 06 48

I will now investigate the cause of the problem by debugging what's going on inside the Puppeteer.

bvogel commented 1 month ago

Thank you so much for the update!