/DG improvements for large datasets (introducing some new /D... commands)

[DE-cr]

addresses https://github.com/fredlcore/BSB-LAN/issues/539

[DE-cr]

Hmm, it should have worked either way:

4. When the user reduces the date range, the data is not fetched from the server again - that's only done when the user extends the date range beyond the most recent data transfer from the server

Today, /DG starts off with /D2023-03-15,2023-03-17 (three days back, starting with today).

Move the start date to a point in time before 2023-03-15 or the end date after 2023-03-17, /DG will do another /Da,b request, to fetch data from the server it clearly hasn't had included in the previous fetch. If the system behaves differently, I'll have to take another look at the javascript.

[fredlcore]

Move the start date to a point in time before 2023-03-15 or the end date after 2023-03-17, /DG will do another /Da,b request

That's basically this line in the code: if(a.value<al||b.value>bl) There's nothing wrong with this because when I set start date from 2023 to 2005, it should load all data from 2005 to 2023 and therefore correctly not load anything again when I move end date from 2023 to 2005. The problem then seems to be in the C++ code, because querying data from 2005 to 2023 just returns the CSV header. Had it returned the data for the requested time frame (basically the same as /D), then the problem wouldn't occur. Only the graph to be drawn wouldn not be more than a point (if at all) if you try to squeeze 18 years in a few thousand pixel. But the legend should show up and after narrowing the timeframe down, the graph should be visible.

[DE-cr]

/DI:

Date;DatalogPosition
2005-08-15;52

Hmm, with that, there should be data available for 2005-08-15 only, at least for /Da,b and therefore also for /DG. /D2005-08-15,2023-03-17 (or /D2005-08-15,2005-08-15) should have returned the same data as /D (or /D1, as of 2023-03-17).

The plot should use all of the width for the data available (i.e. 2005-08-15).

If you experience anything different, I'll have to re-check the code - JavaScript and/or C++.

[fredlcore]

Yes, /DI is correct, there is only data available for that one day. However, /D2005-08-15,2023-03-17 only returns the CSV header. /Dreturns the complete file (consisting of only data from one day).

[fredlcore]

Got it: The C++ code doesn't like data spanning across New Year's. So /D2005-08-15,2005-12-31works, but /D2005-08-15,2006-01-01 does not.

[DE-cr]

What version have you been using? This should have been fixed with https://github.com/fredlcore/BSB-LAN/pull/542/commits/ec51710df840675531ac9c85843f8790629cb4d7

If not, I can take a look at this some time after breakfast.

[fredlcore]

I'm using the this PR's branch's code (top of this page, click on DE-cr:dg-d-improvements-for-large-datasets, then download the zip file with the green button). This change doesn't seem to be in there, but it makes sense, so I'll try it out tomorrow.

[fredlcore]

Ok, tests with the Arduino have also all passed, the only thing I could not check is whether the actual space on the SD card was really freed because the size is given in MB and my data file was just 150kB. But downloading with /D showed a size reduction of 100kB, so that should be fine. Also, transfer speed today was up at over 300kB/s, even though no changes in code were made.

So I guess we're done here, thanks a lot for all your work! I'll merge this after I have made a release version 3.0 in a few days or so, so that all the changes in the pipeline won't potentially affect the currently rather stable situation we have now.

[DE-cr]

Thank you for testing this on Olimex+SD+LAN and on Arduino!

Thank you also for the discussions along the way (mostly in https://github.com/fredlcore/BSB-LAN/issues/539), which definitely have lead to a much better solution compared to my first offering. :)

[fredlcore]

Same here :)

[DE-cr]

@1coderookie let me know if you want me to help you in transferring these into the manual: https://github.com/fredlcore/BSB-LAN/pull/542#issuecomment-1465220464, https://github.com/fredlcore/BSB-LAN/pull/542#issuecomment-1471448906