FlorianRhiem
commented
3 years ago Hey @MayerBjoern and Rene,
thank you for the pull request! While plots can already be uploaded as images, this combination of the data behind the plot and the plot itself can surely come in useful, especially when the data is inserted via the API. However I don't completely understand why preventing manual input of the data would necessary, though. Could you explain why the plotly_chart field needs to be readonly? If there's a good reason for that, maybe a tooltip could explain this when hovering over the textarea?
Also, it looks as if a user could still enter data by altering the textarea using an inspector tool, and the form data would then be interpreted. Is that intentional?
I've tried the PDF export and first got an error that the psutil package is required, after installing that I got another error that the orca executable is required as well. Going by the plotly documentation it seems that the export should instead be done by Kaleido and after installing Kaleido via pip as well, the PDF export worked :) Can you add the psutil and Kaleido packages and their dependencies to requirements.txt?
As the plot filter / plotly_base64_image_from_json function is only used for PDF export, it should export a vector format instead of a raster format. Here is a side-by-side comparison of SVG export (left) and PNG export (right) that shows the much clearer text:
All needed for that is to pass the svg format and use the image/svg+xml mime type:
fig_plot.write_image(image_stream, "svg")
image_stream.seek(0)
return 'data:image/svg+xml;base64,{}'.format(base64.b64encode(image_stream.read()).decode('utf-8'))Going through the code, the plotly data is almost always used in the encoded form. If I didn't miss anything, the only point where it actually needs to be decoded is during validation, when you check that it is a dictionary. If we treat the plotly data as a black-box text format everywhere else, we could avoid a lot of back and forth between decoding and encoding, and it also wouldn't be necessary to change the SQLAlchemy encoder. If we still decode it during validation, the dict-check could still be done along with possible future restrictions (e.g. if a certain plot type were to be required by the schema). What's your opinion on that, do you see any further pros/cons?
MayerBjoern
We introduced a new datatype based on the plotly JSON data format. The idea is to add the capability to store plots, their raw data and visual reprensentation information in the simplest way possible, without having to introduce and manage an own unique system to store different types of raw data and definitions for a multitude of different plot types.
The plotly_chart datatype simply expects the plotly JSON chart schema (https://plotly.com/chart-studio-help/json-chart-schema/#:~:text=The%20Plotly%20JSON%20visualization%20schema,platform%20that%20understands%20the%20schema.) as its data. The JSON structuce is externally verified via the python plotly module.
The demo dataset contains two Samples that show the new datatype in action.
On top of the plotly related changes we also made two general code changes in the process.
We use demjson to decode the JSON plotly data into a python dict. The reason for this is to not loose any floating point accuracy, since demjson creates decimal objects in python (https://docs.python.org/3/library/decimal.html). For this to work, we also change the SQLAlchemy json_serializer in the init.py, since the default json serializer can't handle decimal objects. The sample data in scripts/demo_data/objects/plotly_example_data2.sampledb.json contains an example that needs the decimal object conversion to not loose floating point accuracy. Maybe in the future we can also use this to set up the pint conversions between regular magnitude and base_units to prevent accuracy loss.
We moved the inclusion of jquery to the head of the HTML, so we have it available earlier then the general script_block at the very end of the page. It is just a little bit more convienient, since I atleast don't know how to modify jinja blocks from within an included template within a child template like the objects/view and objects/forms templates.