Closed jackparmer closed 7 years ago
I'm not aware of any formal dashboard JSON spec documentation, but this dashboard is almost exhaustive in its options usage:
"first": {
"boxType": "plot",
"shareKey": null,
"fileId": "jackluo:402"
}
Looks like this is versioned version: 2
, which is great, let's make sure that we can bake that into the Python handling so that we future updates don't need to become backwards incompatible.
create_dashboard()
So far, we've taken the approach of object-based manipulation. The idea would be do something more like:
# Instantiate a new dashboard from scratch
dashboard = Dashboard()
# Instantiate a dashboard from url
dashboard = Dashboard(https://plot.ly/dashboard/jackluo:430)
# Methods to update a dashboard (just examples, not sure on the what methods we really need)
dashboard.add_row()
dashboard.add_column()
dashboard.update_cell(row, col, update)
General question, thoughts on development cycles? Dashboards take quite a while to load if you've got a few plots, I think that we should have an offline integration here, right? I don't mean that we need to have an offline-mode for dashboards, but perhaps just a way to make layout updates and see a super simple preview of the expected layout when you do make the call to Plotly to update?
It'd be sweet if you could have a dashboard.manual_sync = True
flag on instantiation that allows you to sync to plotly still, but it would only happen when you call dashboard.sync()
or something. I just imaging having an annoying time making small tweaks to the layout on a dashboard if I have to wait for a dashboard page to load each time.
Second thought, we could also include a ?no-content
flag or something on dashboards so that you can quickly iterate loading dashboard pages in your browser (or in the notebook) without needing to load Plotly
and retrieve all the plots. Not sure how hard that would be, but I really question how on board folks will be if the dev cycles for pixel pushing layout boxes is super long.
We almost got there with plot-schema, perhaps we can just take the initiative and define an official JSON-schema
for these dashboards? http://json-schema.org/ If we did that in streambed, we get all these amazing validation, generation tools for free in any language we choose (yay standardization --> https://pypi.python.org/pypi/jsonschema). This also assists us with versioning down the line since we just need to match versions to schemas and go from there.
^^ Note, this isn't super hard, the schema for the dashboards is fairly simple. I feel pretty strongly about this and I'd be happy to whip one up.
^^ Also note that I'm pretty sure there is about 0
validation on what goes into a dashboard:
POST https://api.plot.ly/v2/dashboards
{
"content": "{\"rows\": [], \"banner\": {}, \"foobar\": \"anything i want...\"}"
}
Just works for example. We can do the same thing on the backend with jsonschema
as we would in the python api library...
@tarzzz , you wrote the api for dashboards, right? Why do we require the double-encoded content
field there? Is there a reason? This is an inconsistency of ours that I don't really understand. You have to do it for /v2/grids/[fid]/cols
as well, it'd be great if we could either accept something more like:
POST https://api.plot.ly/v2/dashboards
{
"content": {"rows": [], "banner": {}}
}
Or, if we accepted both the above and:
POST https://api.plot.ly/v2/dashboards
{
"content": "{\"rows\": [], \"banner\": {}}"
}
@theengineear : in DRF, JSONField accepts a binary
argument. When set to True, it saves (and retrieves) the data as a JSON string. If set to False
, it saves(and retrieves) data as a python primitive (i.e. dict
in this case). It defaults to False, and we cannot have both (obviously).
That said,
binary=False
) or as a coded JSON string(i.e. binary=True
). I am not sure if it matters much how we save it to DB. Let me know your thoughts on same..How should we save the data to DB?
It should get saved to the DB in the same way it's getting saved right now.
We can override default validator to validate for both dicts, and JSON strings. I can add an issue for this if you agree..
👍 Let's definitely open up an issue, I do find it fairly confusing that there are just a couple endpoints that require this different upload format.
Chris put together a walkthrough that has been very useful: https://plot.ly/~chris/17759/
Rather than being arranged in rows, dashboards are arranged in parent-child relationships. This is probably a more explanatory example of what's going on: https://plot.ly/dashboard/charleyferrari:1281/present
Check the titles to see how they're nesting into each other. F
and S
refer to first and second, so the simplified json structure is something like this:
{
'first': {
'first': 'graph F-F',
'second': {
'first': 'graph F-S-F',
'second': {
'first': {
'first': {
'first': 'graph F-S-S-F-F-F',
'second': 'graph F-S-S-F-F-S'
},
'second': 'graph F-S-S-F-S'
},
'second': 'graph F-S-S-S'
}
}
},
'second': 'empty' // there's always a second empty node here
}
Are you guys planning on abstracting away this structure with this API? Could be a good idea, but I think it's complicated...
With drag and drop this structure "just works", but if someone were building a dashboard programmatically from scratch, I don't think they'd be imagining these sorts of nodes. They'd be thinking in rows or columns, perhaps with horizontal or vertical (respectively) splits in individual cells.
I could see two ways of doing this:
1: We keep the structure as is, encourage users to create a template dashboard using our drag and drop interface, and make it easy for them to swap in new graphs. So, make it easy to do things like update_dashboard_row()
(perhaps making this update_dashboard_cell()
), or create an analog to something like py.get_figure()
for dashboards that allows users to easily bring the dashboard in and just swap out the plot URLs.
2: We abstract away this structure in the API, focusing on building something with rows and columns. IMO, if a user is building a dashboard programmatically from scratch, this is the way they're thinking about it.
I think this is a tradeoff unfortunately. If we abstract the API away it makes it easier to work with from scratch, but harder to use an existing dashboard as a template.
I'm thinking the JSON Schema should look something like:
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Plotly Dashboard JSON Content",
"description": "A JSON representation of the content in a Plotly Dashboard",
"type": "object",
"properties": {
"layout": {"$ref": "#/definitions/container"},
"settings": {
"type": "object",
"properties": {
"backgroundColor": {"$ref": "#/definitions/color"},
"boxBackgroundColor": {"$ref": "#/definitions/color"},
"boxBorderColor": {"$ref": "#/definitions/color"},
"boxHeaderBackgroundColor": {"$ref": "#/definitions/color"},
"foregroundColor": {"$ref": "#/definitions/color"},
"headerBackgroundColor": {"$ref": "#/definitions/color"},
"headerForegroundColor": {"$ref": "#/definitions/color"},
"links": {
"type": "array",
"items": {
"type": "object",
"properties": {
"title": {"$ref": "#/definitions/shortText"},
"url": {"$ref": "#/definitions/url"}
},
"additionalProperties": false
}
},
"logoUrl": {"$ref": "#/definitions/url"},
"title": {"$ref": "#/definitions/shortText"}
},
"additionalProperties": false
},
"version": {"type": "integer", "minimum": 0, "maximum": 1000}
},
"additionalProperties": false,
"definitions": {
"color": {
"oneOf": [
{"$ref": "#/definitions/colorHex"}
]
},
"colorHex": {
"type": "string",
"pattern": "^#([0-9a-fA-F]{3}|[0-9a-fA-F]{6})$"
},
"container": {
"type": "object",
"properties": {
"size": {"$ref": "#/definitions/size"},
"sizeUnit": {"$ref": "#/definitions/sizeUnit"},
"direction": {
"type": "string",
"pattern": "^(vertical|horizontal)$"
},
"first": {
"oneOf": [
{"$ref": "#/definitions/container"},
{"$ref": "#/definitions/leaf"}
]
},
"second": {
"oneOf": [
{"$ref": "#/definitions/container"},
{"$ref": "#/definitions/leaf"}
]
},
"type": {
"type": "string",
"pattern": "^split$"
}
},
"additionalProperties": false
},
"leaf": {
"type": "object",
"properties": {
"size": {"$ref": "#/definitions/size"},
"sizeUnit": {"$ref": "#/definitions/sizeUnit"},
"title": {"$ref": "#/definitions/shortText"},
"text": {"$ref": "#/definitions/shortText"},
"fileId": {
"type": "string",
"pattern": "^[.a-zA-Z0-9_-]+:(?:[0-9]|[1-9]\\d+)$"
},
"boxType": {
"type": "string",
"pattern": "^(empty|plot|text)$"
},
"shareKey": {
"oneOf": [
{"type": "null"},
{"$ref": "#/definitions/shortText"}
]
},
"type": {
"type": "string",
"pattern": "^box$"
}
},
"additionalProperties": false
},
"shortText": {
"type": "string",
"maxLength": 200
},
"size": {
"type": "integer",
"minimum": 0
},
"sizeUnit": {
"type": "string",
"pattern": "^(px|%)$"
},
"url": {
"type": "string",
"maxLength": 2000
}
}
}
Needs cleaning up, but the recursive functionality of the container
was the thing I wanted to show. definitions
actually work really well in JSON Schema
:)
Looks like there may be motivation to do JSON schema versioning with 1-0-0
-style formats:
Interesting read on self-describing schemas:
http://snowplowanalytics.com/blog/2014/05/15/introducing-self-describing-jsons/
Not terribly worried about it, but something to consider.
We could try to share definitions amongst all of our api endpoints (and perhaps that's the right way to go), but I'd prefer just creating a GET /v2/dashboards/schema
endpoint to expose the schema we'd be validating with. For a first pass (and maybe all we need to do), is to just go ahead and return exactly what our backend says to the caller if the validation fails on our backend.
@Kully Here's a first pass at a full spec for the dashboard wrapper. What do ya think?
# plotly/dashboard_objs/dashboard_objs.py
class Container(object):
def __init__(self, box0, box1, split_direction):
pass
class Box(object):
pass
class PlotBox(Box):
pass
class EmptyBox(Box):
pass
class TextBox(Box):
pass
class WebpageBox(Box):
pass
class Dashboard(object):
fid = None
def __init__(self, content):
self.content = content
def _get_box(self, box_id):
box_id_map = self._get_box_id_map()
path = box_id_map[box_id]
return {} # Use the path to get a box and return it.
def _get_box_id_map(self):
# This should map machine-readable ids based on splits to an integer
# box number that a user can easily understand.
#
#
# You should traverse the dashboard content using
# `utils.node_generator` or something (as long as it's consistent) and
# just create a little list as you go and return it.
# return
pass
def get_preview(self, preview_type='html'):
# Should be able to copy ViewLayoutNode:ViewLayoutSplit logic?
# Importantly an ID should be shown in each box.
# Say you have the following dashboard:
# {
# "layout": {
# "direction": "vertical",
# "first": {
# "direction": "vertical",
# "first": {"boxType": "text", "text": "Text.",
# "title": "text title", "type": "box"},
# "second": {
# "direction": "horizontal",
# "first": {"boxType": "plot", "fileId": "local:126",
# "shareKey": null, "title": "plot title",
# "type": "box"},
# "second": {"boxType": "plot", "fileId": "local:108",
# "shareKey": null, "title": "plot title",
# "type": "box"},
# "size": 50,
# "sizeUnit": "%",
# "type": "split"
# },
# "size": 150,
# "sizeUnit": "px",
# "type": "split"
# },
# "second": {"boxType": "empty", "type": "box"},
# "size": 1150,
# "sizeUnit": "px",
# "type": "split"
# },
# "settings": { .. },
# "version": 2
# }
#
# For reference, the layout looks like this:
#
# ------------------------------------------------------------------
# | |
# | 0 |
# | |
# |------------------------------------------------------------------|
# | | |
# | | |
# | | |
# | | |
# | 1 | 2 |
# | | |
# | | |
# | | |
# | | |
# ------------------------------------------------------------------
#
# I'd suggest relating the numbers above to an array of paths locating
# the "leaf" boxes:
# [
# ['first', 'first'], # 0
# ['first', 'second', 'first'], # 1
# ['first', 'second', 'second'] # 2
# ]
#
# Note that I'd rather we *hide* the notion of a container box from
# users. I've thought a fair amount on this, and I ultimately think
# that containers are simply an implementation detail. It's like a
# `div`, just organizational, but it's not what we really want to force
# users to interact with.
pass
def insert(self, box_id, box, side):
# This do something like...
original_box = self._get_box(box_id)
if side == 'right':
container = Container(original_box, box, 'vertical')
elif side == 'left':
container = Container(box, original_box, 'vertical')
elif side == 'below':
container = Container(original_box, box, 'horizontal')
elif side == 'above':
container = Container(box, original_box, 'horizontal')
else:
raise Exception('Boom')
self.replace(box_id, container)
pass
def insert_right(self, box_id, box):
self.insert(box_id, box, 'right')
def insert_left(self, box_id, box):
self.insert(box_id, box, 'left')
def insert_above(self, box_id, box):
self.insert(box_id, box, 'above')
def insert_below(self, box_id, box):
self.insert(box_id, box, 'below')
def move_insert_right(self, source_box_id, destination_box_id):
box = self._get_box(source_box_id)
self.remove(source_box_id)
self.insert_right(destination_box_id, box)
def move_insert_left(self, source_box_id, destination_box_id):
box = self._get_box(source_box_id)
self.remove(source_box_id)
self.insert_left(destination_box_id, box)
def move_insert_above(self, source_box_id, destination_box_id):
box = self._get_box(source_box_id)
self.remove(source_box_id)
self.insert_above(destination_box_id, box)
def move_insert_below(self, source_box_id, destination_box_id):
box = self._get_box(source_box_id)
self.remove(source_box_id)
self.insert_below(destination_box_id, box)
def replace(self, box_id, box):
# Simply replace the box at box_id with the new box.
pass
def remove(self, box_id):
# Remove the box at box_id, this will likely need to cleanup an empty
# container box as well.
pass
def rotate(self, rotation_direction, box_id):
# This should change the split in the parent container of box_id.
pass
def rotate_clockwise(self, box_id):
self.rotate('clockwise', box_id)
def rotate_counterclockwise(self, box_id):
self.rotate('counterclockwise', box_id)
def swap(self, box_id):
# switch which box comes first in the pair.
pass
def resize(self, box_id, direction, size, size_unit):
# This should alter props on the parent container.
pass
def create(self):
# Not sure on the call signature for this, it needs privacy/filename
# opts.
pass
def update(self):
# Not sure on the call signature for this, it needs privacy/filename
# opts.
pass
^^ The underlying format for the dashboard is super clean, it's just a little hard to manage. I've attempted to boil things down into a set a base actions that bind closely to the underlying format and then construct some more user-friendly actions insert_move_right
, etc, etc, that are just compositions.
@charleyferrari, I don't want to abstract away the underlying implementation too much. We'll have fewer headaches the thinner we make this. If folks want something schmancier down the line, we can just build a later ontop of this implementation that does more magic. The current format is perfectly reasonable, it's just a little confusing at first pass. I'd rather we just use names like insert
, remove
, swap
, etc, etc, so that folks don't need to grok the long/confusing ['first', 'second', 'first', 'first, ... ]
.
@jackparmer @Kully @charleyferrari , I think a very important part of this to get right is the visual feedback. This will all feel pretty natural (imo) as long as the flow for users is some thing like:
# create a dashboard
# look at the current layout...
# add a box to the dashboard
# look at the layout *now*...
# add another box
# look at the layout *now*...
# move a box
# look at the layout *now*...
As long as the visual feedback is good and fast, I think this will be a pretty fun experience.
Also, let's hold off on offline mode for now, as a first pass let's just force folks to pass in a url or a fid for plots that are to be added.
Also, also, I spaced on the settings
portion, that will be required somewhere as well. I just want to get this out before I head to bed.
@Kully let's iterate on this, I'd love to hear your thoughts. Note that I'm adding that JSON schema so that you won't need to worry about any sort of validation since the backend will take care of that. Dashboards are relatively small objects that can be passed over the network pretty fast, so there's not really a gain to try and validate locally (plus, we should be controlling the actual object and won't get it wrong). Here's the PR for that https://github.com/plotly/streambed/pull/8989
@theengineear Okay, all this sounds really good. Chris wrote a script for using the API to make dashboards so I'll refer to that as an aid as I go through this.
I'm confused about the paths you mentioned for tracking a specific box in a dashboard layout:
['first', 'first'], # 0
['first', 'second', 'first'], # 1
['first', 'second', 'second'] # 2
What is it based on? Was this in place before we opened up this thread?
^^ Those are literally the paths to the boxes in question:
# {
# "layout": {
# "direction": "vertical",
# "first": {
# "direction": "vertical",
# "first": {"boxType": "text", "text": "Text.",
# "title": "text title", "type": "box"},
# "second": {
# "direction": "horizontal",
# "first": {"boxType": "plot", "fileId": "local:126",
# "shareKey": null, "title": "plot title",
# "type": "box"},
# "second": {"boxType": "plot", "fileId": "local:108",
# "shareKey": null, "title": "plot title",
# "type": "box"},
# "size": 50,
# "sizeUnit": "%",
# "type": "split"
# },
# "size": 150,
# "sizeUnit": "px",
# "type": "split"
# },
# "second": {"boxType": "empty", "type": "box"},
# "size": 1150,
# "sizeUnit": "px",
# "type": "split"
# },
# "settings": { .. },
# "version": 2
# }
#
# For reference, the layout looks like this:
#
# ------------------------------------------------------------------
# | |
# | 0 |
# | |
# |------------------------------------------------------------------|
# | | |
# | | |
# | | |
# | | |
# | 1 | 2 |
# | | |
# | | |
# | | |
# | | |
# ------------------------------------------------------------------
#
# I'd suggest relating the numbers above to an array of paths locating
# the "leaf" boxes:
# [
# ['first', 'first'], # 0
# ['first', 'second', 'first'], # 1
# ['first', 'second', 'second'] # 2
# ]
See how the zeroth box is the object that can be located at dashboard['layout']['first']['first']
and the first box can be located at dashboard['layout']['first']['second']['first']
and the second at dashboard['layout']['first']['second']['second']
?
I'm just saying that users shouldn't need to know about the first
/second
/second
/first
path and whatnot, they should just have to interact with a number.
That said, do you think it'd be confusing that the numbers will change? It's by far simpler than trying to assign and track permanent ids, but users will have to get used to the fact that when you split a box, the numbers are going to move around.
@theengineear Every box is then put into 'first' in order to be put in the layout right?
i.e. dashboard['layout']['first']
is just a statement that box appears in the full frame
In theory layout
is a split
, which means it can contain two the objects first
and second
, either of which may be a split
or a box
.
The dashboard schema may be useful here.
In the schema split
=== container
and box
=== leaf
. I.e., a box
cannot have first
and second
, but a split
can. Likewise, a split
only acts as a container for other containers and/or boxes.
TL;DR...
Every box is then put into 'first' in order to be put in the layout right?
Is this to ask if layout['second']
must always be an empty
box
?
i.e.
dashboard['layout']['first']
is just a statement that box appears in the full frame
I don't really understand this comment since there's no way to know if first
is a box
/leaf
or a split
/container
.
@Kully where's this one stand. Have you been able to get going on it?
Just as an additional heads up, the backend dashboards api will explicitly 400
on you if you:
I.e., when you do implement this, if you can get the request to succeed from the plotly.py api lib, you'll know that the resulting dashboard is at least valid.
@Kully where's this one stand. Have you been able to get going on it?
Yeah, just been a little busy with other tasks. I'm still in early stages, but am working on the insert_box functionality right now, the important first step.
I.e., when you do implement this, if you can get the request to succeed from the plotly.py api lib, you'll know that the resulting dashboard is at least valid.
Okay good to know. Where do I find the dashboard JSON Schema?
I'm still in early stages, but am working on the insert_box functionality right now...
Great! It'd be awesome to get PR up sooner-than-later so that we can all agree that we're going in the right direction. Take your time (obviously), but just keep me in the loop 😸!
Where do I find the dashboard JSON Schema?
Dashboard JSON Schema. Note that the schema is recursive, which can make the validation errors a little hard to parse out. Let me know if you hit any errors that seem overly cryptic.
It's by far simpler than trying to assign and track permanent ids
Okay, I should have read this earlier 😛 I was trying to do something like this, but was having trouble.
Do you think that I should worry about making something that can parse a dashboard from online and make into a Dashboard
class instance? Would probably not be trivial.
Do you think that I should worry about making something that can parse a dashboard from online and make into a Dashboard class instance?
Yes, we'll need this. It's also best to think about this now and not try to jam it in later, it may influence how you organize the Dashboard
class.
Yes, we'll need this. It's also best to think about this now and not try to jam it in later, it may influence how you organize the Dashboard class.
Okay cool. Will do!
Starting a discussion for a wrapper proposal for V2 dashboard creating and updating:
https://api.plot.ly/v2/dashboards#create
I'm not aware of any formal dashboard JSON spec documentation, but this dashboard is almost exhaustive in its options usage:
https://plot.ly/dashboard/jackluo:430/view JSON: https://plot.ly/dashboard/jackluo:430/view.json
The only thing that it is missing is a table (an embedded plotly grid), which this dashboard has:
https://plot.ly/dashboard/jackp:16818 https://plot.ly/dashboard/jackp:16818.json
All of the options for styling embedded plotly grids (through URL query parameters) are here: http://help.plot.ly/add-a-table-to-spectacle-editor/
Creating a dashboard
create_dashboard()
This call to
create_dashboard
would create the first 2 rows of this dashboard, plus a 3rd row with a table and a markdown cell:https://plot.ly/dashboard/jackluo:430/view
Updating a dashboard's top-level attributes
update_dashboard()
Rewrite a top-level attribute of the dashboard (ie something in the "settings" key of https://plot.ly/dashboard/jackluo:430/view.json)
Appending or updating a dashboard row
update_dashboard_row()
Update or append one row at a time by passing the row index and
cc @charleyferrari @cldougl @theengineear @Kully @thejackluo @chriddyp
This was the simplest, no-frills API that I could think of, but there are probably better ways. 👍 if you think this looks good or chime in below to suggest some alternate proposals.
📉 📊 📈 🔢 📈 📊 📉 📊 🔢