Charts plugins development integration in Superset

[trepmag]

Hello,

I successfully implemented a custom chart which work inside a storybook following the repo here: https://github.com/apache-superset/superset-ui-plugins-template.

Now, I'm trying to integrate that custom chart into a superset instance.

Update; I found some indication at https://github.com/apache-superset/superset-ui/blob/master/docs/debugging.md which mention npm link to hook plugins into a Superset instance.

So I did this as follow:

1. Start all services but superset-node from incubator-superset/docker/
2. In incubator-superset/superset-frontend/ trigger npm run dev-server
3. In superset-plugins-template/package/superset-ui-plugin-chart-dummy hit npm link
4. In incubator-superset/superset-frontend/ hit npm link superset-ui-plugin-chart-dummy
5. In incubator-superser/superset-frontend/src/visualizations/presets/MainPreset.js add below the others import DummyChartPlugin from 'superset-ui-plugin-chart-dummy/src';

Now the last step make the dev server return an error:

ERROR in ./node_modules/superset-ui-plugin-chart-dummy/src/DummyChart.tsx 42:16
Module parse failed: Unexpected token (42:16)
You may need an appropriate loader to handle this file type.
|     DummyChart.prototype.render = function () {
|         var _a = this.props, data = _a.data, height = _a.height, width = _a.width;
>         return (<svg style={{ backgroundColor: '#ffe459', borderRadius: 8, height: height, width: width }}>
|         {data.map(function (_a) {
|             var x = _a.x, y = _a.y;
 @ ./node_modules/superset-ui-plugin-chart-dummy/src/index.ts 47:88-111
 @ ./src/visualizations/presets/MainPreset.js
 @ ./src/setup/setupPlugins.ts
 @ ./src/addSlice/App.jsx
 @ ./src/addSlice/index.jsx
 @ multi (webpack)-dev-server/client?http://localhost:9000 (webpack)/hot/dev-server.js babel-polyfill ./src/preamble.js ./src/addSlice/index.jsx

[trepmag]

It appears that incubator-superset/superset-frontent/webpack.config.js needs to be adapted to match what superset-ui-plugins-template do in its own webpack.config.js; added the following rule:

--- a/superset-frontend/webpack.config.js
+++ b/superset-frontend/webpack.config.js
@@ -211,6 +211,27 @@ const config = {
         ],
       },
       {
+        test: /\.(jsx|tsx)?$/,
+        include: /node_modules\/superset\-ui\-plugin\-chart\-dummy\/src/,
+        use: [
+          {
+            loader: 'babel-loader',
+            options: {
+              presets: [
+                ['@babel/preset-env', { useBuiltIns: 'entry', corejs: 3 }],
+                '@babel/preset-react',
+                '@babel/preset-typescript',
+              ],
+              plugins: [
+                '@babel/plugin-proposal-object-rest-spread',
+                '@babel/plugin-proposal-class-properties',
+                '@babel/plugin-syntax-dynamic-import',
+              ]
+            },
+          },
+        ],
+      },
+      {
         test: /\.css$/,
         include: [APP_DIR, /superset[-]ui.+\/src/],
         use: [

[japorized]

Did you get a message about missing @babel/preset-typescript when you're trying to run npm run dev-server in Superset itself?

[trepmag]

Did you get a message about missing @babel/preset-typescript when you're trying to run npm run dev-server in Superset itself?

I dont recall that but you might need to trigger npm run build before...

[spatialbits]

Did you get a message about missing @babel/preset-typescript when you're trying to run npm run dev-server in Superset itself?

I got the same. It also complained about another package: plugin-proposal-export-default-from. I did:

npm install @babel/plugin-proposal-export-default-from --save-dev 
npm install @babel/preset-typescript --save-dev

I also did an npm run build for good measure and compiled without issue.

@trepmag your posts were really helpful.

I've been poking around the superset repos looking to see what needs to be done to make a functional visualization plugin.

Aside from adding to MainPreset.js were there other changes required to get it to appear as an option in the ui chart list?

Looks to me like setupPluginsExtra.ts is where they intend viz plugins to be registered, as opposed to adding to MainPreset.js, but the whole registration/preset flow isn't very clear to me yet.

[japorized]

Things did not really work out that well for me, partly cause I was importing .less files, of which I edited the webpack configs for Storybook. But I resolved my issue now.

I decided to do what the official plugins do, which is to let the main entry be lib/index.js instead of src. This is already specified in package.json. Then in Step 5 above, instead of ending the import with /src, we can have import DummyChartPlugin from 'superset-ui-plugin-chart-dummy'. Lettingsuperset-frontendimport your plugin fromlibshould also remove the need for additional dependencies (like@babel/preset-typescript`, unless if your plugin somehow requires that (despite the build process).

Though to get this to work, you must first build your plugins by running npm run build at the root of the repo, so that it builds your src files into lib (and esm). You might run into the following error:

... Duplicate identifier 'LibraryManagedAttributes'.

This will appear if you have already did npm link in your plugin, which will then create a node_modules file with the dependencies that your plugin needs.

Aside from adding to MainPreset.js were there other changes required to get it to appear as an option in the ui chart list?

Looks to me like setupPluginsExtra.ts is where they intend viz plugins to be registered, as opposed to adding to MainPreset.js, but the whole registration/preset flow isn't very clear to me yet.

I agree. It's not the best, not especially if you want to just package things up in docker. I'm keeping an eye out on item 4 in SIP-38.

But for now, editing the MainPreset.js file should immediately do what you want (getting the viz to show up in the ui chart list).

[spatialbits]

Thanks @japorized that's great, I've got it compiling now.

I agree. It's not the best, not especially if you want to just package things up in docker. I'm keeping an eye out on item 4 in SIP-38.

Same here.

While tinkering around with the plugin I'm running yarn babel --watch, and over in the superset-frontend directory I'm running the dev server. So edits made in the plugin folder get compiled and then trigger a hot reload over in the superset frontend. There's probably a more efficient way to do it but it works and removes a manual build step.

[spatialbits]

Little follow up: I ended up reverting my changes to MainPreset.js as I was able to get the dummy chart plugin appearing in the chart list via the setupPluginsExtra.js hook:

import DummyChartPlugin from 'superset-ui-plugin-chart-dummy'

// For individual deployments to add custom overrides
export default function setupPluginsExtra() {
    new DummyChartPlugin()
        .configure({ key: 'dummychart' })
        .register();
}

[japorized]

Interesting @spatialbits. Thanks for the follow up. Will give that a try on my end as well.

Just a side question, though still on the topic; how does one get the data from the Flask backend into our visualizations? I've poked around with the viz.py file (and I think one other file that I can't remember) but Flask kept telling me that I've made only an empty query.

[spatialbits]

I was getting that too. I ended up adding some controls to the control panel in the DummyChart plugin:

export default class DummyChartPlugin extends ChartPlugin {
  constructor() {
    // What is loaded here is an object that implements the ChartPluginConfig interface 
    super({
      // loadChart: () => import('./DummyChart'), // lazy load it.
      Chart: DummyChart, // or load it directly.
      controlPanel: {
        controlPanelSections: [
          {
            label: t('Query'),
            expanded: true,
            controlSetRows: [
              ['all_columns_x', 'all_columns_y'],
              ['adhoc_filters'],
              ['row_limit', null]
            ],
          },
        ]
      },
      metadata,
      transformProps,
    });
  }
}

Then the all_columns_x and all_columns_y selections get passed into the viz.py backend as form_data kwargs.

Now I'm still not a 100% clear on how query the syntax goes, I think you have to hunt around the other viz.py classes, but in your viz class, you need to override the query_obj metho, and define the query you want there. Mine looks like:

   def query_obj(self):
        form_data = self.form_data
        d = super().query_obj()

        d["is_timeseries"] = self.form_data.get("time_series_option", "not_time")
 for x, y and size"))

        d["columns"] = [form_data.get("all_columns_x"), form_data.get("all_columns_y")]
        return d

Which seems to tell the query builder to select those columns from the db.

You also need to override the get_data method and return something that your chart will understand on the frontend:

def get_data(self, df: pd.DataFrame) -> VizData:
    data = # do something to your dataframe to get the data you want, make it json serializable
    return {'somedata': data}

Then lastly, and back in your plugin, you see the transformProps property on the DummyChartPlugin declaration. That function receives the data returned by api (what you produced in get_data) and puts that into the chart. It's defined in transformProps.ts:

export default function transformProps(chartProps: ChartProps) {
  console.log('Transforming props for Dummy Chart', chartProps);
  const { width, height, formData, queryData } = chartProps;
  const { color } = formData;
  const { data } = queryData;

  return {
    width,
    height,
    color,
    data,
  };
}

Then lastly, in DummyChart.tsx, you modify the DummyChartProps to accept the modified incoming viz data object and use those props in the chart itself:

export type DummyChartProps = {
  height: number;
  width: number;
  data: { somedata: number };
};

export default class DummyChart extends React.PureComponent<DummyChartProps> {
  render() {
    const { data, height, width } = this.props;
    return (
      <div style={{height, width}}>
        <span>{data.somedata}</span>
      </div>
    );
}

There's a lot I still have to work out. Please feel free to point out anything you see wrong there.

[ktmud]

Hi, I took a dab and made importing ts files from plugins work more smoothly: https://github.com/apache/incubator-superset/pull/9326

There's still some issue related to React hot reloading that I wasn't able to resolve. The PR has more details. If someone could help me debug or have ideas for directions, that would be great!

[Adnerw]

Hi All, I'm trying to integrate a plugin within Superset but I'm not succeeding. I followed the steps listed in the readme to add a new plugin created by me.

1. created a plugin
2. run yarn install && build (in the my plugin)
3. enabled the link in the plugin and in the local package to superset-frontend(using a name of the package plugin)
4. insert in the setupPluginsExtra.js file the my plugin new MapChartPlugin().configure({ key: 'map' }).register();
5. run the npm run dev-server command
6. went to the Create a new superset chart page and chose my plugin.

Unfortunately in the explore page i don't see my plugin. After this I have followed your updates posted in this page: https://github.com/apache/incubator-superset/pull/9326 but nothing.

I always have the same error, in the exploration shows me the name of the key with which I registered the plugin and in the console of the browser in the url

localhost:8088/superset/explore_json

returns 500 INTERNAL SERVER ERROR

This is my view after the created of the new chart:

Could someone help me?

Thank you

[japorized]

That really is where the readme ends and we're sort of left with digging around. I apologize that I don't have a clean answer for you, but try looking at the viz.py file in incubator-superset/superset/. You'll see how the other visualizations are hooked up to the Python-side of Superset from there. We can't really help you out too much here cause it depends on the data that your viz needs.

You may run into an empty_query? problem. You can check out @spatialbits' attempt here. Unfortunately, I haven't had a chance to try this myself, so I'm not sure if it will completely resolve the empty_query? problem.

[japorized]

Yeah, I get how you feel cause I was in the same boat. I think the discussion right now on the main Superset repo is that the project is trying to figure out how to easily let 3rd party visualizations be added to Superset without them having to modify Superset in any way, so a plugin system. However, this is not so much of a priority for the project right now, so things on this front are moving at a slower pace.

I apologize that I can’t be of much help but good luck!