MarkLogic Analytics Dashboard Demo

This AngularJS library adds a demo-quality, BI-style dashboard for MarkLogic to Angular 1 applications. It is specifically designed for applications generated by the MarkLogic SlushJS generator. It should be usuable in other Angular 1 applications, but that is not well-tested.

This MarkLogic Analytics Dashboard Demo is intended to demonstrate MarkLogic's ability to perform flexible, efficient aggregate queries. The library is not intended to be a fully-featured BI tool, nor does it cover the entire surface area of MarkLogic's query and analytics API. You can find a more complete introduction to in-database analytics in MarkLogic on the company website. Finally, this is a community-supported library, not officially supported by MarkLogic.

This library is a companion to the analytics-dashboard mlpm module library for adding a BI-style dashboard to Angular applications. This library provides the front end code driving the dashboard, while the mlpm module provides the APIs (REST extensions and modules).

Features

The main features of this MarkLogic Analytics Dashboard Demo include:

A user-friendly GUI for creating pie and bar charts as well as table results for grouping and calculating aggregates.

The user can select existing range indexes as the basis for group-by dimensions or measures for aggregate calculations. The user can also create MarkLogic queries to use as filters (for example, asking for analytics only on orders with a total value over 100). The user can also provide aliases for those dimensions and measures.
The ability to embed those charts on a search page, so that the analytic results are automatically constrained by the user-entered search, including query text and selected facets. This allows a search user to see analytic results and search results side-by-side.
The ability to export chart configurations (themselves stored as documents in MarkLogic) for portability.
The ability to scope analytics queries to a particular collection.

Significant Limitations

The current major limitations of this library include:

No support for joins across documents. Your documents should be in a harmonized state - not shredded across many documents as they would be if imported directly from a relational databased and not harmonized.

For example: an Order should look more like:

{
 'order': {
   'id': 54321,
   'total': 100,
   'customer': {
     name: 'Richie Rich',
     customerId: 12345
   },
   'orderDetails': [
     {
       productName: 'Line of Code',
       productId: 10101,
       unitPrice: 2,
       quantity: 50
     }
   ]
   ...
 }
}

rather than:

{
 'order': {
   id: 54321,
   customerId: 12345
   ...
 }
}

{
 'orderDetail': {
   id: 0001,
   orderId: 54321,
   productId: 10101,
   unitPrice: 2,
   quantity: 50
   ...
 }
}

{
 'customer': {
   id: 12345,
   name: 'Richie Rich'
   ...
 }
}

Only bar chart, pie charts, and tables are currently supported, though this uses Highcharts, which support many others, so this could be improved.
Does not support grouping by dates - though this should be a fairly easy addition when the demand arises.
Lacks the ability to combine several MarkLogic collections as the scope of a query.
Only supports a limited subset of MarkLogic's rich query API when creating document-level filters on analytics reports.
No support for creating range indexes from this tool. You must have already set up range indexes on your data. There are several simple ways to do that, including using the MarkLogic Admin UI.

Getting Started

bower install ml-analytics-dashboard-ng --save

And install the mlpm module:

npm install mlpm --save-dev
mlpm init
mlpm install analytics-dashboard --save

For now, you need to overwrite two modules from the group-by mlpm library (which is a dependency of the above analytics-dashboard library) to allow constraining queries to work. To overwrite, run:

cp {PATH-TO-bower_components}/ml-analytics-dashboard-ng/group-by-json.xqy mlpm_modules/group-by/
cp {PATH-TO-bower_components}/ml-analytics-dashboard-ng/group-by.xqy mlpm_modules/group-by/
# If you are using slush-marklogic-node, that will be:
# cp bower_components/ml-analytics-dashboard-ng/group-by-json.xqy mlpm_modules/group-by/
# cp bower_components/ml-analytics-dashboard-ng/group-by.xqy mlpm_modules/group-by/
# If using slush-marklogic-spring-boot:
# cp src/main/resources/static/bower_components/ml-analytics-dashboard-ng/group-by-json.xqy mlpm_modules/group-by/
# cp src/main/resources/static/bower_components/ml-analytics-dashboard-ng/group-by.xqy mlpm_modules/group-by/

Now you can deploy the mlpm_modules to MarkLogic:

mlpm deploy -H localhost -P 8040 -u admin -p admin

Directives Provided

mlAnalyticsDashboard
- mlAnalyticsDashboardHome
- manageMlAnalyticsDashboard
  - mlAnalyticsNewReport
  - mlAnalyticsDesign
  - mlAnalyticsReportEditor <- TODO, roll this into mlAnalyticsDesign
- mlAnalyticsChart: used to construct and display highcharts
- mlAnalyticsEmbed: used to embed a chart outside the report editor

Example Implementation

Add the mlAnalyticsDashboard module as a dependency to your app.module. For example, in a slush-marklogic-node-generated app, add this to ui/app/app.js (that will be src/main/webapp/app/app.js in a slush-marklogic-spring-boot-generated app):

angular.module('app', [
   # ...
  'ml.analyticsDashboard',
  # ...
]);

Insert the report editor

You can embed the ml-analytics-dashboard report editor in your UI using the mlAnalyticsDashboard directive. Simply add this to your HTML markup:

  <ml-analytics-dashboard></ml-analytics-dashboard>

At the moment, this directive brings up a configuration screen, and it probably makes sense to have on its own page. If you are using angular-ui-router, as in a slush-marklogic-node-generated app, you can add a state definition in ui/app/route/routes.js (src/main/webapp/app/route/routes.js in a [slush-marklogic-spring-boot-generated app](slush-marklogic-spring-boot-generated app)) with something like this:

  .state('root.ml-analytics-dashboard', {
    url: '/ml-analytics-dashboard',
    template: '<ml-analytics-dashboard></ml-analytics-dashboard>'
  })

Then, you will probably also want to create a link to this page. For example, in a slush-generated app, you can add the following line to the navbar of ui/app/root/root.html.

  <li><a href="https://github.com/patrickmcelwee/ml-analytics-dashboard-ng/blob/master/ml-analytics-dashboard">Analytics Dashboard</a></li>

Embed a chart elsewhere on your site

You can embed a chart outside the editor using the mlAnalyticsEmbed directive:

<ml-analytics-embed report-uri="'/ml-analytics-dashboard-reports/example-12345.json'" chart-id="1"></ml-analytics-embed>

Even better, in slush-marklogic-node-generated application, you can embed a chart into a scope with an mlSearchContext, and it will respond to your searches. Simply add the search constraint to your directive call's ml-search attribute:

<ml-analytics-embed report-uri="'/ml-analytics-dashboard-reports/example-12345.json'" chart-id="1" ml-search="ctrl.mlSearch"></ml-analytics-embed>

Finally, you can further configure the chart with a dynamic configuration object. This can allow you, for example, to add additional constraining queries on the fly, or to change the title or chartType of the chart. For example, you may want to create charts by year based on the top 5 values of a particular facet.

The currently supported properties of this object are as follows:

scope.dynamicChartConfig = {
  queryConfig: {
    additionalQueries: []
  },
  chartConfig: {
    title: 'A New Title, perhaps based on a current scope property.',
    chartType: 'pie' // 'column' and 'table' also currently supported
  }
}

You can you use that dynamicChartConfig in the directive (for now, it is only guaranteed to work when you also specify an ml-search context, but if you have a use case for a static chart with dynamicConfig, file a feature request):

<ml-analytics-embed dynamic-config="dynamicChartConfig" report-uri="'/ml-analytics-dashboard-reports/example-12345.json'" chart-id="1" ml-search="ctrl.mlSearch"></ml-analytics-embed>

Troubleshooting

Using path range indexes as a filter

At the moment, you need to declare any namespaces used in a path range index at the app server level. In deploy/ml-config.xml, find the <http-server> element with '${app-name}' and add the following child element:

<namespaces>
  <namespace>
    <prefix>nfe</prefix>
    <namespace-uri>http://www.portalfiscal.inf.br/nfe</namespace-uri>
  </namespace>
</namespaces>

Otherwise, you will get an error like:

Prefix xxx has no namespace binding

Contributing

Contributions welcome! Please see CONTRIBUTING.md.

End-to-end tests

There are some mostly out-of-date tests in the test/ directory, but the best current test suite are the end-to-end Protractor tests in the e2e_spec/ directory. To run these, you need to install Protractor and run a Selenium Server (see full instructions here). Once installed, you can typically start it by running this command:

webdriver-manager start

You will also have to run a MarkLogic server, with data installed using the developing-analytics-dashboard project.

Finally, you will need to start a Node server from the developing-analytics-dashboard directory on port 3000, using the command gulp serve-local.

Once Selenium, MarkLogic, and Node are running, the end-to-end tests will run in a Chrome browser on this command:

protractor e2e_spec/conf.js

patrickmcelwee / ml-analytics-dashboard-ng

readme