Demo Day

[DominicParent]

@ryanmwhitephd, @russellgill, I created this issue to generally discuss/brainstorm what we wanted to do for DemoDay. I created the milestone "DemoDay" to track any issue needing to be resolved for DemoDay.

[ryanmwhitephd]

The effort will focus on the development of the Getting Started guide which includes

• Step-by-step walk through of a complete definition, execution and analysis/interpretation of the outputs data products of an Artemis job.
• Development of a respective table schema that is somewhat representative of scanner/pricing data.
• Development of the example job in code, which also appears in the Getting Started guide.
• Linking of sections in the Getting Started guide to more detailed documentation. For example, the creation of a Business Process Model links to complete documentation on the Artemis Menu in the metadata section of the documentation.
• During the development of the guide and example code, identify areas of improvement, new feature development required to make a useful example, or bugs. Open issues, label ones that are blockers to be sure these are implemented along the way. For example, #15 is required if we want to include an image of the example Business Process Model in the guide.
• We should focus on using as much that exists already and build off of that rather than trying to develop lots of new features. For example, we have limited analytical tools to chain in a menu but they are suitable for demonstration. We are limited to building menus with:
  1. Ingestion of raw data in csv
  2. Chunking data
  3. Converting data to Arrow format
  4. Profiling numerical columns with Tdigests
  5. Filtering columns to produce new Arrow tables
  6. Publishing multiple Arrow tables from filtered data

Getting Started Guide

• Currently in a development branch. Getting Started

Example job

• The current example is examples/distributed_example_2. Example Job.
• An updated example is available which includes the visualization in examples/distributed_dq_example.py. Example Job with visualization

Current Table Schema for generating Csv-like dummy data.

• This is driven by our Excel format layout for defining table schemas found in examples/data/example_product.xls. Example Table Schema

  I'll follow-up with some tasks that I think should be done.

[ryanmwhitephd]

In regards to getting started,

Merge current documentation into master, start a new branch with developments to the guide, example code, and example table schema. Most of the work is postprocessing and examining the outputs.

Outline of Getting Started

• Installation
• What is the definition of an Artemis Job
• Define a Table schema. Refer to Data Synthesis in docs for more information on generating data.
• Define a business process model. Build the execution graph and visualize the model.
• Define the job configuration
• Define/explain algorithms and tools. Refer to Algorithms and Tools section in docs.
• Highlight the use of Properties to retain algorithm, tools, and job configuration in metadata. Refer to the metadata documentation.
• Setup an object store. Use an local path so that outputs can be access after.
• Register the metadata. Viewing the registered content. Viewing the schema (fields) of the input table.
• Process data. Define a large data set, large number of columns and rows (records). Run distributed processing. What is an impressive size of the data to process? Can we show the size of the input dataset and output dataset? Can we process several GB of data in minutes or seconds?
• Checking the output. Demonstrate the Cronus Data management system.
  • What is the input dataset? List the partitions and files.
  • What is the associated input dataset partition and table schema? List the fields.
  • What is the output dataset? List the partitions and files? Show the output table schema.
  • What additional outputs exist. Show the job ids, the log files, the histograms and tdigests.
• Merge the histograms and tdigests in the postprocessing.
• Visualize the histograms and tdigests. Show several tdigests in a web page corresponding to the columns profiled.
• Show other histograms, like total data process, number of files, etc... Do these make sense.

[ryanmwhitephd]

In regards to the table schema @DominicParent we have some previous work that may be a starting point for fake pricing data.

• Include the new distribution providers to generate some price distributions in a fake pricing data set.
• Define in python and in xls, should be equivalent when stored in the protobuf.
• Create a new example starting from distributed_dq_example.

[ryanmwhitephd]

The documentation up to now is now in master.

[DominicParent]

There's a basic version of the new example created in examples/demoday. It is based off distributed_dq_example. The schema files are located in examples/demoday/data. There's also a python file (very rough draft, copied from a test file in tests) that needs to be completed/refined. This work was done through issues #21 and #28.

I talked with @russellgill about getting some of the new distributions in the fake dataset for demoday (I believe issue #24?).