[SCT] Assess SCT system for new technologies

[filipsch]

@ramnathv commented on Wed Jun 14 2017, and @filipsch updated.

User Story

As a product manager, I want to be able to get a sense of the amount of time and effort required to build SCT systems for new technologies, so that I can prioritize and plan development.

There are several new technologies that we might end up supporting over the next year. A critical element of this is the SCT system. Before starting to invest in a new technology, it would be prudent to get a sense of the time and effort involved in rolling out an SCT system. This would require researching into availability of AST parsers and other available systems.

The new technologies under consideration are:

1. SQL Databases

   • MySQL
   • Microsoft SQL Server

2. NoSQL Databases

   • MongoDB

3. Bash

4. Git

Comprehensive SCT support for any new technology is bound to be time consuming. We should get a time estimate of different 'milestones' in the SCT development process:

• Purely regex-based SCT system.
• Minimum viable AST-based system
• Basic automatic messaging
• Advanced automatic messaging (with 'levels'; check X in Y, there's something wrong with Z)

In addition to this:

• limitations that already emerge
• interdependencies between the above languages

when researching these things can be very helpful.

[ramnathv]

@filipsch The order of priority is:

1. SQL databases (MySQL, SQL Server)
2. Bash/Git
3. NoSQL (MongoDB)

[machow]

From pairing w/ @richierocks , it sounds like there may be several Rcpp courses. If we want to do anything other than regex based tests of the c++ parts of the code, we should add it to the list. cc ( @ncarchedi )

[ramnathv]

@machow I have asked @richierocks to research how we can test C++ code. Let us wait till he can get back to us with some concrete references and ideas. Till then, this priority order stays.

[machow]

spreadsheet with details.

Some notes:

• If done as a python course, everything for mongodb could probably be up in less than a week.
• An intro git course seems very fast to implement
• Anything past introductory bash support seems tricky

[filipsch]

Some clarification to @machow's point on MongoDB, after discussing in person:

• Mongo is most commonly used from inside other languages, such as JavaScript and Python.
• A MongoDB course with a mongodb shell, similar to how we did the intro to SQL course (new interface specifically for viewing tables, etc.) doesn't necessarily make a lot of sense.
• MongoDB could work pretty quickly if we just use Python code to work the database, similar to how the relational databases python course was built, or how the 'importing data from databases chapter' from importing data in R was built: here, you're using Python/R to interface to a database. This would greatly reduce the extra work.

FYI: with other engineers we've been discussing how to easily support a wide variety of technologies, going from git/bash, to databases, scala etc. The current thinking is that if we set up a simple python package that handles the commands, and can pipe it into language-specific processes, listening to the output, it could be superfast to support new technologies.

[machow]

closing, but feel free to reopen if there is more to dicuss