Update the structure of the study object to include sub-studies

[miha42-github]

Introduction

To ensure a more straight forward approach to studies we've elected to make a change that would push iterations, questions, themes, etc. into sub-studies. This way it becomes simpler to have a owning parent study object with several sub-studies below the parent. This issue is meant to explore the design change through documentation prior to implementation and include the required tasks to achieve the implementation. @TeraBlitz will make use of these design guides and the reference materials to enable the generation of the appropriate APIs in the backend.

Rationale

The previous design of the API would not have resulted in a simple client. Essentially, the notion of the iteration would have required specific logic for questions, themes, etc. to include the iteration idea which would have resulted in further complication. With the inclusion of the sub-study client parsing will be simplified and more standard across various iterations of the study. Additionally, at this juncture we may elect to make the naming convention a little simpler, but that remains to be seen as we don't want to disturb any work too much for the real backend.

Tasks

• [x] Design the new structure in this issue
• [x] Update the ETL portion of the module to reflect all removals
• [x] Update the ETL portion of the module to reflect the changes/additions
• [x] Update json_server API to account for the changes
• [x] Update the high_level API to account for the changes
• [x] Update CLI tools to account for the changes
• [x] Create an issue in mr_json_server to account for these changes
• [x] Create an issue in mr_caffeine to account for these changes

Current study object structure

For the purposes of providing a starting point the current structure of the study is provided as a reference. This should enable a compare and contrast of the to be design.

{'GUID': '1e14f4ef2520231fc7c421da50b1a1e583c37642517b263587cdce0e5a4d9eca',
  'description': 'Capture what customers want in a product that combines '
                 'storage a compute in a single platform.',
  'document': {'Action': {'1': 'Augment your study with relevant information, '
                               'add interactions and companies, and more '
                               'generally have fun.|Status: In progress',
                          '2': 'Activate your mediumroast.io organization and '
                               'create your first study.|Status: Done',
                          'text': 'To improve completeness of this study the '
                                  'following actions and next steps are '
                                  'documented with appropriate statuses.'},
               'Introduction': 'The mediumroast.io will automatically generate '
                               'key themes, text summaries, and detect '
                               'relevant snippets from the interactions '
                               'associated to a study.\n'
                               'It is then up to you to read, digest, and find '
                               'meaningful points from these interaction data '
                               'and document them here.  In that spirit this '
                               'paragraph should be\n'
                               'rewritten based upon your research into these '
                               'data.',
               'Opportunity': {'1': 'Transparency - Product or service plans '
                                    'that are completely transparent, to all '
                                    'stakeholders and constitutents, are easy '
                                    'to follow and clear in\n'
                                    'direction.  Further inherent transparency '
                                    'enables all involved parties to build '
                                    'community and rally behind your target '
                                    'audience.  The mediumroast.io enables\n'
                                    'the construction of transparent plans.',
                               '2': 'Evidence Based - Too often product, '
                                    'engineering and marketing teams believe '
                                    'in their own vision instead of the vision '
                                    'of their users.  This\n'
                                    'misalignment between internal and '
                                    'external visions is evident when plans, '
                                    'for products and services, lack '
                                    'attributable evidence.  That is because '
                                    'it is\n'
                                    'impossible to show where key elements, '
                                    'leading to the vision, originated from.  '
                                    'Easily constructing evidence based plans '
                                    'is a key value of the mediumroast.io.',
                               '3': 'Community - When different groups have '
                                    'differing opinions on what the future '
                                    'holds for the product or service, the '
                                    'potential for community is\n'
                                    'dashed.  Using both Transparency and '
                                    'Evidence Based plans the mediumroast.io '
                                    'can be the kernel needed to build up a '
                                    'community that is passionate about\n'
                                    'user and market needs instead of internal '
                                    'opinion.',
                               'text': 'Opportunies document the real outcomes '
                                       'from your research into the '
                                       'interactions associated to the study.  '
                                       'It is recommended that you write a '
                                       'quick\n'
                                       'summary related to core things you '
                                       'have found.'}},
  'groups': 'users:studyadmin',
  'id': '1e14f4ef2520231fc7c421da50b1a1e583c37642517b263587cdce0e5a4d9eca',
  'iterations': {'default': {'interactions': {'20160120-HyperConverged-EnlightenTech': {'guid': 'c170dcc2558d731b365b0139a2adb01ae7f6a1f70cc1dc5fcddb56eab222f446',
                                                                                        'state': 'unsummarized'},
                                              '20160215-HyperConverged-SyncTech': {'guid': '89a506bf1728b65513465e95d8756b25a5c84c8dea67c26176d432c3a54b6068',
                                                                                   'state': 'unsummarized'},
                                              '20160219-HyperConverged-ModerationIT': {'guid': '1b111825790e02fbffef4bd83f2c29f57d5699d29787fcaa52346d7ec975d96a',
                                                                                       'state': 'unsummarized'},
                                              '20160624-HyperConverged-PotIT': {'guid': '74f0accf093ac8ebce0cddb6eb983340f82a266fde5eb8ffca9a52425954c3b1',
                                                                                'state': 'unsummarized'},
                                              '20160629-HyperConverged-UK Fancy': {'guid': 'e66df0a89212f89d3c99b20dfa727b05bc9852f0f7293c531c179bfadb4f67c4',
                                                                                   'state': 'unsummarized'},
                                              '20160630-HyperConverged-CreditCo': {'guid': '4d0022b00eac594938d467dda83928822c3e74f3c8d97003c6a0c5b31171d15d',
                                                                                   'state': 'unsummarized'}},
                             'state': 'unthemed_unsummarized',
                             'totalInteractions': 6},
                 'state': 'unthemed_unsummarized',
                 'totalInteractions': 6,
                 'totalIterations': 1},
  'keyQuestions': {'1': {'included': True,
                         'notes': 'A default and universal question that helps '
                                  'in persona development.',
                         'question': 'What is your role in the company?'},
                   '2': {'included': True,
                         'notes': 'Helps in classifying companies interacted '
                                  'with, and is a part of the default '
                                  'questions.',
                         'question': 'Can you provide us a brief description '
                                     'of your company?'}},
  'keyThemeFrequencies': {'1': {'MegaTech': '4'}, '2': {'MegaTech': '4'}},
  'keyThemeQuotes': {'1': {'Mr. Roast, mediumroast.io': 'Here is a quote that '
                                                        'is from the ghost in '
                                                        'the machine.'},
                     '2': {'The Major, MegaTech': 'I can feel this theme in my '
                                                  'ghost.'}},
  'keyThemes': {'1': {'description': 'This is the first example theme assigned '
                                     'to show how themes work',
                      'frequency': '6',
                      'name': 'Theme example 1'},
                '2': {'description': 'A second theme example defined to drive '
                                     'the user experience',
                      'frequency': '8',
                      'name': 'Theme example 2'}},
  'linkedCompanies': {'CreditCo': '8d38b195542a9942fb67a6e167c4163847a2a3685ca857d7aadc6063c35efaa4',
                      'EnlightenTech': '85c78f51bbbbd27746b206f84c53667b745173f6c3c496bbfbe7466700db7502',
                      'ModerationIT': '2d625f1ae96b676097bb1d9ab5b4ca3ce2494a5cfcf0c3d0f2edbaee7304f9da',
                      'PotIT': '78aead5854e4ac5b31411e019392a5558007d313c0393dd5d0146e3872768df8',
                      'SyncTech': '9519f1762be3148d0bd835e2934b98cb02501eeb79fe56ade3302b8d2ea692a3',
                      'UK Fancy': 'c85512474ed7e53a49cb139758b2ec840ed7903b3d0ef67f635247b44979dee4'},
  'linkedInteractions': {'20160120-HyperConverged-EnlightenTech': 'c170dcc2558d731b365b0139a2adb01ae7f6a1f70cc1dc5fcddb56eab222f446',
                         '20160215-HyperConverged-SyncTech': '89a506bf1728b65513465e95d8756b25a5c84c8dea67c26176d432c3a54b6068',
                         '20160219-HyperConverged-ModerationIT': '1b111825790e02fbffef4bd83f2c29f57d5699d29787fcaa52346d7ec975d96a',
                         '20160624-HyperConverged-PotIT': '74f0accf093ac8ebce0cddb6eb983340f82a266fde5eb8ffca9a52425954c3b1',
                         '20160629-HyperConverged-UK Fancy': 'e66df0a89212f89d3c99b20dfa727b05bc9852f0f7293c531c179bfadb4f67c4',
                         '20160630-HyperConverged-CreditCo': '4d0022b00eac594938d467dda83928822c3e74f3c8d97003c6a0c5b31171d15d'},
  'public': False,
  'questions': {'default': {'questions': {'1': {'included': True,
                                                'notes': 'A default and '
                                                         'universal question '
                                                         'that helps in '
                                                         'persona development.',
                                                'question': 'What is your role '
                                                            'in the company?'},
                                          '2': {'included': True,
                                                'notes': 'Helps in classifying '
                                                         'companies interacted '
                                                         'with, and is a part '
                                                         'of the default '
                                                         'questions.',
                                                'question': 'Can you provide '
                                                            'us a brief '
                                                            'description of '
                                                            'your company?'}},
                            'totalQuestions': 2},
                'totalIterations': 1,
                'totalQuestions': 2},
  'studyName': 'HyperConverged',
  'totalCompanies': 6,
  'totalInteractions': 6,
  'totalKeyQuestions': 2,
  'totalKeyThemes': 2}

Updated structure with emphasis on change

Removed attributes

• questions
• keyQuestions
• iterations
• linkedInteractions
• keyThemes
• keyThemeQuotes
• keyThemeFrequencies

Many of these items will be made a part of a sub-study directly or indirectly.

Implementation notes

1. All of the key themes keys in the JSON structure are related. That is to say key theme 1 is related to the number 1 key theme frequency, number 1 key theme quotes, etc.
2. Noise text are nothing more than sentence/text fragments either entered by the system or user which are to be eliminated from all documents in the substudy for various processing like interaction summarization and theming.
3. Every study will start with the default substudy, but not all studies will have more than one substudy.
4. Users can determine the name of the substudy, but not the numerical ID or GUID.

Modified structure and note for proper JSON formatting all single quotes will need to be double quotes.

{
  'GUID': '<GUID>',
  'public': <True_or_False>,
  'studyName': '<name_of_the_study>',
  'description': '<a_description>',
  'document': {'Action': {
                          '1': '<Action_1>: <status>',
                          '2': '<Action_2>: <status>',
                          'text': '<Text introducing the actions>.'},
               'Introduction': '<Findings_Intro>',
               'Opportunity': {
                               '1': '<Opportunity_1>.',
                               '2': '<Opportunity_2>',
                               ...
                               'N': '<Opportunity_N>.',
                               'text': '<Text introducing/summarizing the opportunities>'}},
  'groups': '<group_1>:<group_2>: ... <group_N>',
  'id': '<copy_of_GUID_only_needed_for_json_server>',
  'linkedCompanies': {
                      '<company_name_1>': '<GUID_for_company_name_1>',
                      '<company_name_2>': '<GUID_for_company_name_2>',
                      ...
                      '<company_name_N'>: '<GUID_for_company_name_N>'},
  'totalCompanies': <sum_of_companies>,
  'totalInteractions': <sum_of_all_interactions>,
  'totalKeyQuestions': <sum_of_all_questions>,
  'totalKeyThemes': <sum_of_all_key_themes>,
  'substudies': {
                     'default': {
                         'name': 'Default Substudy',
                         'GUID': '<GUID_of_substudy>,
                         'description': 'The default substudy for the study, and is present for all studies in the mediumroast.',
                         'questions': {
                             '1': {
                                 'included': <True_or_False>,
                                 'notes': '<Notes_free_text>',
                                 'question': '<Question_text>'
                              },
                             '2': {
                                 'included': <True_or_False>,
                                 'notes': '<Notes_free_text>',
                                 'question': '<Question_text>'
                              },
                              ...
                             'N': {
                                 'included': <True_or_False>,
                                 'notes': '<Notes_free_text>',
                                 'question': '<Question_text>'
                              },
                         },
                         'interactions': {
                             '<interaction_name_1>': {
                                 'guid': '<interaction_GUID>',
                                 'state': '<interaction_state_summarized_or_unsummarized>'
                             },
                             '<interaction_name_2>': {
                                 'guid': '<interaction_GUID>',
                                 'state': '<interaction_state_summarized_or_unsummarized>'
                             },
                             ...
                             '<interaction_name_N>': {
                                 'guid': '<interaction_GUID>',
                                 'state': '<interaction_state_summarized_or_unsummarized>'
                             },
                         },
                         'keyThemes': {
                             '1': {
                                 'description': '<key_theme_description>',
                                 'frequency': <int_frequency_of_key_theme>,
                                 'name': '<name_of_key_theme'
                             },
                             '2': {
                                 'description': '<key_theme_description>',
                                 'frequency': <int_frequency_of_key_theme>,
                                 'name': '<name_of_key_theme'
                             },
                             ...
                             'N': {
                                 'description': '<key_theme_description>',
                                 'frequency': <int_frequency_of_key_theme>,
                                 'name': '<name_of_key_theme'
                             }
                         },
                         'keyThemeFrequencies': {
                             '1': {'<company_name': <int_frequency_of_key_theme_attributed_to_company>}, 
                             '2': {'<company_name': <int_frequency_of_key_theme_attributed_to_company>},
                             ...
                             'N': {'<company_name': <int_frequency_of_key_theme_attributed_to_company>},
                         },
                         'keyThemeQuotes': {
                             '1': {'<person_or_org_to_be_quoted>': '<text_of_quote>'},
                             '2': {'<person_or_org_to_be_quoted>': '<text_of_quote>'},
                             ...
                             'N': {'<person_or_org_to_be_quoted>': '<text_of_quote>'}
                         },
                         'noiseText': {
                             '1': '<noise_text_1>',
                             '2': '<noise_text_2>',
                             ...
                             'N': '<noise_text_N>',
                         }
                     },
                     '1': {
                         'name': '<name_of_the_second_sub_study>',
                         'GUID': '<GUID_of_substudy>,
                         'description': '<description_of_the_second_substudy>',
                         'questions': {
                             '1': {
                                 'included': <True_or_False>,
                                 'notes': '<Notes_free_text>',
                                 'question': '<Question_text>'
                              },
                             '2': {
                                 'included': <True_or_False>,
                                 'notes': '<Notes_free_text>',
                                 'question': '<Question_text>'
                              },
                              ...
                             'N': {
                                 'included': <True_or_False>,
                                 'notes': '<Notes_free_text>',
                                 'question': '<Question_text>'
                              },
                         },
                         'interactions': {
                             '<interaction_name_1>': {
                                 'guid': '<interaction_GUID>',
                                 'state': '<interaction_state_summarized_or_unsummarized>'
                             },
                             '<interaction_name_2>': {
                                 'guid': '<interaction_GUID>',
                                 'state': '<interaction_state_summarized_or_unsummarized>'
                             },
                             ...
                             '<interaction_name_N>': {
                                 'guid': '<interaction_GUID>',
                                 'state': '<interaction_state_summarized_or_unsummarized>'
                             },
                         },
                         'keyThemes': {
                             '1': {
                                 'description': '<key_theme_description>',
                                 'frequency': <int_frequency_of_key_theme>,
                                 'name': '<name_of_key_theme'
                             },
                             '2': {
                                 'description': '<key_theme_description>',
                                 'frequency': <int_frequency_of_key_theme>,
                                 'name': '<name_of_key_theme'
                             },
                             ...
                             'N': {
                                 'description': '<key_theme_description>',
                                 'frequency': <int_frequency_of_key_theme>,
                                 'name': '<name_of_key_theme'
                             }
                         },
                         'keyThemeFrequencies': {
                             '1': {'<company_name': <int_frequency_of_key_theme_attributed_to_company>}, 
                             '2': {'<company_name': <int_frequency_of_key_theme_attributed_to_company>},
                             ...
                             'N': {'<company_name': <int_frequency_of_key_theme_attributed_to_company>},
                         },
                         'keyThemeQuotes': {
                             '1': {'<person_or_org_to_be_quoted>': '<text_of_quote>'},
                             '2': {'<person_or_org_to_be_quoted>': '<text_of_quote>'},
                             ...
                             'N': {'<person_or_org_to_be_quoted>': '<text_of_quote>'}
                         },
                         'noiseText': {
                             '1': '<noise_text_1>',
                             '2': '<noise_text_2>',
                             ...
                             'N': '<noise_text_N>',
                         }
                     },
                     ...
                     'N': {
                         ...
                     }
  }
}

[miha42-github]

Base functionality has been validated in the CLI and in mr_json_server. If mr_caffeine requires change then a new PR will be issued.