Sample Issue Comments from Tutor Project

[KingJam]

@snprabhu and I have been thinking about the best way to implement the run time portion of this.

There are basically 2 options:

1. Have the Math API handle doing almost all dialog flow tasks for math type problems. This would include doing functions associated with scoring, feedback, hint generation, NBA, and some LM. It would also require Orchestration to implement a completely new flow. This option is what is described above.
2. Have the Math API do scoring only. Math problems still fall into the flow of Q&A with hints. This option would modify existing services to implement any changes needed for math flow and would only require the Math API to score the answer.

We are still evaluating the best option but welcome feedback.

[KingJam]

This is what I have been thinking of in terms of the domain model and the Math Service API.

Domain Model addition

{
    "activity_id": "main1234567",
    "activity_type": "MATH_SINGLE_NUMERIC_ANSWER",
    "question": "LIMIT[x^2 - 1/x - 1, X->0]",
    "answers": ["-1"],
    "sub_activities": [{
            "activity_id": "abc123",
            "activity_type": "MATH_STEP_HINT",
            "hint_text": "a+b=c",
            "referenced_activity": "main1234567"
        },
        {
            "activity_id": "xyz456",
            "activity_type": "MATH_STEP_HINT",
            "hint_text": "d+e=f",
            "referenced_activity": "main1234567"
        },
        {
            "activity_id": "www789",
            "activity_type": "MATH_PLOT_HINT",
            "hint_url": "https://plot.com/image123.png",
            "referenced_activity": "main1234567"
        }
    ]
}

Note that this structure is similar to what was done on microtutor. I figured that it wouldn't hurt to represent this activity as such since we eventually want to move to this more generic type model. The sub_activities serve as hints to the main activity.

Initial Math API request

{
    "last_activity_id": "main1234567",
    "activity": {
        "activity_id": "main1234567",
        "activity_type": "MATH_SINGLE_NUMERIC_ANSWER",
        "question": "LIMIT[x^2 - 1/x - 1, X->0]",
        "answers": ["-1"],
        "sub_activities": [{
                "activity_id": "abc123",
                "activity_type": "MATH_STEP_HINT",
                "hint_text": "a+b=c",
                "referenced_activity": "main1234567"
            },
            {
                "activity_id": "xyz456",
                "activity_type": "MATH_STEP_HINT",
                "hint_text": "d+e=f",
                "referenced_activity": "main1234567"
            },
            {
                "activity_id": "www789",
                "activity_type": "MATH_PLOT_HINT",
                "hint_url": "https://plot.com/image123.png",
                "referenced_activity": "main1234567"
            }
        ]
    }
}

Note that the last_activity_id maps to the top level activity. This indicates that we are starting this activity for the first time. Also note that there is no student_response that we will see later.

Initial Math API response

{
    "score": "START_QUESTION",
    "activity":  {
          "activity_id": "main1234567",
           "activity_type": "MATH_SINGLE_NUMERIC_ANSWER",
           "question": "LIMIT[x^2 - 1/x - 1, X->0]",
           "answers": ["-1"]
        },
    "feadback": "Try to answer this math problem.",
    "is_done": false
}

Scoring Math API request (Fiirst score)

{
    "last_activity_id": "main1234567",
    "student_response": "17",
    "activity": {
        "activity_id": "main1234567",
        "activity_type": "MATH_SINGLE_NUMERIC_ANSWER",
        "question": "LIMIT[x^2 - 1/x - 1, X->0]",
        "answers": ["-1"],
        "sub_activities": [{
                "activity_id": "abc123",
                "activity_type": "MATH_STEP_HINT",
                "hint_text": "a+b=c",
                "referenced_activity": "main1234567"
            },
            {
                "activity_id": "xyz456",
                "activity_type": "MATH_STEP_HINT",
                "hint_text": "d+e=f",
                "referenced_activity": "main1234567"
            },
            {
                "activity_id": "www789",
                "activity_type": "MATH_PLOT_HINT",
                "hint_url": "https://plot.com/image123.png",
                "referenced_activity": "main1234567"
            }
        ]
    }
}

Note this is similar to the initial request except it has a student_response that will be scored.

Scoring Math API responses

{
    "score": "INCORRECT",
    "activity": {
                "activity_id": "abc123",
                "activity_type": "MATH_STEP_HINT",
                "hint_text": "a+b=c",
                "referenced_activity": "main1234567"
            },
    "feadback": "That's not right. Take a look at the step in solving the problem and try and answer again.",
    "is_done": false
}

Incorrect student response. Show the next hint activity.

{
    "score": "INCORRECT",
    "activity":  {
          "activity_id": "main1234567",
           "activity_type": "MATH_SINGLE_NUMERIC_ANSWER",
           "question": "LIMIT[x^2 - 1/x - 1, X->0]",
           "answers": ["-1"]
        },
    "feadback": "That's not right. Here is the answer.",
    "is_done": true
}

Incorrect student response. No more hints available (is_done=true) so display the answer.

{
    "score": "CORRECT",
    "feadback": "That's right! Good job.",
    "is_done": true
}

Correct student response.

[KingJam]

Tentative time for deployment to Production: Monday February 11, 2019 9 AM EST

Note: It's the Toucan release, but the branch is release/Taurus-1.4.x. This is mostly because we (Srijith) didn't remember the name of the release when we created the release branch. By that time there were significant number of changes.

Required people for deploy

Bruce Srijith Saubhik Vivek (and anyone else he wants to include) Moumita Lisa

Preparing for deploy and general notes (All)

• The Docker image names and tags in prod-deployment.yaml file from cardinal-kubernetes repo on the release/Taurus-1.4.x branch must match those in staging-deployment.yaml.
• If you need to create a new docker images it must be built from the code in the release/Taurus-1.4.x branch. Code built for Toucan will use tag names with the format 1.4.x. Example: registry.ng.bluemix.net/cardinal/next-best-action:1.4.0
• Ensure test/verification assets are in place
• Ensure rollback assets are in place for code and training data
• Ensure services are created in US East to support our move to have Watson Assistant and Redis in US East.
• Tarannum will ensure that PagerDuty alerts are turned off during this deployment. Don't want to have them go off just because we are updating the environment.

Setup the US East services (Srijith):

1. Replicate the Cloudant database.

2. Replicate the Redis keys to the new US East Redis server: https://github.com/compose-ex/python-redis-migrator

   # python3 migrate-redis.py srchost srcpassword srcport desthost destpassword destport --ssldst

   Deploying code to production space (Bruce/Srijith)

3. Set us-east region

   # bx cs region-set us-east

4. From command line, set space target to production-app

   # bx target -s production-app

5. From command line, set cluster environment to cardinal-prod-us-east.

   # bx cs cluster-config cardinal-prod-us-east

6. Verify cluster config context was properly set to cardinal-prod-us-east.

   # kubectl config current-context

7. Apply prod-secrets.yaml file from cardinal-kubernetes repo on the release/Taurus-1.4.x branch.

   # kubectl apply -f prod-secrets.yaml

8. Apply prod-deployment.yaml file from cardinal-kubernetes repo on the release/Taurus-1.4.x branch.

   # kubectl apply -f prod-deployment.yaml

9. Log into IBM Cloud and reduce the number of worker nodes per region from 3 to 2. Unlike in Staging we will stay in all regions. This will allow us to not have to update the ALB.

10. Run jMeter tests to ensure basic functionality. Tests should be run against cluster API (cardinal-prod-us-east.us-east.containers.appdomain.cloud), and CloudFlare (pearson.watson-tutor.ibm.com).

Training the system (Vivek, etc., and Bruce/Srijith)

1. Vivek's team will train DB and Watson Assistant for all titles.

   • This will involve training SRA with the HopsTag model.
   • Creating Watson Assistant services in US East and updating the service configs for each of the titles.

2. Bruce/Srijith will restart SRA to ensure it has latest data loaded.

   # kubectl patch deployment sra-deployment -p "{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"RESTART_DATE\":\"`date +'%s'`\"}}}}}"

   Testing the system (Bruce/Srijith, Lisa, Saubhik, Vivek, etc.)

3. Bruce/Srijith to perform basic jMeter test, on one title, to ensure no errors are returned.

4. Saubhik/Others to run automated tests on all titles.

5. Lisa to run basic performance test

Rollback, if unable to fix system in reasonable amount of time (all)

1. Rollback secret, ingress, service, and deployment using the yaml files in master branch of cardinal-kubernetes
2. Rollback training data
3. Test system using jMeter and automated scripts.

Post deploy procedures

Close the Taurus-1.4.x release for the following repos. This list is complete at the time of this writing. Do this after successful deploy and wait a day, if you can, to make sure we don't need to rollback.

• cardinal-openwhisk-orchestration
• student-feedback
• hint-generator-service
• cardinal-learner-model
• next-best-action
• student-response-analyzer
• cardinal-db-apis
• cardinal-kubernetes

  # git flow release finish Taurus-1.4.x
  # git push origin master
  # git push origin develop
  # git push origin --tags

  If you get merge conflicts against develop branch, create a PR to merge Taurus-1.4.x into develop. Have the dev resolve the merge conflicts and then merge in PR. Pull the latest code to your local machine and then you should be able to successfully do the git flow release finish Musca-1.4.x.