Run IMS Certification Tests for LTI 1.1

[klemay]

Part of https://github.com/hypothesis/lms/issues/705, not blocked by https://github.com/hypothesis/lms/issues/876 but it may not make sense to run the tests until #876 is implemented.

The IMS website is so confusing/complicated that they created a video for members to teach them how to navigate it... I'll try to point y'all to where you need to go.

To run the test:

1. Go to https://www.imsglobal.org/user and log in (credentials in Passpack)
2. Go to https://www.imsglobal.org/lti-v1-certification-suite (if you skip step 1 you will be redirected to the login page and likely won't be able to navigate your way here - so please log in first!)
3. Click on "Process for Tool Providers"
4. You'll be directed to a "wizard" that will walk you through the testing process

The testing process is divided into the following stages:

• Overview
• Configuration (basically you'll create a key and secret to use with their testing suite)
• Tests
• Results (this will tell us if we need to make any changes to get certified)
• Submit (and then we wait for IMS to get back to us)

The certification tests are divided into the following sections:

• Tool registration (5 tests)
• Invalid Launch Requests (9 tests)
• Valid Launch Requests (4 tests)
• Types of User Roles (8 tests)
• Levels of User Information (7 tests)
• Deep Linking Message (2 tests)
• Outcomes 1.0 service (4 tests)
• Tool Consumer Profile (1 test)
• Tool Settings service (4 tests)
• Names and Roles Service (4 tests)

[seanh]

I've run the test suite:

Summary

If I'm reading the certification suite tool correctly, there appear to be several separate certifications here. Some of the certifications apply to our app, and some don't. And there are different tests for each certification:

• 1.0
• 1.1
• 1.2 (doesn't apply to us)
• Deep Linking (doesn't seem to apply to us / I don't understand the tests, see below for details)
• Outcomes-1 (doesn't apply to us until we have released grading for all LMS's, but looks as if it may be required for 1.1 certification, unclear, see below for details)
• Tool Settings (service) (doesn't apply to us)
• Names and Roles (service) (doesn't apply to us)

In order to pass all the tests that the test suite claims "apply to" LTI 1.0 and 1.1 we need to:

1. Fix 7 easy-looking "Invalid Launch Requests" test fails
2. Fix 2 easy-looking "Levels of User Information" test fails (which look like they may both be the same one bug)
3. For LTI 1.1 certification (but not 1.0), if I'm reading the tool correctly, I think we need to finish grading and release it for all LMS's and then run one "Outcomes 1.0 service" test and pass it. It seems odd to me that grading support would be required for LTI 1.1 certification. Surely grading isn't applicable to all apps? But the test suite says it "Applies to: 1.1, Outcomes-1 [required]".

I've created separate issues for each thing we'd need to fix and given them all the LTI 1.1 Certification label.

Details

• Section 1: Invalid Launch Requests: We fail many of the tests in this section either because our app crashes or (mostly) because our app displays its own error page when it's supposed to "return to tool consumer with error message". Even though we have 7 failures in this section, I think they will all be quite quick and easy to fix.

• Section 2: Valid Launch Requests: We passed all these.

• Section 3: Type of User Roles: I think we can fairly claim to pass all these but I did create one issue about how we could improve our app's behavior with anonymous users: https://github.com/hypothesis/lms/issues/1123

• Section 4: Levels of User Information: We failed two tests. Both appear to be the same issue. I think this would be an easy fix.

• Section 5: Deep Linking Message: There are two tests in this section. I don't think our app passes either of them. I don't understand what the tests are supposed to be testing, or what our app is supposed to do here. Apparently our app is supposed to "return an LTI link" in the first test, and in the second test we're supposed to "return the data parameter". I would have to read up on the spec to get a better understanding of what's required to fix our app here. I don't think we need to pass this section for LTI 1.0 & 1.1 certification. I think it's only needed for a separate Deep Linking certification.

• Section 6: Outcomes 1.0 service: Our app does not make use of the outcomes service until we have released grading for all LMS's. Once we've released that we could run these tests. Until then they're inapplicable to our app. But the first test in this section is required for 1.1 certification, if I'm reading the tool correctly.

• Section 7: Tool Consumer Profile: Not applicable to our app. I don't think we need to pass this for LTI 1.1 certification, it's only for LTI 1.2 certification.

Section 8: Tool Settings service**: Not applicable to our app. I don't think we need to pass this for LTI 1.1 certification, it's only for Tool Settings [service] certification.

• Section 9: Names and Roles Service: Not applicable to our app. I don't think we need to pass this for LTI 1.1 certification, it's only for Names and Roles [service] certification.

Full results

Test	Applies to	Status
1  Invalid Launch Requests
1.1  No resource_link_id provided	1.0, 1.1, 1.2	Failed
1.2  No resource_link_id or return URL provided	1.0, 1.1, 1.2	Failed
1.3  Invalid OAuth consumer key	1.0, 1.1, 1.2	Passed
1.4  Invalid OAuth signature	1.0, 1.1, 1.2	Passed
1.5  Invalid LTI version	1.0, 1.1, 1.2	Failed
1.6  Wrong LTI version	1.0, 1.1, 1.2	Failed
1.7  Missing LTI version	1.0, 1.1, 1.2	Failed
1.8  Invalid LTI message type	1.0, 1.1, 1.2	Failed
1.9  Missing LTI message type	1.0, 1.1, 1.2	Failed
2  Valid Launch Requests
2.1  Launch as an instructor	1.0, 1.1, 1.2	Passed
2.2  Launch as an instructor from a different resource link	1.0, 1.1, 1.2	Passed
2.3  Launch as a learner	1.0, 1.1, 1.2	Passed
2.4  Launch as a learner from a different resource link	1.0, 1.1, 1.2	Passed
3  Types of User Roles
3.1  Launch as an instructor using a fully-qualified URN	1.0, 1.1, 1.2	Passed
3.2  Launch as an instructor with a list of roles	1.0, 1.1, 1.2	Passed
3.3  Launch as an instructor using a full URN role in a list	1.0, 1.1, 1.2	Passed
3.4  Launch as a learner using a full URN role in a list	1.0, 1.1, 1.2	Passed
3.5  Launch as a user with no context role	1.0, 1.1, 1.2	Passed
3.6  Launch as a user with an institution role but no context role	1.0, 1.1, 1.2	Passed
3.7  Launch as a user with an institution role which has no corresponding context role	1.0, 1.1, 1.2	Passed
3.8  Launch as a user with an unrecognised role	1.0, 1.1, 1.2	Passed
4  Levels of User Information
4.1  Launch as a learner with email but no name	1.0, 1.1, 1.2	Passed
4.2  Launch as a learner with given/family but no full name	1.0, 1.1, 1.2	Passed
4.3  Launch as a learner with a full name but no given or family names	1.0, 1.1, 1.2	Passed
4.4  Launch as an instructor with no personal information	1.0, 1.1, 1.2	Passed
4.5  Launch as an instructor with no context or personal information apart from the context ID	1.0, 1.1, 1.2	Failed
4.6  Launch as Instructor with no context information	1.0, 1.1, 1.2	Failed
4.7  Launch with only resource_link_id	1.0, 1.1, 1.2	Passed
5  Deep Linking Message
5.1  Create an LTI link	Deep Linking
5.2  Supports data parameter	Deep Linking
6  Outcomes 1.0 service
6.1  replaceResult requests	1.1, Outcomes-1
6.2  readResult requests	1.1, Outcomes-1	OK
6.3  deleteResult requests	1.1, Outcomes-1	OK
6.4  Processing of Gradebook entries	1.1, Outcomes-1
7  Tool Consumer Profile
7.1  Valid request	1.2
8  Tool Settings service
8.1  GET requests	Tool Settings
8.2  PUT requests	Tool Settings
9  Names and Roles Service
9.1  GET Names and Roles container requests	Names and Roles
9.2  GET Names and Roles container requests for a resource link	Names and Roles
9.3  Include 'role' query parameter	Names and Roles
9.4  Include 'limit' query parameter	Names and Roles