Canvas LTI keys are missing for LTI 1.3

[drscottlobo]

Good day, I've just installed a new EC2 instance of Canvas on AWS for LTI 1.3 testing. Canvas is running fine, emails do not send, but otherwise everything seems to work. The issue is with LTI 1.3 launches. I've added my developer key to the site, but the Canvas public keys are empty, so the launch fails. Shouldn't the Turnkey Canvas instance generate these keys as part of the production security and dynamic settings setup? You can see the empty array of public keys here: https://basicsofmusic.courses/api/lti/security/jwks

[JedMeister]

Thanks for your report.

Our latest v18.x Canvas has a number of known issues that appear to stem for the same cause - see #1979. I am unfamiliar with LTI, but have a strong suspicion that it is related.

We're working to resolve the issue/s. I thought that I had suspended access to our Canvas appliance on AWS but I'm guessing that you must have launched it prior to the restriction taking effect?

In the meantime, please contact me directly via our support channel - email to support AT turnkeylinux.org. In your email, please note the method you used to launch your AWS server (i.e. AWS marketplace or TurnKey Hub) and we can discuss how we proceed from here.

Also, thanks again for your report> It bought to my attention that I had not already opened bug reports of the issues that I was already aware of (as I should have already). I have now resolved that and provided as much info as we currently have but please feel free to give further feedback if you have more thoughts.

[drscottlobo]

Hi, thanks for the quick response! I launched it just a day or two ago. So not sure why it was available if it wasn't supposed to be. I'm fairly sure the command to generate keys is in the dynamic_settings.yml.example file that Canvas has in the config folder.

    # to generate a new key, run the following in a Canvas console:
    #
    # key = OpenSSL::PKey::RSA.generate(2048)
    # key.public_key.to_jwk(kid: Time.now.utc.iso8601).to_json

It also has a couple keys in that file, in the Turnkey version of Canvas, has that file been copied to dynamic_settings.yml per the Canvas instructions? Here's the mention from the quickstart: https://github.com/instructure/canvas-lms/wiki/Quick-Start#dynamic-settings-configuration

[JedMeister]

For starters, I should note that Ruby and Ruby-on-Rails (the language & framework for Canvas) are not my strong suit at all. My personal involvement in the development and maintenance of our Canvas appliance has also been somewhat limited. With that disclaimer out of the way...

Our build is based on the Production guide (rather than the "quick start - which runs as development). The default example dynamic_settings.yml config you're referring to only includes config for development - none for production. So we generate our own initial dynamic_settings.yml. But as you may have noticed, it's pretty minimal. When we initially create it it only includes the requirements to run the rce-api server. I'd have to have a closer look at the build code re when/where the secrets conf are added, although the key values themselves are generated on firstboot - so they are unique to your server.

So after having a closer look, I think that you are on the right track re the config you highlight. The issue I'm referring to may well still be a factor, but that config missing from our default likely seems directly related.

Give me a sec and I'll launch one myself and have a poke around.

[JedMeister]

Ok so it took me much longer than I had expected as I hit more issues... :(

I added (what I assume is) the specific relevant config and tried to run the console commands as noted in the config comments to generate unique JWT keys and I can't even get the console to start!? Instead of starting as it should, it's throwing an exception deep in Canvas. From what I can gather something is returning a null value which is triggering the failure.

Just in case it was something required in the dynamic_settings.yml config that I hadn't included, I swapped in the example conf (and manually changed from development to production), but the console still fails with errors. Googling around I managed to find a few similar issues that users were hitting with Canvas, but no recent ones and nothing that helped.

TBH I'm out of my depth now so I'm going to have to pass this on to my colleague who is far more familiar with Ruby/RoR. He's already working on the other issue and will hopefully get that fixed really soon. I'll get him to make this his next priority.

In the meantime, I'm not at all familiar with LTI but I'm guessing that it should work with the default JWT key values so if you want to check that it works, you could just try copying the relevant config to dynamic_settings.yml. Don't use that in production though! AFAICT that should be (note that the leading whitespace matters - i.e. the leading spaces):

 store:
    canvas:
      lti-keys:
        # these are all the same JWK but with different kid
        # to generate a new key, run the following in a Canvas console:
        #
        # key = OpenSSL::PKey::RSA.generate(2048)
        # key.public_key.to_jwk(kid: Time.now.utc.iso8601).to_json
        jwk-past.json: "{\"kty\":\"RSA\",\"e\":\"AQAB\",\"n\":\"uX1MpfEMQCBUMcj0sBYI-iFaG5Nodp3C6OlN8uY60fa5zSBd83-iIL3n_qzZ8VCluuTLfB7rrV_tiX727XIEqQ\",\"kid\":\"2018-05-18T22:33:20Z\",\"d\":\"pYwR64x-LYFtA13iHIIeEvfPTws50ZutyGfpHN-kIZz3k-xVpun2Hgu0hVKZMxcZJ9DkG8UZPqD-zTDbCmCyLQ\",\"p\":\"6OQ2bi_oY5fE9KfQOcxkmNhxDnIKObKb6TVYqOOz2JM\",\"q\":\"y-UBef95njOrqMAxJH1QPds3ltYWr8QgGgccmcATH1M\",\"dp\":\"Ol_xkL7rZgNFt_lURRiJYpJmDDPjgkDVuafIeFTS4Ic\",\"dq\":\"RtzDY5wXr5TzrwWEztLCpYzfyAuF_PZj1cfs976apsM\",\"qi\":\"XA5wnwIrwe5MwXpaBijZsGhKJoypZProt47aVCtWtPE\"}"
        jwk-present.json: "{\"kty\":\"RSA\",\"e\":\"AQAB\",\"n\":\"uX1MpfEMQCBUMcj0sBYI-iFaG5Nodp3C6OlN8uY60fa5zSBd83-iIL3n_qzZ8VCluuTLfB7rrV_tiX727XIEqQ\",\"kid\":\"2018-06-18T22:33:20Z\",\"d\":\"pYwR64x-LYFtA13iHIIeEvfPTws50ZutyGfpHN-kIZz3k-xVpun2Hgu0hVKZMxcZJ9DkG8UZPqD-zTDbCmCyLQ\",\"p\":\"6OQ2bi_oY5fE9KfQOcxkmNhxDnIKObKb6TVYqOOz2JM\",\"q\":\"y-UBef95njOrqMAxJH1QPds3ltYWr8QgGgccmcATH1M\",\"dp\":\"Ol_xkL7rZgNFt_lURRiJYpJmDDPjgkDVuafIeFTS4Ic\",\"dq\":\"RtzDY5wXr5TzrwWEztLCpYzfyAuF_PZj1cfs976apsM\",\"qi\":\"XA5wnwIrwe5MwXpaBijZsGhKJoypZProt47aVCtWtPE\"}"
        jwk-future.json: "{\"kty\":\"RSA\",\"e\":\"AQAB\",\"n\":\"uX1MpfEMQCBUMcj0sBYI-iFaG5Nodp3C6OlN8uY60fa5zSBd83-iIL3n_qzZ8VCluuTLfB7rrV_tiX727XIEqQ\",\"kid\":\"2018-07-18T22:33:20Z\",\"d\":\"pYwR64x-LYFtA13iHIIeEvfPTws50ZutyGfpHN-kIZz3k-xVpun2Hgu0hVKZMxcZJ9DkG8UZPqD-zTDbCmCyLQ\",\"p\":\"6OQ2bi_oY5fE9KfQOcxkmNhxDnIKObKb6TVYqOOz2JM\",\"q\":\"y-UBef95njOrqMAxJH1QPds3ltYWr8QgGgccmcATH1M\",\"dp\":\"Ol_xkL7rZgNFt_lURRiJYpJmDDPjgkDVuafIeFTS4Ic\",\"dq\":\"RtzDY5wXr5TzrwWEztLCpYzfyAuF_PZj1cfs976apsM\",\"qi\":\"XA5wnwIrwe5MwXpaBijZsGhKJoypZProt47aVCtWtPE\"}"

Please let me know if you do try that and have any joy.

I'll post back with progress when I've got something. In the meantime, I suggest that you stop your existing instance - or destroy it altogether.

Either way, please contact me via support (support AT turnkeylinux.org) and so we can discuss refunds.

[JedMeister]

Ok I have copied across the example dynamic_settings.yml with a few edits (as discussed - although the config from my post above should have been enough). Canvas itself seems to be working fine, and there are keys noted via the URL you shared - i.e. /api/lti/security/jwks. FYI here's mine (I'll leave it running for now): https://canvas.jeremydavis.org//api/lti/security/jwks

Although it appears to be working I don't have any way to properly test it. Also ideally fresh keys should be generated and the command noted in the config file is still failing and I don't know why?! I'll continue working on this tomorrow. Wor

It's also worth noting that I have found the issue related to the background jobs failing (see this post: https://github.com/turnkeylinux/tracker/issues/1979#issuecomment-2295874547), but emails still aren't sending? Another user has emails sending fine, so I'm really not sure what's going on there?!