SSL issue with MindTouch Transclusion Script

ymdahi commented 3 years ago

Problem

Pages on LibreTexts that use cross-library transclusions are displaying an SSL-related error and not fetching/displaying the transcluded content. The error:

(unable to fetch text document from uri [status: 0 (UnableToConnect), message: "Error: TrustFailure (Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED)"])

Expected Behaviour

Pages (from other libraries) can be "transcluded" in a book in another library using a Dekiscript template like so: template('CrossTransclude/Web',{'Library':'biz','PageID':4566}); (live example)

The template refernced in this script, CrossTransclude/Web, is defined in the MindTouch Admin Dashboard > Site Administration > Template Directory > Template:CrossTransclude > Web (link to template on Eng)

Possible Causes of Issue

[ ] Let's Encrypt Root Cert Expiration
[ ] batch.libretexts.org being blocked by API call limit ( #152 )
[ ] Cloudflare configuration issues
[ ] api.libretexts.org server configuration issues
[ ] Changes to MindTouch Dekiscript API
[ ] MindTouch SSL Certificate issues

Next Steps

We should attempt to find and define the functions in on the API server that handle the transclusion library. elevate.js seems to be the only place that mentions transclusion:

https://github.com/LibreTexts/Libretext/blob/902d89a989cb7c48bfd8fbe820b8c33e94549d73/API/elevate.js#L185-L214

ymdahi commented 3 years ago

@Miniland1333 : help us henry-one, you're our only hope

ymdahi commented 3 years ago

SSL Certs for both batch and api have been force-renewed on Tues Oct 5. No effect on issue.

Miniland1333 commented 3 years ago

My guess is that it is related to the LetsEncrypt Root Cert swap-over, especially if you only noticed the issue after October 1. To test this theory, try copying the code from CrossTransclude and substitute other LetsEncrypt URLs web.html(web.xml(<<url to another LetsEncrypt page>>)); or web.html(<<url to another LetsEncrypt page>>);

If other letsEncrypt URLs return the same error, then it is a letsEncrypt problem and Mindtouch needs to update the CA on their backend from a decade ago.

ymdahi commented 3 years ago

@drdelmar @ethanaturner Read @Miniland1333 comment above.

@ethanaturner: can you please test this out when you get a chance. I'll be back tomorrow to help with testing.

@Miniland1333: inter-library transclusion works. e.g. same-lib transclusion. But intra-library transclusion doesn't.

For example, on the SocialSci Lib:

this works: web.xml("https://socialsci.libretexts.org/@api/deki/pages/125208/contents");

this doesn't: web.xml("https://human.libretexts.org/@api/deki/pages/16982/contents");

From what I understand, the second one needs to use authenticatedFetch().

Does it still sound like an SSL issue?

Miniland1333 commented 3 years ago

Could you please confirm the not-working example? I was able to get the second piece working normally at https://chem.libretexts.org/Under_Construction/Sandboxes/Henry/1.2%3A_Sandbox

And just to be clear, intra-library transclusion works directly while inter-library uses the endpoint/cross-library on api.libretexts.org

And of course since the issue started occurring after October 1 and involves api.libretexts.org (which uses LetsEncrypt), that is my suspicion at this time.

ymdahi commented 3 years ago

Yeah, sorry I'm seeing that this is actually working now:

web.html(web.xml("https://socialsci.libretexts.org/@api/deki/pages/125208/contents"));
web.html(web.xml("https://human.libretexts.org/@api/deki/pages/16982/contents"));

here: https://socialsci.libretexts.org/Sandboxes/jhalpern/Yasin_test/02%3A_The_Origins_of_Civilization/2.01%3A_Hominids

Why won't template('CrossTransclude/Web',{'Library':'human','PageID':16982}); work if web.html(web.xml("https://human.libretexts.org/@api/deki/pages/16982/contents")) is working?

ymdahi commented 3 years ago

Also the SSL cert for api.LT.org was renewed just yesterday and other services it supports seem to be working fine.

ethanaturner commented 3 years ago

CrossTransclude/Web is passed through the api.LT.org, meanwhile calling the MT API on the lib goes through the MT server hosting it. However, calling the same api.LT.org cross-lib endpoint manually (on my machine) seems to work as expected. So there's something not lining up when poking api.LT.org ....

ymdahi commented 3 years ago

I updated the template for CrossTransclude/Web on ENG to:

var PageID = $PageID ?? 34;
var Library = $Library?? 'chem';
//web.html(web.xml(uri.build("https://api.libretexts.org/endpoint/cross-library/https://"..Library..".libretexts.org",["@api","deki","pages",""..PageID,"contents"])));
web.html(web.xml(uri.build("https://"..Library..".libretexts.org",["@api","deki","pages",""..PageID,"contents"])));

And inter-lib transclusion seems to be working fine (example).

Thoughts...?

ethanaturner commented 3 years ago

That is probably a good way to patch to keep everyone happy/things moving for now! I am still interested in getting to the true bottom of this 🤔

ymdahi commented 3 years ago

@Miniland1333: can you tell us why this is a bad idea.

I feel like this shouldn't be working. Or we are in an echochamber of some sort.

Did MT change the way web.xml fetches?

Miniland1333 commented 3 years ago

Looks functional to me! My guess is that this wasn't working directly when the template was set up a few years ago, but they must have changed something since then.

ymdahi commented 3 years ago

What's puzzling to me is that just going to https://human.libretexts.org/@api/deki/pages/16982/contents in the browser produces an error:

<error>
<exception>MindTouch.Deki.Auth.Exceptions.TokenValidationFailedException</exception>
<resource>System.API.Error.developer-token-validation-failed</resource>
<arguments count="1">
<argument>missing required token</argument>
</arguments>
<message>The request failed API token validation, missing required token</message>
</error>

So accessing the endpoint requires authentication. But how does web.html(web.xml()) bypass that authentication requirement?

ymdahi commented 3 years ago

@Miniland1333: bypassing api.LT.org for the transclusions on MT seems to work fine:

https://eng.libretexts.org/Courses/Prince_Georges_Community_College/INT_2840%3A_Systems_Analysis_and_Project_Management/02%3A_Information_Systems_Development_And_RFP/2.04%3A_Cloud_Computing-_Hype_or_Hope

However, the changes to the CrossTransclude/Web template seem to have no effect on the PDF generator:

https://batch.libretexts.org/print/url=https://eng.libretexts.org/Courses/Prince_Georges_Community_College/INT_2840%3A_Systems_Analysis_and_Project_Management/02%3A_Information_Systems_Development_And_RFP/2.04%3A_Cloud_Computing-_Hype_or_Hope.pdf

I haven't yet been able to find anything in nodePrint that might be causing PDFs to use the 'old way' of transcluding.... thoughts?

ethanaturner commented 3 years ago

@ymdahi I could be wrong, but, if there is nothing in nodePrint using the 'old way' of transcluding, it likely pulls a page with transcluding like normal (in theory working with the 'new way'), but Batch likely hasn't hit its scheduled time to re-compile all of the libs and get rid of those OPENSSL errors in the PDFs.

drdelmar commented 3 years ago

Not sure if Henry shared, but there are amin tools he made for me/us here: https://chem.libretexts.org/Under_Construction/Development_Details/Admin_Tools

The API dashboard has a batch control to force a library update.... I am a little concerned about the bandwidth issue though and blacklisting.

On Fri, Oct 8, 2021 at 10:05 AM Ethan Turner @.***> wrote:

@ymdahi https://github.com/ymdahi I could be wrong, but, if there is nothing in nodePrint using the 'old way' of transcluding, it likely pulls a page with transcluding like normal (in theory working with the 'new way'), but Batch likely hasn't hit its scheduled time to re-compile all of the libs and get rid of those OPENSSL errors in the PDFs.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/LibreTexts/Libretext/issues/153#issuecomment-938873913, or unsubscribe https://github.com/notifications/unsubscribe-auth/AVZI6UOPZMIT54ROJY5M3J3UF4QGPANCNFSM5FQGBJKQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

ymdahi commented 3 years ago

Ah, okay. So it isn't a JIT/on-demand service. This makes more sense.

Here's the schedule for when each lib gets re-compiled:

    '30 0 * * sat': 'bio',
    '30 12 * * sat': 'biz',
    '30 0 * * sun': 'chem',
    '30 12 * * sun': 'eng',
    '30 0 * * mon': 'espanol',
    '30 12 * * mon': 'geo',
    '30 0 * * tue': 'human',
    // '30 12 * * tue': 'law',
    '30 0 * * wed': 'math',
    '30 12 * * wed': 'med',
    '30 0 * * thu': 'phys',
    '30 12 * * thu': 'socialsci',
    '30 0 * * fri': 'stats',
    '30 12 * * fri': 'workforce',

SocialSci was yesterday (after the patch), and it looks like there are no traces of the SSL issue on nodePrint: https://socialsci.libretexts.org/Sandboxes/jhalpern/Yasin_test/02%3A_The_Origins_of_Civilization/2.01%3A_Hominids

So, @drdelmar we can just wait until next week Wednesday for all libraries to update. I agree, forcing an update will probably get us blocked.

Also, any idea why the CHEM lib hasn't updated in 39 days?

Miniland1333 commented 3 years ago

What's puzzling to me is that just going to https://human.libretexts.org/@api/deki/pages/16982/contents in the browser produces an error:
<error>
<exception>MindTouch.Deki.Auth.Exceptions.TokenValidationFailedException</exception>
<resource>System.API.Error.developer-token-validation-failed</resource>
<arguments count="1">
<argument>missing required token</argument>
</arguments>
<message>The request failed API token validation, missing required token</message>
</error>
So accessing the endpoint requires authentication. But how does web.html(web.xml()) bypass that authentication requirement?

This must be why I originally had to use api.libretexts.org and authenticatedFetch(). The endpoints required a deki-token in order to interact with the API, but it looks like Mindtouch has done something in the past 3 years so that their backend is exempt from this authentication requirement.

ymdahi commented 3 years ago

This is resolved for now. Thanks @Miniland1333, @ethanaturner for the effort.

Going to close this ticket.

LibreTexts / Libretext