Testing HTTPS

[tmanternach]

I saw on Dave's Drummer change notes that he posted about HTTPS today. Here is my experience with making this work.

Preface: I am going to assume you already have (or know how to create) an SSL certificate for your domain. If you need a nudge, Google your web server (Apache/nginx) and "Lets Encrypt" for tons of instructions on how to deploy a free one. All of my experience is with a DigitalOcean VPN running Ubuntu and nginx. You can visit my working site here.

I'd love to help anyone else work through this!

1. You need to host the 3 template files listed in the Drummer docs yourself. These are index.html, styles.css, and code.js. They are discussed in the Drummer docs here.
2. These files need to be accessible on the internet via https. I did this using a sub-domain (so you technically would need 2 SSL certificates). I couldn't figure out a clean way to server these from the same domain I am using for Drummer.
3. You need to edit your index.html template file and replace all the references to fargo.io and scripting.com with the s3.amazonaws.com/fargo.io and s3.amazonaws.com/scripting.com equivalents. Here is what my index.html file looks like after those edits. The reason for this is so these files get downloaded via https in your browser. If not, modern browsers don't allow for "mixed content" (the main page loaded via https but script/resources loaded via http).
4. Once your index.html is accessible via https, you need to add a "urlTemplate" OPML header to your blog.opml. This is done by clicking File->Edit OPML Headers in Drummer and clicking the plus button. The value of mine, for example, is "https://static.trevormanternach.com/index.html"
5. You need to configure your web server properly to 1) proxy your main blog to your oldschool.script.com url and 2) serve your 3 template files via https. Here is my nginx config for doing all of this.

Cheers; have a good weekend everyone!

[scripting]

I added a link to this item from the change note about HTTPS.

[sabre23t]

You can visit my working site here.

I just reviewed your blog, tmanternach. It doesn't have the twitter sign on icons, and no retweet buttons for the post. Was that intentional or limitation in https mode?

[tmanternach]

@sabre23t, good catch! Not intentional, no. I'll take a look at this tomorrow and report back.

[am1t]

@tmanternach This is brilliant. I was able to set my blog behind HTTPS too. I also managed to make the same domain serve the static files. Here's the relevant nginx.conf file. And just to document, I also see the Twitter sign on icon and retweet buttons too. So, it is not related to HTTPS for sure.

Thank you for attempting and documenting this! This is my working blog now.

[tmanternach]

@am1t Looks great. Your Twitter sign on icon is even working. I need to see what I messed up on my site for that and the reblog buttons.

Good work with the aliases. I had tried that the other day, but didn't try with a subfolder like you did and I ran into issues.

[jackbaty]

I've just added https support to my blog as well, so thanks for this writeup! A couple of notes from my experience...

I am using a copy of the most recent template from Dave's repo here: https://github.com/scripting/drummerCms. Your example in the gist looks somewhat different (other than the s3 link replacements, of course). I wonder if that's part of why you're not seeing the twitter and version bits at the top.

I was getting mixed-content warnings until I realized my header image attribute was using http://. It's a good idea to check that all referenced files and images use https://.

I changed my urlBlogWebsite attribute to use the https:// version. It works without that change, but all the links will still point to the non-ssl version and this adds an unnecessary redirect when clicking links.

Otherwise, this is awesome, so thanks again. The key for me was learning that there were https:// equivalent URLs in S3 for all of the references (scripting.com, fargo.io, etc.)

Update: It seems the internal links are still using http:// even though I have urlBlogWebsite set as "https://rudimentarylathe.org/". Not sure why, yet.

[tmanternach]

@jackbaty and @am1t, I think I figured out my issues! I had somehow managed to use some really different template files. I'm not even going to pretend to understand how I did that, but I grabbed new ones from that drummerCms repo and everything looks happier now.

I've updated my index.html gist from above to reflect my now corrected template.

Lastly, @jackbaty, to comment on your update about links not being https, I think that isn't a huge deal. Modern browsers try pretty heavily to prefer https, and your web server looks to be properly redirecting a true http:// request to https://

trevormanternach@Trevors-MacBook-Pro ~> curl "http://rudimentarylathe.org/rss.xml" -I
HTTP/1.1 301 Moved Permanently
Server: nginx/1.18.0 (Ubuntu)
Date: Sat, 16 Oct 2021 13:16:45 GMT
Content-Type: text/html
Content-Length: 178
Connection: keep-alive
Location: https://rudimentarylathe.org/rss.xml

[jackbaty]

Another note. I added a new link in Radio3 (after moving the blog to https) and the links stopped showing in the LInks tab (the tab was blank). Suspecting an http/https thing, I changed my urlLinkblogJson attribute from http://radio3.io/users/jackbaty/linkblog.json to https://s3.amazonaws.com/radio3.io/users/jackbaty/linkblog.json and the radio3 links are now showing again. I can't be 100% sure that was the cause and fix (it could have been just coincidence and timing, and I did try doing a hard reload first) but seems likely. I mention it in case anyone runs into the same thing. Worth a shot.

@tmanternach You're right, the links will be redirected properly. It's just a loose end I'd like tied up if possible.

[scripting]

BTW, the template files are already accessible over HTTPS.

If you use these URLs, I will be able to fix any bugs that come up in them and have those changes go through to the HTTPS sites.

1. https://s3.amazonaws.com/scripting.com/code/drummercms/templates/minimal/styles.css

2. https://s3.amazonaws.com/scripting.com/code/drummercms/templates/minimal/code.js

3. https://s3.amazonaws.com/scripting.com/code/drummercms/templates/minimal/index.html

[tmanternach]

@scripting, the main issue is with all of the scripts and stylesheets included in index.html. If you leave that file as-is and try to use HTTPS, it doesn't work, because farg.io and scripting.com aren't available via https directly.

I'm not sure if you'd want to or not, but if you updated index.html to reference//s3.amazonaws.com/scripting.com instead of //scripting.com and //s3.amazonaws.com/fargo.io in place of //fargo.io, then I believe we wouldn't have to host (and edit index.html) these template files ourselves. ;-)

EDIT: There is also a radio.io stylesheet that would need to get updated to its s3 equivalent.

[scripting]

@tmanternach -- of course -- i'll go back into lurk mode. ;-)

I'm juggling too much stuff right now to take this on.

Make minimal changes to make it work, and at some point hopefully i'll be able to pop up for air and review them. If they work and look good i can possible merge them back into the default versions.

And thank you for taking charge here. So far it's a prototype for other collaborative projects I hope to start. :smile:

[scotthansonde]

Since all the cool kids are doing HTTPS these days, I'm doing it too.... https://drumbeat.scotthanson.eu (Don't bookmark that, it's just for testing. 😄 )

I'm using Caddy to reverse proxy, here is the Caddyfile.

drumbeat.scotthanson.eu {
rewrite * /ScottHansonDE{path}
reverse_proxy http://oldschool.scripting.com {
      header_up Host {upstream_hostport}
      header_up X-Forwarded-Host {host}
  }
}

tmpl.scotthanson.eu {
reverse_proxy localhost:1339
}

That's it. The drumbeat.scotthanson.eu with no port specified handles both HTTPS and redirecting HTTP to HTTPS. It also automatically gets the certs from Let's Encrypt. The tmpl.scotthanson.eu is just where I threw the templates in PagePark.

[jackbaty]

At the moment, the About tabs are not loading content on any of the blogs from this thread that are otherwise working well under https, mine included (https://rudimentarylathe.org/?tab=about). See also Amit's at https://ol.amitgawande.com/?tab=about.

I'm posting this in the https thread, because it feels https related due to this error that appears on the console (in latest Safari, macOS):

XMLHttpRequest cannot load http://httpproxy.scripting.com/httpReadUrl?url=http%3A%2F%2Fdrummer.scripting.com%2Fjackbaty%2Fabout.opml&type=text%2Fplain due to access control checks.

I've done the usual hard reload, multiple browser testing with the same result.

[dcoletta]

@papascott I did not know about Caddy but clearly it is very cool. Thanks for that recommendation. However, I tried setting up Caddy the way you described, and ran into mixed-content errors on all the files that are loaded from http://scripting.com. I looked at your site to see how you worked around that, and I can see that it's loading those files from AWS over https instead. Guessing you made your own template which loads them from there? I can see you still have maybe one mixed-content error left?

I guess that copying the files hosted at http://scripting.com to somewhere else where you can host them over https is necessary, but do you have a thought about how to keep them up to date?

[scotthansonde]

@dcoletta I was following @tmanternach's advice at the top of this issue to change the template to link to Dave's files directly on S3. Those are Dave's actual S3 buckets, so we'll catch any changes he makes.

There are a couple of http requests inside the client-side scripts (that's why the About pages aren't working), but that's a task for another day. 😄

[jackbaty]

@papascott You're saying that the failing request is generated from somewhere in the template script (code.js)? I didn't see it there, but if this is something we (as users) can modify ourselves I'm all ears :).

[dcoletta]

@papascott I should really learn to read from the top down instead of the bottom up. The good news is now I know what to do. Thanks!

[dcoletta]

With that extra clue I've got https://cannabiscto.com working. Thanks!

[tmanternach]

At the moment, the About tabs are not loading content on any of the blogs from this thread that are otherwise working well under https, mine included (https://rudimentarylathe.org/?tab=about). See also Amit's at https://ol.amitgawande.com/?tab=about.

@jackbaty, you are on to something here. I think the console message you posted is unrelated, though (the one you posted is probably a difference problem, btw). When I browse to my About tab via https, I get a slightly different message about failing to load a request to http://httpproxy.scripting.com/httpReadUrl.

It looks like Dave has open-sourced the software running at that url in this repo. Just as a proof of concept, I am going to try to run this node program behind an HTTPS endpoint and edit my code.js file (starting at line 426 for those following along) to replace the httpproxy.scripting.com url with my own. This wouldn't be any sort of production fix, just me mostly trying to see what is possible here.

[dcoletta]

I added a favicon to my https-via-Caddy setup. It was non-trivial, because I decided to put the icon resources into PagePark, so they had to be served from the tmpl subdomain. But since the site manifest was also in the tmpl subdomain, I had to enable CORS in order to <link> to it.

This worked, but I'm probably missing something. If there is support for favicons in Drummer, then there's probably a way to do this that doesn't involve hosting my own site manifest.

Or maybe there is an option in Caddy's reverse_proxy command that allows me to write a regular expression for URLs not to proxy.

[jackbaty]

@tmanternach You're absolutely right, I pasted the wrong error from the console, making my report nearly useless, sorry. I've edited the comment with the correct error. Thanks for pointing that out.

[tmanternach]

@jackbaty, I got this to work (proof here), but it is a huge mess that I don't think is sustainable.

I am not actually running the httpproxy node app myself. I am just proxying it (much like the blog itself) through an HTTPS URL (https://static.trevormanternach.com/httpReadUrl proxies to http://httpproxy.scripting.com/httpReadUrl).

I then had to download (and host myself) a separate code.js file. My version of that file is here. I replaced the script src in index.html template (line 19 on mine) to replace "my" version with the original. I then edited "my" basic-code.js file and replaced line 426 with my proxy URL for httpReadUrl.

TLDR: Giant kludge, would not recommend.

[jackbaty]

@tmanternach Nice work, but yeah, seems like a bit too many moving parts :). I'm probably better off waiting to see if Dave knows of a simpler solution.

[scripting]

what's the question, as concisely as possilble?

[dcoletta]

As a Drummer blogger, I'd like my https-hosted Drummer blog to be able to load the content of a tab without a mixed-content error.

[tmanternach]

what's the question, as concisely as possilble?

@scripting, Would it be possible to serve httpproxy.scripting.com via http and https? ;-)

[scripting]

Okay I have no idea why that's something you need me to answer??

I am not the HTTPS dude here, you are. :smile:

[scripting]

okay i think what you mean is would i be willing to host it, somehow with some additional support for https.

is that correct?

it's something that you need to happen on my server?

[tmanternach]

I just answered your question. :-D I'm not actually asking for anything myself right now.

[scripting]

ok cool! that's the best thing possible. I already have far too many things to do! :-)

[am1t]

@jackbaty Actually, I can live without the About page for now. I prefer not to solve the problem for a mostly static page pointing to other pages anyway.

[scripting]

@am1t -- is the problem in the way the template code gets the OPML text for the about outline?

[am1t]

@scripting That looks to be the case. Here's the console log again while loading the About page (mainly for sites deployed on HTTPS) - XMLHttpRequest cannot load http://httpproxy.scripting.com/httpReadUrl?url=http%3A%2F%2Fdrummer.scripting.com%2F_am1t%2Fabout.opml&type=text%2Fplain due to access control checks.

I believe if it is to be solved, it will need to be solved in one of the util packages -- mainly this function readHttpFileThruProxy from whatever I could figure out.

[tmanternach]

@am1t We are all talking about the same thing this morning. Browsers don't let you load non https resources when the main page is https. httpproxy.scripting.com is http only, and that URL is hardcoded in readHttpFileThruProxy function in this file.

The fix is to have that httpproxy server also running on https. Dave just said he has far too many other things to do, I don't blame him. ;-)

[scripting]

i didn't actually say that @tmanternach

[scripting]

i was only trying to understand what the question was. now i understand it.

[scotthansonde]

My two cents worth (as someone who knows just enough JavaScript to be dangerous to himself and others): if we are using our own templates to use with HTTPS, we can load any JavaScript we want. And if there are functions in the standard templates that cause mixed-content errors, we can override those with functions of our own. I'm sure we can figure out how to fix the mixed-content errors without Dave having to change any of his server setups.

[dcoletta]

I'm fine with no About tab for the time being, and if I find I need it, I bet I can figure out how to override readHttpFileThruProxy. Or, if I can't, then probably that's the meta level at which we would want to make a feature request - i.e., making sure JS functions that might need to be overriden are overrideable.

[scripting]

I think I figured out how to get the about outline loaded without having to do anything with the proxy server.

[scripting]

I've got the About tab working in the Clueless Newbie blog without doing a read through the proxy server.

http://clueless.lucky.wtf/?tab=about

If you open up the JS console when loading that page, you'll see a message:

viewAboutTab: using the pre-built aboutOutline.

If you look at the source of the page, you'll see in the head section of the HTML a variable named aboutOutline that has a JavaScript structure with all the data from the about outline.

So the About outline is baked into the page.

The reference to the OPML file for the outline is done on the server, in Old School, where there are no constraints on what can be read, and where HTTPS support is already present.

This does represent a change in functionality, and imho, it's fine -- the about text is baked into the page, that means if you change the text you have to rebuild the blog. This is what many newbies expect (I learned this watching what happened last week).

At the same time I made this change, I swapped in support for the relatively new opml package, it was using a package called daveopml, which I wrote years ago. The new package is what I'm recommending for all devs using OPML in their Node apps, and suggesting it as a prototype for other runtime environments.

https://www.npmjs.com/package/opml?activeTab=readme

Please let me know asap if this works. ;-)

[dcoletta]

I see the About tab, but I'm getting this error when clicking on it:

[Error] ReferenceError: Can't find variable: aboutOutline
    viewAboutTab (code.js:106)
    (anonymous function) (code.js:133)
    forEach
    startTabsIfHomePage (code.js:126)
    startup (code.js:882)
    (anonymous function) (cannabiscto.com:280)
    c (jquery-1.9.1.min.js:3:7979)
    fireWith (jquery-1.9.1.min.js:3:8776)
    ready (jquery-1.9.1.min.js:3:3394)
    H (jquery-1.9.1.min.js:3:817)

I made the about outline public and I have urlAboutOpml set in my blog's headers to be its public URL, http://drummer.scripting.com/CannabisCTO/about.opml.

Was there something else I needed to do?

[scripting]

@dcoletta -- You need to rebuild the blog.

I will add some defensive driving so you don't get this error.

Sorry for the error. ;-)

[dcoletta]

I did rebuild the blog.

[scripting]

@dcoletta -- just guessing that you're not using my template, you're using something that was derived from it?

If so, that template needs to be modified.

Here's the line of code that needs to be added --

var aboutOutline = [%aboutOutline%];

[dcoletta]

That was part of it. But even after I made the change to the index.html template, the value of %aboutOutline% is coming through as undefined.

I can confirm that in the DOM of my page I am picking up the version of code.js you just pushed 20 mins ago.

What am I missing?

[scotthansonde]

OK, I temporarily turned off urlTemplate to make sure I used the default template (I've been burned by that before 😃), rebuilt the blog, and got everything as @scripting said it would be!

1) viewAboutTab message in the console 2) aboutOutline data structure in the source 3) About page rendered as expected.

On Mon, Oct 18, 2021 at 6:18 PM Dave Winer @.***> wrote:

I've got the About tab working in the Clueless Newbie blog without doing a read through the proxy server.

http://clueless.lucky.wtf/?tab=about

If you open up the JS console when loading that page, you'll see a message:

viewAboutTab: using the pre-build aboutOutline.

If you look at the source of the page, you'll see in the head section of the HTML a variable named aboutOutline that has a JavaScript structure with all the data from the about outline.

So the About outline is baked into the page.

The reference to the OPML file for the outline is done on the server, in Old School, where there are no constraints on what can be read, and where HTTPS support is already present.

This does represent a change in functionality, and imho, it's fine -- the about text is baked into the page, that means if you change the text you have to rebuild the blog. This is what many newbies expect (I learned this watching what happened last week).

At the same time I made this change, I swapped in support for the relatively new opml package, it was using a package called daveopml, which I wrote years ago. The new package is what I'm recommending for all devs using OPML in their Node apps, and suggesting it as a prototype for other runtime environments.

https://www.npmjs.com/package/opml?activeTab=readme

Please let me know asap if this works. ;-)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/scripting/drummerSupport/issues/78#issuecomment-945941809, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACCHNHJIIVEIF3UID66226TUHRCEXANCNFSM5GC5U5BQ .

-- Scott Hanson

Email: @.*** Mobile: +49-171-5529568 Johmsweg 9a, 21266 Jesteburg, Germany

Day Job: http://www.mcdonalds-nordheide.de/

[scripting]

@papascott -- i think you;'re saying it's working.

Also at this moment it would be realllllly helpful if people posted links to their blogs.

That is if you want me to help :smile:

[dcoletta]

What am I missing?

I found my problem. Somehow my about.opml public outline had gotten corrupted. Its content had gotten copied from blog.html and some of the headers were wrong and the opml was malformed. (It had a tag that looked like this: <></> and would not render in my browser as it ususally does.

cannabiscto.com

[scotthansonde]

Yes, it works! 😄 To test this I went back to the default blog address without HTTPS: http://oldschool.scripting.com/ScottHansonDE/

[scripting]

@dcoletta -- I know there's at least one bug lurking in there. @papasccott had a report on it yesterday.

I'm hoping we can make this a priority at some point and get reproducible steps.