Open Lonniebiz opened 5 years ago
Ok, let me spell this out more concretely.
There is work being done right now on html templates that would be completely unnecessary if a fundamental flaw was corrected in the standards.
That fundamental flaw is the fact that you cannot load a web page by going directly to a client-side javascript file.
For example, if I created a client-side javascript file, that dynamically generated an html document and css and placed it on a web server at:
www.webpage.com/index.js
The web-broswer would simply show the source code of that file, and it would not run the javascript it contained. That is a fundamental flaw in the standards. A client-side javascript file, that dynamically creates an html document should be a first class citizen of the web and it should not require a static html file to launch a javascript file that does this.
I propose that a client-side javascript file (that generates an html document) become a valid entry point to a website. Right now, such a file is completely ignored. And, if you think about it, I bet you can imagine other work that is being done right now due to the absence of this fundamental thing that should be there.
I predict that one day, what I'm saying in this post will come to fruition.
In the abstract this sounds reasonable, but it's rather unclear what this would mean for HTML's navigate algorithm (which to be clear, is the entry point for page loading)?
You'll have to get a whole lot more concrete than what you got thus far.
https://whatwg.org/faq#adding-new-features might be worth reading.
What's the difference compared to having this file?
<!DOCTYPE html><meta charset="utf-8">
<script>addEventListener('DOMContentLoaded', function () {
document.title = 'Hello index.js';
document.head.appendChild(
document.createElement('style')
).textContent = `
html {
font-family: sans-serif;
font-size: 16px;
text-align: center;
}
`;
document.body.appendChild(
document.createTextNode('Hello body')
);
}, {once: true})</script>
Also bear in mind with a ServiceWorker you can actually create HTML pages at runtime and serve these on demand, meaning you don't really need more than a SW to have a 100% client side JS based site.
What's the difference compared to having this file?
The difference is obvious. That's an html file. A static html file shouldn't be required to load a client-side javascript file that dynamically generates an html document; it is fundamentally unnecessary other than the fact that current standards require it.
@Lonniebiz Just for clarity, browsers are not aware of files or file extensions. It only deals with mime types. The reason why the browser shows the JavaScript source when you navigate to it is because your web server is responding with a text/javascript
mime type. So what you're really doing is proposing a new mime type that triggers essentially about:blank
plus executing a script. Many questions on how this would work.
You could pretty easily configure your web server to respond with text/html
and wrap your JavaScript in a <script>
. You'd need no spec changes to do that.
I would advise against doing so, and am against this proposal, because this is not a good way to deliver web content. The browser can render HTML before it has completed downloading it, which it cannot do with JavaScript. Adopting this proposal would be validating bad practices.
The browser can render HTML before it has completed downloading it, which it cannot do with JavaScript. Adopting this proposal would be validating bad practices.
For a web site that generates its content exclusively from dynamic javascript anyway, being able to provide that javascript as an "entry point first request" saves an unnecessary trip to the server versus downloading an html file with an empty body that then has to make another trip to the webserver to get that javascript it could have gotten on first request.
@Lonniebiz It doesn't have to make another request. I gave an example above about how you an avoid that.
@Lonniebiz It doesn't have to make another request. I gave an example above about how you an avoid that.
To me that is just a work around for something that is more fundamentally missing. There would be little need for server-side tricks, if the client-side supported what I'm proposing.
So you'd still start out with a Document
and such? Doesn't really seem like a fundamental improvement then.
The difference is obvious. That's an html file. A static html file shouldn't be require to load a client-side javascript file that dynamically generates an html document; it is fundamentally unnecessary other than the fact that current standards require it.
A web server could be configured to transform JS files into what @WebReflection wrote whilst streaming the response. It's effectively just boilerplate.
You could potentially have the browser do the same work (just create the minimal DOM and execute the script), but what's the benefit other than avoiding a small amount of boilerplate? The drawbacks seem to outweigh that, in my opinion. For example, it encourages bad accessibility practices of not having fallback HTML, and you can't take advantage of progressive rendering of HTML (because the JS has to be fully loaded, then parsed, then executed).
So you'd still start out with a
Document
and such? Doesn't really seem like a fundamental improvement then.
Yes, a client-side, dynamically generated document (made from scratch with a javascript entry point). We do this type of stuff server-side all the time, why can't the client shoulder some of this burden?
Answer: because the standards don't support a client-side javascript file as a valid entry point to a website.
A web server could be configured to transform JS files into what @WebReflection wrote whilst streaming the response. It's effectively just boilerplate.
The whole point of my proposal is to avoid such server-side tricks.
It is 2019. A client-side javascript file (that dynamically generates an html document) should be a valid entry point to a website (client-side). The web browser should display the html document generated by this client-side javascript file. Javascipt files that do not generate an html document can be treated as they are now.
I think, after the client-side-dynamically-generated-html-document has been displayed, it can be treated as if it is it were a static html file, meaning this feature can be predominantly transparent once supported; it wouldn't have to change much else than where the html document comes from client-side.
You keep saying what the browser should do but failing to explain why. Beyond prototyping, I don't see any benefit at all. Even then, for anything other than a few elements it becomes unwieldy without some kind of tooling and at that point the tooling may as well just create the boilerplate for you.
You keep saying what the browser should do but failing to explain why. Beyond prototyping, I don't see any benefit at all. Even then, for anything other than a few elements it becomes unwieldy without some kind of tooling and at that point the tooling may as well just create the boilerplate for you.
Personally, I don't like creating static html files anymore. I think it stupid that a website driven almost exclusively by javascript, has to generate html files on the server-side just so the browser will be smart enough to know what to do with the javascript files. Seriously, let's skip this middleman: HTML files. Why should servers have to fool the browser into thinking it is receiving some static html file? There's a javascript engine on the client-side that can't do anything without a server-side generated html file. Server-side we are dynamically generating these files. Why can't the client-side support the dynamic creation of an html document from a client-side javascript file?
Answer: because it just doesn't (well, it does, but not as directly as I'm proposing).
How do you set the doctype, meta tags and prefetch in your javascript file?
How do you set the doctype, meta tags and prefetch in your javascript file?
The browser can recognize the deference between "a javascript file directly navigated to" and a "javascript file referenced by an html document". The javascript file would be required to generate a valid html document, including doctype and meta tags. I'm not sure I understand what you mean by prefetch. But, after this document is dynamically created (client-side), it could be treated like any other html document, the fact that it was generated by client-side javascript could be transparently ignored after that dynamic creation.
@Lonniebiz this "site" is 100% driven by Service Worker.
The index is basically this:
<script>navigator.serviceWorker.register('sw.js').then(()=>{location.href="$/index.js"})</script>
Everything else is served via JS. As you know already, GitHub is also for static pages only, so you could do pretty much anything you want.
I haven't bothered using a cache to store pages, and I'm using unpkg service out of the box, so there's room for optimizations but the thing works already.
All used files are here, I hope this helps showing off ServiceWorkers can already be used to generate sites directly via client-side JS only.
<script>navigator.serviceWorker.register('sw.js').then(()=>{location.href="$/index.js"})</script>
Thumbs up for showcasing that, because that's cool. Still, it requires an html script tag, so it is beside the point as far as I'm concerned.
BTW, it is strange how often I come across your posts on the internet. I can't tell you how many times I've had an idea, and by googling about it, I'd come across something you've written that is the same or similar.
What's the downside to wrapping your JS so that it's both valid JS and valid HTML by using https://tc39.github.io/ecma262/#prod-annexB-SingleLineHTMLOpenComment
SingleLineHTMLOpenComment ::
<!--
SingleLineCommentCharsopt
<!-- --><script type="module">
...
// </script>
Currently, you can use client-side javascript to modify an "already existing" html document.
You can even create an html document from scratch using client-side javascript as long as you import that document into an iframe: https://developer.mozilla.org/en-US/docs/Web/API/DOMImplementation/createHTMLDocument
What you can't do, is this:
function makeDocument()
{
var doc = document.implementation.createHTMLDocument("Document Generated By Client-Side Javascript");
doc.documentElement.lang = "en";
var meta = doc.createElement('meta');
meta.setAtribute("charset", "UTF-8");
doc.head.appendChild(meta);
var title = doc.createElement("title");
title.textContent = "Javascript is No Longer a Prisioner of HTML";
doc.head.appendChild(title);
var p = doc.createElement("p");
p.textContent = "This is a clickable paragraph.";
p.addEventListener('click', function(event){ alert("Yes, this click event should work!"); });
doc.body.appendChild(p);
return doc;
}
export let clientSideDynamicallyGeneratedHTMLDocument = makeDocument();
If you navigate to a javascript file, like the one above, the browser will simply show you the source code; it won't run the file.
I believe, that if a client-side javascript module is directly navigated to, and exports a valid html document, the browser should run that javascript and display that document instead of just showing the source code.
It is an unnecessary limitation that javascript must be encapsulated by (or called from) an HTML document to run, and the consequence of that unnecessary limitation, is that I'm constantly making ridiculous html files that look like this:
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>Temporary: Soon To Be Modified by Javascript</title>
<script type="module" src="PrisonerOfHTML.js"></script>
</head>
<body>
Temporary: This content will be completely replaced by javascript.
Have a nice day!
</body>
</html>
Seriously, is such a file, like above, worthy of being a necessity?
@Lonniebiz
I believe, that if a client-side javascript module is directly navigated to, and exports a valid html document, the browser should run that javascript and display that document instead of just showing the source code.
How would you know it exports a valid html document before running the code? Seems undesirable to change the behavior of browsers so it executes all javascript when you navigate to a .js file. It wouldn't even make sense in the majority of cases.
How would you know it exports a valid html document before running the code?
If the javascript file is directly navigated to (and therefore not clicked on from a "view-source:https" page), my idea would be to scan the file for the "createHTMLDocument" string. If that string exists in the file, the browser would then run the code and only display a page if that script ultimately exports an html document.
If the "createHTMLDocument" string is not located, then the browser could just display source code as usual.
Lastly, if does find the "createHTMLDocument" string, but the script doesn't export an html document, then it can display an error.
Wait, I just realized that the creation of the html document could actually happen in an imported module (and not necessarily the file being directly navigated to). So, I guess it would be necessary to look for the "createHTMLDocument" string in all imports too.
However, personally, if the javascript file is navigated to directly, I'm ok with the browser just running the script; it will either work or not just like any other script that is linked-to from an html document "either works or not".
@Lonniebiz re
If the javascript file is directly navigated to (and therefore not clicked on from a "view-source:https" page), my idea would be to scan the file for the "createHTMLDocument" string.
I would oppose anything that would cause Content-type:text/javascript served by existing origins to run code within that origin.
Recasting a widely used inactive content-type as an active content-type will introduce unnecessary vulnerabilities for file hosting services and CDNs that are careful about what they attach HTML, XHTML, and SVG content-types to but are less careful about other widely used content-types.
However, personally, if the javascript file is navigated to directly, I'm ok with the browser just running the script;
There is a long history of vulnerabilities that involve socially engineering users into going to a URL via ads, URL shortneners, open redirectors, and <a href="//cdn.org/people-dont-consistently-read-urls.js">Cute kittens</a>
.
This works for me with python -m SimpleHTTPServer
<!-- --><meta charset="utf8"><script type="module">
import bar from './bar.js';
export default null;
setTimeout(
() => document.body.appendChild(document.createTextNode(bar)),
0);
// </script>
//<!-- --><meta charset="utf8"><script type="module">
export default 'Hello, World!';
// </script>
Without the // at the beginning in bar.js
, I get
Uncaught SyntaxError: HTML comments are not allowed in modules
because of https://tc39.github.io/ecma262/#sec-html-like-comments
B.1.3 HTML-like Comments
The syntax and semantics of 11.4 is extended as follows except that this extension is not allowed when parsing source code using the goal symbol Module:
so if you want to serve a JavaScript file that operates as an entire HTML document, and the polyglot approach is ok for you, then maybe ask github.com/tc39 to remove that exception.
Currently, you can use client-side javascript to modify an "already existing" html document. ... What you can't do, is ...
What does having client-side javascript set the current document enable?
There is a long history of vulnerabilities that involve socially engineering users into going to a URL via ads, URL shortneners, open redirectors, and
<a href="//cdn.org/people-dont-consistently-read-urls.js">Cute kittens</a>
.
I have great respect for you Mike Samuel, because of your contributions to the standards, specifically: Tagged Template Literals.
With that said, I'd like to respectfully point out, that:
<a href="//cdn.org/people-dont-consistently-read-urls.js">Cute kittens</a>
Is very similar to:
<a href="//cdn.org/people-dont-consistently-read-urls.html">Cute kittens</a>
My point is, that malicious javascript that is "navigated to directly" is no more malicious that javascript that automatically runs from the unbarred prison of an html file.
@Lonniebiz
<a href="//cdn.org/people-dont-consistently-read-urls.html">Cute kittens</a>
Yes. Which is why you never upload third-party HTML or SVG alongside your content to your CDN. Many existing services are less careful with other content-types.
Making JS active content breaks the security assumptions underlying existing applications, so is a break-the-web level decision.
See OWASP Unrestricted File Upload:
Attacks on other systems
...
- Upload .html file containing script - victim experiences Cross-site Scripting (XSS)
Note that there is no mention of "JS" or "JavaScript", so this proposal would bypass existing recommendations on safely handling file uploads.
This works for me with
python -m SimpleHTTPServer
Thanks for elaborating on your example. I read your previous attempt to show me this, but I just couldn't understand the specification and how it applied to what your were saying. This is clever.
Still, I wish that, something as fundamental as the "direct navigation to a client-side generated html document" be an acknowledged practice in the standards, and not some brilliant HTML hack that works without passing validation.
Can you think of a way that a javascript file could formally acknowledge that it intends to generate an HTML document, in manner that would circumvent your CDN concerns?
@Lonniebiz
hack that works without passing validation.
The below validates. Comment tokens are allowed before the doctype.
<!-- --><!doctype html><html lang="en"><title>.</title><body><script>
console.log('Hello, World!');
//</script>
Can you think of a way that a javascript file could formally acknowledge that it intends to generate an HTML document, in manner that would circumvent your CDN concerns?
I'm not sure I understand. Obviously we don't want to rely on the payload to tell us whether it's trustworthy.
Any response header that is not generated by existing web servers would suffice. Perhaps Content-type: application/web+javascript
:)
client-side generated html document
I still don't understand why it matters whether the HTMLDocument is created by outer ring JavaScript or by the browser, and I reread your stackoverflow question but found no clues there.
What would letting JavaScript creating the document enable given that it'd have to be via browser APIs that restrict at least some of the document's initial state?
There are parts of the document that cannot be left to outer ring code.
For example, document.origin
has to be monotonically decreasing in privilege over the life of the document and the browser has to restrict its initial value.
If you want something about a document to be mutable to early running code that isn't presently, let's debate that.
I still don't understand why it matters whether the HTMLDocument is created by outer ring JavaScript or by the browser, and I reread your stackoverflow question but found no clues there.
What would letting JavaScript creating the document enable given that it'd have to be via browser APIs that restrict at least some of the document's initial state?
There are parts of the document that cannot be left to outer ring code. For example,
document.origin
has to be monotonically decreasing in privilege over the life of the document and the browser has to restrict its initial value.If you want something about a document to be mutable to early running code that isn't presently, let's debate that.
I'm ok with the browser taking part in the HTMLDocument creation, but I'd like the ability to initiate that creation with client-side javascript in the same manner it gets created using server-side html.
What I wish was possible, is the ability to create a web application without ever leaving my javascript file. I want the ability to write and save a single javascript file, serve it from a web server, and have the web-browser run that file, which then programmaticaly generates the html and css.
Right now, using javascript, on the client-side, you can programmatically generate all the HTML and CSS on a web page except for the HTML wrapper itself . Right now, the only way to create an HTMLDocument is to feed the web browser server-side-html.
So, the closest I can come to achieving my goal, is to hand the web browser a meaninglessly empty html file (to satisfy its 1994 expectations) that either encapsulates or links to my single javascript file.
The whole purpose of my intent is to skip this meaningless middle man: the empty HTML file. If my intent is to generate the HTML using client-side javascript exclusively, then javascript would be the most convenient file encapsulation (not some almost-empty html file designed to satisfy 1994 browser-expectations).
I don't understand all the details of all your concerns, but conceptually it seems likely that many concerns might be handled transparently if the browser would simply treat client-side-generated-html just like it currently treats server-side-generated-html. So, for example, what ever assumptions a browser makes about server-side-html regarding document.origin
, it should make those same assumption about client-side-html that's generated by client-side javascript. The only thing we are changing is the exact source of the html (server-side versus client-side), after the html is produced/received, treat it the same as before.
@Lonniebiz
What I wish was possible, is the ability to create a web application without ever leaving my javascript file. I want the ability to write and save a single javascript file, serve it from a web server, and have the web-browser run that file, which then programmaticaly generates the html and css.
Right now, using javascript, on the client-side, you can programmatically generate all the HTML and CSS on a web page except for the HTML wrapper itself . Right now, the only way to create an HTMLDocument is to feed the web browser server-side-html.
So, the closest I can come to achieving my goal, is to hand the web browser a meaninglessly empty html file (to satisfy its 1994 expectations) that either encapsulates or links to my single javascript file.
Is this a fair restatement of your goal?:
""" A developer can deploy a client-side web application without having to write, check in, or maintain an HTML file. """
If so, none of that requires changing how browsers treat responses from existing servers.
I'm free to configure my server to generate stub HTML using whatever convention I like.
The below treats paths that end in .js.html
as requests for a top-level document where the corresponding .js
file is the start module.
#!/usr/bin/env node
const express = require('express');
const escapeHtml = require('escape-html');
const path = require('path');
const app = express();
const port = 8000;
app.use(express.static(path.join(__dirname, 'static-files')));
// Auto-generate a stub for JS files.
app.get(
/[.]js[.]html$/,
(req, res) => {
const { path } = req;
const jsPath = path.replace(/[.]html$/, '');
res.set('content-type', 'text/html; charset=utf-8');
res.send(`\
<!doctype html>
<html>
<title>Loading</title>
<body>
<script type="module" src="${ escapeHtml(jsPath) }"></script>
</html>`);
});
app.listen(port, () => console.log(`//localhost:${port}`));
Then if I put the JS below in static-files/index.js
and browse to localhost:8000/index.js.html
I get a client-side web application.
document.title = 'Look Ma, no HTML!';
document.body.appendChild(document.createTextNode('Hello, World!'));
export default null;
Is this a fair restatement of your goal?:
""" A developer can deploy a client-side web application without having to write, check in, or maintain an HTML file. """
Yes! You've purified the essence into one sentence and I really like the solution you've provided.
In light of this elegance, promoting a client-side solution is more difficult. My gut tells me that the utility of a client-side solution (to this goal) extends beyond what my imagination immediately brings to mind. However, here's what immediately comes to mind when I compare your server-side solution with my client-side pseudo code:
Portability - With the client-side solution, the file is platform independent, the same file will produce the same page by merely being placed onto any web server; if you move the file from node.js to IIS to nginx to apache, the file produces the same page without the need to perform any url-rewrite code on the underling technology. All that is needed is contained by a single file (independence, encapsulation, portability). As an extreme example, you could even attach this same file to an email, and the recipient could receive a fully functional web application by simply opening it in their browser. That type of portability can be achieved with html, but it wouldn't meet my goal that you refined so well.
I really love Tagged Template literals, I think they are the key to solving the HTML-DOM impedance mismatch that complicates web development today. Long-run, I see myself putting more html into template literals and Tagged Template literals than into static html files. Because of this, my gut tells me there is way more utility in serving javascript files directly (without html wrappers) than I can currently imagine. This is why I started this thread in the unusually abstract way I did. I was hoping that someone else might be able to see something more concretely persuasive than my own speculation. Perhaps there is just nothing to see.
The server-side solution you provided is going to save me mountains of tedious html work. It is amazing how much unnecessary work you can eliminate by tackling a problem at its most fundamental level. Likewise, I think that "not being able to directly navigated to a client-side javascript file that produces its own html wrapper programmatically" is a fundamental lacking. And one day, someone is going to come up with reasons persuasive enough to bring this concept into fruition.
Yeah. The portability is a good point. In my experience, the JS file that initializes the document often is more entangled with the application that more ancillary files.
If you pick an extension (something better than .js.html
that doesn't overlap) and document the server side approach and or the polyglot hack and enough people like it, you'll find it a lot easier later to convince browser vendors to standardize it. The non-overlap is what would allow it to be accepted widely without having to prove backwards compatibility with existing security assumptions, which means that anything that ends with .js
or .mjs
would be a problem.
Bikeshedding: .js.webapp
I also hate the ugly grammer of html, but the browser's responsibilitt is to browse different mimes of resource, NOT JUST text/html. Browser can display image, text, audio, pdf.... Although not friendly, showing the plain text of text/javascript instead of executing it is what things SHOULD be.
Only browsing text/html will load the VM library and DOM, but other resources have nothing to do with the VM.
Maybe in the future, here comes a new mime like webpage/javascript or something, the content of that is pure javascript code or even WebAssembly! Then we can load webpage from javascript directly.
I like this idea, but the DOM Centirc mindset will be hard to break. At their heart, browsers are viewers for "traditional" documents - text files. Over time they have become much more. Writing a form driven application for browers is in many ways like writing applications in MSWord with macros - it's not a "document", but it needs to be crammed into a document. Then we need to deal with document artifacts, like how white-space is handled. The Window should support multiple "viewer" types, one being DOM/window.document. But we could have options like window.svg or window.mediaplayer. In these cases there would be no document
/DOM, another view engine would replace it. The problem is that functionality that could go on window
(having nothing to do with a "actual document") continues to get placed on document
. Being able to start from a non-html source would be a step twords ditching the DOM altogether.
Very interesting discussion. I have recently been developing a system that is javascript-centric. I arrived at it after realizing that javascript was the only container type that could hold all other types meaningfully within a single file, minimizing the need for round-trips to the server. I also discovered that javascript couldn't be used as the outermost container, though, and had to place it in an HTML wrapper file.
Thanks to @WebReflection and @mikesamuel for the clever techniques mentioned here, I'll be evaluating them for my development, in hopes that the javascript-first idea @Lonniebiz is promoting can someday come to fruition in some form. As @mikesamuel pointed out, it would only take a new mime-type and browser recognition of that mime-type to implement it, and, due to mimetypes-from-file-extensions conversion databases, probably a new file extension to avoid migration issues.
What about JS files loaded/imported ? Have they to return the JS string or execute it?
fetch('/index.js').then( res=> res.text() )
This is the same exactly as typing the URL in the adress bar, so what should do the browser? Execute the JS and get the result as text, or get the JS as text to be ( by ex. ) transpiled and executed manually?
The problem of changing a spec is that it have to be retro-compatible with all the infinte already existing behaviors...
A "gessing" algorythm to if the .js is to execute or not is waaaaaay more complicated as telling the browser explicitly to execute it by surrounding the JS by a <script>[JS]</script>
...
So if you want to change world wide web behavior because you don't want to add this minimal 2 tags surrounding your JS, the simplicity principle will say that breaking X behaviors to add only 1 is worthless ...
What you may not understand is that changing this behavior is changing it everywhere, not only for your use case (that is legit ! but in minority)
This is no different that anything else added to the spec. It would begin to work in the latest version of all currently developed web browsers at some point, and the older versions wouldn't be expected to support it. We're not taking anything away here, we are simply adding a feature. I'd love this to be accomplished without adding a new mime-type, but if @mikesamuel suggests that's needed, he's probably right.
Imagine how much extra work is being done right now because of the fact that a browser cannot load a web page directly from a javascript file. It must be html.
Read this: https://stackoverflow.com/questions/55789770/loading-a-web-page-directly-from-a-javascript-file-exclusively
Can you think of anything, right now, that is being developed by standards committees, that could be completely eliminated, if the browser would support client-side javascript as an entry-point to a web page?
Often, people do a lot of work on something because they do not realize a more fundamental flaw, that if corrected, would eliminate mountains of work.
The readers of what I'm writing right now, are abstract thinkers. Just think about what I'm saying....