Use progressive enhancement for the frontend

[alexmuller]

I noticed that some parts of the frontend, like the header links / login status only work when JavaScript is enabled because they're using client-side includes.

I think Warehouse would be faster and more resilient to failure if it was built using progressive enhancement.

Would you welcome pull requests that reduce the dependence on JavaScript when it's not needed? Is there a reason for the existing approach, like not wanting every request to need a session?

[alexmuller]

I've just realised that PyPI is a heavy user of Fastly which explains the decision to rely heavily on caching at the CDN. I also found some related comments on #979.

I'd still be interested in trying to find a way to solve these issues while providing a good experience for all users. I'll keep having a think.

[alexmuller]

Ah I've also just found #923 so I'd love to know what didn't work with edge-side includes :(

Sorry for the number of comments!

[dstufft]

If I remember correctly, and I could be wrong, the issue we had with ESIs were:

• It was difficult to emulate them correctly in local development, meaning we had errors that only appeared after the site was deployed, which wasn't an ideal situation.
• It ironically broke the ability to cache pages on the clientside correctly. The top level http request (the one that contained the ESI tags to other resources) could not include any content or be influenced by the ESI included resource other than by simple inclusion. This meant that the CacheControl header would be incorrect for this page, as well as the ETag header IIRC. (Since the Etag header didn't get updated and the browser saw the entire page as a single object, but only got the CacheControl header for the top level page).
• Pages that had ESI enabled were not able to be compressed and had to be served to end clients uncompressed. For smaller sites this probably wouldn't be an issue, but PyPI serves >1PB of traffic a month, and although the bulk of that is not the UI, every little bit helps.

The combination of these factors meant that the use of ESI here made the site as a whole less scalable since it effectively ruled out client side caching and compression for most UI pages and made it harder to actually develop the site.

The HTML include is sort-of "progressive", in a not really progressive kind of way. In that I mean that the pages work for their primary use case without JS, they just lose the extra user specific bits that get mixed in on each page. If I remember all of the places this pattern has been used, the most painful of these would be the lack of a Login/Logout link in the top header. It's possible we could do something better here. Perhaps a fallback to the logged out view if Javascript is disabled or the network request to fetch the snippet fails? That would provide a not as good experience as JS (since they'd appeared logged out, even if they were logged in) but they'd get a button that they can press to login, and ideally that could possible take them to some page that is naturally user specific, so doesn't use JS to handle the session using bits.

That's a lot of words to basically say, I think we'd be happy to look at options here to either reduce the use of JS or provide better fallbacks than silently do nothing. However the bar is pretty high, and I'm personally not to fussed about requiring logged in users to have JS enabled (even modern screen readers and the like can handle JS just fine) but that doesn't mean we can't do better!

[alexmuller]

Thank you very much for taking the time to write a really detailed reply, I know you're busy.

I'll have a think and maybe after launch have some more concrete suggestions for ways to improve this.

[brainwane]

@alexmuller, Thanks for suggesting this.

Update, in case you didn't already know: The folks working on Warehouse have gotten funding to concentrate on improving and deploying Warehouse, and have kicked off work towards our development roadmap -- the most urgent task is to improve Warehouse to the point where we can redirect pypi.python.org to pypi.org so the site is more sustainable and reliable.

Since this feature isn't something that the legacy site has, I've moved it to a future milestone. But we're interested in your suggestions. Also, we'll be at the PyCon North America sprints in May in Cleveland and we're hoping to also have a Warehouse sprint at EuroPython in July in Edinburgh, and that would be a great time to talk about it in person. And of course you can also talk with us in this issue, or on #pypa-dev on Freenode, or on the pypa-dev mailing list.

Thanks and sorry again for the wait.