Noisy and potentially inappropriate warning to update to 0.2 version of amp-mustache

[AcidRaZor]

I just got called in by exco (who handles the marketing budget on Google, so has access to Search console) who received an AMP validation error (actually just a warning) that amp-mustache says needs to be at 0.2 and not 0.1

This is on our PRODUCTION. I got crapped out from a dizzy height because they thought I made changes that broke our client-facing site.

This is just unacceptable guys! UNACCEPTABLE. Not only do you introduce bugs on LIVE that BREAKS features on our end for 18+ hours, you do this now too??????

You have no idea how, in detail, I had to explain that this is a warning and doesn't break the site at all. I faced immediate dismissal!!!!!!

Jesus Christ, you guys seriously have no chill when it comes to developers do you?

How about this. If I reference 0 point fking ONE, I reference 0 point fking ONE. If it works and support for the component on that version gets deprecated, MY problem. MINE ALONE. It's a choice to use and not upgrade on live until QA/UAT is finished!

What nerve!!!!!!!!

[dreamofabear]

Wow.. I'm sorry that happened to you. It must've been tough to deal with.

AMP only has a few channels of communication with developers with varying levels of coverage, with the AMP validator and search console being key for "push" communication. And as you mentioned, validator warnings have no tangible impact and can be safely ignored.

We do want to encourage developers to adopt new versions of extensions when they are available. Version 0.2 of amp-mustache is faster and has a 40% smaller file size (while also supporting more element types e.g. SVG). Keeping AMP pages in the wild up-to-date is important for our users to make sure they can have the best possible experience on the web.

[jpettitt]

DO NOT UPDATE TO 0.2 IT FAILS VALIDATION

Update: the validation issue has been resolved 0.2 passes validation

See #17138

[cramforce]

We'll follow up with a postmortem on this change.

CC @Gregable for visibility

[Gregable]

Validator rules were updated to simultaneously add support for 0.2 and add a deprecation warning for 0.1. These rules get released at different schedules to different Google products. In this case, Search Console picked up the rules first a few hours ago and began displaying the deprecation warning. The Javascript Validator interfaces, such as the extension you noticed, picked up the change a few minutes ago.

The message in Search Console is a deprecation warning. Both 0.1 and 0.2 versions are supported and will continue to be as long as there is significant usage of 0.1 in documents visible to Google Search crawl. The only case where backwards incompatible changes to validation rules affecting a large number of documents are introduced is for security vulnerabilities.

[Gregable]

For more information on the release skew, please follow https://github.com/ampproject/amphtml/issues/17138

[aghassemi]

@AcidRaZor Although I understand your frustration, this is a professional community with a code of conduct that all participants are required to adhere to.

Strong and inappropriate language not only is in violation of our CoC, it also does not help advance your argument and negatively impacts the community. I would like to ask you to please stick to technical aspects and impact assessment of issues you are facing when raising concerns and void inappropriate and rude language.

[AcidRaZor]

@choumx Thank you for the reply and keeping this topic open for discussion. Indeed the most frustrating and stressful time in any meeting that I've ever been in, and I've been doing this for 18 years now, so have had some insane ones... this one definitely topped all of them. I'm actually thinking perhaps to preemptively start looking for a different position, because I fear this won't be the last time this happens and I'm looking like I'm making excuses for failure.

The only thing your documentation says on ampproject.org (other than the reduction in size and support for svg) is "Internal improvements may cause minor breaking changes; we recommend testing your pages before adoption". We're programmers, not the marketing team clicking and dragging objects. More clarity is always welcome. I can read technical documents, changes and code. I'm sure most programmers can.

I'm subscribed to Google Chrome Developers as well as The AMP Channel on youtube (https://www.youtube.com/channel/UCXPBsjgKKG2HqsKBhWA4uQw/videos). No announcement. Nothing. I watch youtube at home every day to catch up on development stuff, you had the perfect opportunity to use your "advocates" to do a 2 minute video to inform of the upcoming change.

My suggestion would be to perhaps use some of your AMP channels, like ampbyexample.com or ampproject.org, where there's easy-to-reference notes for everyone to reference on a Blog post on the homepage of some kind to announce changes.

I would have loved being able to point to a piece of content with the 0.2 upgrade / deprecation warning of 0.1 so I could show exco that it was indeed Google at the helm of the changes and not me. But announcing changes shouldn't automatically translate into immediate deprecation warnings for everyone who uses amp-mustache 0.1 either... right?

I know you would like people to adopt the newer version as soon as possible with multiple nice benefits, but there needs to be a time-period between announcing changes, to deprecation warnings. I really don't know how you do it in your environment, but pushing out deprecation warnings before any documentation or announcements made live, or giving developers a chance to disseminate the information and test on their own QA/UAT environments first, would have been way more helpful than what happened.

So think of the channels and how each knocks on to the next to come up with a roll-out plan that doesn't "push" notifies developers via their boss' boss when he gets e-mailed errors, thinking the site is broken and they're wasting money on ads (I don't get access to Search console, now they have less reason to give permission)

I realize you guys are trying to be agile and super-awesome, but there comes a point of adoption in any new technology where you can't just afford to push "whatever" live and expect immediate adoption and widespread use, especially, after months, of development to get a production environment ready that adopts AMP at it's core, and not just a "slap on add-on" to try and benefit from the Cache to get better rankings. In Africa, AMP is very useful considering the huge cellphone usage.

I am sure you will have tons of developers willing to sign up to a be "beta" testers on a branch they can push their live environments to if they feel brave, so that those souls can be cannon-fodder for the rest before the proverbial crap hits the fan. You guys have proven yourself quick on the draw when it comes to live bugs, imagine not having that additional stress to know that bug is still in the beta-phase before it hits live? Then slow down the rate of what you release changes at, and give us the option to adopt the newer versions. It would really not have taken my testers long to approve the 0.2 change, provided the minor breaking changes internally didn't affect our implementation....

Hope this makes sense? English isn't my first language.

@aghassemi Thank you for understanding my frustration; If you made me lose my job, I can't sue for damages or compensation, we do not work together, and I censored my curse words (** does not magically appear by themselves). I had a point to make. I made it.

Unfortunately I'm not some kind of big corporate business you can schmooze over with big shiny promises from Google. This is my day job and I am employed by someone not affiliated with Google, pays my salary, and holds me accountable, not you.

So frankly, I really can't be arsed to be professional when my livelihood is threatened like this. I'd much rather put my family first, not Google or beautifying my speech so that you're not offended. Feel different? Cool, I will stop using AMP and actively advocate against the technology. Force new developer's to learn all optimization techniques, DevOps or otherwise, to be able to push out sites just as fast (if not faster) for mobile.

How unprofessional is it for a company to push code live, that breaks everyone else's sites who may implement it, enough for e-commerce shopping to grind to a halt? Google thinks it's okay.

I'd rather curse and be unprofessional than have the axe hang above my head regarding this issue that I BARELY turned around in my favor on Friday.

But at Google you guy's don't care... right... as long as everyone is professional, it doesn't matter who loses their jobs? Cool.

If I were, at any point lucky enough to just fall into another job, maybe I wouldn't have cared. Told exco to f**k-off and ask @aghassemi on github why they're getting these warnings. Unfortunately I'm not even in a 1st world country, nor do I exhibit such disrespect to my superiors, so I can't even begin to stress to you how this would have impacted my beautiful wife and 2 year-old son would have been.

Why treat this as some kind of alpha "we push anything live because we need to impress the big boys to adopt this technology" while your snake-oil salesman go from conference to conference, convincing big corporate that AMP should be their focus and it is stable enough to develop in?

0.1, sure... indicates we're not even in a 1.0 release. I will point that out in my next Dev funnel meeting, and seriously put to table, even if it means I have to work for free, overtime, to move technology away from anything Google controls, because the "Caching" benefit doesn't outweigh my own job-security, sanity or ability to choose when, and which features, are pushed to live at any point in time.

I don't have to explain myself to a bunch of people who shows clear disregard for other developers and systems and processes everyone (mostly) follows in the I.T world these days. If I were some hobbyist trying to implement this on my own site, whatever. Push changes to live while drunk and doing blow off a hooker's ass for all I care. But this isn't a personal project.

I haven't had this many things go wrong on a project in a very long time. Not even my junior's can push to live directly. Yet, here we are, with Google just fumbling/falling over each other as if it's some kind of free-for-all to "push stuff live" competition where validation's lag behind Google search sending out deprecation warnings while documentation isn't even updated or anything announced on any formal channels. Did you guys turn into Microsoft, and Microsoft into how you used to operate somehow?

And if @Gregable remarks on the process is correct (which I'm sure it is), you should perhaps think of how you want to get information out to those developers who gives you the benefit of doubt and implements this technology world-wide on production systems (which I have suggested). It shouldn't push breaking changes or warnings to their live systems please? Or give us the ability to choose whether or not a version upgrade will happen right the very second you throw down your god-hammer and decided "it will be so" or not.

You do realize developers, programmers, the guys who build web applications, are responsible for implementation of these technologies, correct? So why shy away from detail? Sure, I can spend most of my time just monitoring github and constantly check if anything changed, but monitoring what you do and how you run this project, isn't my job.

You aren't developing ANY of this for the man-on-the-street that needs a GUI to drag-and-drop pretty pictures to their websites. If that is your end-goal, holy crap are you approaching it all wrong. Your time would be better spent writing some kind of engine that automatically takes any html input, and spits out AMPified components to make everything load faster, instead of different components for other developers to implement.

Again. Thanks for understanding, but I don't think you realize the gravity of what happened. These kinds of things, internally to Google, may be acceptable. But not when it affects me, my lively hood or my ability to take care and provide for my family.

[jpettitt]

We are aware the search console may still be flagging the 0.2 version as an error in some cases. This can be safely ignored and we're working on getting the messaging fixed.

[Gregable]

@jpettitt Search Console never flagged 0.2 as an error since the original change landed which added support for 0.2. There is no issue there.

Search console began flagging 0.1 as a deprecation warning July 27. This is also correct, as 0.2 became the recommended version at that time.

UPDATE: I was unaware of a separate issue when I posted this reply.

[Gregable]

@AcidRaZor, no breaking changes were pushed. AMP validation went from supporting one version of amp-mustache two versions, including the version that was previously supported. This change was strictly backwards compatible.

[craigscott29]

In light of all this, what would be the odds of offering different ways to link to the amp JS files?

Example: amp-mustache-latest.js and amp-mustache-stable.js

Latest being the the latest in-progress version, stable being the latest stable release (ie; v0.2 in this case).

I would 100% run latest in my sandbox and stable on my main website. Obviously leave the option open for people to hard write in the version so as to vet that change themselves. Maybe completely unnecessary, but just a thought. Myself and one person other manage our website and we can jump on things pretty quickly so I'm not too concerned about letting something like a -latest or -stable take control of the file versioning for me.

[Gregable]

@aghassemi for a better answer.

@craigscott29 , are you aware that -latest already exists? I think the answer is that -latest is always a stable version, it's not used as a beta channel mechanism.

[aghassemi]

@craigscott29 we already support -latest however when we do a version bump, it pretty much always carries breaking changes with it (since we mostly make backward-compatible changes in the stable ever-green 0.1 version) so it is quite risky to use -latest in production.

[craigscott29]

Thank you @Gregable and @aghassemi for the explanation. I thought I had seen -latest in my Google travels late last week but then could not find it again so I chalked it up to some end of the week losing it.

[AcidRaZor]

@Gregable The powers that be didn't see it as a warning. Not sure how client-facing you are, but normal people (with no I.T background and no inclination to learn) sees a warning as an error and vice verse. They just care that something is wrong and will affect their bottom line.

The changes that have gone live before, breaking functionality, was with navigateTo. Also had to explain it wasn't an error introduced by myself or my team, and Google was already on-top of fixing it.

Targeting specific versions would help, but not if it stays at 0.1 forever, as it would be difficult to know that 0.1 was updated with changes/bug fixes or not. I'm pretty confused as to why it seems there is no standard followed here. I've used Semantic versioning (Major.Minor.Patch) since early 2000.

If "latest" or "stable" won't work, at least go, 0.1.20180803?

We could then periodically check changes on github or any of the websites, youtube channels or even https://amphtml.wordpress.com/ which I found today, that also doesn't mention anything about 0.2 mustache.

Based off of the changes/fixes/upgrades, we could then target 0.1 which will always give us the latest version/build or 0.1.20180501 to target a previous, stable and tested build on the systems we implement AMP on.

A rolling window of 12 months could be applied to any older version being deprecated, and Search console will be PERFECT to notify us of any pages or entire sections of our site that needs to be looked at, in case we haven't looked at it in THAT long.

It will definitely reduce the noise and increase peace-of-mind for the developers that implement your technology, to be able to target a previously known stable build for their project, which gives them time to make the necessary changes to be compatible with future ones.

Thank you for being part of this conversation.

[Gregable]

@AcidRaZor We are looking at a few areas in Search Console's interface that may cause unnecessary confusion over what is a warning and what is not. In particular, my understanding is that the email sent to you simply showed "New AMP Issues detected..." while the Search Console UI is clearer on the type (warning vs. error) and detail of the issue.

If you have other feedback that we can bring to the Search Console team to help make these warnings clearer in the future, please feel free to share that with us or Google Search Console directly.

Since publishers reference the version number directly in their own pages, those versions are major versions only, so that publishers do not need to update all documents on a weekly basis, for example. You can use -latest on all extensions, but it is not recommended for production since you may pick up breaking changes without realizing it.

The deprecation warning does not mean that the version is unsupported. We use this mechanism to communicate the start of the deprecation process, not the end of it. Older versions of extensions will not be removed if they are used by greater than 0.1% of crawler accessible AMP pages.

[dreamofabear]

As part of the post-mortem action items, we've updated and clarified our versioning policy in #17370.

Moving forward, all extension version deprecations will follow a strict deprecations process including announcement on public channels, community discussion, and constrained timelines to deprecation/invalidation. For example:

A version must not be deprecated until a new version is released and stable for at least 1 month. A version must not be invalidated until it has been deprecated for at least 1 year.

Closing this issue. Sorry again for the alarm and thanks for your patience and engagement with the AMP community.

[futwick]

AMP only has a few channels of communication with developers with varying levels of coverage

I'd hoped to find some RSS feeds to notify us when new version of tags become available. A feed for each tag (so I can subscribe to just the tags I use) and/or a combined feed containing updates for all tags. Alternatively an API which returns latest and stable version numbers would suffice.