search

iamSahdeep / seo_renderer

A Flutter Web Plugin to display Text Widget as Html for SEO purpose
MIT License
114 stars 15 forks source link

Google punishment chances? #1

Open denisviana opened 3 years ago

denisviana commented 3 years ago

There is any chance of a punishment from Google if I use this package?

iamSahdeep commented 3 years ago

There should not be, I have seen some comments on issue as well that people are using this kind of technique to make a workaround. For example in this comment

They are entirely using a different template for their website and seems like working for them so far and we are not even changing things entirely. So if we are not manipulating the data and using it as it should be there should be no consequences.

But to be on safer side I have asked the same at google SEO help community. Let's see what input they will provide. https://support.google.com/webmasters/thread/115430508/will-this-be-considered-cloaking-or-anything-which-may-lead-to-penalising-my-site?hl=en

denisviana commented 3 years ago

Thank you.

If I can ask another question... Is there any downside to using it?

iamSahdeep commented 3 years ago

downside as in?

denisviana commented 3 years ago

downside using this package... Any important limitation, any case where it should not be used?

iamSahdeep commented 3 years ago

Honestly, It's hard to tell right now as it's still in development, there will a lot of use cases which will come soon, limitations would be visible at that time. Apart from it I would recommend to simply just try this out for some time on any other non important projects in terms of SEO.

hazarbelge commented 2 years ago

The post in the Google SEO Help Community is locked right now. Are there any new information about the situation? I think it is safe, but to be safer I've decided to wait.

Levi-Lesches commented 2 years ago

Unfortunately, this package seems to qualify as "cloaking" under the Google SEO guidelines and can get your app/site heavily penalized or even removed from Google Search. Some quotes from Google's Cloaking Guidelines:

Cloaking is considered a violation of Google's Webmaster Guidelines because it provides our users with different results than they expected. Some examples of cloaking include:

  • Serving a page of HTML text to search engines, while showing a page of images to users

Cloaking is essentially showing different content to users than Googlebot

If you were to look through the code of your app or web server, would you find something that deliberately checks for a user-agent of Googlebot specifically or Googlebot's IP address specifically?.... That's the stuff that's high-risk.

Are you treating users roughly the same way as you're treating Googlebot?

The code here violates these principles, as it explicitly checks the user-agent to check for Googlebot/other scrapers and shows different HTML tags based on their presence. https://github.com/iamSahdeep/seo_renderer/blob/2e2a3e25e3f4d73a603301911f22ca3aba1f42b3/lib/helpers/robot_detector_web.dart#L24-L38

While it may seem like essentially adding "alt-text" to your Text widgets, to the Googlebot you are returning completely different HTML content, and that is the crux of the issue,

However, simply checking for Googlebot in the user-agent isn't enough to be considered cloaking. From Google's page on Dynamic Rendering:

Dynamic rendering is not cloaking

Googlebot generally doesn't consider dynamic rendering as cloaking. As long as your dynamic rendering produces similar content, Googlebot won't view dynamic rendering as cloaking.

When you're setting up dynamic rendering, your site may produce error pages. Googlebot doesn't consider these error pages as cloaking and treats the error as any other error page.

Using dynamic rendering to serve completely different content to users and crawlers can be considered cloaking. For example, a website that serves a page about cats to users and a page about dogs to crawlers can be considered cloaking.

This means you can use some pre-rendering service to pre-render your page and display roughly the same content you would for a human to Google's crawlers.

ubergaijin commented 2 years ago

I'm so confused by Google Search Console recommendations, they are quite vague and even contradict themselves. On one hand they say (in their video on cloaking) that even the hash of the page should be identical for a bot and a human agent, and your code shouldn't try to identify that the agent is a bot by any mean... and on the other hand they say that dynamic rendering is not considered cloaking as long as the content is "similar". What does qualify as similar? Where do they draw the line and who is doing it? Is it googlebot or do they do some manual audit to compare the pages? Does a text only html page with no formatting counts as "similar content" as long as the same text is displayed on both versions?

iamSahdeep commented 2 years ago

@Levi-Lesches Thanks for the input, but as @dossiman stated, there is still a lot of confusion on the penalization part.

Also we are not showing Dogs instead of Cats xD, well that depends on us, developers itself. It also depends on us how much similar we are making these divs similar to our actual content, showing every image/text/link etc on the page. Not sure how much points it add to not consider this as cloaking.

Levi-Lesches commented 2 years ago

What does qualify as similar?

I imagine this is to protect cases where the page updates in real-time (such as ads or stats). In this case, every query of the page will produce slightly different content, be it a user or bot. As for extent, I'm sure that as long as the structure of the page remains the same, it is okay to change the content within.

Is it Googlebot or do they do some manual audit to compare the pages?

Perhaps some cases and appeals are reviewed manually, but given the scale of the internet, you have to assume most of this is automatic. That's why I believe it's safe to assume the content can change slightly but the overall structure has to be the same since the bot isn't going to put that much effort into analyzing the page at such a high level. In any case, given how the content is rendered (on top of the original), it would probably trip up the OCR in some cases.

Regardless of whether a page with this package may slip under the cracks, the official answer given by Google is that changing the content of the page can lead to possible cloaking penalties. Adding elements to the DOM changes the content of the page (see the image on the README) and is not rendered the same as Flutter does. That is very different from dynamic rendering, which renders all the elements exactly as a user would see it before sending it to the crawler. Additionally, more complex layouts such as alignment and coloring cannot be accurately represented using this package, which further distances this from dynamic rendering.