search

gbif / portal-feedback

User feedback for the GBIF API, website and published data. You can ask questions here. 🗨❓
30 stars 16 forks source link

Deleted datasets show up in google searches #5423

Closed gbif-portal closed 2 months ago

gbif-portal commented 2 months ago

Deleted datasets show up in google searches

There are 2 IRMNG datasets showing up in google searches, one of them this deleted dataset since 2016.

The page already contains:

Maybe there is sth else that can be done to prevent google from indexing deleted dataset pages?

User: See in registry - Send email System: Safari 17.5.0 / Mac OS X 10.15.7 Referer: https://www.gbif.org/dataset/05b2f719-f3df-4e60-9e69-3060d9f5f950 Window size: width 1759 - height 540 API log&_a=(columns:!(_source),filters:!(),index:'3390a910-fcda-11ea-a9ab-4375f2a9d11c',interval:auto,query:(language:kuery,query:''),sort:!())) Site log&_a=(columns:!(_source),filters:!(),index:'5c73f360-fce3-11ea-a9ab-4375f2a9d11c',interval:auto,query:(language:kuery,query:''),sort:!())) System health at time of feedback: OPERATIONAL

mdoering commented 2 months ago

try searching for IRMNG and GBIF

ahahn-gbif commented 2 months ago

@mdoering, this is not a bug as such: the page does exist, as we do delete the data, but not the metadata of deleted datasets. This is by design, so that existing links do not just go out of scope. We would need to inform search indexes to omit specific pages, but have not looked into this so far

mdoering commented 2 months ago

Yes, I didn't want to challenge the logical deletion and keeping tombstones. It is just that these deleted datasets show up in google searches and have confused users - we should tell google not to do so.

mdoering commented 2 months ago

@MortenHofft @thomasstjerne could we easily return a HTTP 410 Gone response in the response of the deleted pages?

MortenHofft commented 2 months ago

We return a no index no follow. <meta name="robots" content="noindex nofollow"> If google index it then they aren't playing nicely. Asking google and others to not even look at it will, according to their documentation, mean that the page WILL show up in search results, because they might know of its existence from other pages linking to it. So they will just show a link with no description. But since they aren't allowed to look at it, then they won't read the noindex tag.

Perhaps they have changed guidelines. Or perhaps they just do what they want.

The sitemap is generated using our search API so it shouldn't be their either since that does not include deleted datasets

MortenHofft commented 2 months ago

Reading above comment I see it. There is a typo!! 😱 there is a space instead of a comma. Obviously that is unreadable by anyone. That is probably the reason. It should be <meta name="robots" content="noindex,nofollow">

I'm deploying and asking google to reindex the page

MortenHofft commented 2 months ago

It is correct now, but it might take a while for Google to notice. I've tested that one page and it behaves as expected in the search console