search

DOAJ / doaj

The Directory of Open Access Journals - website and directory software
Apache License 2.0
55 stars 16 forks source link

Staging URLs appearing in Google search #598

Closed dommitchell closed 9 years ago

dommitchell commented 9 years ago

5th one down. Neither staging nor test should appear in Google

staging

emanuil-tolev commented 9 years ago

Hm, strange. I've no idea how they've indexed that. I can't tell them what to index though :). But I might be able to tell them not to index something, I'll check the various ways of doing that.

Maybe we'll need to put up a basic authentication shield so only humans with a simple user and password combo can see test and staging. I assume this OK with you?

Steven-Eardley commented 9 years ago

Do we use a robots file to determine crawler access? Making test and staging stricter than live should help. http://en.m.wikipedia.org/wiki/Robots_exclusion_standard

emanuil-tolev commented 9 years ago

Don't think we serve a specific one. But yes, this will be the solution unless I can somehow tell Google just not to crawl and never index this and this URL (which we can prove we own). I didn't want to introduce more environment-specific differences in the repo at first if possible :).

emanuil-tolev commented 9 years ago

I've a solution (password-protecting) but my main internet connection's down, currently awaiting repairs, and the mobile is making it very difficult to do anything over the secure link required to implement the password thing. I'll try this tomorrow. (+ I'll need to figure out exactly how to ask google to remove the already indexed pages from the search results, I don't fancy removing them one by one in webmaster tools... they will go away in probably 1-2 weeks after the password protection is in, but still)

emanuil-tolev commented 9 years ago

... this has been obsoleted by the fact staging.doaj.cottagelabs.com no longer works (redirects to cottagelabs.com) due to the new infrastructure slotting into place. We won't have staging for a little while so this should not be an issue. However @dommitchell I'll leave this issue open so we can remember to lock test and staging with a password.

emanuil-tolev commented 9 years ago

@richard-jones added to PVOD project milestone just so we don't forget to do it (and that's likely the next project that will use test and staging servers).

emanuil-tolev commented 9 years ago

646 and #647 take care of the issues that arose here.