search

appirio-tech / topcoder-app

topcoder-app is an Angular application that contains dashboard, profile, settings, login/registration, and design submissions.
35 stars 59 forks source link

Open up forums.topcoder.com to robots #933

Open dmessing opened 7 years ago

dmessing commented 7 years ago

We should allow Google to index our forums (obviously still following our rules and regulations

birdofpreyru commented 7 years ago

@ajefts Where do we have the Forum related code? I guess, this issue is a simple modification of robot.txt in there? What are the rules and regulations to follow, is it necessary to still restrict robots access to some info?

ajefts commented 7 years ago

I believe this is the robots.txt that forums is using: https://apps.topcoder.com/robots.txt

But that looks ok...

ajefts commented 7 years ago

I just added apps.topcoder.com to our google webmaster console so we can see what's going on. It appears to be crawling. It might help us to describe the page structure to google in here to help it crawl and index....

birdofpreyru commented 7 years ago

Do you mean to setup an automatic generation of sitemap for the forums, which can be then fed to Google.Webmaster to improve the crawling?

ajefts commented 7 years ago

I connected the apps.topcoder.com domain to the webmaster console and I started to describe the page structure to it with page highlighting. I'll monitor it's indexing activity...

On Wed, Jan 18, 2017 at 3:26 PM, birdofpreyru notifications@github.com wrote:

Do you mean to setup an automatic generation of sitemap for the forums, which can be then fed to Google.Webmaster to improve the crawling?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/appirio-tech/topcoder-app/issues/933#issuecomment-273591443, or mute the thread https://github.com/notifications/unsubscribe-auth/ADorrlRhU7dVwYxzJXRRxrn3adF9GDcOks5rTnVcgaJpZM4LmeRS .