Remove Index, Follow Tags From Core Page Source

[DonnaScriptTechs]

From @Elshara on March 24, 2017 3:34

(Thanks for reporting an issue! Please make sure you click the link above to view the issue guidelines, then fill out the blanks below.)

What are the steps to reproduce this issue?

1. … Go to any front end page of your site not listed in the administration panel.
2. … Click view page source.
3. … It will display the command index, follow, which is basically a search engine command to ignore all private content and index it anyways as well as follow any external or internal links provided on such pages.

What happens?

… Search engines have back door access to all the items of your website. Even if the htaccess rules are set to block them from doing so. Which they aren't by default.

What were you expecting to happen?

… Some respect to be given to member profile and content privacy. As cached results of pages are out there.

Any logs, error output, etc?

(If it’s long, please paste to https://ghostbin.com/ and insert the link here.)

Paste Log Here

Any other comments?

… I'm kind of surprised at how desperate such a command looks when it is embedded and hidden away in the page source. Not even able to comment it out or anything using custom meta data area in admin panel.

What versions of software are you using?

Operating System: … Windows 10 personal. Linux web server.

SocialEngine PHP Version: … 4.8.13

Copied from original issue: SocialEngine/phpv4-issues#624

[DonnaScriptTechs]

Does adding those pages to the robots.txt work to stop them from indexing?

[DonnaScriptTechs]

From @Elshara on March 30, 2017 2:41

I do not believe so. The pages themselves carry the index follow attributes on them. So even if Google visited the pages in the robots.txt file or any other search engine, they'd be able to view them no problem because of the attributes. It would actually achieve the opposite effect.

On 29/03/2017, DonnaB notifications@github.com wrote:

Does adding those pages to the robots.txt work to stop them from indexing?

-- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: https://github.com/SocialEngine/phpv4-issues/issues/624#issuecomment-290188385

[DonnaScriptTechs]

From @skylucrew on April 2, 2017 2:46

I couldn't find so far any way to add no index and no follow to some page, included custom pages. I think the ability to edit index follow attributes should be a basic feature for a cms like socialengine. Any suggestion please how to to do that?

Thanks

[DonnaScriptTechs]

From @Elshara on April 2, 2017 5:8

It's called per page custom meta data. Social Engine doesn't have the ability to edit this yet. You can add global meta data through the general settings area, which is where the meta tags of such can be placed. You could also do it manually through URL specification if you have an apache server and the appropriate .htaccess file installed per module directory.

On 01/04/2017, skylucrew notifications@github.com wrote:

I couldn't find so far any way to add no index and no follow to some page, included custom pages. I think the ability to edit index follow attributes should be a basic feature for a cms like socialengine. Any suggestion please how to to do that?

Thanks

-- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: https://github.com/SocialEngine/phpv4-issues/issues/624#issuecomment-290960970

[DonnaScriptTechs]

From @skylucrew on April 2, 2017 11:41

Thanks for your reply. Ok about global meta, but that's for general purpose not to set no index e.g. for a specif page, as you told. I couldn't find any instructions on the web about the method you specified through htacess. What do you mean for appropriate .htaccess? What I am trying to do actually is to set no index for a custom html page, not for a module page. Maybe you could help me and provide me some link on how to do that? Yes, I think I do have an apache server. Is there any way to edit the header only for a specif page in social engine and set those attributes in a simple way?

I would appreciate it a lot..;-) Thanks

[DonnaScriptTechs]

From @Elshara on April 2, 2017 15:41

Yes there is a way. I apologize for the confusion. It's on the robots.txt file where you'll want to do the actual editing. The .htaccess file would be for changes you wish to make, say, if you wanted to enforce no viewing or indexing by anyone, not even visitors. To make the robots.txt file change, just specify the path to the page itself. User-agent: *

Disallow: /path/to/file Replace the /path/to/file text with the path of the page file. Save changes and it should be deindexed. You may want to upload your robots.txt results to a robots.txt tester like Google has to see if the changes are successful.

On 02/04/2017, skylucrew notifications@github.com wrote:

Thanks for your reply. Ok about global meta, but that's for general purpose not to set no index e.g. for a specif page, as you told. I couldn't find any instructions on the web about the method you specified through htacess. What do you mean for appropriate .htaccess? What I am trying to do actually is to set no index for a custom html page, not for a module page. Maybe you could help me and provide me some link on how to do that? Yes, I think I do have an apache server. Is there any way to edit the header only for a specif page in social engine and set those attributes in a simple way?

I would appreciate it a lot..;-) Thanks

-- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: https://github.com/SocialEngine/phpv4-issues/issues/624#issuecomment-290981095

[DonnaScriptTechs]

From @skylucrew on April 4, 2017 11:53

Thanks for your reply, but unfortunatly robot.txt it's not the solution. As you can read below from google developers: https://support.google.com/webmasters/answer/93710?hl=en "Important! For the noindex meta tag to be effective, the page must not be blocked by a robots.txt file. If the page is blocked by a robots.txt file, the crawler will never see the noindex tag, and the page can still appear in search results, for example if other pages link to it."

I couldn't find any solution yet. We should find another way to modify the header for a specific page. That's frustrating.

[DonnaScriptTechs]

From @Elshara on April 5, 2017 7:24

That's why I brought the problem up here. I really don't understand why certain parts of SE function so primitively. This is one of them.

On 04/04/2017, skylucrew notifications@github.com wrote:

Thanks for your reply, but unfortunatly robot.txt it's not the solution. As you can read below from google developers: https://support.google.com/webmasters/answer/93710?hl=en "Important! For the noindex meta tag to be effective, the page must not be blocked by a robots.txt file. If the page is blocked by a robots.txt file, the crawler will never see the noindex tag, and the page can still appear in search results, for example if other pages link to it."

I couldn't find any solution yet. We should find another way to modify the header for a specific page. That's frustrating.

-- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: https://github.com/SocialEngine/phpv4-issues/issues/624#issuecomment-291477218

[DonnaScriptTechs]

My question about the robots.txt was to see if that worked at all for now. Not to avoid this issue. Just as a temp fix.

Do you want a setting for admins to choose whether to add the nofollow?

Should it be added automatically to certain pages?

Can you detail any more areas where it should be automatically added or if there needs to be a choice? Perhaps a checkbox on each page to add nofollow.

Please note that I am not a core developer but am trying to get more information for this.

[DonnaScriptTechs]

From @gsf00001 on April 5, 2017 16:46

==>"Should it be added automatically to certain pages? ... Perhaps a checkbox on each page ..."

My thinking is not automatically (I guess you can see this common theme from me based on other threads - I believe very little (nothing?) should be decided for ADMINs. So my vote is for a setting by page in Layout Editor. I'm assuming that part of the function of all this is to prevent private data from being indexed regardless of this possible checkbox/setting?

[DonnaScriptTechs]

From @Elshara on April 10, 2017 23:43

Yes. That's the best use case scenario for this. I would personally opt for an option to include it or not individually. Be it a check box, or just a setting. Would do fine.

On 05/04/2017, gsf00001 notifications@github.com wrote:

==>"Should it be added automatically to certain pages? ... Perhaps a checkbox on each page ..."

My thinking is not automatically (I guess you can see this common theme from me based on other threads - I believe very little (nothing?) should be decided for ADMINs. So my vote is for a setting by page in Layout Editor. I'm assuming that part of the function of all this is to prevent private data from being indexed regardless of this possible checkbox/setting?

-- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: https://github.com/SocialEngine/phpv4-issues/issues/624#issuecomment-291923239

[DonnaScriptTechs]

From @skylucrew on April 12, 2017 18:25

==> Do you want a setting for admins to choose whether to add the nofollow? We need an option to add custom code to the header of each page (in layout editor), just like we can add custom code to general header in admin panel (general settings > Head Scripts/Styles). That's because we need not just nofollow but also other meta tag attributes, especially noindex attribute. This way we could add any code we need to a specific page. WONDERFUL ==> Perhaps a checkbox on each page to add nofollow. This would the last option because you would add only a few meta tag. But if this is the only option at least you should add: index, noindex; follow, nofollow; nosnippet, noarchive, date

==> Should it be added automatically to certain pages? Not automatically, It must be decided by ADMINs

[DonnaScriptTechs]

From @Elshara on April 12, 2017 19:12

I agree. I think this would solve the problem. Having full meta data access for each page. So we can get rid of the default site name and description on all pages and replace it with what the page actually represents.

On 12/04/2017, skylucrew notifications@github.com wrote:

==> Do you want a setting for admins to choose whether to add the nofollow? We need an option to add custom code to the header of each page (in layout editor), just like we can add custom code to general header in admin panel (general settings > Head Scripts/Styles). That's because we need not just nofollow but also other meta tag attributes, especially noindex attribute. This way we could add any code we need to a specific page. WONDERFUL ==> Perhaps a checkbox on each page to add nofollow. This would the last option because you would add only a few meta tag. But if this is the only option at least you should add: index, noindex; follow, nofollow; nosnippet, noarchive, date

==> Should it be added automatically to certain pages? Not automatically, It must be decided by ADMINs

-- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: https://github.com/SocialEngine/phpv4-issues/issues/624#issuecomment-293665874