search

superfeedr / documentation

The Superfeedr Documentation
http://documentation.superfeedr.com
15 stars 8 forks source link

Pull requests from site every 5 to 10 minutes #90

Closed 7thcirclegit closed 1 year ago

7thcirclegit commented 8 years ago

I manage numerous sites and my logs are showing polling every 5 to 10 minutes. I submitted a request through the support form, but no response.

7thcirclegit commented 8 years ago

I guess everyone took time off with the sale and all http://myinforms.com/en-us/a/35437359-medium-acquires-superfeedr-fights-for-open-web/

julien51 commented 8 years ago

Hey... sorry about the delay! The sale did not help but mostly I did not get an email from github about this issue :/ sorry about that!

Please tell us more. Are we causing trouble by polling too often? Do you mind sharing the feed URL?

7thcirclegit commented 8 years ago

I have several feeds, some are incorrect (http instead of https), and others are just pulling quite often. I am happy to have them indexed, and would even be happy to have access to configure the below feeds and my others that are not being queried currently. All should be https, and if you need I can send log files with timestamps.

www.7thcircledesigns.com feed-id:172109348 modelblog.2hypeenterprises.com feed-id:176106148 www.attogproductions.com feed-id:169696374 photoblog.ericleuthardt.com feed-id:176113404 www.fotografiadellalucerossa.com feed-id:179754460 cozyegg.cozyeggdesigns.com feed-id:195292062 cozystyle.cozyeggdesigns.com feed-id:195293170

julien51 commented 8 years ago

Thanks. Unfortunately, even if we changed the URLs on our end our customers would subscribe to the non-https url again. You should really setup a redirection on your end to avoid losing SEO juice.

7thcirclegit commented 8 years ago

I have the redirects in place and they are working, but with the number of queries x2 for the redirects it is excessive.

7thcirclegit commented 8 years ago

Here is an update of the logs from 1 site for this morning.

198.58.103.160 - - [22/Jun/2016:07:02:24 -0500] "GET /feed/ HTTP/1.1" 301 211 "-" "Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touch - feed-id:179754460" 198.58.102.96 - - [22/Jun/2016:07:13:46 -0500] "GET /feed/ HTTP/1.1" 301 211 "-" "Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touch - feed-id:179754460" 198.58.102.96 - - [22/Jun/2016:07:25:12 -0500] "GET /feed/ HTTP/1.1" 301 211 "-" "Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touch - feed-id:179754460" 198.58.103.114 - - [22/Jun/2016:07:36:47 -0500] "GET /feed/ HTTP/1.1" 301 211 "-" "Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touch - feed-id:179754460" 198.58.102.95 - - [22/Jun/2016:07:44:41 -0500] "GET /feed/ HTTP/1.1" 301 211 "-" "Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touch - feed-id:179754460" 50.116.28.209 - - [22/Jun/2016:07:52:41 -0500] "GET /feed/ HTTP/1.1" 301 211 "-" "Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touch - feed-id:179754460" 198.58.103.114 - - [22/Jun/2016:08:01:04 -0500] "GET /feed/ HTTP/1.1" 301 211 "-" "Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touch - feed-id:179754460" 198.58.103.160 - - [22/Jun/2016:08:09:09 -0500] "GET /feed/ HTTP/1.1" 301 211 "-" "Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touch - feed-id:179754460" 50.116.30.23 - - [22/Jun/2016:08:17:14 -0500] "GET /feed/ HTTP/1.1" 301 211 "-" "Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touch - feed-id:179754460" 198.58.103.114 - - [22/Jun/2016:08:25:15 -0500] "GET /feed/ HTTP/1.1" 301 211 "-" "Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touch - feed-id:179754460" 198.58.103.28 - - [22/Jun/2016:08:33:16 -0500] "GET /feed/ HTTP/1.1" 301 211 "-" "Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touch - feed-id:179754460" 198.58.96.215 - - [22/Jun/2016:08:41:17 -0500] "GET /feed/ HTTP/1.1" 301 211 "-" "Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touch - feed-id:179754460" 198.58.103.36 - - [22/Jun/2016:08:49:15 -0500] "GET /feed/ HTTP/1.1" 301 211 "-" "Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touch - feed-id:179754460" 198.58.103.28 - - [22/Jun/2016:08:57:16 -0500] "GET /feed/ HTTP/1.1" 301 211 "-" "Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touch - feed-id:179754460" 198.58.103.36 - - [22/Jun/2016:09:05:35 -0500] "GET /feed/ HTTP/1.1" 301 211 "-" "Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touch - feed-id:179754460" 198.58.102.49 - - [22/Jun/2016:09:13:32 -0500] "GET /feed/ HTTP/1.1" 301 211 "-" "Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touch - feed-id:179754460" 198.58.99.82 - - [22/Jun/2016:09:21:45 -0500] "GET /feed/ HTTP/1.1" 301 211 "-" "Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touch - feed-id:179754460" 198.58.102.95 - - [22/Jun/2016:09:29:50 -0500] "GET /feed/ HTTP/1.1" 301 211 "-" "Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touch - feed-id:179754460" 198.58.102.96 - - [22/Jun/2016:09:37:54 -0500] "GET /feed/ HTTP/1.1" 301 211 "-" "Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touch - feed-id:179754460" 198.58.96.215 - - [22/Jun/2016:09:50:01 -0500] "GET /feed/ HTTP/1.1" 301 211 "-" "Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touch - feed-id:179754460" 198.58.103.91 - - [22/Jun/2016:09:58:00 -0500] "GET /feed/ HTTP/1.1" 301 211 "-" "Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touch - feed-id:179754460" 50.116.30.23 - - [22/Jun/2016:10:07:03 -0500] "GET /feed/ HTTP/1.1" 301 211 "-" "Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touch - feed-id:179754460" 198.58.96.215 - - [22/Jun/2016:10:14:58 -0500] "GET /feed/ HTTP/1.1" 301 211 "-" "Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touch - feed-id:179754460" 198.58.99.82 - - [22/Jun/2016:10:23:18 -0500] "GET /feed/ HTTP/1.1" 301 211 "-" "Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touch - feed-id:179754460" 198.58.102.158 - - [22/Jun/2016:10:31:24 -0500] "GET /feed/ HTTP/1.1" 301 211 "-" "Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touch - feed-id:179754460" 50.116.30.23 - - [22/Jun/2016:10:39:19 -0500] "GET /feed/ HTTP/1.1" 301 211 "-" "Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touch - feed-id:179754460" 198.58.103.36 - - [22/Jun/2016:10:47:19 -0500] "GET /feed/ HTTP/1.1" 301 211 "-" "Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touch - feed-id:179754460" 198.58.102.155 - - [22/Jun/2016:10:55:27 -0500] "GET /feed/ HTTP/1.1" 301 211 "-" "Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touch - feed-id:179754460" 198.58.102.158 - - [22/Jun/2016:11:04:22 -0500] "GET /feed/ HTTP/1.1" 301 211 "-" "Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touch - feed-id:179754460" 50.116.30.23 - - [22/Jun/2016:11:12:27 -0500] "GET /feed/ HTTP/1.1" 301 211 "-" "Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touch - feed-id:179754460" 198.58.103.160 - - [22/Jun/2016:11:20:44 -0500] "GET /feed/ HTTP/1.1" 301 211 "-" "Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touch - feed-id:179754460" 198.58.102.156 - - [22/Jun/2016:11:28:39 -0500] "GET /feed/ HTTP/1.1" 301 211 "-" "Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touch - feed-id:179754460" 50.116.30.23 - - [22/Jun/2016:11:36:50 -0500] "GET /feed/ HTTP/1.1" 301 211 "-" "Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touch - feed-id:179754460" 198.58.102.96 - - [22/Jun/2016:11:44:49 -0500] "GET /feed/ HTTP/1.1" 301 211 "-" "Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touch - feed-id:179754460" 198.58.103.92 - - [22/Jun/2016:11:52:48 -0500] "GET /feed/ HTTP/1.1" 301 211 "-" "Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touch - feed-id:179754460" 50.116.30.23 - - [22/Jun/2016:12:01:03 -0500] "GET /feed/ HTTP/1.1" 301 211 "-" "Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touch - feed-id:179754460" 198.58.103.28 - - [22/Jun/2016:12:09:04 -0500] "GET /feed/ HTTP/1.1" 301 211 "-" "Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touch - feed-id:179754460" 198.58.103.28 - - [22/Jun/2016:12:17:03 -0500] "GET /feed/ HTTP/1.1" 301 211 "-" "Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touch - feed-id:179754460" 198.58.102.96 - - [22/Jun/2016:12:24:59 -0500] "GET /feed/ HTTP/1.1" 301 211 "-" "Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touch - feed-id:179754460" 198.58.102.158 - - [22/Jun/2016:12:33:00 -0500] "GET /feed/ HTTP/1.1" 301 211 "-" "Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touch - feed-id:179754460" 198.58.102.96 - - [22/Jun/2016:12:44:58 -0500] "GET /feed/ HTTP/1.1" 301 211 "-" "Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touch - feed-id:179754460"

7thcirclegit commented 8 years ago

I see that the fix for me reporting this issue was for you to update the system to not send a user agent. I also see that you must be on vacation again since it has been almost 3 weeks with no response.

julien51 commented 8 years ago

Eric, I am very sorry for the delay. I was not on vacation but just crawling under a huge list of TODO things as well as trying to move halway around the glob with 2 young kids :(

That said, we did not change the User Agent at all. Can you clarify things?