Reduce HTTP Requests - Githubissues

From @Elshara on July 26, 2015 2:41

Social engine supports a system that has a ton of http requests built in referencing every database call, many css and javascript files and other codes depending on each widget that is installed. Not only is it loading php code, but a bunch of other unnecessary files as well, constantly making database requests and slowing down the site. Although stable, it is probably slower than word press on a good day and there are many things that can be improved with this, the solution is more simpler than you may think.

Eliminating the dependent cms aspect to things will speed up things. It isn't just about cache, cache shouldn't be necessary if the files themselves made more server requests than external file requests per a database query or session. Social engine ought to do away with mootools and use jquery or an external javascript library to make things load faster. Do away with zend template engine and install something a lot faster. Twig, smarty, blade, phalcon, jinja, tenjin, vanilla js, mustache, node.js, Nunjucks, ur/web haskell, jade, Dwoo Php, Fry Php, Tornado, Mako, Webpy, Amazon Dynamo, Jango, Pyramid, Erb.
Make blocks more accessible. Currently, any module that has to connect to a database has to do so through several entries at once. If there was a core database file it could simply read user data from at its core and then just add a layout and a controller, it would be much easier than having to specify custom templates.
Include 1 CSS file. A huge percentage of social engine load times relies on images, and there are a lot, especially if you have a lot of entries per module instance let alone content and or its attachments. Make css not inline, so the only css that needs to load is for what page you are actually viewing, reducing the need to have more than 1 global css file. Ideally, there should not be any inline css files to load. Javascript on the other hand should have a global file and then add in ones required for content management as needed. They are a little harder to track down, especially when having to reference specific templates. Reducing the need to have templates, but instead have a global template with different files referencing core functions per instance therefore would generate all the css necessary for that page.
Make code a little less compressed, more simple. Looking at source code such as menus, you see a lot of instances where it references a different type of object located in another file. Independent dependencies slow things down because it loads things from the same page even though the files are actually very different, which is why so many database queries tend to be called. It is very time consuming because the code takes about 100 times as long to download than the actual content, and that includes simultaneously loading several iframes and rss feeds on one page, mainly due to image requests, but also due to extra unneeded file queries from within a given document assuming and depending on the need for a members potential call to action.
This can be proven by looking at the page source code. Generally the line numbers are in the hundreds, but for the page load time, should be several thousand lines in actuality. Gzip compression takes care of this to a certain degree however it makes no difference to how fast the page actually loads. Google averages about 6 to 12 seconds with just default plugins installed and not very much content at all. Most platforms load pages at about the 0.5 second mark, so you can see where social engine is lagging big time, especially when it comes to relying on third party items like a chat to work while the server is constantly processing a lot more http requests than it has to. At peak times, it can even delay a script from refreshing at all, especially when relying on ajax calls to do so like activity feed updates and notifications.
Too many dependency checks resulting in potential zend errors cause problems. Such checks relies on queries to ensure all files are in place before the page can load at all. The more widgets you have, the longer this takes, especially if the widgets are third party plugins. Ironically tags take the most time to load, because that as well as search makes content queries last longer provinding it has to find all available links to compile into a page. This is especially true when referencing things like the members profile information fields. The tags take too long of a query to process so they time out, resulting in no links or a time out to actually process a post once awaiting a response.
Double seo header information exists on all pages. Not only is it loading header information for global seo, but for specific page seo as well and compiling it into one place in the header, even though it is served from one file, as a template, makes extra queries to get the info even though it isn't as fatal as say queries referencing functions users may not wish to perform. Those take the longest, and image caching isn't a priority when this occurs.
Speaking of image caching, the temporary directory never gets deleted or updated. This should be a requirement via cron job and database dump. The reason for this is so that when files do load, a lot more server memory can handle the requests, rather than checking temporary files to see if they are in fact up to date. On first page load, this takes the most resources to complete after timing out.

Copied from original issue: SocialEngine/phpv4-issues#194

From @meetgerry on August 2, 2015 4:21

I second Elshara, when it comes to PgeSpeed or YSlow, SE is a big failure. The above is what any page analysis for SE will show.

From @wtoalabi on August 13, 2015 17:38

And I "Third" this! Please can you guys look into and weave out your magic wand once again! Thanks!!!

From @Elshara on August 14, 2015 3:10

We have to look at the framework in which our sites are hosted on. Doesn't matter whether Nginx or Apache or any cdn for that matter hosting it, where ever the more file transfers has to be made, the site is as slow as a 1995 dial up connection on a good day. Partially is because zend db is relying on just that. Queries to the database any time any action or even a simple page request is made. Trust me this is bad news because if the database is hosted locally on the same server your site is hosted on, and you have a lot of traffic coming in trying to do things not just look, it puts a lot of storage space on the database and your server. Of course the up side to this slowness is 3 main points. First, no matter what you do, you're guaranteed to have the same level of up time and there's 0 possibility for a memory leak. One of the advantages of just using file caching in the first place, you have to rely however, on every single component of your site working correctly otherwise zend will make the entire page if not the entire site where ever the faulty component even if it's a single coding character error is misplaced appear shut down. That's how you know ever request goes through the database first to ensure everything is preloaded. The bad news is, every millisecond this occurs, it sends more http requests to the client and so it makes the site a resource and bandwidth hog not just on social engine, but on mobiles and computers as well. In fact, streaming takes up less bandwidth then the functionality of a social engine website. Never mind the statistics, there's so many seo areas social engine could improve upon but it relies on way too many files let alone database queries for each file loaded half in duplicates the other half in incripted areas to get the job done. No wonder it takes for ever to load someone's home page, especially with third party add ons installed. One coding error could bring your whole site down, and you won't even know where the error originated. That could be as simple as a sintax error or as complex as a whole line of code. This proves the database is being loaded per table instructions and in combination with archaic javascript, it eventually loads the page. Javascript I might add that's been around since social engine 4.0.0 if not earlier. Not that there is an alternative to this mind you, other than to replace the entire framework and database requirements, however should one exist for a php based website, it will be from the following list of alternatives I found that rather takes care of this little horde of problems somewhat nicely.

On 13/08/2015, wtoalabi notifications@github.com wrote:

And I "Third" this! Please can you guys look into and weave out your magic wand once again! Thanks!!!

Reply to this email directly or view it on GitHub: https://github.com/SocialEngine/phpv4-issues/issues/194#issuecomment-130774085

From @Elshara on August 14, 2015 3:55

Here's a list of articles I found relating to this. Specifically alternatives to zend framework. https://free-alternative.net/apps/zend-framework http://www.masterzendframework.com/php/why-kohana-is-an-excellent-alternative-to-zend-framework http://phpframeworks.com/ http://mashable.com/2014/04/04/php-frameworks-build-applications/ http://www.sitepoint.com/best-php-framework-2015-sitepoint-survey-results/ http://www.lynda.com/CakePHP-tutorials/MVC-Frameworks-Building-PHP-Web-Applications/315196-2.html http://www.phptherightway.com/#frameworks https://github.com/codeguy/php-the-right-way/wiki/Frameworks http://www.slimframework.com/ Here's some actual feedback during the production of Zend Framework. Alternative suggestions are scattered throughout comments. http://www.koch.ro/blog/index.php?/archives/108-Why-Zend-Framework-sucks.-Part-1infinite-Exceptions-Missuse.html http://dorkage.net/blog/2009/08/26/alternatives-to-zend-framework/ http://www.alexhudson.com/2012/03/24/a-fond-farewell-to-zend-framework/ Hope this helps. I'll keep looking for some more.

On 13/08/2015, Elshara sourceminded@gmail.com wrote:

We have to look at the framework in which our sites are hosted on. Doesn't matter whether Nginx or Apache or any cdn for that matter hosting it, where ever the more file transfers has to be made, the site is as slow as a 1995 dial up connection on a good day. Partially is because zend db is relying on just that. Queries to the database any time any action or even a simple page request is made. Trust me this is bad news because if the database is hosted locally on the same server your site is hosted on, and you have a lot of traffic coming in trying to do things not just look, it puts a lot of storage space on the database and your server. Of course the up side to this slowness is 3 main points. First, no matter what you do, you're guaranteed to have the same level of up time and there's 0 possibility for a memory leak. One of the advantages of just using file caching in the first place, you have to rely however, on every single component of your site working correctly otherwise zend will make the entire page if not the entire site where ever the faulty component even if it's a single coding character error is misplaced appear shut down. That's how you know ever request goes through the database first to ensure everything is preloaded. The bad news is, every millisecond this occurs, it sends more http requests to the client and so it makes the site a resource and bandwidth hog not just on social engine, but on mobiles and computers as well. In fact, streaming takes up less bandwidth then the functionality of a social engine website. Never mind the statistics, there's so many seo areas social engine could improve upon but it relies on way too many files let alone database queries for each file loaded half in duplicates the other half in incripted areas to get the job done. No wonder it takes for ever to load someone's home page, especially with third party add ons installed. One coding error could bring your whole site down, and you won't even know where the error originated. That could be as simple as a sintax error or as complex as a whole line of code. This proves the database is being loaded per table instructions and in combination with archaic javascript, it eventually loads the page. Javascript I might add that's been around since social engine 4.0.0 if not earlier. Not that there is an alternative to this mind you, other than to replace the entire framework and database requirements, however should one exist for a php based website, it will be from the following list of alternatives I found that rather takes care of this little horde of problems somewhat nicely.

On 13/08/2015, wtoalabi notifications@github.com wrote:

And I "Third" this! Please can you guys look into and weave out your magic wand once again! Thanks!!!

Reply to this email directly or view it on GitHub: https://github.com/SocialEngine/phpv4-issues/issues/194#issuecomment-130774085

From @meetgerry on August 26, 2015 10:37

Wish if someone from the SE team comments something here !

From @Elshara on August 26, 2015 19:14

hmm

On 26/08/2015, meetgerry notifications@github.com wrote:

Wish if someone from the SE team comments something here !

Reply to this email directly or view it on GitHub: https://github.com/SocialEngine/phpv4-issues/issues/194#issuecomment-134940048

From @pulsatemedia on August 26, 2015 19:34

Web performance is a complex, individual and multifaceted thing.

In my experience a core SE install runs very snappy on a properly configured server that's in proportion with the size and activity level of your community. I could be wrong, but I doubt every http request requires a DB query. Each page, maybe depending on the content and exactly how you've cached it. But each http request?

When you add third party scripts and don't take time to remove many of the pointless widgets they provide, that slows things down for sure.

That's not to say that improvements in the core would hurt. But I don't think that's the problem. On the slow SE websites I have worked on, their poor performance is not due to anything in the core. It's more to do with the widgets they have configured and their server setup. Social networks are complex, and very individual in how they are configured and setup. There's no magic fix for performance. Drastic changes to the core brings in a bunch of variables as well as breaking the entire third party developer ecosystem it's a big risk to take.

Remember pagespeed , yslow etc are largely built to show you a best case scenario for a site that is relatively static. ie, wordpress and gives recommendations in that regard. If you have a facebook profile, on firefox use the yslow extension and get a report on your facebook profile page. You might be surprised at the results. When you consider facebook are using bespoke hardware and software as well as HHVM to serve php you'll find that it's not an easy task to present they many objects that cannot be combined in html quickly to the end user. HTTP2/SPDY helps a lot in that regard.

Performance tuning your SE install is just like any other site, but it's more complex. You need to look, in detail at the waterfalls, make changes and iterate. For the most part, the biggest hangup from the core is TTFB. But with the correct server, DB and opcode caching that can be managed quite well down to 0.2-0.5s. The rest is up to the website owner in terms of the modules they install, how they configure them etc.

In short, performance wise, the SE Core is "OK". It's the other variables that are leading to SE site owners down the path of slow performance to the point that visitors are bouncing off their site. At least, that's been my experience.

From @Elshara on August 27, 2015 16:51

No kidding. In short, se should take a look at some of the demo sites of third party add ons. If someone had a hundred scripts running off of one core, they all still load all at once. Now even if that was divided equally say 2 scripts per core, including social engine default add ons, and you had about 3 or 4 different core scripts running, that's more http requests. It's not loading things in just one file, that's including the many css style sheets required for each plugin plus the core of the script itself. Especially when it comes to third party themes, and templates, be very careful of these, as in its current state, se php is delicately balanced by perfect code. As it relies on the database for a verification each time an http request is made, one coding error can throw the whole site down particularly where global variables both in files of the core and global scripts as well as local files of the particularly requested files and its contents are found. Truth is, zend db is one of the most clunky frameworks ever built. It isn't like social engine to build off of ruby on rails. Relying on pure php might make things less complex, but then again, it might not make it just as easy to manage. The core script because of so many http requests, relies on way too many core files to run even a simple page not found error 404 script. This could point to one of two things. Either the core scripting template files are searching for the folder itself making sure it is valid then opening the document while sending more http requests to the database and local server to do this while reading the code just for the page to return its okay signal, or because there's so many themes overriding one another, the code has to be processed which takes so long, the last of every document has to load first before it can be fully received which could make a huge speed decrease in loading times. Either way you look at this major issue, it is unlikely it will be looked at. Due to the increase in page loading problems, many of which have to do with images and high bandwidth processes, the framework itself needs a major over hall. The only good news about all of this is that in the event of a major let down, we still have our handy little ftp details on hand to fix something as well as ssh access for more advanced configurations. Needless to say, core files including themes should only make an appearance if they are needed, not because they simply belong in a global loading center. This points to less control over how widgets are displayed as this should be done from an inline perspective, and more control over what actually belongs to a widget including its direct dependencies which should be localized, not globalized. In order to do this, an alternative framework may have to be put in place, though it may not be as easy as simply providing one through a third party module, simply because it could override the core files of the script itself. Yes, social engine is more complicated to code than it is to implement, unfortunately.

On 26/08/2015, pulsatemedia notifications@github.com wrote:

Web performance is a complex, individual and multifaceted thing.

In my experience a core SE install runs very snappy on a properly configured server that's in proportion with the size and activity level of your community. I could be wrong, but I doubt every http request requires a DB query. Each page, maybe depending on the content and exactly how you've cached it. But each http request?

When you add third party scripts and don't take time to remove many of the pointless widgets they provide, that slows things down for sure.

That's not to say that improvements in the core would hurt. But I don't think that's the problem. On the slow SE websites I have worked on, their poor performance is not due to anything in the core. It's more to do with the widgets they have configured and their server setup. Social networks are complex, and very individual in how they are configured and setup. There's no magic fix for performance. Drastic changes to the core brings in a bunch of variables as well as breaking the entire third party developer ecosystem it's a big risk to take.

Remember pagespeed , yslow etc are largely built to show you a best case scenario for a site that is relatively static. ie, wordpress and gives recommendations in that regard. If you have a facebook profile, on firefox use the yslow extension and get a report on your facebook profile page. You might be surprised at the results. When you consider facebook are using bespoke hardware and software as well as HHVM to serve php you'll find that it's not an easy task to present they many objects that cannot be combined in html quickly to the end user. HTTP2/SPDY helps a lot in that regard.

Performance tuning your SE install is just like any other site, but it's more complex. You need to look, in detail at the waterfalls, make changes and iterate. For the most part, the biggest hangup from the core is TTFB. But with the correct server, DB and opcode caching that can be managed quite well down to 0.2-0.5s. The rest is up to the website owner in terms of the modules they install, how they configure them etc.

In short, performance wise, the SE Core is "OK". It's the other variables that are leading to SE site owners down the path of slow performance to the point that visitors are bouncing off their site. At least, that's been my experience.

Reply to this email directly or view it on GitHub: https://github.com/SocialEngine/phpv4-issues/issues/194#issuecomment-135147855

From @gsf00001 on February 14, 2017 16:29

I just began using SEAO's Minify Plugin which reduced the # .http requests and total size of .js/.css files considerably. My page load times don't seem to be any better (if so, negligible) quite possibly due to the overhead processing for Minify. Not sure if this is a good/bad/neutral tradeoff, but can't tell since my site isn't live yet.

From @gsf00001 on May 5, 2017 15:31

Thanks Donna - glad to see that.

Elshara and pulsatemedia make some excellent points. When I compare my page load speed (with about 150 Plugins including 5-6 Cores and of course multiple .css/.js) to SE's Demo, it's a huge difference (yes, I realize SE is probably using PHP 7.1 with SE 4.9.x and I'm still on PHP 5.6). But SE's Demo is almost instant page loads, my site is about 7 secs for initial page load. Even with reduced # widgets on a page, it's still many many files (but SEAO's Minify has reduced that considerably).

So this may be more of a 3rd-Party thing, but it seems there may be a more efficient way of combining things (like what Minify attempts to do, but not using Minify since it seems the reduced requests is offset by the added server processing so the net load time is negligible). For example - since some 3rd-Party Devs don't use their own core, why do so many others - licensing? If so, that could be handled with changing the entire licensing system (as per other posts).

As other posts also discuss updating libraries to more current (or replacing with more current/better software), my hope is that the entire SE will be looked at from top to bottom and reengineered from the ground up (well, almost). And yes, this is a huge undertaking - one I'd be happy to pre-purchase my 5.0 upgrade for :)

From @pulsatemedia on May 6, 2017 19:37

Performance depends on ton of things. You need to work on front end and backend. On the backend, how many active users you have, and what they are doing has a big impact. The more dynamic their activity is, the less caching is going to help for them because you're hitting the DB. Most SE sites are ghosttowns though, so that's generally not the issue. It's that they have insufficient hosting for even the base install and tack a bunch of plugins on top of that.

Frontend, you need to consider two things:

1) Reduce the clutter of widgets on your pages (Important for UX anyway and most SE sites need a LOT of work here). Be aggressive about removing them. Less is more. This of course helps with DB load too.

2) Use cloudflare, even the free version. You get minification, object caching, IP masquerading as well as all the speed advantages of HTTP2. Don't add yet another plugin to do something that can be done elsewhere.. You can use all CF optimizations except rocketloader. By default, rocketloader will likely cause issues with your JS. If you have the means you can create a custom setup for rocketloader to define specific scripts to include and exclude in optimization.

Backend gets much more complex. Caching (in it's many forms), DB table optimization, DB engine optimization, web server optimization etc etc etc. and are all very individual.

Given how long 4.9 took to come out, I don't see 5.x coming quickly, and then you have to wait for third party vendors to catch up. That takes a while because they are too busy shipping more broken plugins instead of fixing the buggy stuff they have already shipped. The latter is the main reason I've ditched socialengine. Quality is not a focus of the aftermarket ecosystem of SE in my experience. Most of them don't even maintain a changelog.

If your project has a real use case for a social network (Hint a "social network for xxx" is not a value proposition for people anymore, they don't care), SE is still a good bet. But if you want to do much outside of that, and or plan on having content to drive people to your site want something more geared towards that.

I know SE have made the case that SE is not a CMS (Even though they asked us to vote for it as one?!), but the reality is that original content drives traffic, traffic drives (potential) users. If the experience of posting and viewing content is poor, people will bounce.

SE hasn't kept up with the times and the quality in the aftermarket ecosystem has sunk it for me.

SocialEngine / phpv4-feature-requests

Reduce HTTP Requests #8