search

wpsharks / comet-cache

An advanced WordPress® caching plugin inspired by simplicity.
https://cometcache.com
GNU General Public License v3.0
75 stars 17 forks source link

Feature Request: CDN #55

Closed jaswrks closed 9 years ago

jaswrks commented 10 years ago

@raamdev I'd like to request the following feature in QC Pro.

Here is a clip from a custom plugin that I've used for quite awhile now. Perhaps it could serve as a starting point for a whole new set of functionality that's offered by QC Pro when it comes to CDNs.

add_filter('pre_option_upload_path', 'cdn::uploads_path');
add_filter('pre_option_upload_url_path', 'cdn::uploads_url_path');

class cdn // Content delivery network for this site.
{
    public static function uploads_path()
        {
            return '/amazon-s3-bucket-mount';
            // Files uploaded through the WP Media Library go here.
        }

    public static function uploads_url_path()
        {
            return 'http://d1v41qemfjie0l.cloudfront.net/';
            // Files served by WordPress that reside in my `upload_path`; are served with this URL as the base.
        }
}

This code assumes that an S3 Bucket has been mounted locally, so it won't work unless a site owner can do that. Ideally, Quick Cache Pro could simplify this part further and make it easier for a novice site owner to set things up on their own.

Other Considerations...

A QC user recently wrote...

I just installed your quick cache plugin last night and I just wanted to give you a quick suggestion that I think would improve sales significantly. I was looking at the difference in the free and pro versions and seeing if I wanted to upgrade, the thing that would have made me pay the $15 upgrade right away I didn't see in the pro version. That feature is CDN image rewriting. The feature is present in both your top 2 competitors for free but your product is the easiest to use of the 3 products. I googled for the feature and found many people requesting the same feature from your product saying it's the only thing that it's missing. I also tried several other plugins along with your plugin to accomplish this feature and they wouldn't point the images to my Amazon Cloudfront CDN properly while your plugin cached the pages. As a first time user that spent an hour analyzing solutions to speed up Wordpress last night I wanted to let you know that you would have the perfect combination product if you added that one feature as caching and CDN are the top two methods to increase speed, why not make them into one super easy plugin? To keep things as simple as you have them all I would want is a single field to enter the new address to change the local image path into the CDN image path.

TODOs

Slobork commented 10 years ago

Hi,

I want to say that I've tried the CDN-Linker plugin with the Quick Cache and it mixes great. Maybe you can add its functionality to your plugin, I know that one of your competitor already does this. This is the CDN plugin - https://github.com/wmark/CDN-Linker

Alternatively you can just keep your plugin compatible with the CDN-Linker.

raamdev commented 10 years ago

@Sboq Thank you very much for that info! I'll be working on this soon and will try to maintain support for that plugin as well.

bridgeport commented 10 years ago

One thing I noticed with the various CDN plugins (whether they use output buffering regex or not) is that they don't provide the ability to switch CDN URL's based on whether the current page is HTTP or HTTPS (SSL). This can become an issue if you're using a CNAME CDN (e.g., cdn.example.com), but also need to CDNify your HTTPS pages. If your SSL pages aren't pointing to a URL that has a valid security certificate, you'll get browser security errors.

For example, using the CloudFront CDN with origin pull, you'd have to use uglySubdomain.cloudfront.net for SSL pages, as the cdn.example.com CNAME wouldn't match the SSL certificate. Now, amazon does offer SSL CNAME certificates for around $600/month, but that's probably cost-prohibitive for a lot of folks. You could probably get it to work locally if you acquired a wildcard SSL certificate, but that's more expensive than a non-wildcard. So just using the ugly SSL CDN URL is the least expensive option.

What's interesting with Quick Cache implementing CDN support (SSL page or not) is that CDN URL conversion could be performed whether or not caching is being performed. For instance, if someone excludes certain pages from caching, it's probable that they'll still want their static files CDNified. So Quick Cache would be more of a hybrid plugin where the CDNification operates independently from the caching.

Slobork commented 10 years ago

@raamdev I'm configuring a new site and I wonder should I install the existing CDN plugin or wait for your built-in with QC. It's not that I want to be pushy, I just wonder how fast this can be expected?

raamdev commented 10 years ago

@Sboq I don't have any precise ETA, but it will be at least a few weeks before I have this feature integrated, tested, and ready for release.

Slobork commented 10 years ago

@raamdev Well since you've decided to implement the CDN into the Quick Cache I have to say that I cheer for this feature to come fast because I'll have to reconfigure things later when it becomes available. The challenge would probably be with Multisite where QC is designed to be network active, while CDN plugin should be enabled and configured site by site. That's the reason the current situation with 2 separate plugins is working very well. I just wanted to remind you about this.

Edit: Maybe setting the CDN plugin to work like a plugin for the QC could be a good approach, then it's functionality can be accessed on a site by site basis.

Slobork commented 10 years ago

@raamdev it's 2 months since the hint about the CDN addition. Is this feature any time close? Thanks.

raamdev commented 10 years ago

@Sboq My apologies for the delay. I'm a bit behind on things, but this feature, along with Branched Cache Structure, are at the top of my list for new features that I'm working on for Quick Cache.

jaswrks commented 10 years ago

This code assumes that an S3 Bucket has been mounted locally, so it won't work unless a site owner can do that. Ideally, Quick Cache Pro could simplify this part further and make it easier for a novice site owner to set things up on their own.

@raamdev In my original feature request I noted the above.

I see there is an easy solution to this, which I will describe briefly...

Using the AWS-SDK (which we can redistribute w/ Quick Cache); we can add a stream wrapper like s3://[bucket]/uploads. That would be (pretty much) the same as having an S3 bucket mounted locally, but in a much more compatible way that can be accomplished via PHP code alone.

See: http://docs.aws.amazon.com/aws-sdk-php/guide/latest/feature-s3-stream-wrapper.html

So with that in place... the example I posted earlier might look like this...

<?php
require_once 'aws.phar';
use Aws\S3\S3Client;

add_action('init', 'cdn::init');
add_filter('pre_option_upload_path', 'cdn::uploads_path');
add_filter('pre_option_upload_url_path', 'cdn::uploads_url_path');

class cdn // Content delivery network for this site.
{
    public static function init()
        {
            $client = S3Client::factory(array(
                                            'key'    => '<aws access key>',
                                            'secret' => '<aws secret key>'
                                        ));
            $client->registerStreamWrapper(); // Register `s3://` stream wrapper.
        }

    public static function uploads_path()
        {
            return 's3://[bucket]/uploads';
            // Files uploaded through the WP Media Library go here.
        }

    public static function uploads_url_path()
        {
            return 'http://d1v41qemfjie0l.cloudfront.net/';
            // Files served by WordPress that reside in my `upload_path`; are served with this URL as the base.
        }
}
jaswrks commented 10 years ago

@raamdev The AWS-SDK can be downloaded here. http://aws.amazon.com/sdkforphp/

raamdev commented 10 years ago

@JasWSInc Perfect! Thanks so much. I'll work on getting this integrated so that we can start testing it for a future release.

jaswrks commented 10 years ago

@raamdev I just ran some tests against the S3 stream wrapper in my work on another project. There are two issues that I ran into which I thought I would share with you here.

  1. The WordPress filters I mentioned previously in this topic: pre_option_upload_path and pre_option_upload_url_path; will NOT work with an s3:// stream wrapper.

    WP forces an ABSPATH prefix on the values. So this won't work because you will end up with /abs/path/to/wordpress/s3://bucket/uploads. Not good.

    The solution is to filter the entire array of data that WP uses. See filter upload_dir here: https://github.com/WordPress/WordPress/blob/master/wp-includes/functions.php#L1658

    A quick example...

     add_filter('upload_dir', 'cdn::upload_dir');

 class cdn // Content delivery network for this site.
 {
     public static function upload_dir($details)
         {
             $details['basedir'] = 's3://[BUCKET]/uploads';
             $details['path']    = $details['basedir'].$details['subdir'];

             $details['baseurl'] = 'http://d1v41qemfjie0l.cloudfront.net/uploads';
             $details['url']     = $details['baseurl'].$details['subdir'];

             return $details; // After CDN filters.
         }
 }

  2. You MUST use the latest release of the AWS-SDK. I was testing this the other day and ran into a major limitation in the S3 stream wrapper related to mkdir() not working properly. I contacted AWS and they've had the issue corrected in the latest release (available today).

    See: https://github.com/aws/aws-sdk-php/releases

raamdev commented 10 years ago

@JasWSInc Thanks! That will definitely come in handy when I tackle CDN support. I'm currently working on #3.

ethanpil commented 10 years ago

+! on CDN. I use CDN-Linker and its great, but if we can combine that with S3 support for the cache files... AMAZING!!

raamdev commented 10 years ago

Adding this to the Future Release milestone to work on next, after the Next Release.

raamdev commented 10 years ago

Also requested here: http://wordpress.org/support/topic/cdn-support?replies=1

raamdev commented 10 years ago

Punting to Future Release milestone so that the bugs in the Next Release milestone can be released sooner rather than later. CDN integration will be my top priority feature to work on for the following release.

ioerjfoige0439i commented 10 years ago

Not sure if my comment showed up.

Two questions for you:

  1. Is this CDN feature going to be as good as W3 Total Cache's CDN with MaxCDN? I intend to use it with MaxCDN.
  2. Will Quick Cache & the CDN feature be fully compatible with s2Member and iDevAffiliate? (because W3 Total Cache is not 100% compatible with both)

Thanks!

raamdev commented 10 years ago

Is this CDN feature going to be as good as W3 Total Cache's CDN with MaxCDN? I intend to use it with MaxCDN.

Yes.

Will Quick Cache & the CDN feature be fully compatible with s2Member and iDevAffiliate? (because W3 Total Cache is not 100% compatible with both)

Yes. It will be fully compatible with s2Member and iDevAffiliate.

ioerjfoige0439i commented 10 years ago

Raam you're going to make me cry bro. Lol, I'm so excited. Now I can finally get rid of W3 Total Cache! Weeee!

jaswrks commented 10 years ago

:crying_cat_face: : I am already there. Tears of happiness brother. I can't wait!

ethanpil commented 10 years ago

Very excited here on this as well...

But to completely replace other (bloated) caching plugins, would need #47 too!! (Just a little plug and reminder!! :))))

ioerjfoige0439i commented 10 years ago

@raamdev @jaswsinc Holy moly, DreamSpeed CDN (Part of DreamObjects) just came out! Can you please make sure your release (I'm assuming is on the 19th) is compatible with this as well!! It's much better than MaxCDN so we're going to be switching over to it and I really want to delete as many plugins as I'm using as possible so we're only using WebSharks products and iDevAffiliate. Here's the link: https://www.dreamhost.com/cloud/cdn/ Thank you!! And here's a wiki on it http://wiki.dreamhost.com/DreamSpeed_CDN_Overview

raamdev commented 10 years ago

Request for Comments

What is the intention for Quick Cache CDN integration? @jaswsinc your initial comment when you opened this feature request indicated the following:

I would like to be able to serve WordPress media from a CDN of my choosing. For instance, so I can cache static media at edge locations with CloudFront.

But "static media" here implies only media uploaded through the WordPress Media Library, correct?

There are lots of ways Quick Cache could utilize CDN integration:

I feel that it would be nice to eventually support all of these, but obviously that will take some time, planning, and testing. I'm thinking it will make sense to start with CDN support for hosting attachments (files uploaded through the Media Library) and then work on the other features, maybe in the order listed above.

Thoughts?

jaswrks commented 10 years ago

@raamdev I'm going to respond to this privately and share my thoughts about what we could do here.

jaswrks commented 10 years ago

https://websharks.slack.com/files/jaswsinc/F02FWJPEZ/quick_cache_cdn_integration

jaswrks commented 10 years ago

Referencing: https://github.com/MaxCDN/php-maxcdn Also referencing: https://docs.maxcdn.com/

jaswrks commented 10 years ago

Referencing: http://docs.aws.amazon.com/AmazonCloudFront/latest/APIReference/Welcome.html Also referencing: http://docs.aws.amazon.com/aws-sdk-php/guide/latest/installation.html Also referencing: http://docs.aws.amazon.com/aws-sdk-php/latest/ Creating a distro: http://amzn.to/1opSyYJ

jaswrks commented 10 years ago

Quick update: This branch of QCP now contains a composer.json file. When we work on this repo locally we'll need to remember to run cd quick-cache-pro/quick-cache-pro && composer install to install the MaxCDN SDK and AWS SDK dependencies that I've added as part of the work on this issue.

If you don't have Composer, you can get this via Homebrew or Mac Ports (I think).

jaswrks commented 10 years ago

Another option, is that I could just push the vendor directory that I have locally into this repo. There are MANY files in that directory installed via Composer, but that would put everything (except the sub-modules for the HTML Compressor) into the repo for QCP.

jaswrks commented 10 years ago

I'm going to leave this issue as-is for now. That way @raamdev can have a chance to review what's been added thus far. Maybe we could chat about this a little also and see how we might want to proceed.

jaswrks commented 10 years ago
jaswrks commented 10 years ago
ioerjfoige0439i commented 10 years ago

Great to see the progress on this! Keeping an eye out. I switched from dreamhost to inmotion hosting because they have SSD drives and you can definitely tell the difference. All I need to make my site complete now is quick cache with MaxCDN :-)! Weeeee!!! Hopefully it works similar to how W3TC works where it caches as much as possible in the wp-content folder and elsewhere with options to not cache the directories or files of your choice.

jaswrks commented 10 years ago
jaswrks commented 10 years ago

The URL/content filtering phase of this project has been completed. A first pass anyway. Further testing will be necessary once the CDN integration is closer to completion overall, but I feel good about the content/URL filtering portion of it for now.

Suggested Next Steps...
omriamos commented 10 years ago

+1 for CDN support :) I was actually trying to use both QuickCache and each one of its 2 competitors (with caching disabled, just the CDN feature enabled) together in order to have the "CDN Rewrite" feature, but with no luck... supercache's cdn feature doesn't work at all when its caching feature is disabled, and w3tc works great with cdn enabled and caching disabled - but it also completely disables/overrides QuickCache...

Basically I already have a cdn domain (I'm using it as a subdomain: cdn.domain.com) and all I'm looking for the the "url rewrite" feature that will point static files in wp-content and wp-include from domain.com to cdn.domain.com . Please make sure to include this "generic mirror" CDN mode

Hope to see this feature in QuickCache Pro soon...

Thanks!

ioerjfoige0439i commented 10 years ago

Hey @jaswsinc @raamdev will the price of Quick Cache go up with the CDN release? Thank you.

jaswrks commented 10 years ago

@ioerjfoige0439i I'd say that is likely, yes. A bit, anyway :-)

jaswrks commented 10 years ago

TODO:

ioerjfoige0439i commented 10 years ago

@jaswsinc @raamdev Okay but what if you've already paid for the $15 license? We still have to pay more money to upgrade?

jaswrks commented 10 years ago

@ioerjfoige0439i It's too early to say for sure, but there are no plans to do this at the moment :-)

ioerjfoige0439i commented 10 years ago

@jaswsinc @raamdev Okay nice! I'll make sure to tell my customers to go buy the $15 license right now then so they don't have to pay more later. Thanks!

omriamos commented 10 years ago

+1 for the CDN-Linker plugin - works great together with QC Pro.

TheBlogHouse commented 10 years ago

+1 for CNAME CDN support (e.g., cdn.example.com) or CDN-Linker plugin comaptibility

ioerjfoige0439i commented 9 years ago

trifecta Any updates guys? Here's the TRIFECTA pictured above: Quick Cache / Zen Cache perfectly integrated with: s2Member iDevAffiliate MaxCDN

ioerjfoige0439i commented 9 years ago

Let me know your thoughts on my above Trifecta @clavaque @raamdev @BruceCaldwell @jaswsinc @ethanpil @Reedyseth I think this would be the dream for all owners of Quick Cache / Zen Cache. trifecta

ioerjfoige0439i commented 9 years ago

iDevAffiliate works with s2Member now already but unfortunately, iDevAffiliate and s2Member don't work together properly when W3 Total Cache is turned on, but W3 Total Cache does MaxCDN perfectly. If W3 Total Cache worked properly with iDevAffiliate and s2Member then I would just use it instead because all that's missing is MaxCDN. If Quick Cache / Zen Cache could work nicely and fully tested together with all 3 then that would eliminate our need for W3 Total Cache completely and allow everyone to see the full potential of all these technologies working together!

ioerjfoige0439i commented 9 years ago

Of course being that iDevAffiliate has it's own seperate install folder, but just that it can properly communicate with s2Member when Quick Cache / Zen Cache is turned on and connected to MaxCDN. iDevAffiliate would not need to be using any CDN features because it is pretty lite weight ( I mean unless you wanted to do that in the future but for right now that's not very important ).

raamdev commented 9 years ago

@ioerjfoige0439i thank you for the ideas and the suggestions. :)

We already have plans to launch ZenCache with MaxCDN support within the next few weeks / early January. We'll ensure that ZenCache + s2Member + iDevAffiliate work together as expected!