search

wimleers / fileconveyor

File Conveyor is a daemon written in Python to detect, process and sync files. In particular, it's designed to sync files to CDNs. Amazon S3 and Rackspace Cloud Files, as well as any Origin Pull or (S)FTP Push CDN, are supported. Originally written for my bachelor thesis at Hasselt University in Belgium.
https://wimleers.com/fileconveyor
The Unlicense
341 stars 95 forks source link

image_optimizer not optimizing PNGs? #165

Open csdougliss opened 10 years ago

csdougliss commented 10 years ago

I have fileconveyor finally working, connecting to Rackspace CDN (UK).

I have verified an original PNG is 65.5 KB in size (just an example). After uploading/processing by fileconveyor it is still 65.5 KB in size.

If I run the file through tinypng.org it is 20.6 KB after processing. Also, will the files be smaller or should they be the same in comparison to tinypng?

I installed django-cumulus 1.0.10.

Here is some output from the command line:

2014-04-28 09:24:10,585 - Arbitrator                - WARNING  - File Conveyor is initializing.
2014-04-28 09:24:10,587 - Arbitrator                - WARNING  - Loaded config file.
2014-04-28 09:24:10,608 - Arbitrator                - WARNING  - Created 'cloudfiles' transporter for the 'rackspace' server.
2014-04-28 09:24:10,608 - Arbitrator                - WARNING  - Server connection tests succesful!
2014-04-28 09:24:10,609 - Arbitrator                - WARNING  - Setup: created transporter pool for the 'rackspace' server.
2014-04-28 09:24:10,610 - Arbitrator                - WARNING  - Setup: initialized 'pipeline' persistent queue, contains 0 items.
2014-04-28 09:24:10,610 - Arbitrator                - WARNING  - Setup: initialized 'files_in_pipeline' persistent list, contains 0 items.
2014-04-28 09:24:10,611 - Arbitrator                - WARNING  - Setup: initialized 'failed_files' persistent list, contains 0 items.
2014-04-28 09:24:10,611 - Arbitrator                - WARNING  - Setup: initialized 'files_to_delete' persistent list, contains 0 items.
2014-04-28 09:24:10,612 - Arbitrator                - WARNING  - Setup: moved 0 items from the 'files_in_pipeline' persistent list into the 'pipeline' persistent queue.
2014-04-28 09:24:10,612 - Arbitrator                - WARNING  - Setup: connected to the synced files DB. Contains metadata for 0 previously synced files.
2014-04-28 09:24:10,615 - Arbitrator                - WARNING  - Setup: initialized FSMonitor.
2014-04-28 09:24:10,616 - Arbitrator                - WARNING  - Fully up and running now.
2014-04-28 09:25:50,221 - Arbitrator                - WARNING  - Created 'cloudfiles' transporter for the 'rackspace' server.
2014-04-28 09:25:50,982 - Arbitrator                - WARNING  - Created 'cloudfiles' transporter for the 'rackspace' server.
2014-04-28 09:25:51,391 - Arbitrator                - WARNING  - Created 'cloudfiles' transporter for the 'rackspace' server.
2014-04-28 09:25:52,198 - Arbitrator                - WARNING  - Created 'cloudfiles' transporter for the 'rackspace' server.
2014-04-28 09:25:52,602 - Arbitrator                - WARNING  - Created 'cloudfiles' transporter for the 'rackspace' server.
2014-04-28 09:25:53,407 - Arbitrator                - WARNING  - Created 'cloudfiles' transporter for the 'rackspace' server.
2014-04-28 09:25:53,811 - Arbitrator                - WARNING  - Created 'cloudfiles' transporter for the 'rackspace' server.
2014-04-28 09:25:54,618 - Arbitrator                - WARNING  - Created 'cloudfiles' transporter for the 'rackspace' server.
2014-04-28 09:25:55,022 - Arbitrator                - WARNING  - Created 'cloudfiles' transporter for the 'rackspace' server.
2014-04-28 09:25:55,831 - Arbitrator                - WARNING  - Created 'cloudfiles' transporter for the 'rackspace' server.
2014-04-28 09:26:12,483 - Arbitrator                - WARNING  - Synced: '/var/www/vhosts/magento-capistrano-dev/current/skin/adminhtml/default/default/oauth-simple.css' (CREATED).
2014-04-28 09:26:12,487 - Arbitrator                - WARNING  - Synced: '/var/www/vhosts/magento-capistrano-dev/current/skin/adminhtml/default/default/aw_all/images/store.png' (CREATED).
2014-04-28 09:26:13,105 - Arbitrator                - WARNING  - Synced: '/var/www/vhosts/magento-capistrano-dev/current/skin/adminhtml/default/default/aw_all/images/readme.png' (CREATED).
2014-04-28 09:26:13,320 - Arbitrator                - WARNING  - Synced: '/var/www/vhosts/magento-capistrano-dev/current/skin/adminhtml/default/default/aw_all/images/window/btn-close.gif' (CREATED).

Here is my config.xml:

<?xml version="1.0" encoding="UTF-8"?>
<config>
  <!-- Sources -->
  <sources ignoredDirs="CVS:.git">
    <source name="media" scanPath="/var/www/vhosts/magento-capistrano-dev/current/media" />
    <source name="skin" scanPath="/var/www/vhosts/magento-capistrano-dev/current/skin" />
  </sources>

  <!-- Servers -->
  <servers>
    <server name="rackspace" transporter="cloudfiles">
      <username>username</username>
      <api_key>key</api_key>
      <container>MagentoEESync</container>
    </server>
  </servers>

  <!-- Rules -->
  <rules>
    <rule for="media" label="media">
      <filter>
        <paths>blfa_files:captcha:cart_icons:catalog:categories:css:css_secure:customer:dhl:downloadable:import:js:minified:product_icons:rma_item:tmp:videos:wysiwyg:xmlconnect</paths>
        <extensions>ico:js:css:gif:png:jpg:jpeg:svg:swf:avi:mp4:webm:pdf</extensions>
      </filter>
      <processorChain>
        <processor name="image_optimizer.KeepFilename" />
      </processorChain>
      <destinations>
        <destination server="rackspace" path="media" />
      </destinations>
    </rule>
    <rule for="skin" label="skin">
      <filter>
        <!--<paths>skin</paths>-->
        <extensions>ico:js:css:gif:png:jpg:jpeg:svg:swf:avi:mp4:webm:pdf</extensions>
      </filter>
      <processorChain>
        <processor name="image_optimizer.KeepFilename" />
      </processorChain>
      <destinations>
        <destination server="rackspace" path="skin" />
      </destinations>
    </rule>
  </rules>

My cumulus settings.py:

import cloudfiles
from cloudfiles.consts import default_cdn_ttl

from django.conf import settings

CUMULUS = {
    'API_KEY': None,
    'AUTH_URL': 'uk_authurl',
    'CNAMES': None,
    'CONTAINER': None,
    'SERVICENET': False,
    'TIMEOUT': 5,
    'TTL': default_cdn_ttl,  # 86400s (24h), python-cloudfiles default
    'USE_SSL': False,
    'USERNAME': None,
    'STATIC_CONTAINER': None,
    'FILTER_LIST': [],
    'HEADERS': {},
    'GZIP_CONTENT_TYPES': [],
}

if hasattr(settings, 'CUMULUS'):
    CUMULUS.update(settings.CUMULUS)

# set auth_url to the actual URL string in the cloudfiles module
CUMULUS['AUTH_URL'] = getattr(cloudfiles, CUMULUS['AUTH_URL'])

# backwards compatibility for old-style cumulus settings
if not hasattr(settings, 'CUMULUS') and hasattr(settings, 'CUMULUS_API_KEY'):
    import warnings
    warnings.warn(
        "settings.CUMULUS_* is deprecated; use settings.CUMULUS instead.",
        PendingDeprecationWarning
    )

    CUMULUS.update({
        'API_KEY': getattr(settings, 'CUMULUS_API_KEY'),
        'CNAMES': getattr(settings, 'CUMULUS_CNAMES', None),
        'CONTAINER': getattr(settings, 'CUMULUS_CONTAINER'),
        'SERVICENET': getattr(settings, 'CUMULUS_USE_SERVICENET', False),
        'TIMEOUT': getattr(settings, 'CUMULUS_TIMEOUT', 5),
        'TTL': getattr(settings, 'CUMULUS_TTL', default_cdn_ttl),
        'USERNAME': getattr(settings, 'CUMULUS_USERNAME'),
    })

My django_settings.py:

# Dummy settings for `django-storages`.
SECRET_KEY='random guid'
MEDIA_URL=''
MEDIA_ROOT=''
# `backends/ftp.py`
FTP_STORAGE_LOCATION=''
# `backends/sftp.py`
SFTP_STORAGE_HOST=''
# django-cumulus
#CUMULUS['USERNAME'] = '';
#CUMULUS['CUMULUS_API_KEY'] = '';
#CUMULUS['CONTAINER'] = '';
CUMULUS_API_KEY = '';
CUMULUS_USERNAME = '';
CUMULUS_CONTAINER = '';

I also see the occasional read timeout?

2014-04-28 10:05:49,124 - Arbitrator                - WARNING  - Synced: '/var/www/vhosts/magento-capistrano-dev/current/skin/frontend/enterprise/default/images/i_availability_only_arrow.gif' (CREATED).
2014-04-28 10:05:49,493 - Arbitrator.Transporter    - ERROR    - The transporter 'Cloud Files' has failed while transporting the file '/var/www/vhosts/magento-capistrano-dev/current/skin/adminhtml/default/default/images/ajax-loader.gif' (action: 1). Error: 'The read operation timed out'.
2014-04-28 10:05:49,543 - Arbitrator                - WARNING  - Retry queue -> 'failed_files' persistent list: '/var/www/vhosts/magento-capistrano-dev/current/skin/adminhtml/default/default/images/ajax-loader.gif'. Retrying later.
2014-04-28 10:05:50,032 - Arbitrator.Transporter    - ERROR    - The transporter 'Cloud Files' has failed while transporting the file '/var/www/vhosts/magento-capistrano-dev/current/skin/adminhtml/default/default/images/widget/reports__product_widget_viewed.gif' (action: 1). Error: 'The read operation timed out'.
2014-04-28 10:05:50,142 - Arbitrator.Transporter    - ERROR    - The transporter 'Cloud Files' has failed while transporting the file '/var/www/vhosts/magento-capistrano-dev/current/skin/adminhtml/default/default/images/xmlconnect/design_default/banner_ipad_l.png' (action: 1). Error: 'The read operation timed out'.

As you can see I am syncing the media and skin folders.

Is there a way to omit a subfolder from the media folder? I've setup media and skin seperately so it is not a major issue.

Any ideas? thanks