search

wimleers / fileconveyor

File Conveyor is a daemon written in Python to detect, process and sync files. In particular, it's designed to sync files to CDNs. Amazon S3 and Rackspace Cloud Files, as well as any Origin Pull or (S)FTP Push CDN, are supported. Originally written for my bachelor thesis at Hasselt University in Belgium.
https://wimleers.com/fileconveyor
The Unlicense
341 stars 95 forks source link

some folders being excluded? #166

Open csdougliss opened 10 years ago

csdougliss commented 10 years ago

Inside my media folder I have CSS, downloadable folder etc. which is excluded from my rackspace CDN sync - why is this?

Here is my config.xml:

<?xml version="1.0" encoding="UTF-8"?>
<config>
  <!-- Sources -->
  <sources ignoredDirs="CVS:.git">
    <source name="media" scanPath="/var/www/vhosts/magento-capistrano-dev/current/media" />
    <source name="skin" scanPath="/var/www/vhosts/magento-capistrano-dev/current/skin" />
  </sources>

  <!-- Servers -->
  <servers>
    <server name="rackspace" transporter="cloudfiles">
      <username>username</username>
      <api_key>key</api_key>
      <container>MagentoEESync</container>
    </server>
  </servers>

  <!-- Rules -->
  <rules>
    <rule for="media" label="media">
      <filter>
        <paths>blfa_files:captcha:cart_icons:catalog:categories:css:css_secure:customer:dhl:downloadable:import:js:minified:product_icons:rma_item:tmp:videos:wysiwyg:xmlconnect</paths>
        <extensions>ico:js:css:gif:png:jpg:jpeg:svg:swf:avi:mp4:webm:pdf</extensions>
      </filter>
      <processorChain>
        <processor name="image_optimizer.KeepFilename" />
      </processorChain>
      <destinations>
        <destination server="rackspace" path="media" />
      </destinations>
    </rule>
    <rule for="skin" label="skin">
      <filter>
        <!--<paths>skin</paths>-->
        <extensions>ico:js:css:gif:png:jpg:jpeg:svg:swf:avi:mp4:webm:pdf</extensions>
      </filter>
      <processorChain>
        <processor name="image_optimizer.KeepFilename" />
      </processorChain>
      <destinations>
        <destination server="rackspace" path="skin" />
      </destinations>
    </rule>
  </rules>
sqlite3 persistent_data.db 
SQLite version 3.8.2 2013-12-06 14:53:30
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> .table
failed_files_list     pipeline_list       
files_to_delete_list  pipeline_queue      
sqlite> select count(*) from pipeline_queue;
8923
sqlite> select count(*) from pipeline_list;
48
sqlite> select count(*) from failed_files_list;
16

When running verify.py I get a crash after 5 files:

  1% (444/26719)Traceback (most recent call last):
  File "./verify.py", line 20, in <module>
    conn.request("HEAD", parsed.path)
  File "/usr/lib/python2.7/httplib.py", line 973, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 1007, in _send_request
    self.endheaders(body)
  File "/usr/lib/python2.7/httplib.py", line 969, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python2.7/httplib.py", line 829, in _send_output
    self.send(msg)
  File "/usr/lib/python2.7/httplib.py", line 805, in send
    self.sock.sendall(data)
  File "/usr/lib/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xfc' in position 48: ordinal not in range(128)