search

wimleers / fileconveyor

File Conveyor is a daemon written in Python to detect, process and sync files. In particular, it's designed to sync files to CDNs. Amazon S3 and Rackspace Cloud Files, as well as any Origin Pull or (S)FTP Push CDN, are supported. Originally written for my bachelor thesis at Hasselt University in Belgium.
https://wimleers.com/fileconveyor
The Unlicense
341 stars 95 forks source link

fileconveyor not syncronizing #158

Open lseica opened 10 years ago

lseica commented 10 years ago

We have a server with several containers and installed fileconveyor on our devices to syncronize the image files. The sync happened for the firs 146 files, but now it's not working anymore.

We tried to install it on a second device, but this time it's not syncronizing any files.

we have both the server and the containers at Chicago ORD

this is an error that we get :

Arbitrator.Transporter - ERROR - The transporter 'Cloud Files' has failed while transporting the file '/var/www/dev.mjs/sites/default/files/item_raw/2_ALT_PICS/W41255581_002_alt1.jpg' (action: 1). Error: 'container_name'. 2013-10-2

right now the arbitrator.py is running, but nothing is happening and if i restart it :

logs:

2013-10-25 12:22:51,393 - Arbitrator - WARNING - File Conveyor is initializing. 2013-10-25 12:22:51,393 - Arbitrator - INFO - Loading config file. 2013-10-25 12:22:51,401 - Arbitrator.Config - INFO - Parsing sources. 2013-10-25 12:22:51,402 - Arbitrator.Config - INFO - Parsing servers. 2013-10-25 12:22:51,402 - Arbitrator.Config - INFO - Parsing rules. 2013-10-25 12:22:51,402 - Arbitrator - WARNING - Loaded config file. 2013-10-25 12:22:51,579 - Arbitrator - WARNING - Created 'cloudfiles' transporter for the 'Rackspace Cloud Files' server. 2013-10-25 12:22:51,579 - Arbitrator - WARNING - Server connection tests succesful! 2013-10-25 12:22:51,592 - Arbitrator - WARNING - Setup: created transporter pool for the 'Rackspace Cloud Files' server. 2013-10-25 12:22:51,592 - Arbitrator - INFO - Setup: collected all metadata for rule 'Images' (source: 'devmjs'). 2013-10-25 12:22:51,634 - Arbitrator - WARNING - Setup: initialized 'pipeline' persistent queue, contains 33004 items. 2013-10-25 12:22:51,636 - Arbitrator - WARNING - Setup: initialized 'files_in_pipeline' persistent list, contains 68 items. 2013-10-25 12:22:51,644 - Arbitrator - WARNING - Setup: initialized 'failed_files' persistent list, contains 901 items. 2013-10-25 12:22:51,645 - Arbitrator - WARNING - Setup: initialized 'files_to_delete' persistent list, contains 0 items

Sincerely, Seica Laszlo

david-wilkins commented 10 years ago

I'm having the same issue. Here's my config.xml:

        <?xml version="1.0" encoding="UTF-8"?>
        <config>
          <!-- Sources -->
          <sources ignoredDirs="">
            <source name="test" scanPath="/var/www/test" />
          </sources>
          <!-- Servers -->
          <servers>
            <server name="Rackspace Cloud Files" transporter="cloudfiles">
              <username>[my user id]</username>
              <api_key>[my api key]</api_key>
              <container>test</container>
            </server>
          </servers>
         <!-- Rules -->
          <rules>
            <rule for="test" label="Test">
              <destinations>
                <destination server="Rackspace Cloud Files" path="test" />
              </destinations>
            </rule>
          </rules>
        </config>

Amd here's the log file output:

        2013-11-12 19:06:01,419 - Arbitrator                - WARNING  - File Conveyor is initializing.
        2013-11-12 19:06:01,420 - Arbitrator                - INFO     - Loading config file.
        2013-11-12 19:06:01,421 - Arbitrator.Config         - INFO     - Parsing sources.
        2013-11-12 19:06:01,421 - Arbitrator.Config         - INFO     - Parsing servers.
        2013-11-12 19:06:01,421 - Arbitrator.Config         - INFO     - Parsing rules.
        2013-11-12 19:06:01,422 - Arbitrator                - WARNING  - Loaded config file.
        2013-11-12 19:06:01,452 - Arbitrator                - WARNING  - Created 'cloudfiles' transporter for the  Rackspace Cloud Files' server.
        2013-11-12 19:06:01,452 - Arbitrator                - WARNING  - Server connection tests succesful!
        2013-11-12 19:06:01,453 - Arbitrator                - WARNING  - Setup: created transporter pool for the 'Rackspace Cloud Files' server.
        2013-11-12 19:06:01,454 - Arbitrator                - INFO     - Setup: collected all metadata for rule 'Test' (source: 'test').
        2013-11-12 19:06:01,594 - Arbitrator                - WARNING  - Setup: initialized 'pipeline' persistent queue, contains 0 items.
        2013-11-12 19:06:01,602 - Arbitrator                - WARNING  - Setup: initialized 'files_in_pipeline' persistent list, contains 0 items.
        2013-11-12 19:06:01,914 - Arbitrator                - WARNING  - Setup: initialized 'failed_files' persistent list, contains 0 items.
        2013-11-12 19:06:01,934 - Arbitrator                - WARNING  - Setup: initialized 'files_to_delete' persistent list, contains 0 items.
        2013-11-12 19:06:01,935 - Arbitrator                - WARNING  - Setup: moved 0 items from the 'files_in_pipeline' persistent list into the 'pipeline' persistent queue.
        2013-11-12 19:06:02,013 - Arbitrator                - WARNING  - Setup: connected to the synced files DB. Contains metadata for 0 previously synced files.
        2013-11-12 19:06:02,030 - Arbitrator.FSMonitor      - INFO     - FSMonitor class used: FSMonitorPolling.
        2013-11-12 19:06:02,030 - Arbitrator                - WARNING  - Setup: initialized FSMonitor.
        2013-11-12 19:06:02,030 - Arbitrator                - INFO     - Setup: monitoring '/var/www/test' (test).
        2013-11-12 19:06:02,031 - Arbitrator                - INFO     - Cleaned up the working directory '/tmp/fileconveyor'.
        2013-11-12 19:06:02,031 - Arbitrator                - WARNING  - Fully up and running now.
        2013-11-12 19:06:02,238 - Arbitrator.FSMonitor      - INFO     - Generating missed events for '/var/www/test' (event mask: None).
        2013-11-12 19:06:02,266 - Arbitrator.FSMonitor      - INFO     - Detected 'CREATED' event for '/var/www/test/testfile.txt' through generate_missed_events (for monitored path '/var/www/test').
        2013-11-12 19:06:02,267 - Arbitrator.FSMonitor      - INFO     - Done generating missed events for '/var/www/test' (event mask: None).
        2013-11-12 19:06:02,539 - Arbitrator                - INFO     - Discover queue -> pipeline queue: '/var/www/test/testfile.txt'.
        2013-11-12 19:06:02,550 - Arbitrator                - INFO     - Pipeline queue -> filter queue: '/var/www/test/testfile.txt'.
        2013-11-12 19:06:02,550 - Arbitrator                - INFO     - Filtering: '/var/www/test/testfile.txt' matches the 'Test' rule for the 'test' source!
        2013-11-12 19:06:02,551 - Arbitrator                - INFO     - Filter queue -> transport queue: '/var/www/test/testfile.txt' (rule: 'Test').
        2013-11-12 19:06:02,551 - Arbitrator                - WARNING  - Created 'cloudfiles' transporter for the 'Rackspace Cloud Files' server.
        2013-11-12 19:06:02,552 - Arbitrator                - INFO     - Transport queue: '/var/www/test/testfile.txt' to transfer to server 'Rackspace Cloud Files' with transporter #0 (of 1), place 1 in the queue.
        2013-11-12 19:06:03,053 - Arbitrator.Transporter    - DEBUG    - Running the transporter 'Cloud Files' to sync '/var/www/test/testfile.txt'.
        2013-11-12 19:06:03,460 - Arbitrator.Transporter    - ERROR    - The transporter 'Cloud Files' has failed while transporting the file '/var/www/test/testfile.txt' (action: 1). Error: 'test'.

I could really use some help here.

david-wilkins commented 10 years ago

I notice in transporter_cloudfiles.py there is a line: from cumulus.storage import CloudFilesStorage

I'm assuming now that django-cumulus is a pre-req?

More info when I have it

david-wilkins commented 10 years ago

Yes I was missing the pre-requisite django portions...It's syncing now, most of the time