search

wimleers / fileconveyor

File Conveyor is a daemon written in Python to detect, process and sync files. In particular, it's designed to sync files to CDNs. Amazon S3 and Rackspace Cloud Files, as well as any Origin Pull or (S)FTP Push CDN, are supported. Originally written for my bachelor thesis at Hasselt University in Belgium.
https://wimleers.com/fileconveyor
The Unlicense
341 stars 95 forks source link

ERROR - The transporter 'cumulus' has failed while transporting the file #131

Closed sobi3ch closed 11 years ago

sobi3ch commented 11 years ago

I've double check config.xml file with my cloudfiles Rackspace settings everything looks ok. How can I check more deeply what's happening with this problem?? .log file showing me exactly same data

What's mean 'static' in path destination in config.xml settings Here is my config: https://gist.github.com/4155726

STAGING (~/fileconveyor/fileconveyor) 17:20 piotrek (no-delete)*$ sudo python arbitrator.py /home/piotrek/fileconveyor/fileconveyor/filter.py:10: DeprecationWarning: the sets module is deprecated from sets import Set, ImmutableSet 2012-11-27 17:24:39,556 - Arbitrator - WARNING - File Conveyor is initializing. 2012-11-27 17:24:39,559 - Arbitrator - WARNING - Loaded config file. /usr/lib/python2.6/site-packages/Django-1.4.2-py2.6.egg/django/conf/init.py:110: DeprecationWarning: The SECRET_KEY setting must not be empty. warnings.warn("The SECRET_KEY setting must not be empty.", DeprecationWarning) 2012-11-27 11:24:39,670 - Arbitrator - WARNING - Created 'cumulus' transporter for the 'rackspace' server. 2012-11-27 11:24:39,671 - Arbitrator - WARNING - Server connection tests succesful! 2012-11-27 11:24:39,715 - Arbitrator - WARNING - Setup: created transporter pool for the 'rackspace' server. 2012-11-27 11:24:39,774 - Arbitrator - WARNING - Setup: initialized 'pipeline' persistent queue, contains 40803 items. 2012-11-27 11:24:39,776 - Arbitrator - WARNING - Setup: initialized 'files_in_pipeline' persistent list, contains 48 items. 2012-11-27 11:24:39,777 - Arbitrator - WARNING - Setup: initialized 'failed_files' persistent list, contains 62 items. 2012-11-27 11:24:39,778 - Arbitrator - WARNING - Setup: initialized 'files_to_delete' persistent list, contains 0 items. 2012-11-27 11:24:40,124 - Arbitrator - WARNING - Setup: moved 48 items from the 'files_in_pipeline' persistent list into the 'pipeline' persistent queue. 2012-11-27 11:24:40,258 - Arbitrator - WARNING - Moved 20 items from the 'failed_files' persistent list into the 'pipeline' persistent queue. 2012-11-27 11:24:40,260 - Arbitrator - WARNING - Setup: connected to the synced files DB. Contains metadata for 0 previously synced files. 2012-11-27 11:24:40,335 - Arbitrator - WARNING - Setup: initialized FSMonitor. 2012-11-27 11:24:40,337 - Arbitrator - WARNING - Fully up and running now. 2012-11-27 11:24:40,981 - Arbitrator - WARNING - Created 'cumulus' transporter for the 'rackspace' server. 2012-11-27 11:24:40,986 - Arbitrator - WARNING - Created 'cumulus' transporter for the 'rackspace' server. 2012-11-27 11:24:40,990 - Arbitrator - WARNING - Created 'cumulus' transporter for the 'rackspace' server. 2012-11-27 11:24:41,942 - Arbitrator - WARNING - Created 'cumulus' transporter for the 'rackspace' server. 2012-11-27 11:24:41,983 - Arbitrator.Transporter - ERROR - The transporter 'cumulus' has failed while transporting the file '/var/www/versions/phoenix109-piotrek/sites/default/files/orlando-ballooon.gif' (action: 1). Error: ''. 2012-11-27 11:24:41,990 - Arbitrator.Transporter - ERROR - The transporter 'cumulus' has failed while transporting the file '/var/www/versions/phoenix109-piotrek/sites/default/files/universal_new_years_celebration.jpg' (action: 1). Error: ''. 2012-11-27 11:24:41,994 - Arbitrator.Transporter - ERROR - The transporter 'cumulus' has failed while transporting the file '/var/www/versions/phoenix109-piotrek/sites/default/files/walt_disney_home_page3.jpg' (action: 1). Error: ''. 2012-11-27 11:24:42,234 - Arbitrator - WARNING - Retry queue -> 'failed_files' persistent list: '/var/www/versions/phoenix109-piotrek/sites/default/files/orlando-ballooon.gif'. Retrying later. 2012-11-27 11:24:42,306 - Arbitrator - WARNING - Retry queue -> 'failed_files' persistent list: '/var/www/versions/phoenix109-piotrek/sites/default/files/universal_new_years_celebration.jpg'. Retrying later. 2012-11-27 11:24:42,379 - Arbitrator - WARNING - Retry queue -> 'failed_files' persistent list: '/var/www/versions/phoenix109-piotrek/sites/default/files/walt_disney_home_page3.jpg'. Retrying later. 2012-11-27 11:24:42,446 - Arbitrator.Transporter - ERROR - The transporter 'cumulus' has failed while transporting the file '/var/www/versions/phoenix109-piotrek/sites/default/files/clearwater_beach2.jpg' (action: 1). Error: ''. 2012-11-27 11:24:42,498 - Arbitrator.Transporter - ERROR - The transporter 'cumulus' has failed while transporting the file '/var/www/versions/phoenix109-piotrek/sites/default/files/disneyscaliforniagrill_0.jpg' (action: 1). Error: ''. 2012-11-27 11:24:42,500 - Arbitrator.Transporter - ERROR - The transporter 'cumulus' has failed while transporting the file '/var/www/versions/phoenix109-piotrek/sites/default/files/islands_of_adventure2.jpg' (action: 1). Error: ''. 2012-11-27 11:24:42,505 - Arbitrator.Transporter - ERROR - The transporter 'cumulus' has failed while transporting the file '/var/www/versions/phoenix109-piotrek/sites/default/files/gatorland-alligator.jpg' (action: 1). Error: ''. 2012-11-27 11:24:42,711 - Arbitrator - WARNING - Retry queue -> 'failed_files' persistent list: '/var/www/versions/phoenix109-piotrek/sites/default/files/clearwater_beach2.jpg'. Retrying later. 2012-11-27 11:24:42,786 - Arbitrator - WARNING - Retry queue -> 'failed_files' persistent list: '/var/www/versions/phoenix109-piotrek/sites/default/files/disneyscaliforniagrill_0.jpg'. Retrying later. 2012-11-27 11:24:42,861 - Arbitrator - WARNING - Retry queue -> 'failed_files' persistent list: '/var/www/versions/phoenix109-piotrek/sites/default/files/islands_of_adventure2.jpg'. Retrying later. 2012-11-27 11:24:42,933 - Arbitrator - WARNING - Retry queue -> 'failed_files' persistent list: '/var/www/versions/phoenix109-piotrek/sites/default/files/gatorland-alligator.jpg'. Retrying later. 2012-11-27 11:24:42,945 - Arbitrator.Transporter - ERROR - The transporter 'cumulus' has failed while transporting the file '/var/www/versions/phoenix109-piotrek/sites/default/files/blizzard_beach_default_image.jpg' (action: 1). Error: ''. 2012-11-27 11:24:43,001 - Arbitrator.Transporter - ERROR - The transporter 'cumulus' has failed while transporting the file '/var/www/versions/phoenix109-piotrek/sites/default/files/tickvip.jpg' (action: 1). Error: ''. 2012-11-27 11:24:43,010 - Arbitrator.Transporter - ERROR - The transporter 'cumulus' has failed while transporting the file '/var/www/versions/phoenix109-piotrek/sites/default/files/kennedy_free_kidsbanner-_otd_ticket_page2.jpg' (action: 1). Error: ''. 2012-11-27 11:24:43,011 - Arbitrator.Transporter - ERROR - The transporter 'cumulus' has failed while transporting the file '/var/www/versions/phoenix109-piotrek/sites/default/files/mickeysnotsoscaryhalloween_1.jpg' (action: 1). Error: ''. 2012-11-27 11:24:43,273 - Arbitrator - WARNING - Retry queue -> 'failed_files' persistent list: '/var/www/versions/phoenix109-piotrek/sites/default/files/blizzard_beach_default_image.jpg'. Retrying later. 2012-11-27 11:24:43,347 - Arbitrator - WARNING - Retry queue -> 'failed_files' persistent list: '/var/www/versions/phoenix109-piotrek/sites/default/files/tickvip.jpg'. Retrying later. 2012-11-27 11:24:43,429 - Arbitrator - WARNING - Retry queue -> 'failed_files' persistent list: '/var/www/versions/phoenix109-piotrek/sites/default/files/kennedy_free_kidsbanner-_otd_ticket_page2.jpg'. Retrying later. 2012-11-27 11:24:43,500 - Arbitrator - WARNING - Retry queue -> 'failed_files' persistent list: '/var/www/versions/phoenix109-piotrek/sites/default/files/mickeysnotsoscaryhalloween_1.jpg'. Retrying later. 2012-11-27 11:24:43,522 - Arbitrator.Transporter - ERROR - The transporter 'cumulus' has failed while transporting the file '/var/www/versions/phoenix109-piotrek/sites/default/files/1_0.jpg' (action: 1). Error: ''. 2012-11-27 11:24:43,854 - Arbitrator - WARNING - Retry queue -> 'failed_files' persistent list: '/var/www/versions/phoenix109-piotrek/sites/default/files/1_0.jpg'. Retrying later. 2012-11-27 11:24:44,004 - Arbitrator.Transporter - ERROR - The transporter 'cumulus' has failed while transporting the file '/var/www/versions/phoenix109-piotrek/sites/default/files/gatorland_default_image.jpg' (action: 1). Error: ''. 2012-11-27 11:24:44,012 - Arbitrator.Transporter - ERROR - The transporter 'cumulus' has failed while transporting the file '/var/www/versions/phoenix109-piotrek/sites/default/files/magic_kingdom_default_image_16.jpg' (action: 1). Error: ''. 2012-11-27 11:24:44,157 - Arbitrator - WARNING - Retry queue -> 'failed_files' persistent list: '/var/www/versions/phoenix109-piotrek/sites/default/files/gatorland_default_image.jpg'. Retrying later. 2012-11-27 11:24:44,228 - Arbitrator - WARNING - Retry queue -> 'failed_files' persistent list: '/var/www/versions/phoenix109-piotrek/sites/default/files/magic_kingdom_default_image_16.jpg'. Retrying later. 2012-11-27 11:24:44,494 - Arbitrator.Transporter - ERROR - The transporter 'cumulus' has failed while transporting the file '/var/www/versions/phoenix109-piotrek/sites/default/files/legoland6_0.jpg' (action: 1). Error: ''. 2012-11-27 11:24:44,512 - Arbitrator.Transporter - ERROR - The transporter 'cumulus' has failed while transporting the file '/var/www/versions/phoenix109-piotrek/sites/default/files/go-orlando-card-sm.gif' (action: 1). Error: ''. 2012-11-27 11:24:44,609 - Arbitrator - WARNING - Retry queue -> 'failed_files' persistent list: '/var/www/versions/phoenix109-piotrek/sites/default/files/legoland6_0.jpg'. Retrying later. 2012-11-27 11:24:44,710 - Arbitrator - WARNING - Retry queue -> 'failed_files' persistent list: '/var/www/versions/phoenix109-piotrek/sites/default/files/go-orlando-card-sm.gif'. Retrying later. 2012-11-27 11:24:44,997 - Arbitrator.Transporter - ERROR - The transporter 'cumulus' has failed while transporting the file '/var/www/versions/phoenix109-piotrek/sites/default/files/magic_kingdom_default_image.jpg' (action: 1). Error: ''. 2012-11-27 11:24:45,043 - Arbitrator.Transporter - ERROR - The transporter 'cumulus' has failed while transporting the file '/var/www/versions/phoenix109-piotrek/sites/default/files/epcot.jpg' (action: 1). Error: ''.

(.. and continue like that)

wimleers commented 11 years ago
  1. Please enable DEBUG logging.
  2. Are you trying to make it work with Rackspace CloudFiles UK or US?
sobi3ch commented 11 years ago
  1. UK

rest in a minute

sobi3ch commented 11 years ago

now is loading ok by it dosn't reading any file. And I'm using no-delete branch made by chrisivens

STAGING (~/fileconveyor/fileconveyor) 15:44 piotrek (no-delete)*$ python arbitrator.py /home/piotrek/fileconveyor/fileconveyor/filter.py:10: DeprecationWarning: the sets module is deprecated from sets import Set, ImmutableSet 2012-11-28 15:44:49,249 - Arbitrator - WARNING - File Conveyor is initializing. 2012-11-28 15:44:49,250 - Arbitrator - INFO - Loading config file. 2012-11-28 15:44:49,252 - Arbitrator.Config - INFO - Parsing sources. 2012-11-28 15:44:49,252 - Arbitrator.Config - INFO - Parsing servers. 2012-11-28 15:44:49,252 - Arbitrator.Config - INFO - Parsing rules. 2012-11-28 15:44:49,253 - Arbitrator - WARNING - Loaded config file. 2012-11-28 09:44:49,368 - Arbitrator - WARNING - Created 'cumulus' transporter for the 'rackspace' server. 2012-11-28 09:44:49,368 - Arbitrator - WARNING - Server connection tests succesful! 2012-11-28 09:44:49,369 - Arbitrator - WARNING - Setup: created transporter pool for the 'rackspace' server. 2012-11-28 09:44:49,370 - Arbitrator - INFO - Setup: collected all metadata for rule 'JS, CSS, Images' (source: 'drupal'). 2012-11-28 09:44:49,371 - Arbitrator - WARNING - Setup: initialized 'pipeline' persistent queue, contains 0 items. 2012-11-28 09:44:49,372 - Arbitrator - WARNING - Setup: initialized 'files_in_pipeline' persistent list, contains 0 items. 2012-11-28 09:44:49,373 - Arbitrator - WARNING - Setup: initialized 'failed_files' persistent list, contains 0 items. 2012-11-28 09:44:49,374 - Arbitrator - WARNING - Setup: initialized 'files_to_delete' persistent list, contains 0 items. 2012-11-28 09:44:49,374 - Arbitrator - WARNING - Setup: moved 0 items from the 'files_in_pipeline' persistent list into the 'pipeline' persistent queue. 2012-11-28 09:44:49,375 - Arbitrator - WARNING - Setup: connected to the synced files DB. Contains metadata for 0 previously synced files. 2012-11-28 09:44:49,459 - Arbitrator.FSMonitor - INFO - FSMonitor class used: FSMonitorInotify. 2012-11-28 09:44:49,460 - Arbitrator - WARNING - Setup: initialized FSMonitor. 2012-11-28 09:44:49,460 - Arbitrator - INFO - Setup: monitoring '/var/www/versions/phoenix109-piotrek' (drupal). 2012-11-28 09:44:49,461 - Arbitrator - INFO - Cleaned up the working directory '/tmp/fileconveyor'. 2012-11-28 09:44:49,461 - Arbitrator - WARNING - Fully up and running now.

sobi3ch commented 11 years ago

I don't follow it starts working suddenly. Probably I've change something, but I'm working on this second day so I'm not aware what it can be.. basicly I've try to install this from scrach 2 times more.

Queastion: is it normal I need to wait long minutes after [Fully up and running now.] message and start syncing?

sobi3ch commented 11 years ago

also during demon start I've this message

/home/piotrek/fileconveyor/fileconveyor/filter.py:10: DeprecationWarning: the sets module is deprecated from sets import Set, ImmutableSet

what's wrong? Seems for me dons't impact on the workflow..

sobi3ch commented 11 years ago

ok closing.. thank's for all

chrisivens commented 11 years ago

I think I need to take a look at my branch and get a pull request sorted pretty urgently. I can then deprecate all the stuff in my branch so no-one uses it once the UK integration is in the main repo. Apologies if there happens to be some duff code in my stuff that has caused you headache. I haven't tested it fully so there is a possibility.

wimleers commented 11 years ago

:)

sobi3ch commented 11 years ago

@chrisivens no worries, thanks for reply

sobi3ch commented 11 years ago

BTW can someone explain me why I can see any information about uploading when I'm running fileconveyor on new container? After "Fully up and running now" message nothings happening.. once (when I install demon for the first time) it was showing me strait away error message for each file that was unable to upload. Is it normal?

Does fileconveyor is in "silent" mode during proper uploading?

sobi3ch commented 11 years ago

correct me if I'm wrong. Seems to uploading files each time I'm chaing someting (createing folder, copying/updating files). So do I need to purge something or change in config to get work with this initial upload?

I've change *.db files in setings.py and container name in with source name, rule for, server name and destination server. And still working only when I changing something in filter folder.