search

wimleers / fileconveyor

File Conveyor is a daemon written in Python to detect, process and sync files. In particular, it's designed to sync files to CDNs. Amazon S3 and Rackspace Cloud Files, as well as any Origin Pull or (S)FTP Push CDN, are supported. Originally written for my bachelor thesis at Hasselt University in Belgium.
https://wimleers.com/fileconveyor
The Unlicense
341 stars 95 forks source link

'Settings' object has no attribute 'CUMULUS' #161

Open dasginganinja opened 11 years ago

dasginganinja commented 11 years ago

Hello all. I've done some searching on this issue and haven't been able to turn up too much so I figured I would finally ask in here. I've been trying to get File Conveyor working with Rackspace's Cloud Files but have been having horrible luck doing so. I installed originally with the instructions from INSTALL.txt and when starting up an instance with a fully configured config.xml file I got these errors while using "transport = 'cloudfiles'".

/usr/local/src/fileconveyor/fileconveyor# python arbitrator.py
/usr/local/src/fileconveyor/fileconveyor/filter.py:10: DeprecationWarning: the sets module is deprecated
  from sets import Set, ImmutableSet
2013-11-25 17:03:39,878 - Arbitrator                - WARNING  - File Conveyor is initializing.
2013-11-25 17:03:39,888 - Arbitrator                - INFO     - Loading config file.
2013-11-25 17:03:39,890 - Arbitrator.Config         - INFO     - Parsing sources.
2013-11-25 17:03:39,890 - Arbitrator.Config         - INFO     - Parsing servers.
2013-11-25 17:03:39,894 - Arbitrator.Config         - INFO     - Parsing rules.
2013-11-25 17:03:39,894 - Arbitrator                - WARNING  - Loaded config file.
Traceback (most recent call last):
  File "arbitrator.py", line 1185, in <module>
    run_file_conveyor()
  File "arbitrator.py", line 1168, in run_file_conveyor
    arbitrator = Arbitrator(os.path.join(FILE_CONVEYOR_PATH, "config.xml"), restart)
  File "arbitrator.py", line 142, in __init__
    transporter_class = self._import_transporter(transporter_name)
  File "arbitrator.py", line 1148, in _import_transporter
    module = __import__(module_name, globals(), locals(), ["TRANSPORTER_CLASS"], -1)
  File "/usr/local/src/fileconveyor/fileconveyor/transporters/transporter_cloudfiles.py", line 2, in <module>
    from cumulus.storage import CloudFilesStorage
  File "/usr/local/lib/python2.6/dist-packages/cumulus/storage.py", line 11, in <module>
    from cumulus.settings import CUMULUS
  File "/usr/local/lib/python2.6/dist-packages/cumulus/settings.py", line 33, in <module>
    if "FILTER_LIST" in settings.CUMULUS.keys():
  File "/usr/local/lib/python2.6/dist-packages/django/conf/__init__.py", line 55, in __getattr__
    return getattr(self._wrapped, name)
AttributeError: 'Settings' object has no attribute 'CUMULUS'

My Python 2.6 dist packages looks like so:

/usr/local/src/fileconveyor/fileconveyor# ls /usr/local/lib/python2.6/dist-packages/ -1
Babel-1.3-py2.6.egg-info
Django-1.6-py2.6.egg-info
argparse-1.2.1-py2.6.egg-info
argparse.py
argparse.pyc
babel
boto
boto-1.6b-py2.6.egg-info
cloudfiles
cssutils
cssutils-0.9.10-py2.6.egg-info
cumulus
distribute-0.6.49-py2.6.egg
django
django_cumulus-1.0.11-py2.6.egg-info
easy-install.pth
encutils
fileconveyor-0.3_dev-py2.6.egg
fileconveyor.egg-link
httplib2
httplib2-0.8-py2.6.egg-info
iso8601
iso8601-0.1.8-py2.6.egg-info
keyring
keyring-3.2.1-py2.6.egg-info
mock-1.0.1-py2.6.egg-info
mock.py
mock.pyc
novaclient
os_diskconfig_python_novaclient_ext
os_diskconfig_python_novaclient_ext-0.1.1-py2.6.egg-info
os_networksv2_python_novaclient_ext-0.21-py2.6.egg-info
os_networksv2_python_novaclient_ext.py
os_networksv2_python_novaclient_ext.pyc
os_virtual_interfacesv2_python_novaclient_ext-0.14-py2.6.egg-info
os_virtual_interfacesv2_python_novaclient_ext.py
os_virtual_interfacesv2_python_novaclient_ext.pyc
pbr
pbr-0.5.23-py2.6.egg-info
pip
pip-1.4.1-py2.6.egg-info
prettytable-0.7.2-py2.6.egg-info
prettytable.py
prettytable.pyc
pyinotify-0.9.4-py2.6.egg-info
pyinotify.py
pyinotify.pyc
pyrax
pyrax-1.4.10-py2.6.egg-info
python_cloudfiles-1.7.10-py2.6.egg-info
python_novaclient-2.15.0-py2.6.egg-info
python_swiftclient-1.8.0-py2.6.egg-info
pytz
pytz-2013.8-py2.6.egg-info
rackspace_auth_openstack
rackspace_auth_openstack-1.1-py2.6.egg-info
rackspace_novaclient-1.4-py2.6.egg-info
rax_default_network_flags_python_novaclient_ext
rax_default_network_flags_python_novaclient_ext-0.1.3-py2.6.egg-info
rax_scheduled_images_python_novaclient_ext
rax_scheduled_images_python_novaclient_ext-0.2.1-py2.6.egg-info
requests
requests-2.0.1-py2.6.egg-info
setuptools-0.6c11-py2.6.egg-info
setuptools.pth
simplejson
simplejson-3.3.1-py2.6.egg-info
six-1.4.1-py2.6.egg-info
six.py
six.pyc
swiftclient

Can somebody point me in the right direction?

Thanks!

dasginganinja commented 11 years ago

Here is what my config.xml looks like:

/usr/local/src/fileconveyor/fileconveyor# cat config.xml
<?xml version="1.0" encoding="UTF-8"?>
<config>
  <!-- Sources -->
  <sources ignoredDirs="">
    <source name="test" scanPath="/var/www" documentRoot="/var/www" basePath="/" />
  </sources>

  <!-- Servers -->
  <servers>
    <server name="rackspace" transporter="cloudfiles">
      <username>###</username>
      <api_key>###</api_key>
      <container>###</container>
    </server>
  </servers>
 <!-- Rules -->
  <rules>
    <rule for="test" label="Media or images only">
              <filter>
                <paths>sites/all:sites/default/files</paths>
                <extensions>js:ico:gif:png:jpg:jpeg:svg:swf:css</extensions>
              </filter>
              <processorChain>
              </processorChain>
              <destinations>
                <destination server="rackspace" path="" />
              </destinations>
            </rule>
  </rules>
</config>
dasginganinja commented 11 years ago

I've got a little bit more of an update for you.

I brought up a brand new Vagrant Debian Squeeze box, installed git, pip, and File Conveyor per INSTALL.txt. After configuring config.xml for cloudfiles I am still getting the following error after a 

root@vagrant-debian-squeeze:~# python src/fileconveyor/fileconveyor/arbitrator.py
/root/src/fileconveyor/fileconveyor/filter.py:10: DeprecationWarning: the sets module is deprecated
  from sets import Set, ImmutableSet
2013-11-26 08:31:08,075 - Arbitrator                - WARNING  - File Conveyor is initializing.
2013-11-26 08:31:08,081 - Arbitrator                - WARNING  - Loaded config file.
Traceback (most recent call last):
  File "src/fileconveyor/fileconveyor/arbitrator.py", line 1185, in <module>
    run_file_conveyor()
  File "src/fileconveyor/fileconveyor/arbitrator.py", line 1168, in run_file_conveyor
    arbitrator = Arbitrator(os.path.join(FILE_CONVEYOR_PATH, "config.xml"), restart)
  File "src/fileconveyor/fileconveyor/arbitrator.py", line 142, in __init__
    transporter_class = self._import_transporter(transporter_name)
  File "src/fileconveyor/fileconveyor/arbitrator.py", line 1148, in _import_transporter
    module = __import__(module_name, globals(), locals(), ["TRANSPORTER_CLASS"], -1)
  File "/root/src/fileconveyor/fileconveyor/transporters/transporter_cloudfiles.py", line 2, in <module>
    from cumulus.storage import CloudFilesStorage
  File "/usr/local/lib/python2.6/dist-packages/cumulus/storage.py", line 11, in <module>
    from cumulus.settings import CUMULUS
  File "/usr/local/lib/python2.6/dist-packages/cumulus/settings.py", line 33, in <module>
    if "FILTER_LIST" in settings.CUMULUS.keys():
  File "/usr/local/lib/python2.6/dist-packages/django/conf/__init__.py", line 55, in __getattr__
    return getattr(self._wrapped, name)
AttributeError: 'Settings' object has no attribute 'CUMULUS'

Here's the output from pip:

root@vagrant-debian-squeeze:~# pip freeze
Babel==1.3
Django==1.6
argparse==1.2.1
boto==1.6b
cssutils==0.9.10
distribute==0.6.49
django-cumulus==1.0.11
-e git+https://github.com/wimleers/fileconveyor@ca3585edbdcc75973c35bd8ee4d52601153bf6a3#egg=fileconveyor-master
httplib2==0.8
iso8601==0.1.8
keyring==3.2.1
mock==1.0.1
os-diskconfig-python-novaclient-ext==0.1.1
os-networksv2-python-novaclient-ext==0.21
os-virtual-interfacesv2-python-novaclient-ext==0.14
pbr==0.5.23
prettytable==0.7.2
pyinotify==0.9.4
pyrax==1.4.10
python-apt==0.7.100.1-squeeze1
python-cloudfiles==1.7.10
python-novaclient==2.15.0
python-swiftclient==1.8.0
pytz==2013.8
rackspace-auth-openstack==1.1
rackspace-novaclient==1.4
rax-default-network-flags-python-novaclient-ext==0.1.3
rax-scheduled-images-python-novaclient-ext==0.2.1
reportbug==4.12.6
requests==2.0.1
simplejson==3.3.1
six==1.4.1
wsgiref==0.1.2
omnia-ibrahim commented 10 years ago

I have the same errors, any help?

dasginganinja commented 10 years ago

None here. I'm still waiting on Wim Leers to respond. I'm assuming that this is because one of the packages is at a non-working version.

Can somebody who has a working copy of fileconveyor please put up a pip listing of the currently installed modules? We might be able to determine which module is the cause of the current problem that we are all experiencing with getting this module working.

Again, this problem is only for the select few of us that will be utilizing cloudfiles. Signs point to the cumulus module. I think I'm going to start from there as I need to have this fixed by the end of the day.

omnia-ibrahim commented 10 years ago

What other alternatives if this is not fixed? i want to use cloud files

chrisivens commented 10 years ago

When I was last using fileconveyor, it was at the time when the cloudfiles stuff was just a pull request so it's a while ago. Having said that, I did get it running and unfortunately I haven't got access to that machine any more. On looking at the docs for django-cumulus though, it does mention the CUMULUS object.

CUMULUS = {
    'USERNAME': 'YourUsername',
    'API_KEY': 'YourAPIKey',
    'CONTAINER': 'ContainerName'
}

Now I suspect that it'll need adding to the django-settings.py file but this is just a hunch and I can no longer test with any Rackspace stuff without setting up a new account.

omnia-ibrahim commented 10 years ago

I tried adding the CUMULUS object in both the django-settings in fileconveyor and also in the django settings file itself and both didn't work :(

chrisivens commented 10 years ago

My old fork of fileconveyor has this in the django_settings.py file. I suspect I've missed out the api key etc:

SECRET_KEY='change me for something better'
# Dummy settings for `django-storages`.
MEDIA_URL=''
MEDIA_ROOT=''
# `backends/ftp.py`
FTP_STORAGE_LOCATION=''
# `backends/sftp.py`
SFTP_STORAGE_HOST=''
# django-cumulus
#CUMULUS['USERNAME'] = '';
#CUMULUS['CUMULUS_API_KEY'] = '';
#CUMULUS['CONTAINER'] = '';
CUMULUS_API_KEY = '';
CUMULUS_USERNAME = '';
CUMULUS_CONTAINER = '';
CUMULUS = {
  'AUTH_URL' : 'uk_authurl'
}

It doesn't look like my settings.py file differs much. You can look over my fork if you wish but I warn you, it's deliberately broken to prevent local deletions propagating to Cloudfiles. Oh, and it's quite a bit behind.

omnia-ibrahim commented 10 years ago

its almost the same as mine, i have a feeling its not related to the settings in fileconveyor, but something with django and its settings, but i have no idea why its not working

dasginganinja commented 10 years ago

Good morning all! I don't have any updates on my end other than the fact that I've tried multiple versions of cumulus, django, etc. and still can't get it working. I created a gist (https://gist.github.com/dasginganinja/7769677) with some outputs that might be helpful in diagnosing. In the gist I give the setup done to the box to get it up and running, the strace output for starting the arbitrator, and a pip freeze to show what python modules are actually being used.

Hopefully this helps somebody diagnose further! Here's a quick shortcut to the strace output: https://gist.github.com/dasginganinja/7769677/raw/696b112bf25a967ecab63cf08330ac4f6569425b/Strace+Output.txt

dasginganinja commented 10 years ago

Okay here's another little update. I went ahead and put the CUMULUS object into /usr/lib/local/python2.6/dist-packages/django/conf/global_settings.py with my correct settings. Here is the output when I run now:

tom@dev2-lamp:/tmp$ sudo vim /usr/local/lib/python2.6/dist-packages/django/conf/global_settings.py
tom@dev2-lamp:/tmp$ python /usr/local/lib/python2.6/dist-packages/fileconveyor-0.3_dev-py2.6.egg/fileconveyor/arbitrator.py
/usr/local/lib/python2.6/dist-packages/fileconveyor-0.3_dev-py2.6.egg/fileconveyor/filter.py:10: DeprecationWarning: the sets module is deprecated
  from sets import Set, ImmutableSet
2013-12-03 10:22:22,742 - Arbitrator                - WARNING  - File Conveyor is initializing.
2013-12-03 10:22:22,748 - Arbitrator                - WARNING  - Loaded config file.
Traceback (most recent call last):
  File "/usr/local/lib/python2.6/dist-packages/fileconveyor-0.3_dev-py2.6.egg/fileconveyor/arbitrator.py", line 1185, in <module>
    run_file_conveyor()
  File "/usr/local/lib/python2.6/dist-packages/fileconveyor-0.3_dev-py2.6.egg/fileconveyor/arbitrator.py", line 1168, in run_file_conveyor
    arbitrator = Arbitrator(os.path.join(FILE_CONVEYOR_PATH, "config.xml"), restart)
  File "/usr/local/lib/python2.6/dist-packages/fileconveyor-0.3_dev-py2.6.egg/fileconveyor/arbitrator.py", line 142, in __init__
    transporter_class = self._import_transporter(transporter_name)
  File "/usr/local/lib/python2.6/dist-packages/fileconveyor-0.3_dev-py2.6.egg/fileconveyor/arbitrator.py", line 1162, in _import_transporter
    self.logger.error("The Transporter module '%s' was found, but its Transporter class '%s' could not be found."  % (module_name, classname))
UnboundLocalError: local variable 'classname' referenced before assignment

As you can see, at this point this error is now what is being shown in #162.

omnia-ibrahim commented 10 years ago

i think he is asking for the cloud files storage, i added it to settings DEFAULT_FILE_STORAGE = 'cumulus.storage.CloudFilesStorage', but also didn't work

dasginganinja commented 10 years ago

At this point all I think we need to do is find out why 'classname' isn't getting set. I'm sure that will open a whole other can of worms but that's the next step.

omnia-ibrahim commented 10 years ago

you reached anything about this class issue? i couldn't solve it :s

dasginganinja commented 10 years ago

I went back through and took a look at the PIP Install log for this program and noticed the following lines:


  Running setup.py install for python-cloudfiles
    cloudfiles/__init__.py:90: DeprecationWarning: python-cloudfiles has been deprecated as of August 1, 2013. Please see https://github.com/openstack/python-swiftclient.
      DeprecationWarning)
    /tmp/pip_build_root/python-cloudfiles/setup.py:11: DeprecationWarning: python-cloudfiles has been deprecated as of August 1, 2013. Please see https://github.com/openstack/python-swiftclient.
      DeprecationWarning)
    warning: no previously-included files found matching 'python_cloudfiles.egg-info'

I wonder if this has something to do with it....

ihuro commented 10 years ago

Update django-cumulus to 1.0.10: easy_install "django-cumuls==1.0.10" or pip install django-cumulus==1.0.10

In setup.py the dependencies must be >=1.0.5 and <=1.0.10 for now

omnia-ibrahim commented 10 years ago

The error is now gone thanksss :)

But i have another issue, when saving images i got this error: Filter queue: dropped 'files/blue-search-icon_2.png' because it doesn't match any rules

stifboy commented 10 years ago

I have put the cumulus object into django/conf/global_settings.py with my settings but now I am getting this error

Last login: Fri Mar 28 22:28:25 2014 from 196.201.228.134 [root@af-server ~]# sudo python src/fileconveyor/fileconveyor/arbitrator.py /root/src/fileconveyor/fileconveyor/filter.py:10: DeprecationWarning: the sets m odule is deprecated from sets import Set, ImmutableSet 2014-03-28 22:34:38,206 - Arbitrator - WARNING - File Conveyor i s initializing. 2014-03-28 22:34:38,210 - Arbitrator - WARNING - Loaded config f ile. Traceback (most recent call last): File "src/fileconveyor/fileconveyor/arbitrator.py", line 1185, in run_file_conveyor() File "src/fileconveyor/fileconveyor/arbitrator.py", line 1168, in run_file_con veyor arbitrator = Arbitrator(os.path.join(FILE_CONVEYOR_PATH, "config.xml"), rest art) File "src/fileconveyor/fileconveyor/arbitrator.py", line 142, in init transporter_class = self._import_transporter(transporter_name) File "src/fileconveyor/fileconveyor/arbitrator.py", line 1162, in _import_tran sporter self.logger.error("The Transporter module '%s' was found, but its Transporte r class '%s' could not be found." % (module_name, classname)) UnboundLocalError: local variable 'classname' referenced before assignment [root@af-server ~]#

I have already updated django-cumulus to 1.0.10 and still get the error.