yeti-2.0a quickstart error

[andreisirghi]

When I try to rune yeti-2.0a using scripts/quickstart.sh I've got a strange error:

  File "/Library/Python/2.7/site-packages/django/core/management/__init__.py", line 453, in execute_from_command_line
    utility.execute()
  File "/Library/Python/2.7/site-packages/django/core/management/__init__.py", line 392, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/Library/Python/2.7/site-packages/django/core/management/__init__.py", line 263, in fetch_command
    app_name = get_commands()[subcommand]
  File "/Library/Python/2.7/site-packages/django/core/management/__init__.py", line 109, in get_commands
    apps = settings.INSTALLED_APPS
  File "/Library/Python/2.7/site-packages/django/conf/__init__.py", line 53, in __getattr__
    self._setup(name)
  File "/Library/Python/2.7/site-packages/django/conf/__init__.py", line 48, in _setup
    self._wrapped = Settings(settings_module)
  File "/Library/Python/2.7/site-packages/django/conf/__init__.py", line 132, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/Library/Python/2.7/site-packages/django/utils/importlib.py", line 35, in import_module
    __import__(name)
  File "/Users/macuser4/Documents/work/yeti-2.0a/yeti/__init__.py", line 6, in <module>
    import StixXml111QueryHandler
  File "/Users/macuser4/Documents/work/yeti-2.0a/yeti/StixXml111QueryHandler.py", line 4, in <module>
    from taxii_services.base_taxii_handlers import BaseXmlQueryHandler
  File "build/bdist.macosx-10.8-intel/egg/taxii_services/__init__.py", line 7, in <module>
AttributeError: 'module' object has no attribute 'setup'

I've installed all dependencies including taxii-services yeti-1.1.0 works ok

Is there a bug or the deployment/run procedure is different?

[MarkDavidson]

What version of Django you are running? The django.setup() function, which django-taxii-services uses, was added in (I think) Django 1.7.

If you are running a version of Django older than 1.7, try updating pip install django --upgrade. If that fixes your problem, I can add an issue to check the Django version in software and display a more useful error message. Something like "You are using an unsupported version of Django, please update".

Thank you. -Mark

[andreisirghi]

I've used django 1.5.1 but after upgrading to django 1.7 a new error is shown:

  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/Library/Python/2.7/site-packages/django/core/management/__init__.py", line 385, in execute_from_command_line
    utility.execute()
  File "/Library/Python/2.7/site-packages/django/core/management/__init__.py", line 345, in execute
    settings.INSTALLED_APPS
  File "/Library/Python/2.7/site-packages/django/conf/__init__.py", line 46, in __getattr__
    self._setup(name)
  File "/Library/Python/2.7/site-packages/django/conf/__init__.py", line 42, in _setup
    self._wrapped = Settings(settings_module)
  File "/Library/Python/2.7/site-packages/django/conf/__init__.py", line 98, in __init__
    % (self.SETTINGS_MODULE, e)
ImportError: Could not import settings 'yeti.settings' (Is it on sys.path? Is there an import error in the settings file?): No module named constants

[MarkDavidson]

I'm guessing libtaxii is the dependency issue here. Are you on libtaxii 1.1.103?

Clearly the dependencies are not checked very well. I'm going to look into the best way to solve this (maybe the dependencies are not clearly spelled out).

I've opened https://github.com/TAXIIProject/django-taxii-services/issues/14 to look into how dependencies are addressed in django-taxii-services.

Thank you for raising these issues! -Mark

[andreisirghi]

Yep, is hard to identify required libs, at least you can add an updated requirements.txt as for release 1.1.0. But, even after upgrading libtaxii to 1.1.103 I've got a new error:

File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/Library/Python/2.7/site-packages/django/core/management/__init__.py", line 385, in execute_from_command_line
    utility.execute()
  File "/Library/Python/2.7/site-packages/django/core/management/__init__.py", line 345, in execute
    settings.INSTALLED_APPS
  File "/Library/Python/2.7/site-packages/django/conf/__init__.py", line 46, in __getattr__
    self._setup(name)
  File "/Library/Python/2.7/site-packages/django/conf/__init__.py", line 42, in _setup
    self._wrapped = Settings(settings_module)
  File "/Library/Python/2.7/site-packages/django/conf/__init__.py", line 94, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/Users/macuser4/Documents/work/yeti-new/yeti/__init__.py", line 6, in <module>
    import StixXml111QueryHandler
  File "/Users/macuser4/Documents/work/yeti-new/yeti/StixXml111QueryHandler.py", line 1345, in <module>
    register_query_handler(StixXml111QueryHandler, name="StixXml111QueryHandler")
  File "/Users/macuser4/Documents/work/yeti-new/taxii_services/management.py", line 64, in register_query_handler
  File "/Library/Python/2.7/site-packages/django/db/models/manager.py", line 92, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/Library/Python/2.7/site-packages/django/db/models/query.py", line 422, in get_or_create
    return self.get(**lookup), False
  File "/Library/Python/2.7/site-packages/django/db/models/query.py", line 345, in get
    clone = self.filter(*args, **kwargs)
  File "/Library/Python/2.7/site-packages/django/db/models/query.py", line 691, in filter
    return self._filter_or_exclude(False, *args, **kwargs)
  File "/Library/Python/2.7/site-packages/django/db/models/query.py", line 709, in _filter_or_exclude
    clone.query.add_q(Q(*args, **kwargs))
  File "/Library/Python/2.7/site-packages/django/db/models/sql/query.py", line 1287, in add_q
    clause, require_inner = self._add_q(where_part, self.used_aliases)
  File "/Library/Python/2.7/site-packages/django/db/models/sql/query.py", line 1314, in _add_q
    current_negated=current_negated, connector=connector)
  File "/Library/Python/2.7/site-packages/django/db/models/sql/query.py", line 1138, in build_filter
    lookups, parts, reffed_aggregate = self.solve_lookup_type(arg)
  File "/Library/Python/2.7/site-packages/django/db/models/sql/query.py", line 1076, in solve_lookup_type
    _, field, _, lookup_parts = self.names_to_path(lookup_splitted, self.get_meta())
  File "/Library/Python/2.7/site-packages/django/db/models/sql/query.py", line 1339, in names_to_path
    field, model, direct, m2m = opts.get_field_by_name(name)
  File "/Library/Python/2.7/site-packages/django/db/models/options.py", line 416, in get_field_by_name
    cache = self.init_name_map()
  File "/Library/Python/2.7/site-packages/django/db/models/options.py", line 445, in init_name_map
    for f, model in self.get_all_related_m2m_objects_with_model():
  File "/Library/Python/2.7/site-packages/django/db/models/options.py", line 563, in get_all_related_m2m_objects_with_model
    cache = self._fill_related_many_to_many_cache()
  File "/Library/Python/2.7/site-packages/django/db/models/options.py", line 570, in _fill_related_many_to_many_cache
    for obj, model in parent._meta.get_all_related_m2m_objects_with_model():
  File "/Library/Python/2.7/site-packages/django/db/models/options.py", line 563, in get_all_related_m2m_objects_with_model
    cache = self._fill_related_many_to_many_cache()
  File "/Library/Python/2.7/site-packages/django/db/models/options.py", line 577, in _fill_related_many_to_many_cache
    for klass in self.apps.get_models():
  File "/Library/Python/2.7/site-packages/django/utils/lru_cache.py", line 101, in wrapper
    result = user_function(*args, **kwds)
  File "/Library/Python/2.7/site-packages/django/apps/registry.py", line 168, in get_models
    self.check_models_ready()
  File "/Library/Python/2.7/site-packages/django/apps/registry.py", line 131, in check_models_ready
    raise AppRegistryNotReady("Models aren't loaded yet.")
django.core.exceptions.AppRegistryNotReady: Models aren't loaded yet.

taxii-services 0.1.1 libtaxii 1.1.103 Django 1.7

[MarkDavidson]

So the dependencies I will fix by adding install_requires = ['libtaxii>=1.1.103', 'django>=1.7.0', 'python-dateutil>=1.4.1', 'lxml>=2.2.3'] to setup.py. I'll push out an update release to django-taxii-services with that once we get your issues resolved.

Related to your error, I've found two related pieces of information:

• https://docs.djangoproject.com/en/dev/releases/1.7/
• http://stackoverflow.com/questions/25244631/models-arent-loaded-yet-error-while-populating-in-django1-8-and-python2-7-8 (This is why the django.setup() call was added)

Let me do some digging to see if I can figure out what the root cause is.

Are you getting this error from running python manage.py runserver ?

[andreisirghi]

I'm getting it in both cases: python manage.py syncdb or python manage.py runserver. I'll try to find a fix and inform you in any case.

[MarkDavidson]

What happens if you comment out register_query_handler(StixXml111QueryHandler, name="StixXml111QueryHandler") in yeti/StixXml111QueryHandler.py ?

That appears to be the line that causes the error, and it's not needed to run YETI (yet) - it's a test for the TAXII Query extension point..

Edit: Fixed filename for yeti/StixXml111QueryHandler.py

[andreisirghi]

it works!, Thanks a lot!!!

[MarkDavidson]

Nice to hear! I've opened an issue for what you're seeing: https://github.com/TAXIIProject/django-taxii-services/issues/15.

I'm not able to reproduce it myself, so the resolution might have to wait. If you know how to reproduce it, would you be able to list those steps on the django-taxii-services tracker?

In the interim, I'll do an update release of YETI to include that commented out line, since it's not a necessary part of YETI just yet.

[MarkDavidson]

I've just done an update release of django-taxii-services (https://pypi.python.org/pypi/taxii-services/0.1.2) and YETI (https://github.com/TAXIIProject/yeti/releases/tag/2.0a2). Hopefully these update releases resolve your issue(s).

I'll close this issue in a bit unless there are further comments.

[andreisirghi]

the server seems to work, I re-tried to reproduce django.core.exceptions.AppRegistryNotReady: Models aren't loaded yet.. But even after uncommenting that line and removing database, everything was started.

[MarkDavidson]

Closing the issue since this appears to be fixed. Please keep the bug reports coming!

[jargcomp]

I am having the same issue mentioned in the thrid comment except I am in version 2.0a2

administrator@ubuntu-taxii:~/Documents/yeti-2.0a2$ ./scripts/quickstart.sh Traceback (most recent call last): File "manage.py", line 10, in execute_from_command_line(sys.argv) File "/usr/local/lib/python2.7/dist-packages/django/core/management/init.py", line 385, in execute_from_command_line utility.execute() File "/usr/local/lib/python2.7/dist-packages/django/core/management/init.py", line 345, in execute settings.INSTALLED_APPS File "/usr/local/lib/python2.7/dist-packages/django/conf/init.py", line 46, in getattr self._setup(name) File "/usr/local/lib/python2.7/dist-packages/django/conf/init.py", line 42, in _setup self._wrapped = Settings(settings_module) File "/usr/local/lib/python2.7/dist-packages/django/conf/init.py", line 98, in init % (self.SETTINGS_MODULE, e) ImportError: Could not import settings 'yeti.settings' (Is it on sys.path? Is there an import error in the settings file?): No module named taxii_services.base_taxii_handlers Traceback (most recent call last): File "manage.py", line 10, in execute_from_command_line(sys.argv) File "/usr/local/lib/python2.7/dist-packages/django/core/management/init.py", line 385, in execute_from_command_line utility.execute() File "/usr/local/lib/python2.7/dist-packages/django/core/management/init.py", line 345, in execute settings.INSTALLED_APPS File "/usr/local/lib/python2.7/dist-packages/django/conf/init.py", line 46, in getattr self._setup(name) File "/usr/local/lib/python2.7/dist-packages/django/conf/init.py", line 42, in _setup self._wrapped = Settings(settings_module) File "/usr/local/lib/python2.7/dist-packages/django/conf/init.py", line 98, in init % (self.SETTINGS_MODULE, e) ImportError: Could not import settings 'yeti.settings' (Is it on sys.path? Is there an import error in the settings file?): No module named taxii_services.base_taxii_handlers

my versions are as follows: Name: libtaxii Version: 1.1.103 Name: Django Version: 1.7

[MarkDavidson]

@jargcomp

It seems that the error is that you don't have the taxii_services library installed.

Quick Note - I just now pushed an updated taxii_services to PyPI (v0.2), and I will be pushing a new version of YETI shortly. YETI 2.0a2 should be paired with taxii_services 0.1.2.

YETI 2.0a3 (which I will push shortly) should be paired with taxii_services 0.2.

I am realizing that the README.md file does not list taxii_services as a dependency, which I'll update.

-Mark

Edit: Did that help? Let me know if that fixes what you're seeing or not.

[jargcomp]

@MarkDavidson, tried out 2.0a3 after adding the taxi_services and it worked like a charm. Thanks!

[MarkDavidson]

@jargcomp, Great! Let me know if you run into anything else or have any more questions. YETI and taxii_services are still a bit of a work in progress, so there may be some non-obvious things.

[bworrell]

@MarkDavidson, I'm not sure if this is a terrible idea or not so feel free to shoot it down :) What do you think about putting a quick import sanity check in manage.py that verifies that taxii_services can be imported and that the correct version is installed.

Something like... (warning: this is off the top of my head)

from pkg_resources import parse_version

MIN_TAXII_SERVICES_VERSION = "0.1.2"
ERR_TAXII_SERVICES = "Please install taxii_services >= v%s" % MIN_TAXII_SERVICES_VERSION

try:
    import taxii_services
except ImportError:
    sys.stderr.write(ERR_TAXII_SERVICES)
    sys.exit(1)

if parse_version(taxii_services.__version__) < parse_version(MIN_TAXII_SERVICES_VERSION):
    raise Exception(ERR_TAXII_SERVICES)

Maybe it's only there for while everything is in alpha and still being developed heavily. I dunno, just an idea!

[bworrell]

Oh, and adding a requirements.txt might be helpful.

[MarkDavidson]

Probably both are a good, especially for the time when taxii_services is quickly evolving.