Importing tags with >999 files fails

[dbrgn]

When using the importswid command on Dropbox/BA/swid-tags-full-jfu.txt, it fails on several packages with the following trace:

Traceback (most recent call last):
  File "./manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/home/danilo/.virtualenvs/strongtnc/lib/python2.7/site-packages/django/core/management/__init__.py", line 399, in execute_from_command_line
    utility.execute()
  File "/home/danilo/.virtualenvs/strongtnc/lib/python2.7/site-packages/django/core/management/__init__.py", line 392, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/danilo/.virtualenvs/strongtnc/lib/python2.7/site-packages/django/core/management/base.py", line 242, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/home/danilo/.virtualenvs/strongtnc/lib/python2.7/site-packages/django/core/management/base.py", line 285, in execute
    output = self.handle(*args, **options)
  File "/mnt/data/Projects/ba/strongTNC/apps/swid/management/commands/importswid.py", line 38, in handle
    tag = utils.process_swid_tag(tag_xml)
  File "/home/danilo/.virtualenvs/strongtnc/lib/python2.7/site-packages/django/db/transaction.py", line 371, in inner
    return func(*args, **kwargs)
  File "/mnt/data/Projects/ba/strongTNC/apps/swid/utils.py", line 73, in process_swid_tag
    tag.files = file_pks
  File "/home/danilo/.virtualenvs/strongtnc/lib/python2.7/site-packages/django/db/models/fields/related.py", line 842, in __set__
    manager.add(*value)
  File "/home/danilo/.virtualenvs/strongtnc/lib/python2.7/site-packages/django/db/models/fields/related.py", line 583, in add
    self._add_items(self.source_field_name, self.target_field_name, *objs)
  File "/home/danilo/.virtualenvs/strongtnc/lib/python2.7/site-packages/django/db/models/fields/related.py", line 658, in _add_items
    new_ids = new_ids - set(vals)
  File "/home/danilo/.virtualenvs/strongtnc/lib/python2.7/site-packages/django/db/models/query.py", line 96, in __iter__
    self._fetch_all()
  File "/home/danilo/.virtualenvs/strongtnc/lib/python2.7/site-packages/django/db/models/query.py", line 854, in _fetch_all
    self._result_cache = list(self.iterator())
  File "/home/danilo/.virtualenvs/strongtnc/lib/python2.7/site-packages/django/db/models/query.py", line 1065, in iterator
    for row in self.query.get_compiler(self.db).results_iter():
  File "/home/danilo/.virtualenvs/strongtnc/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 713, in results_iter
    for rows in self.execute_sql(MULTI):
  File "/home/danilo/.virtualenvs/strongtnc/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 786, in execute_sql
    cursor.execute(sql, params)
  File "/home/danilo/.virtualenvs/strongtnc/lib/python2.7/site-packages/django/db/backends/util.py", line 69, in execute
    return super(CursorDebugWrapper, self).execute(sql, params)
  File "/home/danilo/.virtualenvs/strongtnc/lib/python2.7/site-packages/django/db/backends/util.py", line 53, in execute
    return self.cursor.execute(sql, params)
  File "/home/danilo/.virtualenvs/strongtnc/lib/python2.7/site-packages/django/db/utils.py", line 99, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/home/danilo/.virtualenvs/strongtnc/lib/python2.7/site-packages/django/db/backends/util.py", line 53, in execute
    return self.cursor.execute(sql, params)
  File "/home/danilo/.virtualenvs/strongtnc/lib/python2.7/site-packages/django/db/backends/sqlite3/base.py", line 451, in execute
    return Database.Cursor.execute(self, query, params)
django.db.utils.OperationalError: too many SQL variables

This appears to be an SQLite issue. Apparently SQLite does not allow more than 999 parameters to be present. If more than 999 files are assigned to a tag, all those PKs are sent to the DB in a single query.

We could start splitting the queries into smaller chunks. Although I still think that SQLite should be avoided and that the docs and README should discourage its use.

[dbrgn]

Added a workaround. Please review, @cfaessler or @d22.

Additionally, this commit adds status output to the importswid command and fixes some encoding issues.

[cfaessler]

Upon trying the importswid command a got the following exception:

Traceback (most recent call last):
  File "C:\Program Files (x86)\JetBrains\PyCharm 3.1.1\helpers\pycharm\django_manage.py", line 23, in <module>
    run_module(manage_file, None, '__main__', True)
  File "C:\Python27\Lib\runpy.py", line 176, in run_module
    fname, loader, pkg_name)
  File "C:\Python27\Lib\runpy.py", line 82, in _run_module_code
    mod_name, mod_fname, mod_loader, pkg_name)
  File "C:\Python27\Lib\runpy.py", line 72, in _run_code
    exec code in run_globals
  File "C:\git\BA\strongTNC\manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "C:\Users\chrigi\tnc-django\lib\site-packages\django\core\management\__init__.py", line 399, in execute_from_command_line
    utility.execute()
  File "C:\Users\chrigi\tnc-django\lib\site-packages\django\core\management\__init__.py", line 392, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "C:\Users\chrigi\tnc-django\lib\site-packages\django\core\management\base.py", line 242, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "C:\Users\chrigi\tnc-django\lib\site-packages\django\core\management\base.py", line 285, in execute
    output = self.handle(*args, **options)
  File "C:/git/BA/strongTNC\apps\swid\management\commands\importswid.py", line 39, in handle
    self.stdout.write('Processed {0}'.format(tag).encode(self.stdout.encoding, 'replace'))
TypeError: encode() argument 1 must be string, not None