search

wooey / Wooey

A Django app that creates automatic web UIs for Python scripts.
http://wooey.readthedocs.org
BSD 3-Clause "New" or "Revised" License
2.13k stars 184 forks source link

Having issues with running any scripts #244

Closed setsimmo closed 4 years ago

setsimmo commented 6 years ago

Hi,

Just setup wooey for the first time and imported a simple script, but can't seem to get it to run anything. Whenever I run the script, it fails to ever complete or error out, and I see the following output to stdout by the runserver proc:

[22/May/2018 14:37:58] "GET /jobs/5/jsonhtml HTTP/1.1" 500 23741
Internal Server Error: /jobs/5/jsonhtml
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/sqlite3/base.py", line 328, in execute
    return Database.Cursor.execute(self, query, params)
sqlite3.OperationalError: no such table: wooey_cache_table

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python3.6/site-packages/django/views/generic/base.py", line 68, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/django/views/generic/base.py", line 88, in dispatch
    return handler(request, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/django/views/generic/detail.py", line 117, in get
    return self.render_to_response(context)
  File "/usr/local/lib/python3.6/site-packages/wooey/views/wooey_celery.py", line 248, in render_to_response
    'stdout': context['job_info']['job'].get_stdout(),
  File "/usr/local/lib/python3.6/site-packages/wooey/models/core.py", line 268, in get_stdout
    rt = self.get_realtime().get('stdout')
  File "/usr/local/lib/python3.6/site-packages/wooey/models/core.py", line 261, in get_realtime
    out = cache.get(self.get_realtime_key())
  File "/usr/local/lib/python3.6/site-packages/django/core/cache/backends/db.py", line 62, in get
    "WHERE cache_key = %%s" % table, [key])
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/utils.py", line 79, in execute
    return super(CursorDebugWrapper, self).execute(sql, params)
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.6/site-packages/django/db/utils.py", line 94, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/usr/local/lib/python3.6/site-packages/django/utils/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/sqlite3/base.py", line 328, in execute
    return Database.Cursor.execute(self, query, params)
django.db.utils.OperationalError: no such table: wooey_cache_table
NoneType: None
NoneType: None
Chris7 commented 6 years ago

Thanks for reporting this issue. Can you:

  1. Post your user_settings.py
  2. Your your OS
  3. The output of pip freeze
setsimmo commented 6 years ago
  1. Just using the default user_settings.py: 
    
import errno
import os
from .wooey_settings import *
# This file is where the user can override and customize their installation of wooey

Wooey Apps - add additional apps here after the initial install (remember to follow everything by a comma)

INSTALLED_APPS += ( )

Whether to allow anonymous job submissions, set False to disallow 'guest' job submissions

WOOEY_ALLOW_ANONYMOUS = True

Celery related options

INSTALLED_APPS += ( 'django_celery_results', 'kombu.transport.filesystem', )

This stores the results of tasks. For larger sites, a database may become slow and other solutions

such as redis should be considered.

CELERY_RESULT_BACKEND = 'django-db'

This should absolutely be changed to a non-filesystem based broker for production deployments!

http://docs.celeryproject.org/en/latest/getting-started/brokers/

CELERY_BROKER_URL = 'filesystem://'

This function exists just to ensure the filesystem has the correct folders

def ensure_path(path): try: os.makedirs(path) except Exception as e: if e.errno == errno.EEXIST: pass else: raise return path

broker_dir = ensure_path(os.path.join(BASE_DIR, '.broker')) CELERY_BROKER_TRANSPORT_OPTIONS = { "data_folder_in": ensure_path(os.path.join(broker_dir, "out")), "data_folder_out": ensure_path(os.path.join(broker_dir, "out")), "data_folder_processed": ensure_path(os.path.join(broker_dir, "processed")), }

CELERY_TRACK_STARTED = True WOOEY_CELERY = True CELERY_SEND_EVENTS = True CELERY_IMPORTS = ('wooey.tasks',)

A cache interface. This provides realtime updates for scriots and should definitely be changed

to use something like redis or memcached in production

WOOEY_REALTIME_CACHE = 'default' CACHES = { 'default': { 'BACKEND': 'django.core.cache.backends.db.DatabaseCache', 'LOCATION': 'wooey_cache_table', } }

Things you most likely do not need to change

the directory for uploads (physical directory)

MEDIA_ROOT = os.path.join(BASE_DIR, 'user_uploads')

the url mapping

MEDIA_URL = '/uploads/'

the directory to store our webpage assets (images, javascript, etc.)

STATIC_ROOT = os.path.join(BASE_DIR, 'static')

the url mapping

STATIC_URL = '/static/'

Here is a setup example for production servers

A postgres database -- for multiple users a sqlite based database is asking for trouble

DATABASES = {

'default': {

'ENGINE': 'django.db.backends.postgresql_psycopg2',

for production environments, these should be stored as environment variables

I also recommend the django-heroku-postgresify package for a super simple setup

'NAME': os.environ.get('DATABASE_NAME', 'wooey'),

'USER': os.environ.get('DATABASE_USER', 'wooey'),

'PASSWORD': os.environ.get('DATABASE_PASSWORD', 'wooey'),

'HOST': os.environ.get('DATABASE_URL', 'localhost'),

'PORT': os.environ.get('DATABASE_PORT', '5432')

}

}

A better celery broker -- using RabbitMQ (these defaults are from two free rabbitmq Heroku providers)

CELERY_BROKER_URL = os.environ.get('AMQP_URL') or \

os.environ.get('RABBITMQ_BIGWIG_TX_URL') or \

os.environ.get('CLOUDAMQP_URL', 'amqp://guest:guest@localhost:5672/')

CELERY_BROKER_POOL_LIMIT = 1

CELERYD_CONCURRENCY = 1

CELERY_TASK_SERIALIZER = 'json'

CELERY_TASK_ACKS_LATE = True

#

for production environments, django-storages abstracts away much of the difficulty of various storage engines.

Here is an example for hosting static and user generated content with S3

from boto.s3.connection import VHostCallingFormat

#

INSTALLED_APPS += (

'storages',

'collectfast',

)

We have user authentication -- we need to use https (django-sslify)

NOTE: This is MIDDLEWARE and not MIDDLEWARE_CLASSES in Django 1.10+!

if not DEBUG:

MIDDLEWARE_CLASSES = ['sslify.middleware.SSLifyMiddleware']+list(MIDDLEWARE_CLASSES)

SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')

#

ALLOWED_HOSTS = (

'localhost',

'127.0.0.1',

"wooey.herokuapp.com",# put your site here

)

#

AWS_CALLING_FORMAT = VHostCallingFormat

#

AWS_ACCESS_KEY_ID = environ.get('AWS_ACCESS_KEY_ID', '')

AWS_SECRET_ACCESS_KEY = environ.get('AWS_SECRET_ACCESS_KEY', '')

AWS_STORAGE_BUCKET_NAME = environ.get('AWS_STORAGE_BUCKET_NAME', '')

AWS_AUTO_CREATE_BUCKET = True

AWS_QUERYSTRING_AUTH = False

AWS_S3_SECURE_URLS = True

AWS_FILE_OVERWRITE = False

AWS_PRELOAD_METADATA = True

AWS_S3_CUSTOM_DOMAIN = environ.get('AWS_S3_CUSTOM_DOMAIN', '')

#

GZIP_CONTENT_TYPES = (

'text/css',

'application/javascript',

'application/x-javascript',

'text/javascript',

)

#

AWS_EXPIREY = 60 60 7

AWS_HEADERS = {

'Cache-Control': 'max-age=%d, s-maxage=%d, must-revalidate' % (AWS_EXPIREY,

AWS_EXPIREY)

}

#

STATIC_URL = 'http://%s.s3.amazonaws.com/' % AWS_STORAGE_BUCKET_NAME

MEDIA_URL = '/user-uploads/'

#

STATICFILES_STORAGE = DEFAULT_FILE_STORAGE = 'wooey.wooeystorage.CachedS3BotoStorage'

WOOEY_EPHEMERAL_FILES = True

AUTHENTICATION_BACKEND = 'django.contrib.auth.backends.ModelBackend'


2. CentOS 7

3.

amqp==2.2.2 anyjson==0.3.3 appdirs==1.4.3 arrow==0.10.0 billiard==3.5.0.3 celery==4.1.1 certifi==2017.4.17 clinto==0.2.1 Django==1.11.13 django-auth-ldap==1.5.0 django-autoslug==1.9.3 django-celery==3.2.2 django-celery-results==1.0.1 esp-snow-cli==0.9.4 esp-snow-lib==0.9.7 feedparser==5.2.1 jsonfield==2.0.2 kombu==4.2.0 ldap==1.0.2 ldap3==2.5 lxml==4.1.0 pyasn1==0.4.2 pyasn1-modules==0.2.1 python-dateutil==2.6.1 python-ldap==3.0.0 pytz==2018.4 pyvmomi==6.5.0.2017.5.post1 PyYAML==3.12 pyzabbix==0.7.4 requests==2.13.0 six==1.11.0 terminaltables==3.1.0 vine==1.1.4 wooey==0.10.0 xmltodict==0.11.0 zabbix-api==0.4 ZabbixSender==0.2.7

Chris7 commented 6 years ago

Oh, I bet it is because the command python manage.py createcachetable has not been run. Try running that. It may need to be added to the bootstrap script.

Chris7 commented 6 years ago

No, the bootstrap script does run it (https://github.com/wooey/Wooey/blob/94b5bdd670175c9817e6bdac9be5f28e8cdd0f5a/wooey/backend/command_line.py#L113). But can you try running it again?

setsimmo commented 6 years ago

After running the command I don't get the error anymore! But my job still never completes/does anything:

[23/May/2018 20:08:19] "GET / HTTP/1.1" 200 23622
[23/May/2018 20:08:20] "GET /jobs/queue/all/json HTTP/1.1" 200 102
[23/May/2018 20:08:35] "GET /scripts/the-easy-button-tm/ HTTP/1.1" 200 232154
[23/May/2018 20:08:36] "GET /jobs/queue/all/json HTTP/1.1" 200 102
[23/May/2018 20:08:45] "POST /scripts/the-easy-button-tm/ HTTP/1.1" 200 52
[23/May/2018 20:08:45] "GET /jobs/6/ HTTP/1.1" 200 34495
[23/May/2018 20:08:45] "GET /jobs/queue/all/json HTTP/1.1" 200 342
[23/May/2018 20:08:48] "GET /jobs/6/jsonhtml HTTP/1.1" 200 123
[23/May/2018 20:08:52] "GET /jobs/6/jsonhtml HTTP/1.1" 200 123
[23/May/2018 20:08:54] "GET /jobs/6/jsonhtml HTTP/1.1" 200 123
[23/May/2018 20:08:57] "GET /jobs/6/jsonhtml HTTP/1.1" 200 123
[23/May/2018 20:09:00] "GET /jobs/6/jsonhtml HTTP/1.1" 200 123
[23/May/2018 20:09:00] "GET /jobs/queue/all/json HTTP/1.1" 200 342

It just gets stuck in the queue with no output or conclusion...

To clarify, this was working properly for me in wooey 0.9.x, I upgraded and rebuilt the project "wooify -p easy_button" after moving the old project to easy_button_old. So it's starting with a clean slate, but not working properly.

Chris7 commented 6 years ago

Did you migrate after upgrading via python manage.py migrate?

setsimmo commented 6 years ago

Yes, I did.

To add another detail -- I am currently running wooey with Python 3.6.5 (didn't see in the documentation anywhere the recommended version of Python).

Chris7 commented 6 years ago

I will test on that when I have some time. Here are some ideas: 1) You have some packages from your previous install that shouldn't interfere, but might (django-celery is one). Can you try with a clean environment? 2) How are you running celery? 3) Can you look at the broker messages?