converting Unix style filter, stdin->process->stdout

[hsum]

Is there a way to convert a script that takes a (csv) file source and outputs another (csv) file? Adding argparse isn't a problem, but I guess a file upload would be needed? It would also be great to have an http response to deliver the resulting csv.

Any nice ways to get this done?

thank you

[Chris7]

If you use a file-like field in argparse, Wooey will automatically setup a file upload field for you. Similarly, if you use the argparse standards for outputs, that will be available to download in the Wooey output. Also, any files generated in the temporary folder Wooey makes to run the script will be collected to be stored.

Does that answer your question?

[hsum]

@Chris7 Thank you very much. I've been using argparse for years and never knew about FileType. I added arguments to support input and output.

parser.add_argument('--input', type=argparse.FileType('r'))

parser.add_argument('--output', type=argparse.FileType('w'))

The script works as expected.

In Wooey, input using FileType worked exactly as you described. Unfortunately, I didn't get a download link to the output no matter what I specified in the 'output' field. After submitting, the job remains in waiting state.

Do you have any hints on how to debug this to find out why it's not finishing? I ran strace and it's just polling (select call).

[Chris7]

Could you attach your script or a parsed down version that reproduces the problem? Would also be good to know how you are setting up celery or running everything synchronously.

[hsum]

I stripped out some row filtering: roster.py

I'm running it using the dev server with defaults. I had assumed it wouldn't use Celery, but maybe I was wrong? (.ve) 2019-10-09 18:01:27 [hsum@x1c5 roster]$ grep CELERY roster/settings/*.py roster/settings/user_settings.py:CELERY_RESULT_BACKEND = 'django-db' roster/settings/user_settings.py:CELERY_BROKER_URL = 'filesystem://' roster/settings/user_settings.py:CELERY_BROKER_TRANSPORT_OPTIONS = { roster/settings/user_settings.py:CELERY_TRACK_STARTED = True roster/settings/user_settings.py:WOOEY_CELERY = True roster/settings/user_settings.py:CELERY_SEND_EVENTS = True roster/settings/user_settings.py:CELERY_IMPORTS = ('wooey.tasks',) roster/settings/user_settings.py:# CELERY_BROKER_URL = os.environ.get('AMQP_URL') or \ roster/settings/user_settings.py:# CELERY_BROKER_POOL_LIMIT = 1 roster/settings/user_settings.py:# CELERYD_CONCURRENCY = 1 roster/settings/user_settings.py:# CELERY_TASK_SERIALIZER = 'json' roster/settings/user_settings.py:# CELERY_TASK_ACKS_LATE = True roster/settings/wooey_settings.py:WOOEY_CELERY_APP_NAME = 'wooey.celery' roster/settings/wooey_settings.py:WOOEY_CELERY_TASKS = 'wooey.tasks'

Please let me know if you see something. Thank you again.

[Chris7]

That script works fine for me. If WOOEY_CELERY is set to True, that means Wooey expects celery to run the jobs for it. Are you running celery?

[hsum]

@Chris7 Whoops! That's exactly the problem. I didn't start celery as per documentation. Thank you.

Would it be possible for detect Celery not running and logging.warning() it? Would it be possible to have a custom management/command to runserver and start up celery?

thank you again.

[Chris7]

There is a custom Procfile to run both things using honcho (though that only exists in the docs, it might be useful to provide it with the bootstrap).

With respect to checking if celery is running, there may be a way but I'm worried the implementations will rely on the broker used add more confusion when it only sometimes works.