search

fossasia / open-event-server

The Open Event Organizer Server to Manage Events https://test-api.eventyay.com
https://api.eventyay.com
GNU General Public License v3.0
2.98k stars 1.89k forks source link

Only the first task works on Celery #4775

Closed mayank8318 closed 6 years ago

mayank8318 commented 6 years ago

Describe the bug The task to export the event as a zip fails.

To Reproduce Steps to reproduce the behavior:

  1. Use /v1/events/{event_identifier}/export/json API to start the export (Make sure Celery is working)
  2. You will get the stack trace in the terminal at this point in time
  3. Use /v1/tasks/{task_id} to check the result of the task. It will return "FAILURE"

Expected behavior Zipping should work as stated.

Stacktrace

[2018-05-24 17:31:42,874: ERROR/MainProcess] Task export.event[d23f127a-2d47-4958-a9c2-214fa53560e7] raised unexpected: OperationalError('(psycopg2.OperationalError) SSL SYSCALL error: EOF detected\n',)
Traceback (most recent call last):
  File "/home/mayank/anaconda3/envs/openevent3/lib/python3.6/site-packages/celery/app/trace.py", line 240, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/mayank/OpenSource/open-event-server/app/api/helpers/request_context_task.py", line 46, in __call__
    result = call()
  File "/home/mayank/OpenSource/open-event-server/app/api/helpers/request_context_task.py", line 32, in <lambda>
    call = lambda: super(RequestContextTask, self).__call__(*args, **kwargs)
  File "/home/mayank/anaconda3/envs/openevent3/lib/python3.6/site-packages/celery/app/trace.py", line 438, in __protected_call__
    return self.run(*args, **kwargs)
  File "/home/mayank/OpenSource/open-event-server/app/api/helpers/tasks.py", line 60, in export_event_task
    event = safe_query(db, Event, 'id', event_id, 'event_id')
  File "/home/mayank/OpenSource/open-event-server/app/api/helpers/db.py", line 46, in safe_query
    record = self.session.query(model).filter(getattr(model, column_name) == value).one()
  File "/home/mayank/anaconda3/envs/openevent3/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2814, in one
    ret = self.one_or_none()
  File "/home/mayank/anaconda3/envs/openevent3/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2784, in one_or_none
    ret = list(self)
  File "/home/mayank/anaconda3/envs/openevent3/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2855, in __iter__
    return self._execute_and_instances(context)
  File "/home/mayank/anaconda3/envs/openevent3/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2878, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/home/mayank/anaconda3/envs/openevent3/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 945, in execute
    return meth(self, multiparams, params)
  File "/home/mayank/anaconda3/envs/openevent3/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 263, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/home/mayank/anaconda3/envs/openevent3/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1053, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/home/mayank/anaconda3/envs/openevent3/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1189, in _execute_context
    context)
  File "/home/mayank/anaconda3/envs/openevent3/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1402, in _handle_dbapi_exception
    exc_info
  File "/home/mayank/anaconda3/envs/openevent3/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/home/mayank/anaconda3/envs/openevent3/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 186, in reraise
    raise value.with_traceback(tb)
  File "/home/mayank/anaconda3/envs/openevent3/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1182, in _execute_context
    context)
  File "/home/mayank/anaconda3/envs/openevent3/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 470, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) SSL SYSCALL error: EOF detected
 [SQL: 'SELECT events.id AS events_id, events.identifier AS events_identifier, events.name AS events_name, events.external_event_url AS events_external_event_url, events.logo_url AS events_logo_url, events.starts_at AS events_starts_at, events.ends_at AS events_ends_at, events.timezone AS events_timezone, events.latitude AS events_latitude, events.longitude AS events_longitude, events.location_name AS events_location_name, events.searchable_location_name AS events_searchable_location_name, events.description AS events_description, events.original_image_url AS events_original_image_url, events.thumbnail_image_url AS events_thumbnail_image_url, events.large_image_url AS events_large_image_url, events.icon_image_url AS events_icon_image_url, events.organizer_name AS events_organizer_name, events.is_map_shown AS events_is_map_shown, events.has_organizer_info AS events_has_organizer_info, events.organizer_description AS events_organizer_description, events.is_sessions_speakers_enabled AS events_is_sessions_speakers_enabled, events.privacy AS events_privacy, events.state AS events_state, events.event_type_id AS events_event_type_id, events.event_topic_id AS events_event_topic_id, events.event_sub_topic_id AS events_event_sub_topic_id, events.ticket_url AS events_ticket_url, events.code_of_conduct AS events_code_of_conduct, events.schedule_published_on AS events_schedule_published_on, events.is_ticketing_enabled AS events_is_ticketing_enabled, events.deleted_at AS events_deleted_at, events.payment_country AS events_payment_country, events.payment_currency AS events_payment_currency, events.paypal_email AS events_paypal_email, events.is_tax_enabled AS events_is_tax_enabled, events.can_pay_by_paypal AS events_can_pay_by_paypal, events.can_pay_by_stripe AS events_can_pay_by_stripe, events.can_pay_by_cheque AS events_can_pay_by_cheque, events.can_pay_by_bank AS events_can_pay_by_bank, events.can_pay_onsite AS events_can_pay_onsite, events.cheque_details AS events_cheque_details, events.bank_details AS events_bank_details, events.onsite_details AS events_onsite_details, events.created_at AS events_created_at, events.pentabarf_url AS events_pentabarf_url, events.ical_url AS events_ical_url, events.xcal_url AS events_xcal_url, events.is_sponsors_enabled AS events_is_sponsors_enabled, events.discount_code_id AS events_discount_code_id \nFROM events \nWHERE events.id = %(id_1)s'] [parameters: {'id_1': '2'}]

Additional details (please complete the following information):

bhaveshAn commented 6 years ago

For me GET /v1/tasks/{task_id} gives

"state": "WAITING"
schedutron commented 6 years ago

For me, celery gives this error for the task GET request:

  File "/open-event-server/app/api/helpers/tasks.py", line 64, in export_event_task
    path = event_export_task_base(event_id, settings)
  File "/open-event-server/app/api/exports.py", line 69, in event_export_task_base
    path = export_event_json(event_id, settings)
  File "/open-event-server/app/api/helpers/export_helpers.py", line 209, in export_event_json
    fp.write(data_str)
TypeError: write() argument must be str, not bytes
mayank8318 commented 6 years ago

@bhaveshAn Make sure celery is working @schedutron That's weird. So there is some problem definitely. I will look into it

iamareebjamal commented 6 years ago

@schedutron Must be for python 3.5

Needs to be fixed though.

schedutron commented 6 years ago

@iamareebjamal @mayank8318 Perhaps a simple call to data_str.decode('utf-8') may fix the error I get.

iamareebjamal commented 6 years ago

No, there are two errors here, the error for this issue is not related to byte str fiasco

mayank8318 commented 6 years ago

@schedutron Are you using the right Python version?

iamareebjamal commented 6 years ago

Python version is 3+ so it needs to be fixed @mayank8318 This doesn't happen in 3.6 but doesn't mean it shouldn't be fixed for 3.5

mayank8318 commented 6 years ago

@iamareebjamal Should I open another issue for this or do it in this one itself? Also, what would be the best way to handle this?

iamareebjamal commented 6 years ago

As we can't gather all methods which have byte str problems due to lack of types in Python, we'll have to fix this as errors pop up

And yes, create a new issue. Change bytes to str in all instances of file.write

mayank8318 commented 6 years ago

@iamareebjamal Okay :)

mayank8318 commented 6 years ago

After #4785 the following happens :

Python 3.5

schedutron commented 6 years ago

If you export a different event after successfully exporting the first event, do you get the same error?

mayank8318 commented 6 years ago

@schedutron Yes same error

mayank8318 commented 6 years ago

@iamareebjamal I tried solutions from https://github.com/celery/celery/issues/3238, https://github.com/celery/celery/issues/1564, https://github.com/celery/celery/issues/634 but nothing worked. Am I missing any implementation details?

iamareebjamal commented 6 years ago

I think this has to do with detached session https://stackoverflow.com/questions/31121948/flask-sqlalchemy-cant-reconnect-until-invalid-transaction-is-rolled-back

iamareebjamal commented 6 years ago

https://stackoverflow.com/questions/45607541/flask-sqlalchemy-error-invalid-transaction

mayank8318 commented 6 years ago

@iamareebjamal I am getting OperationalError: (psycopg2.OperationalError) SSL error: decryption failed or bad record mac consistently

mayank8318 commented 6 years ago

Real Problem While trying to solve this issue I realized that the reason this was happening was that second task onwards the old database connections which were used in the first task were being reused thus causing OperationalError: (psycopg2.OperationalError) SSL error: decryption failed or bad record mac. This happens for any Celery task which uses the database.

Reference https://github.com/celery/celery/issues/3238

mayank8318 commented 6 years ago

@srv-twry @bhaveshAn @pradeepgangwar @Kreijstal this should be a high priority issue since all our tasks use celery and currently, you need to restart celery after every task. I tried a few solutions but none worked. Let's solve it together :smile: UPDATE - Tried https://stackoverflow.com/questions/37801766/correctly-managing-postgresql-connections-in-celery-task-for-flask-sqlalchemy-an too.

dr0pdb commented 6 years ago

@mayank8318 Can you try this one: https://stackoverflow.com/questions/25360136/flask-with-create-app-sqlalchemy-and-celery?rq=1

mayank8318 commented 6 years ago

@srv-twry Nope - 

[2018-06-14 16:01:47,397: ERROR/MainProcess] Task export.event[ec6e075e-e576-4389-a20f-8f8eadf50719] raised unexpected: AttributeError("'Celery' object has no attribute 'app'",)
Traceback (most recent call last):
  File "/home/mayank/anaconda3/envs/openevent3/lib/python3.6/site-packages/celery/app/trace.py", line 240, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/mayank/OpenSource/open-event-server/app/api/helpers/request_context_task.py", line 46, in __call__
    result = call()
  File "/home/mayank/OpenSource/open-event-server/app/api/helpers/request_context_task.py", line 32, in <lambda>
    call = lambda: super(RequestContextTask, self).__call__(*args, **kwargs)
  File "/home/mayank/anaconda3/envs/openevent3/lib/python3.6/site-packages/celery/app/trace.py", line 438, in __protected_call__
    return self.run(*args, **kwargs)
  File "/home/mayank/OpenSource/open-event-server/app/api/helpers/tasks.py", line 70, in export_event_task
    with celery.app.app_context():
AttributeError: 'Celery' object has no attribute 'app'
mayank8318 commented 6 years ago

I think we need scoped db sessions, something like this -> http://www.prschmid.com/2013/04/using-sqlalchemy-with-celery-tasks.html