plazi / lycophron

Batch uploader to Zenodo
Creative Commons Zero v1.0 Universal
2 stars 5 forks source link

Add processing status for each record #21

Open alejandromumo opened 1 year ago

alejandromumo commented 1 year ago

The current data model supports that each record has a "status", thus allowing the user to understand the current progress on its uploads.

The application supports it but there is currently an error when accessing the database inside a celery task:

  File "/Users/alejandromumo/.virtualenvs/lycophron/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1964, in _exec_single_context
    self.dialect.do_execute(
  File "/Users/alejandromumo/.virtualenvs/lycophron/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 747, in do_execute
    cursor.execute(statement, parameters)
MemoryError
[2023-03-27 15:36:41,853: ERROR/MainProcess] Pool callback raised exception: MemoryError('Process got: ')
Traceback (most recent call last):
  File "/Users/alejandromumo/.virtualenvs/lycophron/lib/python3.9/site-packages/billiard/pool.py", line 1796, in safe_apply_callback
    fun(*args, **kwargs)
  File "/Users/alejandromumo/.virtualenvs/lycophron/lib/python3.9/site-packages/celery/worker/request.py", line 730, in on_success
    return self.on_failure(retval, return_ok=True)
  File "/Users/alejandromumo/.virtualenvs/lycophron/lib/python3.9/site-packages/celery/worker/request.py", line 545, in on_failure
    raise MemoryError(f'Process got: {exc}')
MemoryError: Process got:

it seems that the engine (sqlite) is executing the cursor to fetch the data and somehow fails. The, celery returns a MemoryError.