A clear and concise description of what the bug is.
As I was trying to upload BAM files, I got an error message for the first upload, because I forgot to create the required subdirectory bam_files.
This is in general not a problem, unless we are re-deploying the application. While we need the parent HOST_DATA_DIR (DATA_PATH) directory to exist, maybe subdirectories don't need to raise an error, and can be created on startup if they don't exists?
Output or error messages.
5087c2e04ead 2024-05-28 09:02:43 [WARNING] scimodom.services.file | Failed to create file '/data/bam_files/hSLg7AzyWeZ7_2bc4069a-9daa-4c09-abfe-664c520f2510_test.bam.bai': [Errno 21] Is a directory: '/data/bam_files/hSLg7AzyWeZ7_2bc4069a-9daa-4c09-abfe-664c520f2510_test.bam.bai' - discarding file.
5087c2e04ead 2024-05-28 09:02:43 [WARNING] scimodom.services.file | Failed to delete '/data/bam_files/hSLg7AzyWeZ7_2bc4069a-9daa-4c09-abfe-664c520f2510_test.bam.bai': [Errno 21] Is a directory: '/data/bam_files/hSLg7AzyWeZ7_2bc4069a-9daa-4c09-abfe-664c520f2510_test.bam.bai'.
5087c2e04ead 2024-05-28 09:02:43 [ERROR] scimodom | Exception on /api/v0/bam_file/hSLg7AzyWeZ7/test.bam.bai [POST]
5087c2e04ead Traceback (most recent call last):
5087c2e04ead File "/app/venv/lib/python3.11/site-packages/scimodom/services/file.py", line 78, in create_or_update_bam_file
5087c2e04ead bam_file = self.get_bam_file(dataset, name)
5087c2e04ead ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
5087c2e04ead File "/app/venv/lib/python3.11/site-packages/scimodom/services/file.py", line 90, in get_bam_file
5087c2e04ead ).one()
5087c2e04ead ^^^^^
5087c2e04ead File "/app/venv/lib/python3.11/site-packages/sqlalchemy/engine/result.py", line 1810, in one
5087c2e04ead return self._only_one_row(
5087c2e04ead ^^^^^^^^^^^^^^^^^^^
5087c2e04ead File "/app/venv/lib/python3.11/site-packages/sqlalchemy/engine/result.py", line 752, in _only_one_row
5087c2e04ead raise exc.NoResultFound(
5087c2e04ead sqlalchemy.exc.NoResultFound: No row was found when one was required
5087c2e04ead
5087c2e04ead During handling of the above exception, another exception occurred:
5087c2e04ead
5087c2e04ead Traceback (most recent call last):
5087c2e04ead File "/app/venv/lib/python3.11/site-packages/flask/app.py", line 1473, in wsgi_app
5087c2e04ead response = self.full_dispatch_request()
5087c2e04ead ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
5087c2e04ead File "/app/venv/lib/python3.11/site-packages/flask/app.py", line 882, in full_dispatch_request
5087c2e04ead rv = self.handle_user_exception(e)
5087c2e04ead ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
5087c2e04ead File "/app/venv/lib/python3.11/site-packages/flask_cors/extension.py", line 178, in wrapped_function
5087c2e04ead return cors_after_request(app.make_response(f(*args, **kwargs)))
5087c2e04ead ^^^^^^^^^^^^^^^^^^
5087c2e04ead File "/app/venv/lib/python3.11/site-packages/flask/app.py", line 880, in full_dispatch_request
5087c2e04ead rv = self.dispatch_request()
5087c2e04ead ^^^^^^^^^^^^^^^^^^^^^^^
5087c2e04ead File "/app/venv/lib/python3.11/site-packages/flask/app.py", line 865, in dispatch_request
5087c2e04ead return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) # type: ignore[no-any-return]
5087c2e04ead ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
5087c2e04ead File "/app/venv/lib/python3.11/site-packages/flask_cors/decorator.py", line 130, in wrapped_function
5087c2e04ead resp = make_response(f(*args, **kwargs))
5087c2e04ead ^^^^^^^^^^^^^^^^^^
5087c2e04ead File "/app/venv/lib/python3.11/site-packages/flask_jwt_extended/view_decorators.py", line 170, in decorator
5087c2e04ead return current_app.ensure_sync(fn)(*args, **kwargs)
5087c2e04ead ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
5087c2e04ead File "/app/venv/lib/python3.11/site-packages/scimodom/api/bam_file.py", line 45, in post_bam_file
5087c2e04ead file_service.create_or_update_bam_file(
5087c2e04ead File "/app/venv/lib/python3.11/site-packages/scimodom/services/file.py", line 81, in create_or_update_bam_file
5087c2e04ead self._create_bam_file(dataset, name, data_stream, max_size)
5087c2e04ead File "/app/venv/lib/python3.11/site-packages/scimodom/services/file.py", line 137, in _create_bam_file
5087c2e04ead self._stream_to_file(data_stream, path, max_size)
5087c2e04ead File "/app/venv/lib/python3.11/site-packages/scimodom/services/file.py", line 66, in _stream_to_file
5087c2e04ead self._handle_upload_error(exc, path)
5087c2e04ead File "/app/venv/lib/python3.11/site-packages/scimodom/services/file.py", line 128, in _handle_upload_error
5087c2e04ead raise exception
5087c2e04ead File "/app/venv/lib/python3.11/site-packages/scimodom/services/file.py", line 54, in _stream_to_file
5087c2e04ead with open(path, "wb") as fp:
5087c2e04ead ^^^^^^^^^^^^^^^^
5087c2e04ead IsADirectoryError: [Errno 21] Is a directory: '/data/bam_files/hSLg7AzyWeZ7_2bc4069a-9daa-4c09-abfe-664c520f2510_test.bam.bai'
Additional context
The error is not reproducible, once the target directory is created. In this case, it is automatically created after the first failed upload.
A clear and concise description of what the bug is.
As I was trying to upload BAM files, I got an error message for the first upload, because I forgot to create the required subdirectory
bam_files.This is in general not a problem, unless we are re-deploying the application. While we need the parent
HOST_DATA_DIR(DATA_PATH) directory to exist, maybe subdirectories don't need to raise an error, and can be created on startup if they don't exists?Output or error messages.
Additional context
The error is not reproducible, once the target directory is created. In this case, it is automatically created after the first failed upload.
What browser were you using?
Firefox, Chrome
What version of Sci-ModoM were you using?
dev