search

jic-dtool / dservercore

Given a dataset UUID returns a URI to where the dataset is stored
MIT License
4 stars 5 forks source link

WIP: Mongo refactor additions #29

Closed jotelha closed 1 year ago

jotelha commented 1 year ago

For now, reading search plugin-related env variables in core server config.

Should move those config parts to the plugins themselves, just as with the other server plugins, i.e. https://github.com/livMatS/dtool-lookup-server-plugin-scaffolding/blob/c867622f7518f94d4349dcefd0418aaee7a57989/dtool_lookup_server_plugin_scaffolding/config.py.

The README additions document how to start up an instance with mongo search and retrieve plugins.

Following the modified README protocol, I am able to launch with flask run. Will check for proper operation in the next few days.

Running the core server's tests yields four failures,

$ pytest 
=============================================================================================== test session starts ===============================================================================================
platform linux -- Python 3.8.10, pytest-7.2.0, pluggy-1.0.0
rootdir: /mnt/dat2/git/dtool/dtool-lookup-server, configfile: setup.cfg, testpaths: tests
plugins: cov-4.0.0
collected 54 items                                                                                                                                                                                                

tests/test_admin_user_routes.py .F                                                                                                                                                                          [  3%]
tests/test_base_uri_routes.py .F                                                                                                                                                                            [  7%]
tests/test_cli.py ......                                                                                                                                                                                    [ 18%]
tests/test_config.py .                                                                                                                                                                                      [ 20%]
tests/test_config_routes.py ..                                                                                                                                                                              [ 24%]
tests/test_dataset_routes.py .....FF....                                                                                                                                                                    [ 44%]
tests/test_lookup_datasets_by_user_and_uuid.py .                                                                                                                                                            [ 46%]
tests/test_permission_routes.py ..                                                                                                                                                                          [ 50%]
tests/test_sql_dataset_utils.py .                                                                                                                                                                           [ 51%]
tests/test_sql_list_datasets_by_user.py .                                                                                                                                                                   [ 53%]
tests/test_summary_of_datasets_by_user.py .                                                                                                                                                                 [ 55%]
tests/test_timestamp_consistency.py .                                                                                                                                                                       [ 57%]
tests/test_user_routes.py .                                                                                                                                                                                 [ 59%]
tests/test_utils_auth.py ......                                                                                                                                                                             [ 70%]
tests/test_utils_base_uri_management.py .                                                                                                                                                                   [ 72%]
tests/test_utils_dataset_info_is_valid.py .....                                                                                                                                                             [ 81%]
tests/test_utils_get_annotations_from_uri_by_user.py .                                                                                                                                                      [ 83%]
tests/test_utils_get_manifest_from_uri_by_user.py .                                                                                                                                                         [ 85%]
tests/test_utils_get_readme_from_uri_by_user.py .                                                                                                                                                           [ 87%]
tests/test_utils_permission_management.py .                                                                                                                                                                 [ 88%]
tests/test_utils_preprocess_query_base_uris.py .                                                                                                                                                            [ 90%]
tests/test_utils_register_dataset.py ....                                                                                                                                                                   [ 98%]
tests/test_utils_user_management.py .                                                                                                                                                                       [100%]

==================================================================================================== FAILURES =====================================================================================================
______________________________________________________________________________________________ test_list_user_route _______________________________________________________________________________________________

tmp_app_with_users = <FlaskClient <Flask 'dtool_lookup_server'>>

    def test_list_user_route(tmp_app_with_users):  # NOQA

        headers = dict(Authorization="Bearer " + snowwhite_token)
        r = tmp_app_with_users.get(
            "/admin/user/list",
            headers=headers,
        )
>       assert r.status_code == 200
E       assert 500 == 200
E        +  where 500 = <WrapperTestResponse streamed [500 INTERNAL SERVER ERROR]>.status_code

tests/test_admin_user_routes.py:75: AssertionError
------------------------------------------------------------------------------------------------ Captured log call ------------------------------------------------------------------------------------------------
ERROR    dtool_lookup_server:app.py:1741 Exception on /admin/user/list [GET]
Traceback (most recent call last):
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/flask/app.py", line 2525, in wsgi_app
    response = self.full_dispatch_request()
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/flask/app.py", line 1822, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/flask_cors/extension.py", line 165, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/flask/app.py", line 1820, in full_dispatch_request
    rv = self.dispatch_request()
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/flask/app.py", line 1796, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/flask_smorest/pagination.py", line 199, in wrapper
    result, status, headers = unpack_tuple_response(func(*args, **kwargs))
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/flask_smorest/response.py", line 90, in wrapper
    func(*args, **kwargs)
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/flask_jwt_extended/view_decorators.py", line 154, in decorator
    return current_app.ensure_sync(fn)(*args, **kwargs)
  File "/mnt/dat2/git/dtool/dtool-lookup-server/dtool_lookup_server/user_admin_routes.py", line 64, in list_users
    return query.paginate(
TypeError: paginate() takes 1 positional argument but 4 were given
____________________________________________________________________________________________ test_base_uri_list_route _____________________________________________________________________________________________

tmp_app_with_data = <FlaskClient <Flask 'dtool_lookup_server'>>

    def test_base_uri_list_route(tmp_app_with_data):  # NOQA

        headers = dict(Authorization="Bearer " + snowwhite_token)
        r = tmp_app_with_data.get(
            "/admin/base_uri/list",
            headers=headers,
        )
>       assert r.status_code == 200
E       assert 500 == 200
E        +  where 500 = <WrapperTestResponse streamed [500 INTERNAL SERVER ERROR]>.status_code

tests/test_base_uri_routes.py:71: AssertionError
------------------------------------------------------------------------------------------------ Captured log call ------------------------------------------------------------------------------------------------
ERROR    dtool_lookup_server:app.py:1741 Exception on /admin/base_uri/list [GET]
Traceback (most recent call last):
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/flask/app.py", line 2525, in wsgi_app
    response = self.full_dispatch_request()
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/flask/app.py", line 1822, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/flask_cors/extension.py", line 165, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/flask/app.py", line 1820, in full_dispatch_request
    rv = self.dispatch_request()
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/flask/app.py", line 1796, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/flask_smorest/pagination.py", line 199, in wrapper
    result, status, headers = unpack_tuple_response(func(*args, **kwargs))
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/flask_smorest/response.py", line 90, in wrapper
    func(*args, **kwargs)
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/flask_jwt_extended/view_decorators.py", line 154, in decorator
    return current_app.ensure_sync(fn)(*args, **kwargs)
  File "/mnt/dat2/git/dtool/dtool-lookup-server/dtool_lookup_server/base_uri_routes.py", line 60, in base_uri_list
    return query.paginate(
TypeError: paginate() takes 1 positional argument but 4 were given
___________________________________________________________________________________________ test_dataset_register_route ___________________________________________________________________________________________

tmp_app_with_users = <FlaskClient <Flask 'dtool_lookup_server'>>

    def test_dataset_register_route(tmp_app_with_users):  # NOQA

        from dtool_lookup_server.utils import (
            get_admin_metadata_from_uri,
            get_readme_from_uri_by_user,
            lookup_datasets_by_user_and_uuid,
        )

        base_uri = "s3://snow-white"
        uuid = "af6727bf-29c7-43dd-b42f-a5d7ede28337"
        uri = "{}/{}".format(base_uri, uuid)
        dataset_info = {
            "base_uri": base_uri,
            "uuid": uuid,
            "uri": uri,
            "name": "my-dataset",
            "type": "dataset",
            "readme": "---\ndescription: test dataset",
            "manifest": {
                "dtoolcore_version": "3.7.0",
                "hash_function": "md5sum_hexdigest",
                "items": {
                    "e4cc3a7dc281c3d89ed4553293c4b4b110dc9bf3": {
                        "hash": "d89117c9da2cc34586e183017cb14851",
                        "relpath": "U00096.3.rev.1.bt2",
                        "size_in_bytes": 5741810,
                        "utc_timestamp": 1536832115.0
                    }
                }
            },
            "creator_username": "olssont",
            "frozen_at": "1536238185.881941",
            "annotations": {"software": "bowtie2"},
            "tags": ["rnaseq"],
            "number_of_items": 1,
            "size_in_bytes": 5741810,
        }

        for token in [dopey_token, sleepy_token]:
            headers = dict(Authorization="Bearer " + sleepy_token)
            r = tmp_app_with_users.post(
                "/dataset/register",
                headers=headers,
                data=json.dumps(dataset_info),
                content_type="application/json"
            )
            assert r.status_code == 401

        headers = dict(Authorization="Bearer " + grumpy_token)
        r = tmp_app_with_users.post(
            "/dataset/register",
            headers=headers,
            data=json.dumps(dataset_info),
            content_type="application/json"
        )
>       assert r.status_code == 201
E       assert 500 == 201
E        +  where 500 = <WrapperTestResponse streamed [500 INTERNAL SERVER ERROR]>.status_code

tests/test_dataset_routes.py:294: AssertionError
------------------------------------------------------------------------------------------------ Captured log call ------------------------------------------------------------------------------------------------
ERROR    dtool_lookup_server:app.py:1741 Exception on /dataset/register [POST]
Traceback (most recent call last):
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/flask/app.py", line 2525, in wsgi_app
    response = self.full_dispatch_request()
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/flask/app.py", line 1822, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/flask_cors/extension.py", line 165, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/flask/app.py", line 1820, in full_dispatch_request
    rv = self.dispatch_request()
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/flask/app.py", line 1796, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/webargs/core.py", line 594, in wrapper
    return func(*args, **kwargs)
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/flask_smorest/arguments.py", line 82, in wrapper
    return func(*f_args, **f_kwargs)
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/flask_smorest/response.py", line 90, in wrapper
    func(*args, **kwargs)
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/flask_jwt_extended/view_decorators.py", line 154, in decorator
    return current_app.ensure_sync(fn)(*args, **kwargs)
  File "/mnt/dat2/git/dtool/dtool-lookup-server/dtool_lookup_server/dataset_routes.py", line 137, in register
    dataset_uri = register_dataset(dataset)
  File "/mnt/dat2/git/dtool/dtool-lookup-server/dtool_lookup_server/utils.py", line 551, in register_dataset
    search.register_dataset(dataset_info.copy())
  File "/mnt/dat2/git/dtool/dtool-lookup-server-search-plugin-mongo/dtool_lookup_server_search_plugin_mongo/utils_search.py", line 155, in register_dataset
    return _register_dataset_descriptive_metadata(self.collection, dataset_info)
  File "/mnt/dat2/git/dtool/dtool-lookup-server-search-plugin-mongo/dtool_lookup_server_search_plugin_mongo/utils_search.py", line 58, in _register_dataset_descriptive_metadata
    exists = collection.find_one(query)
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/pymongo/collection.py", line 1452, in find_one
    for result in cursor.limit(-1):
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/pymongo/cursor.py", line 1248, in next
    if len(self.__data) or self._refresh():
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/pymongo/cursor.py", line 1165, in _refresh
    self.__send_message(q)
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/pymongo/cursor.py", line 1052, in __send_message
    response = client._run_operation(
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/pymongo/_csot.py", line 105, in csot_wrapper
    return func(self, *args, **kwargs)
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1330, in _run_operation
    return self._retryable_read(
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/pymongo/_csot.py", line 105, in csot_wrapper
    return func(self, *args, **kwargs)
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1448, in _retryable_read
    return func(session, server, sock_info, read_pref)
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1326, in _cmd
    return server.run_operation(
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/pymongo/server.py", line 100, in run_operation
    message = operation.get_message(read_preference, sock_info, use_cmd)
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/pymongo/message.py", line 388, in get_message
    request_id, msg, size, _ = _op_msg(
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/pymongo/message.py", line 692, in _op_msg
    return _op_msg_uncompressed(flags, command, identifier, docs, opts)
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/bson/binary.py", line 267, in from_uuid
    raise ValueError(
ValueError: cannot encode native uuid.UUID with UuidRepresentation.UNSPECIFIED. UUIDs can be manually converted to bson.Binary instances using bson.Binary.from_uuid() or a different UuidRepresentation can be configured. See the documentation for UuidRepresentation for more information.
______________________________________________________________________________ test_dataset_register_route_when_created_at_is_string ______________________________________________________________________________

tmp_app_with_users = <FlaskClient <Flask 'dtool_lookup_server'>>

    def test_dataset_register_route_when_created_at_is_string(tmp_app_with_users):  # NOQA

        from dtool_lookup_server.utils import (
            get_admin_metadata_from_uri,
            lookup_datasets_by_user_and_uuid,
        )

        base_uri = "s3://snow-white"
        uuid = "af6727bf-29c7-43dd-b42f-a5d7ede28337"
        uri = "{}/{}".format(base_uri, uuid)
        dataset_info = {
            "base_uri": base_uri,
            "uuid": uuid,
            "uri": uri,
            "name": "my-dataset",
            "type": "dataset",
            "readme": "---\ndescription: test dataset",
            "manifest": {
                "dtoolcore_version": "3.7.0",
                "hash_function": "md5sum_hexdigest",
                "items": {
                    "e4cc3a7dc281c3d89ed4553293c4b4b110dc9bf3": {
                        "hash": "d89117c9da2cc34586e183017cb14851",
                        "relpath": "U00096.3.rev.1.bt2",
                        "size_in_bytes": 5741810,
                        "utc_timestamp": 1536832115.0
                    }
                }
            },
            "creator_username": "olssont",
            "frozen_at": "1536238185.881941",
            "created_at": "1536238185.881941",
            "number_of_items": 1,
            "size_in_bytes": 5741810,
            "annotations": {"software": "bowtie2"},
            "tags": ["rnaseq"],
        }

        headers = dict(Authorization="Bearer " + grumpy_token)
        r = tmp_app_with_users.post(
            "/dataset/register",
            headers=headers,
            data=json.dumps(dataset_info),
            content_type="application/json"
        )
>       assert r.status_code == 201
E       assert 500 == 201
E        +  where 500 = <WrapperTestResponse streamed [500 INTERNAL SERVER ERROR]>.status_code

tests/test_dataset_routes.py:432: AssertionError
------------------------------------------------------------------------------------------------ Captured log call ------------------------------------------------------------------------------------------------
ERROR    dtool_lookup_server:app.py:1741 Exception on /dataset/register [POST]
Traceback (most recent call last):
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/flask/app.py", line 2525, in wsgi_app
    response = self.full_dispatch_request()
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/flask/app.py", line 1822, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/flask_cors/extension.py", line 165, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/flask/app.py", line 1820, in full_dispatch_request
    rv = self.dispatch_request()
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/flask/app.py", line 1796, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/webargs/core.py", line 594, in wrapper
    return func(*args, **kwargs)
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/flask_smorest/arguments.py", line 82, in wrapper
    return func(*f_args, **f_kwargs)
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/flask_smorest/response.py", line 90, in wrapper
    func(*args, **kwargs)
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/flask_jwt_extended/view_decorators.py", line 154, in decorator
    return current_app.ensure_sync(fn)(*args, **kwargs)
  File "/mnt/dat2/git/dtool/dtool-lookup-server/dtool_lookup_server/dataset_routes.py", line 137, in register
    dataset_uri = register_dataset(dataset)
  File "/mnt/dat2/git/dtool/dtool-lookup-server/dtool_lookup_server/utils.py", line 551, in register_dataset
    search.register_dataset(dataset_info.copy())
  File "/mnt/dat2/git/dtool/dtool-lookup-server-search-plugin-mongo/dtool_lookup_server_search_plugin_mongo/utils_search.py", line 155, in register_dataset
    return _register_dataset_descriptive_metadata(self.collection, dataset_info)
  File "/mnt/dat2/git/dtool/dtool-lookup-server-search-plugin-mongo/dtool_lookup_server_search_plugin_mongo/utils_search.py", line 58, in _register_dataset_descriptive_metadata
    exists = collection.find_one(query)
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/pymongo/collection.py", line 1452, in find_one
    for result in cursor.limit(-1):
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/pymongo/cursor.py", line 1248, in next
    if len(self.__data) or self._refresh():
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/pymongo/cursor.py", line 1165, in _refresh
    self.__send_message(q)
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/pymongo/cursor.py", line 1052, in __send_message
    response = client._run_operation(
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/pymongo/_csot.py", line 105, in csot_wrapper
    return func(self, *args, **kwargs)
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1330, in _run_operation
    return self._retryable_read(
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/pymongo/_csot.py", line 105, in csot_wrapper
    return func(self, *args, **kwargs)
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1448, in _retryable_read
    return func(session, server, sock_info, read_pref)
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1326, in _cmd
    return server.run_operation(
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/pymongo/server.py", line 100, in run_operation
    message = operation.get_message(read_preference, sock_info, use_cmd)
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/pymongo/message.py", line 388, in get_message
    request_id, msg, size, _ = _op_msg(
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/pymongo/message.py", line 692, in _op_msg
    return _op_msg_uncompressed(flags, command, identifier, docs, opts)
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/bson/binary.py", line 267, in from_uuid
    raise ValueError(
ValueError: cannot encode native uuid.UUID with UuidRepresentation.UNSPECIFIED. UUIDs can be manually converted to bson.Binary instances using bson.Binary.from_uuid() or a different UuidRepresentation can be configured. See the documentation for UuidRepresentation for more information.
================================================================================================ warnings summary =================================================================================================
tests/test_admin_user_routes.py: 2 warnings
tests/test_base_uri_routes.py: 2 warnings
tests/test_cli.py: 6 warnings
tests/test_config.py: 1 warning
tests/test_config_routes.py: 2 warnings
tests/test_dataset_routes.py: 11 warnings
tests/test_lookup_datasets_by_user_and_uuid.py: 1 warning
tests/test_permission_routes.py: 2 warnings
tests/test_sql_dataset_utils.py: 1 warning
tests/test_sql_list_datasets_by_user.py: 1 warning
tests/test_summary_of_datasets_by_user.py: 1 warning
tests/test_timestamp_consistency.py: 1 warning
tests/test_user_routes.py: 1 warning
tests/test_utils_auth.py: 6 warnings
tests/test_utils_base_uri_management.py: 1 warning
tests/test_utils_get_annotations_from_uri_by_user.py: 1 warning
tests/test_utils_get_manifest_from_uri_by_user.py: 1 warning
tests/test_utils_get_readme_from_uri_by_user.py: 1 warning
tests/test_utils_permission_management.py: 1 warning
tests/test_utils_preprocess_query_base_uris.py: 1 warning
tests/test_utils_register_dataset.py: 4 warnings
tests/test_utils_user_management.py: 1 warning
  /mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/flask_marshmallow/__init__.py:115: DeprecationWarning: The 'db' attribute is deprecated and will be removed in Flask-SQLAlchemy 3.1. The extension is registered directly as 'app.extensions["sqlalchemy"]'.
    db = app.extensions["sqlalchemy"].db

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

---------- coverage: platform linux, python 3.8.10-final-0 -----------
Name                                       Stmts   Miss  Cover   Missing
------------------------------------------------------------------------
dtool_lookup_server/__init__.py              109     20    82%   56, 66, 87, 96, 105, 114, 128, 146, 148, 157, 159, 165-167, 177, 210-217
dtool_lookup_server/base_uri_routes.py        31      1    97%   56
dtool_lookup_server/cli.py                   124     38    69%   58-65, 72-77, 92-95, 168-181, 188-214
dtool_lookup_server/config.py                 45      2    96%   11, 42
dtool_lookup_server/config_routes.py          15      0   100%
dtool_lookup_server/dataset_routes.py        123     19    85%   129, 132, 138, 157, 165-167, 180, 183, 187, 191-193, 207, 210, 214, 218-220
dtool_lookup_server/date_utils.py             12      0   100%
dtool_lookup_server/permission_routes.py      30      2    93%   37-38
dtool_lookup_server/schemas.py                56      0   100%
dtool_lookup_server/sql_models.py             53      3    94%   34, 66, 96
dtool_lookup_server/user_admin_routes.py      31      1    97%   60
dtool_lookup_server/user_routes.py            19      0   100%
dtool_lookup_server/utils.py                 254     36    86%   57-59, 83-97, 101, 107, 113-145, 546, 554, 582
dtool_lookup_server/utils_auth.py             44      0   100%
------------------------------------------------------------------------
TOTAL                                        946    122    87%

============================================================================================= short test summary info =============================================================================================
FAILED tests/test_admin_user_routes.py::test_list_user_route - assert 500 == 200
FAILED tests/test_base_uri_routes.py::test_base_uri_list_route - assert 500 == 200
FAILED tests/test_dataset_routes.py::test_dataset_register_route - assert 500 == 201
FAILED tests/test_dataset_routes.py::test_dataset_register_route_when_created_at_is_string - assert 500 == 201
==================================================================================== 4 failed, 50 passed, 49 warnings in 6.29s ====================================================================================

I encountered the UUID-related failure a couple of months ago, mentioned in https://github.com/jic-dtool/dtool-lookup-server/pull/27.

Then, I simply worked around it with https://github.com/jic-dtool/dtool-lookup-server/pull/27/commits/31e71b6d812bbbe3dd92bdf46e0f8372665240b9.

With the refactor, this issue falls into the scope of the mongo search plugin. Will need to look into https://pymongo.readthedocs.io/en/stable/examples/uuid.html#handling-uuid-data again and figure out how to handle it properly.

tjelvar-olsson commented 1 year ago

@jotelha you are correct that the plugin in environment variables should move into the respective plugin packages.