Bits and pieces on openapi implementation

jotelha commented 2 years ago

This is an old pull request I had still open on Luis' fork (https://github.com/ljyanesm/dtool-lookup-server/pull/1), with a few points not discussed yet. Below I copy over what I still find relevant. A few of the issues described arose when playing with auto-generating Python API from the openapi specs (https://github.com/jotelha/dtool-lookup-server/tree/0de6f54d0912c79740c9089228f2c5c764babf26/devel).

I think the server core should not depend on specific storage brokers like "dtool_irods", "dtool_s3", "dtool_ecs".
When working within these openapi boundaries, what's not clear to me is how to handle arbitrarily structured data? I.e. for the config of the server, that's a key-value dict, https://github.com/jic-dtool/dtool-lookup-server/blob/master/dtool_lookup_server/config.py, that would also present plugin-specific configs nested below toplevel, again with possibly arbitrarily nested objects. The schema to capture this would clearly just be the general JSON or yaml specifications, and that's the case for a dataset's README.yml as well (if it's not just handled as a string). Clearly, one does not want to prescribe a narrow schema here, but still wants that reflected somehow in the documentation. Are empty schemas like those https://github.com/jotelha/dtool-lookup-server/blob/eca0305dbe1ed1575e51f2b5367e72dbca6b1cab/dtool_lookup_server/schemas.py#L87-L92 legitimate?
Standardized spelling of Uri to URI in models and schema for all occurrences.
Different cases in the URI acronym like BaseURI, BaseUri, BaseURISchema, and BaseUriSchema in sql_models.py and schemas.py caused troubles for autogenerated Python client API. For now, resorted to suffix SQLAlchemy for the schema defined within sql_models.py due to a lack of imagination. Do we actually need the different schema in sql_models.BaseURISQLAlchemySchema(ma.SQLAlchemyAutoSchema) and schemas.BaseURISchema(Schema) ?
Originally, sql_models.Dataset(db.Model) defines columns frozen_at and created_at as db.DateTime(), its method as_dict() converts those with dtoolcore.utils.timestamp(self.frozen_at). This, however, results in the autogenerated client API to expect a serialized datetime object and to throw an exception when receiving float. Treat created_at and frozen_at as floats in schema with https://github.com/jotelha/dtool-lookup-server/blob/9d1b8cc751f1d1e143bae10d9454d86be438dc13/dtool_lookup_server/sql_models.py#L122-L128 as suggested below https://marshmallow.readthedocs.io/en/latest/quickstart.html#implicit-field-creation. Not sure whether this is legitimate.

To avoid failure of tests on register route,

ERROR    dtool_lookup_server:app.py:1455 Exception on /dataset/register [POST]
Traceback (most recent call last):
File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/flask/app.py", line 2077, in wsgi_app
response = self.full_dispatch_request()
File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/flask/app.py", line 1525, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/flask_cors/extension.py", line 165, in wrapped_function
return cors_after_request(app.make_response(f(*args, **kwargs)))
File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/flask/app.py", line 1523, in full_dispatch_request
rv = self.dispatch_request()
File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/flask/app.py", line 1509, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/webargs/core.py", line 452, in wrapper
return func(*args, **kwargs)
File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/flask_smorest/arguments.py", line 82, in wrapper
return func(*f_args, **f_kwargs)
File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/flask_smorest/response.py", line 90, in wrapper
func(*args, **kwargs)
File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/flask_jwt_extended/view_decorators.py", line 127, in decorator
return current_app.ensure_sync(fn)(*args, **kwargs)
File "/mnt/dat2/git/dtool/dtool-lookup-server/dtool_lookup_server/dataset_routes.py", line 148, in register
dataset_uri = register_dataset(dataset_info)
File "/mnt/dat2/git/dtool/dtool-lookup-server/dtool_lookup_server/utils.py", line 686, in register_dataset
register_dataset_descriptive_metadata(dataset_info.copy())
File "/mnt/dat2/git/dtool/dtool-lookup-server/dtool_lookup_server/utils.py", line 632, in register_dataset_descriptive_metadata
_register_dataset_descriptive_metadata(collection, dataset_info)
File "/mnt/dat2/git/dtool/dtool-lookup-server/dtool_lookup_server/utils.py", line 658, in _register_dataset_descriptive_metadata
exists = collection.find_one(query)
File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/pymongo/collection.py", line 1419, in find_one
for result in cursor.limit(-1):
File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/pymongo/cursor.py", line 1248, in next
if len(self.__data) or self._refresh():
File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/pymongo/cursor.py", line 1165, in _refresh
self.__send_message(q)
File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/pymongo/cursor.py", line 1052, in __send_message
response = client._run_operation(
File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1267, in _run_operation
return self._retryable_read(
File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1371, in _retryable_read
return func(session, server, sock_info, read_pref)
File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1263, in _cmd
return server.run_operation(
File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/pymongo/server.py", line 100, in run_operation
message = operation.get_message(read_preference, sock_info, use_cmd)
File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/pymongo/message.py", line 375, in get_message
request_id, msg, size, _ = _op_msg(
File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/pymongo/message.py", line 673, in _op_msg
return _op_msg_uncompressed(flags, command, identifier, docs, opts)
File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/bson/binary.py", line 267, in from_uuid
raise ValueError(
ValueError: cannot encode native uuid.UUID with UuidRepresentation.UNSPECIFIED. UUIDs can be manually converted to bson.Binary instances using bson.Binary.from_uuid() or a different UuidRepresentation can be configured. See the documentation for UuidRepresentation for more information.

treat UUIDs as string for now, keep https://pymongo.readthedocs.io/en/stable/examples/uuid.html#handling-uuid-data in mind.

jotelha commented 1 year ago

An instance of this branch runs at https://demo.dtool.dev/, with openapi docs at https://demo.dtool.dev/lookup/doc/swagger, would like to discuss.

jotelha commented 1 year ago

Broke what's left here down into several small pull requests,

Closing this one.

jic-dtool / dservercore

Bits and pieces on openapi implementation #27