search

Neoteroi / mkdocs-plugins

Plugins for MkDocs.
MIT License
116 stars 9 forks source link

Error in serializing datetime #35

Closed foxpluto closed 1 year ago

foxpluto commented 1 year ago

I have a quite long openapi file with a lot of date-time like this one:

      - description: Start time stamp of the returned data interval
        example: 2022-08-17T18:00:00Z
        in: query
        name: fromInstant
        required: false
        schema:
          format: date-time
          type: string

but I have this error:

ERROR    -  Error reading page 'openapi/openapi.md': Object of type datetime is not JSON serializable
Traceback (most recent call last):
  File "/opt/homebrew/bin/mkdocs", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/mkdocs/__main__.py", line 234, in serve_command
    serve.serve(dev_addr=dev_addr, livereload=livereload, watch=watch, **kwargs)
  File "/opt/homebrew/lib/python3.11/site-packages/mkdocs/commands/serve.py", line 83, in serve
    builder(config)
  File "/opt/homebrew/lib/python3.11/site-packages/mkdocs/commands/serve.py", line 76, in builder
    build(config, live_server=live_server, dirty=dirty)
  File "/opt/homebrew/lib/python3.11/site-packages/mkdocs/commands/build.py", line 308, in build
    _populate_page(file.page, config, files, dirty)
  File "/opt/homebrew/lib/python3.11/site-packages/mkdocs/commands/build.py", line 177, in _populate_page
    page.markdown = config.plugins.run_event(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/mkdocs/plugins.py", line 520, in run_event
    result = method(item, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/neoteroi/mkdocs/oad/__init__.py", line 44, in on_page_markdown
    return self.rx.sub(self._replacer, markdown)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/neoteroi/mkdocs/oad/__init__.py", line 34, in _replacer
    return handler.write()
           ^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/openapidocs/mk/v3/__init__.py", line 417, in write
    return self._writer.write(
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/openapidocs/mk/jinja.py", line 109, in write
    return template.render(data, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/jinja2/environment.py", line 1301, in render
    self.environment.handle_exception()
  File "/opt/homebrew/lib/python3.11/site-packages/jinja2/environment.py", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "/opt/homebrew/lib/python3.11/site-packages/openapidocs/mk/v3/views_mkdocs/layout.html", line 14, in top-level template code
    {% include "partial/path-items.html" %}
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/openapidocs/mk/v3/views_mkdocs/partial/path-items.html", line 29, in top-level template code
    {%- include "partial/request-responses.html" %}
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/openapidocs/mk/v3/views_mkdocs/partial/request-responses.html", line 25, in top-level template code
    {% include "partial/content-examples.html" %}
^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/openapidocs/mk/v3/views_mkdocs/partial/content-examples.html", line 4, in top-level template code
    {{handler.write_content_example(example, content_type) | indent(4) | safe}}
^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/openapidocs/mk/v3/__init__.py", line 494, in write_content_example
    return example_handler.write(example.value)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/openapidocs/mk/contents.py", line 33, in write
    return json.dumps(value, ensure_ascii=False, indent=4)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
          ^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/json/encoder.py", line 202, in encode
    chunks = list(chunks)
             ^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/json/encoder.py", line 432, in _iterencode
    yield from _iterencode_dict(o, _current_indent_level)
  File "/opt/homebrew/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/json/encoder.py", line 406, in _iterencode_dict
    yield from chunks
  File "/opt/homebrew/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/json/encoder.py", line 439, in _iterencode
    o = _default(o)
        ^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/json/encoder.py", line 180, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type datetime is not JSON serializable

Is this a bug?

Regards, S.

RobertoPrevato commented 1 year ago

Hi @foxpluto Thank You for reporting this issue, it is definitely a bug. It's happening because:

  1. YAML parsers parse dates automatically into instances of datetime (unlike JSON parsing standard)
  2. I am using the built-in json.dumps method to serialize, which is in my opinion silly, throwing exception for common data types like UUID and datetime and other built-in types like dataclass, etc.

Good news: I even created in another library a method to avoid this kind of scenario, so to fix the error I would only need to replace the built-in json.dumps with essentials.json.dumps.

It uses a more user-friendly JSONEncoder class that handles those types.

foxpluto commented 1 year ago

Wonderful !!! I'll wait the fix to test it.

Thanks, S.

foxpluto commented 1 year ago

no ETA for this fix?

Sorry to bother...

RobertoPrevato commented 1 year ago

Hi @foxpluto Sorry for taking longer than I promised at first, in the last weeks I was focused again on my web framework.

I fixed the bug and published a new version of the library where the bug happens 1.0.6. You can upgrade the essentials-openapi dependency and the error has been resolved.

However, while writing tests and the fix, I also realized that maintaining the exact format of the datetime as in the examples is not obvious because datetimes are parsed automatically by YAML, then serialized using a certain format for JSON (the built-in json.dumps throws exception probably exactly for this reason, that the desired serialization format for dates and times is not obvious).

To maintain the exact format you set for your examples, like 2022-08-17T18:00:00Z, you have two options:

  1. change your YAML to have example datetimes defined as strings and avoid automatic parsing (quote the values) - this way they are kept as strings and displayed in JSON exactly the same. Note that this would have worked already before I applied the fix.
      - description: Start time stamp of the returned data interval
-       example: 2022-08-17T18:00:00Z
+       example: '2022-08-17T18:00:00Z'
        in: query
        name: fromInstant
        required: false
        schema:
          format: date-time
          type: string
  1. keep your YAML fragments as they are, and set this environmental variable with the desired datetime format: OPENAPI_DATETIME_FORMAT="{YOUR_DESIRED_FORMAT_FOR_DATETIME_strftime}". This is something I added to 1.0.6.
foxpluto commented 1 year ago

Wonderful !!!

At least I have my API documentation in order on my site. I have used your second option with:

OPENAPI_DATETIME_FORMAT="%Y%m%dT%H%M%S.%fZ"

And works perfectly.

Thanks

RobertoPrevato commented 1 year ago

You're welcome 😄 I leave this issue open to recall about documenting this feature, when I get the time. Now I'm still dedicating time to my web framework.