Bug: DocumentPost.created raises TypeError

[andreheuer]

Thank you for constantly improving and updating the library! It is very helpful!

However, I am facing an issue with the "created" value when posting a document as a TypeError is being raised. I have used the following code:

    await paperless_andre.initialize()
    doc_post = DocumentPost(
        title="Hallo",
        document=open(
            "some-file.pdf",
            "rb",
        ),
        created=datetime.now(),
    )

    print(await paperless_andre.documents.create(doc_post))

    await paperless_andre.close()

When executing this code, the following TypeError is being raised:

Traceback (most recent call last):
  File "[...]/.venv/lib/python3.11/site-packages/aiohttp/formdata.py", line 145, in _gen_form_data
    part = payload.get_payload(
           ^^^^^^^^^^^^^^^^^^^^
  File "[...]/.venv/lib/python3.11/site-packages/aiohttp/payload.py", line 72, in get_payload
    return PAYLOAD_REGISTRY.get(data, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[...]/.venv/lib/python3.11/site-packages/aiohttp/payload.py", line 119, in get
    raise LookupError()
aiohttp.payload.LookupError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "[...]/migration.py", line 134, in <module>
    asyncio.run(main())
  File "/opt/homebrew/Cellar/python@3.11/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "[...]/migration.py", line 59, in main
    print(await paperless_andre.documents.create(doc_post))
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[...]/.venv/lib/python3.11/site-packages/pypaperless/controllers/documents.py", line 142, in create
    res = str(await self._paperless.request_json("post", url, data=form))
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[...]/.venv/lib/python3.11/site-packages/pypaperless/__init__.py", line 223, in request_json
    async with self.generate_request(method, endpoint, **kwargs) as res:
  File "/opt/homebrew/Cellar/python@3.11/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/contextlib.py", line 204, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "[...]/.venv/lib/python3.11/site-packages/pypaperless/__init__.py", line 212, in generate_request
    async with self._session.request(method, url, **kwargs) as res:
  File "[...]/.venv/lib/python3.11/site-packages/aiohttp/client.py", line 1187, in __aenter__
    self._resp = await self._coro
                 ^^^^^^^^^^^^^^^^
  File "[...]/.venv/lib/python3.11/site-packages/aiohttp/client.py", line 541, in _request
    req = self._request_class(
          ^^^^^^^^^^^^^^^^^^^^
  File "[...]/.venv/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 333, in __init__
    self.update_body_from_data(data)
  File "[...]/.venv/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 546, in update_body_from_data
    body = body()
           ^^^^^^
  File "[...]/.venv/lib/python3.11/site-packages/aiohttp/formdata.py", line 170, in __call__
    return self._gen_form_data()
           ^^^^^^^^^^^^^^^^^^^^^
  File "[...]/.venv/lib/python3.11/site-packages/aiohttp/formdata.py", line 149, in _gen_form_data
    raise TypeError(
TypeError: Can not serialize value type: <class 'datetime.datetime'>
 headers: {}
 value: datetime.datetime(2024, 1, 5, 16, 35, 41, 571802)
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x104763dd0>
Unclosed connector
connections: ['[(<aiohttp.client_proto.ResponseHandler object at 0x104754670>, 451375.45515175)]']
connector: <aiohttp.connector.TCPConnector object at 0x104763e10>

I have also tried date.now() instead of datetime.now() without any difference :-(

Any ideas?

BTW: the created property is also not covered in the tests.

[andreheuer]

The same issue appears also for the document_type with following error:

[2024-01-05 16:52:01,909: ERROR/ForkPoolWorker-8] Task Rechnung an Paperless senden[538881f7-451a-4440-aba4-4cb942baf173] raised unexpected: TypeError("Can not serialize value type: <class 'int'>\n headers: {}\n value: 4")
Traceback (most recent call last):
  File "[...]/.venv/lib/python3.11/site-packages/aiohttp/formdata.py", line 145, in _gen_form_data
    part = payload.get_payload(
           ^^^^^^^^^^^^^^^^^^^^
  File "[...]/.venv/lib/python3.11/site-packages/aiohttp/payload.py", line 72, in get_payload
    return PAYLOAD_REGISTRY.get(data, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[...]/.venv/lib/python3.11/site-packages/aiohttp/payload.py", line 119, in get
    raise LookupError()
aiohttp.payload.LookupError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "[...]/.venv/lib/python3.11/site-packages/celery/app/trace.py", line 477, in trace_task
    R = retval = fun(*args, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^
  File "[...]/.venv/lib/python3.11/site-packages/celery/app/trace.py", line 760, in __protected_call__
    return self.run(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[...]/pkv_web/pdfimport/tasks.py", line 23, in send_rechnung_to_paperless
    asyncio.run(asend_rechnung_to_paperless(rechnung_id))
  File "/opt/homebrew/Cellar/python@3.11/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "[...]/pkv_web/pdfimport/tasks.py", line 32, in asend_rechnung_to_paperless
    async with paperless:
  File "[...]/.venv/lib/python3.11/site-packages/pypaperless/__init__.py", line 263, in __aexit__
    raise exc
  File "[...]/pkv_web/pdfimport/tasks.py", line 57, in asend_rechnung_to_paperless
    task_id = await paperless.documents.create(document_post)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[...]/.venv/lib/python3.11/site-packages/pypaperless/controllers/documents.py", line 142, in create
    res = str(await self._paperless.request_json("post", url, data=form))
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[...]/.venv/lib/python3.11/site-packages/pypaperless/__init__.py", line 221, in request_json
    async with self.generate_request(method, endpoint, **kwargs) as res:
  File "/opt/homebrew/Cellar/python@3.11/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/contextlib.py", line 204, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "[...]/.venv/lib/python3.11/site-packages/pypaperless/__init__.py", line 211, in generate_request
    async with self._session.request(method, url, **kwargs) as res:
  File "[...]/.venv/lib/python3.11/site-packages/aiohttp/client.py", line 1187, in __aenter__
    self._resp = await self._coro
                 ^^^^^^^^^^^^^^^^
  File "[...]/.venv/lib/python3.11/site-packages/aiohttp/client.py", line 541, in _request
    req = self._request_class(
          ^^^^^^^^^^^^^^^^^^^^
  File "[...]/.venv/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 333, in __init__
    self.update_body_from_data(data)
  File "[...]/.venv/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 546, in update_body_from_data
    body = body()
           ^^^^^^
  File "[...]/.venv/lib/python3.11/site-packages/aiohttp/formdata.py", line 170, in __call__
    return self._gen_form_data()
           ^^^^^^^^^^^^^^^^^^^^^
  File "[...]/.venv/lib/python3.11/site-packages/aiohttp/formdata.py", line 149, in _gen_form_data
    raise TypeError(
TypeError: Can not serialize value type: <class 'int'>
 headers: {}
 value: 4

I think this is related to this call . Based on the aiohttp documentation of add_field, it only supports string or binary values. However, the DocumentPost class uses int´s and date´s, which are not converted.

[tb1337]

Thank you, I gonna check that. Still figuring out how to better test requests, since it is my first python project ever.

[andreheuer]

For testing purposes, I would recommend to make real requests against a Paperless instance. This would allow to find issues as I did. Might be an option would be to use the Paperless demo instance (or at least the data in it)? https://demo.paperless-ngx.com/

[tb1337]

Already thought about that too, but I can't build a ci/test pipeline which utilizes data out of my control. If the paperless team changes that data, every ci would fail, too risky for me. That's the reason for making use of preassembled test data and mocking that in. Gonna find another, or at least an additional way.

[andreheuer]

Just found out the following: when creating a post as follows:

    doc_post = DocumentPost(
        title="Hallo",
        document=open(
            "[...].pdf",
            "rb",
        ),
        tags=["175"],
        created="2023-01-02",
    )

I do not get a TypeError when posting the document. However, this contradicts to the definition DocumentPost as datetime (but it works ;-))

I would also like to contribute, but I am struggling with setting up the dev environment :(

[tb1337]

Locally I‘ve got a new branch with a devcontainer. That should help as it’s setting everything up for you. I will push it tomorrow. You only need docker and visual studio code on your machine 😅

Played around with end2end tests against a real paperless-ngx, but I think it’s way too much for a little api wrapper. I will overhaul the testing suite and mock aiohttp instead.

[tb1337]

Okay, I‘m almost done with improving stuff. Implemented a Fake PaperlessAPI instead of patching responses, well, it enables better tests with different versions of a Paperless API.

The TypeError you mentioned is actually a bug. When developed that locally against a dev Paperless, it worked fine. I did a lesser change to the code and that added the bug.

On the dev branch I moved the aiohttp.FormData stuff away from the documents controller, right into the Paperless class. It will handle the correct conversion centrally (in the next commit). And after that we have a new release, including workflows. 😅

[andreheuer]

Locally I‘ve got a new branch with a devcontainer. That should help as it’s setting everything up for you. I will push it tomorrow. You only need docker and visual studio code on your machine 😅

Played around with end2end tests against a real paperless-ngx, but I think it’s way too much for a little api wrapper. I will overhaul the testing suite and mock aiohttp instead.

I have tried to use the container, however it is not starting. I am building the container with `docker build . -t pypaperless' and then 'docker run pypaperless'. Are there additional params required? Can you provide a docker-compose file?