Support for Polars

[ghost]

Polars is an up and coming alternative for Pandas which already has more then 17.000 GitHub Stars: https://pola-rs.github.io/polars-book/

I wonder if it would be possible to provide an export_to_polars and import_from_polars function without too much effort?

As a workaround Polars could already be used with polars.from_pandas, but this involves copying the whole data.

[tkilias]

Hi xyxz-web,

export_to_pandas is basically a special implementation of export_to_callback, so the quickest thing would implementing a call back which reads the raw data into polaris.

Here is an example for export_to_callback https://github.com/exasol/pyexasol/blob/master/examples/b02_import_export.py.

Here you can see how the export_to_pandas callback is called https://github.com/exasol/pyexasol/blob/63e97d84b08ae6452b7275aa90199c6fc3f371e3/pyexasol/connection.py#L273

Here you can see how the callback is implemented. https://github.com/exasol/pyexasol/blob/63e97d84b08ae6452b7275aa90199c6fc3f371e3/pyexasol/callback.py#L36

It basically reads the CSV coming from the pipe with pandas.read_csv. I think, polars provides almost the same API. https://pola-rs.github.io/polars/py-polars/html/reference/api/polars.read_csv.html

Hope that already helps.

[tkilias]

Hi @xyxz-web,

We discussed this internally, and Polars is currently not our roadmap for this project. I hope, you can write your own callback with the of my previous explanation, should you need it. For that reason, I will close this ticket for the moment.

[ghost]

Hi @tkilias ,

thanks for your answer! In case I would implement this, could it be merged to the master branch?

[littleK0i]

Ok, here is the CSV-parsing call in Polars: https://pola-rs.github.io/polars/py-polars/html/reference/api/polars.read_csv.html

Implementation is trivial, but we have a few interesting questions to answer:

Decimal type? Polars supports Decimal type natively: https://pola-rs.github.io/polars/py-polars/html/reference/api/polars.Decimal.html

It might be nice to use it automatically for DECIMAL columns.

try_parse_dates? I think it should be enabled by default, as long as it can fit full range of Exasol TIMESTAMP values.

rechunk? It is enabled by default. Not sure if it should stay enabled.

[tkilias]

Hi,

If you want to implement it and have it merged, I would suggest making polars an optional python dependency, such that user can decide if he would like to install it, because I am not sure about the size of polars and compatibility with different platforms.

[littleK0i]

This seems to be working: https://github.com/exasol/pyexasol/compare/master...polars

The only problem is related to warnings, which I do not know how to mute:

Polars found a filename. Ensure you pass a path to the file instead of a python file object when possible for best performance.
Polars found a filename. Ensure you pass a path to the file instead of a python file object when possible for best performance.

Pyexasol writes into pipe, but polars thinks it's a file.

Output:

shape: (10_000, 8)
┌─────────┬────────────┬───────────┬────────────┬───────────┬─────────────┬────────────┬───────────┐
│ USER_ID ┆ USER_NAME  ┆ REGISTER_ ┆ LAST_VISIT ┆ IS_FEMALE ┆ USER_RATING ┆ USER_SCORE ┆ STATUS    │
│ ---     ┆ ---        ┆ DT        ┆ _TS        ┆ ---       ┆ ---         ┆ ---        ┆ ---       │
│ i64     ┆ str        ┆ ---       ┆ ---        ┆ i64       ┆ f64         ┆ f64        ┆ str       │
│         ┆            ┆ date      ┆ datetime[μ ┆           ┆             ┆            ┆           │
│         ┆            ┆           ┆ s]         ┆           ┆             ┆            ┆           │
╞═════════╪════════════╪═══════════╪════════════╪═══════════╪═════════════╪════════════╪═══════════╡
│ 0       ┆ KTSU80DIWH ┆ 2018-08-1 ┆ 2018-05-20 ┆ 0         ┆ 0.04        ┆ 8.07       ┆ PENDING   │
│         ┆            ┆ 9         ┆ 19:19:04.3 ┆           ┆             ┆            ┆           │
│         ┆            ┆           ┆ 71         ┆           ┆             ┆            ┆           │
│ 1       ┆ O0PH2A7OM5 ┆ 2018-03-0 ┆ 2018-10-03 ┆ 1         ┆ 0.26        ┆ 46.13      ┆ DISABLED  │
│         ┆            ┆ 2         ┆ 07:59:06.0 ┆           ┆             ┆            ┆           │
│         ┆            ┆           ┆ 91         ┆           ┆             ┆            ┆           │
│ 2       ┆ E19GGXPHZE ┆ 2018-04-2 ┆ 2018-05-14 ┆ 0         ┆ 0.28        ┆ null       ┆ SUSPENDED │
│         ┆            ┆ 3         ┆ 15:58:59.7 ┆           ┆             ┆            ┆           │
│         ┆            ┆           ┆ 27         ┆           ┆             ┆            ┆           │
│ 3       ┆ NDNSL7VWKL ┆ 2018-05-2 ┆ 2018-03-08 ┆ 0         ┆ 0.84        ┆ 10.75      ┆ ACTIVE    │
│         ┆            ┆ 0         ┆ 15:39:04.0 ┆           ┆             ┆            ┆           │
│         ┆            ┆           ┆ 26         ┆           ┆             ┆            ┆           │
│ …       ┆ …          ┆ …         ┆ …          ┆ …         ┆ …           ┆ …          ┆ …         │
│ 9996    ┆ 0WY4RNSIYV ┆ 2018-01-2 ┆ 2018-01-24 ┆ 0         ┆ 0.61        ┆ 44.9       ┆ ACTIVE    │
│         ┆            ┆ 0         ┆ 00:33:56.4 ┆           ┆             ┆            ┆           │
│         ┆            ┆           ┆ 00         ┆           ┆             ┆            ┆           │
│ 9997    ┆ CEVORWHH6P ┆ 2018-03-0 ┆ 2018-12-31 ┆ 1         ┆ 0.56        ┆ 31.23      ┆ DISABLED  │
│         ┆            ┆ 7         ┆ 11:19:13.0 ┆           ┆             ┆            ┆           │
│         ┆            ┆           ┆ 58         ┆           ┆             ┆            ┆           │
│ 9998    ┆ M8K2YDG7JR ┆ 2018-10-1 ┆ 2018-05-10 ┆ 0         ┆ 0.59        ┆ 21.12      ┆ SUSPENDED │
│         ┆            ┆ 8         ┆ 09:04:13.9 ┆           ┆             ┆            ┆           │
│         ┆            ┆           ┆ 55         ┆           ┆             ┆            ┆           │
│ 9999    ┆ CJY3XA21LU ┆ 2018-07-2 ┆ 2018-02-03 ┆ 0         ┆ 0.84        ┆ 60.88      ┆ DISABLED  │
│         ┆            ┆ 9         ┆ 21:06:53.5 ┆           ┆             ┆            ┆           │
│         ┆            ┆           ┆ 95         ┆           ┆             ┆            ┆           │
└─────────┴────────────┴───────────┴────────────┴───────────┴─────────────┴────────────┴───────────┘
EXPORTED 10000 rows in 1.13350248336792s
IMPORTED 0 rows in 0.059839725494384766s
shape: (5_000, 2)
┌─────────┬────────────┐
│ USER_ID ┆ USER_NAME  │
│ ---     ┆ ---        │
│ i64     ┆ str        │
╞═════════╪════════════╡
│ 8192    ┆ 11MPTFV6GF │
│ 8193    ┆ KYVDZY4JJC │
│ 8194    ┆ G6L6A4QRLQ │
│ 8195    ┆ 7CGA3VXMGD │
│ …       ┆ …          │
│ 8188    ┆ IVV5VI3T2Z │
│ 8189    ┆ 5SE7LA5AUR │
│ 8190    ┆ 1YIWZZEPQE │
│ 8191    ┆ ILLJ1XAW78 │
└─────────┴────────────┘
EXPORTED 5000 rows in 1.0486383438110352s

[littleK0i]

Also, I've double-checked DateTime type. It supports the full range of values from 0001-01-01 to 9999-12-31 23:59:59.999.

No issues with that.

[raayu83]

I wonder if there has been any progress on this lately.

[raayu83]

Hello,

since polars passed version 1.0 recently... Are there any plans to finalize this? I can help with some tasks, but I can't run the tests locally because I'm on Windows.

[Nicoretti]

Hi @raayu83,

as far as I understand, this was closed because it is not planned to be implemented. However, I will go and bring this up with our PM. In case this issue is not reopened or linked to a new one, you can safely assume that it is still not planned.

best Nico

[raayu83]

Would really love to see Polars support, since it is probably the next big thing after Pandas. The 1.0 release was mentioned in many places.

[Nicoretti]

@raayu83 Good point, I agree. I can tell you that we are generally working on supporting and providing more integrations. In the realm of DataFrames, most notably since the beginning of this year, Ibis has been supporting Exasol, which can be a viable alternative to Pandas depending on the use cases at hand.

That said, I will definitely bring your issue forward to our PM, but the decision regarding prioritization is not up to me.

[Nicoretti]

Hi @raayu83,

I just had a discussion with the respective PM. He also agrees that this would be a nice feature, but sadly, it is currently not high enough priority to schedule.

Regarding your suggestion/offer:

I can help with some tasks, but I can't run the tests locally because I'm on Windows.

The feedback was that we will provide assistance for an external PR, including help with test setup and potential project-specific hurdles, as well as code reviews and feedback. However, we can't provide any implementation effort.

Side Note: The CI for PRs is running tests on a Linux image, so running tests within the CI in the context of the PR would be an option.

best Nico

[raayu83]

Hi Nico,

great to hear you are open for a contribution! I can't promise any timeline, but I'm willing to tackle this once possible. Do you have some docs on how to setup the dev environment on windows? This would be very helpful in the process. I have access to WSL2 and Docker for Windows in case it is necessary.

[Nicoretti]

@raayu83 not yet. However, if you have Poetry installed, setting up the basic project should be straightforward.

:information_source: On Windows, you may encounter issues with one or some of the development (test) dependencies (e.g. itde integration-test-docker-environment). For starters, you could disable them locally if you don't want to run the tests locally. If you do want to run the tests locally and encounter specific problems, we can tackle them then.

Here is the workflow for setting up the Poetry project:

1. cd into the checkout directory (<pyexsol-root>)
2. Create a Poetry environment/shell by running poetry shell
3. Install the dependencies by running poetry install --sync Note: This step may fail due to dev-dependency issues on Windows. In that case, you need to fix them. Alternatively, you can choose not to install any development dependencies by executing either poetry install --sync --no-dev or poetry install --sync --without dev.
4. You should now have a working environment. Depending on the available installed dependencies, you can run tasks listed with nox -l.

Hope that's enough to get you started. (If not, let me know)

best Nico

[Nicoretti]

I have access to WSL2 and Docker for Windows in case it is necessary.

Haven't worked with wsl2, but if you install poetry and run everything within WSL2 it may even work "exactly" like on Linux.

[raayu83]

I've tried multiple routes getting the test environment working, allow of which failed at nox -s db-start.

1. PowerShell: Error because the logfile can't be created
2. WSL: Error because the port can't be bound (will try upgrading docker desktop)
3. MacOS with colima: The process starts, but aborts with an error. Here is the log:
   Click to expand

nox -s db-start                                             
nox > Running session db-start
nox > itde spawn-test-environment --environment-name test --database-port-forward 8563 --bucketfs-port-forward 2580 --docker-db-image-version 7.1.17 --db-mem-size 4GB
PrepareDockerNetworkForTestEnvironment_d91fa6010b(job_id=2024_07_10_19_47_28_1_SpawnTestEnvironmentWithDockerDB, no_cache=False, environment_name=test, network_name=db_network_test, attempt=0): Exception in run: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
Traceback (most recent call last):
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/urllib3/connectionpool.py", line 789, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/urllib3/connectionpool.py", line 495, in _make_request
    conn.request(
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/urllib3/connection.py", line 398, in request
    self.endheaders()
  File "/opt/homebrew/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/http/client.py", line 1331, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/opt/homebrew/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/http/client.py", line 1091, in _send_output
    self.send(msg)
  File "/opt/homebrew/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/http/client.py", line 1035, in send
    self.connect()
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/docker/transport/unixconn.py", line 26, in connect
    sock.connect(self.unix_socket)
FileNotFoundError: [Errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/requests/adapters.py", line 667, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/urllib3/connectionpool.py", line 843, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/urllib3/util/retry.py", line 474, in increment
    raise reraise(type(error), error, _stacktrace)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/urllib3/util/util.py", line 38, in reraise
    raise value.with_traceback(tb)
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/urllib3/connectionpool.py", line 789, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/urllib3/connectionpool.py", line 495, in _make_request
    conn.request(
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/urllib3/connection.py", line 398, in request
    self.endheaders()
  File "/opt/homebrew/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/http/client.py", line 1331, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/opt/homebrew/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/http/client.py", line 1091, in _send_output
    self.send(msg)
  File "/opt/homebrew/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/http/client.py", line 1035, in send
    self.connect()
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/docker/transport/unixconn.py", line 26, in connect
    sock.connect(self.unix_socket)
urllib3.exceptions.ProtocolError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/docker/api/client.py", line 223, in _retrieve_server_version
    return self.version(api_version=False)["ApiVersion"]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/docker/api/daemon.py", line 181, in version
    return self._result(self._get(url), json=True)
                        ^^^^^^^^^^^^^^
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/docker/utils/decorators.py", line 44, in inner
    return f(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/docker/api/client.py", line 246, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/requests/sessions.py", line 602, in get
    return self.request("GET", url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/requests/adapters.py", line 682, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/exasol_integration_test_docker_environment/lib/base/base_task.py", line 239, in run
    task_generator = self.run_task()
                     ^^^^^^^^^^^^^^^
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/exasol_integration_test_docker_environment/lib/test_environment/prepare_network_for_test_environment.py", line 28, in run_task
    self.network_info = self.create_docker_network()
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/exasol_integration_test_docker_environment/lib/test_environment/prepare_network_for_test_environment.py", line 39, in create_docker_network
    self.remove_container(self.test_container_name)
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/exasol_integration_test_docker_environment/lib/test_environment/prepare_network_for_test_environment.py", line 76, in remove_container
    with self._get_docker_client() as docker_client:
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/exasol_integration_test_docker_environment/lib/docker/__init__.py", line 14, in __enter__
    self._client = docker.from_env(**self.kwargs)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/docker/client.py", line 94, in from_env
    return cls(
           ^^^^
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/docker/client.py", line 45, in __init__
    self.api = APIClient(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/docker/api/client.py", line 207, in __init__
    self._version = self._retrieve_server_version()
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/docker/api/client.py", line 230, in _retrieve_server_version
    raise DockerException(
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
[pid 78549] Worker Worker(salt=1044532436, workers=5, host=MinivonNiklaus, username=klst, pid=78540) failed    PrepareDockerNetworkForTestEnvironment_d91fa6010b(job_id=2024_07_10_19_47_28_1_SpawnTestEnvironmentWithDockerDB, no_cache=False, environment_name=test, network_name=db_network_test, attempt=0)
Traceback (most recent call last):
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/urllib3/connectionpool.py", line 789, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/urllib3/connectionpool.py", line 495, in _make_request
    conn.request(
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/urllib3/connection.py", line 398, in request
    self.endheaders()
  File "/opt/homebrew/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/http/client.py", line 1331, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/opt/homebrew/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/http/client.py", line 1091, in _send_output
    self.send(msg)
  File "/opt/homebrew/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/http/client.py", line 1035, in send
    self.connect()
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/docker/transport/unixconn.py", line 26, in connect
    sock.connect(self.unix_socket)
FileNotFoundError: [Errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/requests/adapters.py", line 667, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/urllib3/connectionpool.py", line 843, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/urllib3/util/retry.py", line 474, in increment
    raise reraise(type(error), error, _stacktrace)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/urllib3/util/util.py", line 38, in reraise
    raise value.with_traceback(tb)
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/urllib3/connectionpool.py", line 789, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/urllib3/connectionpool.py", line 495, in _make_request
    conn.request(
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/urllib3/connection.py", line 398, in request
    self.endheaders()
  File "/opt/homebrew/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/http/client.py", line 1331, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/opt/homebrew/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/http/client.py", line 1091, in _send_output
    self.send(msg)
  File "/opt/homebrew/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/http/client.py", line 1035, in send
    self.connect()
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/docker/transport/unixconn.py", line 26, in connect
    sock.connect(self.unix_socket)
urllib3.exceptions.ProtocolError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/docker/api/client.py", line 223, in _retrieve_server_version
    return self.version(api_version=False)["ApiVersion"]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/docker/api/daemon.py", line 181, in version
    return self._result(self._get(url), json=True)
                        ^^^^^^^^^^^^^^
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/docker/utils/decorators.py", line 44, in inner
    return f(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/docker/api/client.py", line 246, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/requests/sessions.py", line 602, in get
    return self.request("GET", url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/requests/adapters.py", line 682, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/luigi/worker.py", line 210, in run
    new_deps = self._run_get_new_deps()
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/luigi/worker.py", line 147, in _run_get_new_deps
    requires = next(task_gen)
               ^^^^^^^^^^^^^^
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/exasol_integration_test_docker_environment/lib/base/dependency_logger_base_task.py", line 103, in run
    yield from self.write_dependencies_for_generator(
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/exasol_integration_test_docker_environment/lib/base/dependency_logger_base_task.py", line 59, in write_dependencies_for_generator
    element = next(task_generator)
              ^^^^^^^^^^^^^^^^^^^^
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/exasol_integration_test_docker_environment/lib/base/stoppable_base_task.py", line 31, in run
    raise exception
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/exasol_integration_test_docker_environment/lib/base/stoppable_base_task.py", line 27, in run
    yield from task_generator
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/exasol_integration_test_docker_environment/lib/base/timeable_base_task.py", line 120, in run
    yield from task_generator
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/exasol_integration_test_docker_environment/lib/base/base_task.py", line 248, in run
    raise e
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/exasol_integration_test_docker_environment/lib/base/base_task.py", line 239, in run
    task_generator = self.run_task()
                     ^^^^^^^^^^^^^^^
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/exasol_integration_test_docker_environment/lib/test_environment/prepare_network_for_test_environment.py", line 28, in run_task
    self.network_info = self.create_docker_network()
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/exasol_integration_test_docker_environment/lib/test_environment/prepare_network_for_test_environment.py", line 39, in create_docker_network
    self.remove_container(self.test_container_name)
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/exasol_integration_test_docker_environment/lib/test_environment/prepare_network_for_test_environment.py", line 76, in remove_container
    with self._get_docker_client() as docker_client:
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/exasol_integration_test_docker_environment/lib/docker/__init__.py", line 14, in __enter__
    self._client = docker.from_env(**self.kwargs)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/docker/client.py", line 94, in from_env
    return cls(
           ^^^^
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/docker/client.py", line 45, in __init__
    self.api = APIClient(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/docker/api/client.py", line 207, in __init__
    self._version = self._retrieve_server_version()
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/docker/api/client.py", line 230, in _retrieve_server_version
    raise DockerException(
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
ERROR:root:Task SpawnTestEnvironmentWithDockerDB_e7e74bb23a(job_id=2024_07_10_19_47_28_1_SpawnTestEnvironmentWithDockerDB, no_cache=False, docker_db_image_name=exasol/docker-db, docker_db_image_version=7.1.17, db_user=sys, create_certificates=False, additional_db_parameter=[], environment_name=test) failed. failed target exists.
ERROR:root:Going to abort the task SpawnTestEnvironmentWithDockerDB_e7e74bb23a(job_id=2024_07_10_19_47_28_1_SpawnTestEnvironmentWithDockerDB, no_cache=False, docker_db_image_name=exasol/docker-db, docker_db_image_version=7.1.17, db_user=sys, create_certificates=False, additional_db_parameter=[], environment_name=test)
ERROR:luigi-interface.PrepareDockerNetworkForTestEnvironment:PrepareDockerNetworkForTestEnvironment_d91fa6010b(job_id=2024_07_10_19_47_28_1_SpawnTestEnvironmentWithDockerDB, no_cache=False, environment_name=test, network_name=db_network_test, attempt=0): Error during removing container test_container_test: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory')):
ERROR:luigi-interface.PrepareDockerNetworkForTestEnvironment:PrepareDockerNetworkForTestEnvironment_d91fa6010b(job_id=2024_07_10_19_47_28_1_SpawnTestEnvironmentWithDockerDB, no_cache=False, environment_name=test, network_name=db_network_test, attempt=0): Error during removing container db_container_test: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory')): 
ERROR:luigi-interface.PrepareDockerNetworkForTestEnvironment:PrepareDockerNetworkForTestEnvironment_d91fa6010b(job_id=2024_07_10_19_47_28_1_SpawnTestEnvironmentWithDockerDB, no_cache=False, environment_name=test, network_name=db_network_test, attempt=0): Error during removing container db_network_test: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
The command failed after 0.461334 s with:

Task failure message: Task SpawnTestEnvironmentWithDockerDB_e7e74bb23a(job_id=2024_07_10_19_47_28_1_SpawnTestEnvironmentWithDockerDB, no_cache=False, docker_db_image_name=exasol/docker-db, docker_db_image_version=7.1.17, db_user=sys, create_certificates=False, additional_db_parameter=[], environment_name=test) (or any of it's subtasks) failed.
Following task failures were caught during the execution:
- PrepareDockerNetworkForTestEnvironment_d91fa6010b:
    Traceback (most recent call last):
      File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/urllib3/connectionpool.py", line 789, in urlopen
        response = self._make_request(
                   ^^^^^^^^^^^^^^^^^^^
      File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/urllib3/connectionpool.py", line 495, in _make_request
        conn.request(
      File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/urllib3/connection.py", line 398, in request
        self.endheaders()
      File "/opt/homebrew/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/http/client.py", line 1331, in endheaders
        self._send_output(message_body, encode_chunked=encode_chunked)
      File "/opt/homebrew/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/http/client.py", line 1091, in _send_output
        self.send(msg)
      File "/opt/homebrew/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/http/client.py", line 1035, in send
        self.connect()
      File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/docker/transport/unixconn.py", line 26, in connect
        sock.connect(self.unix_socket)
    FileNotFoundError: [Errno 2] No such file or directory

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/requests/adapters.py", line 667, in send
        resp = conn.urlopen(
               ^^^^^^^^^^^^^
      File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/urllib3/connectionpool.py", line 843, in urlopen
        retries = retries.increment(
                  ^^^^^^^^^^^^^^^^^^
      File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/urllib3/util/retry.py", line 474, in increment
        raise reraise(type(error), error, _stacktrace)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/urllib3/util/util.py", line 38, in reraise
        raise value.with_traceback(tb)
      File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/urllib3/connectionpool.py", line 789, in urlopen
        response = self._make_request(
                   ^^^^^^^^^^^^^^^^^^^
      File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/urllib3/connectionpool.py", line 495, in _make_request
        conn.request(
      File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/urllib3/connection.py", line 398, in request
        self.endheaders()
      File "/opt/homebrew/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/http/client.py", line 1331, in endheaders
        self._send_output(message_body, encode_chunked=encode_chunked)
      File "/opt/homebrew/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/http/client.py", line 1091, in _send_output
        self.send(msg)
      File "/opt/homebrew/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/http/client.py", line 1035, in send
        self.connect()
      File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/docker/transport/unixconn.py", line 26, in connect
        sock.connect(self.unix_socket)
    urllib3.exceptions.ProtocolError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/docker/api/client.py", line 223, in _retrieve_server_version
        return self.version(api_version=False)["ApiVersion"]
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/docker/api/daemon.py", line 181, in version
        return self._result(self._get(url), json=True)
                            ^^^^^^^^^^^^^^
      File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/docker/utils/decorators.py", line 44, in inner
        return f(self, *args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^
      File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/docker/api/client.py", line 246, in _get
        return self.get(url, **self._set_request_timeout(kwargs))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/requests/sessions.py", line 602, in get
        return self.request("GET", url, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
        resp = self.send(prep, **send_kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/requests/sessions.py", line 703, in send
        r = adapter.send(request, **kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/requests/adapters.py", line 682, in send
        raise ConnectionError(err, request=request)
    requests.exceptions.ConnectionError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

    The above exception was the direct cause of the following exception:

    Traceback (most recent call last):
      File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/exasol_integration_test_docker_environment/lib/base/stoppable_base_task.py", line 27, in run
        yield from task_generator
      File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/exasol_integration_test_docker_environment/lib/base/timeable_base_task.py", line 120, in run
        yield from task_generator
      File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/exasol_integration_test_docker_environment/lib/base/base_task.py", line 248, in run
        raise e
      File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/exasol_integration_test_docker_environment/lib/base/base_task.py", line 239, in run
        task_generator = self.run_task()
                         ^^^^^^^^^^^^^^^
      File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/exasol_integration_test_docker_environment/lib/test_environment/prepare_network_for_test_environment.py", line 28, in run_task
        self.network_info = self.create_docker_network()
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/exasol_integration_test_docker_environment/lib/test_environment/prepare_network_for_test_environment.py", line 39, in create_docker_network
        self.remove_container(self.test_container_name)
      File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/exasol_integration_test_docker_environment/lib/test_environment/prepare_network_for_test_environment.py", line 76, in remove_container
        with self._get_docker_client() as docker_client:
      File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/exasol_integration_test_docker_environment/lib/docker/__init__.py", line 14, in __enter__
        self._client = docker.from_env(**self.kwargs)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/docker/client.py", line 94, in from_env
        return cls(
               ^^^^
      File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/docker/client.py", line 45, in __init__
        self.api = APIClient(*args, **kwargs)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/docker/api/client.py", line 207, in __init__
        self._version = self._retrieve_server_version()
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/docker/api/client.py", line 230, in _retrieve_server_version
        raise DockerException(
    docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

nox > Command itde spawn-test-environment --environment-name test --database-port-forward 8563 --bucketfs-port-forward 2580 --docker-db-image-version 7.1.17 --db-mem-size 4GB failed with exit code 1
nox > Session db-start failed.

[raayu83]

btw, would be ok to reopen the issue?

Also, which tools do you use to for code quality and formatting right now? Personally I prefer ruff check and ruff format (black codestyle based formatter) and pre-commit to ensure commits are clean.

[raayu83]

I've blindly edited the code for now. Am I right to assume that in order for the github workflows to run I need to open a pull request, even if I'm not yet finished?

[raayu83]

Got tests running on github. They fail because of issues with the TextIOWrapper: "OSError: No such device (os error 19)". I wonder if I'm doing something wrong or if this could be related to https://github.com/pola-rs/polars/issues/17428.

[Nicoretti]

Hi @raay83

btw, would be ok to reopen the issue?

Fine for me. I'll assign you to the issue, so the team will be aware that the issue is taken care of by someone outside.

Also, which tools do you use to for code quality and formatting right now?

Right now, we are using black (If you are interested in the details see python-toolbox). That being said, we are just in the process of adopting/integrating pyexasol into Exasol standard toolset and setup (we have taken maintenance "recently"). So far, the focus has been to switch to poetry and add a basic suite of integration tests. The formatter, etc. are on the agenda but not the next up in the list, so my suggestions would be:

• Only apply the formatter to the changset or
• Isolate your additions mostly within one file which you are free to format however you like. or
• Create a PR adding the ruff dependency and configuration which also I can do then a reformat of the entire project which can be excluded from diffs and afterwards you can work from there (start the "next pr" with the feature)

The main thing I want to avoid is increasing or adding noise to the diffs.

Am I right to assume that in order for the github workflows to run I need to open a pull request, even if I'm not yet finished?

Yes, if you are working on your fork, it should just work if you create a PR. On our repo approval for the runs are also required (I'll should get automatic notification on that though).

Got tests running on Github. They fail because of issues with the TextIOWrapper: "OSError: No such device (os error 19)". I wonder if I'm doing something wrong or if this could be related to pola-rs/polars#17428.

Could be, yes but can't tell for sure.

[Nicoretti]

3. MacOS with colima:

The integration-test-docker-environment (docker-db) does not work on ARM.

[raayu83]

Ok I need a working local database for debugging this properly... Can't use my company's Exasol for now... I've been able to get the Docker container up and running using colima in x86 mode.

The final lines of the container logs say:

[2024-07-11 19:46:48.670915 +00:00] stage6: Current node id is '11'
[2024-07-11 19:46:48.672450 +00:00] stage6: All nodes are online: ('n11',)
[2024-07-11 19:46:48.673110 +00:00] stage6: Init done marker in mode 'finish' done with file '/exa/etc/init_done' done.
[2024-07-11 19:46:48.737499 +00:00] stage6: All stages finished.
[2024-07-11 19:46:50.515920 +00:00] root child 366 (python3_cos) returned with state 0.

So I assume it is running properly?

Docker PS says the following:

CONTAINER ID   IMAGE                     COMMAND                  CREATED          STATUS          PORTS                      NAMES
b8b138c8b27c   exasol/docker-db:latest   "/opt/exasol/cos-8.4…"   18 minutes ago   Up 18 minutes   127.0.0.1:9563->8563/tcp   exasoldb

So I assume I shoud be able to connect with dsn 127.0.01:9563, usersys and password exasol.

However I get the following error when trying to connect with pyexasol (encryption disabled): Could not connect to Exasol: Connection to remote host was lost.

Do you have any hints on what I might be doing wrong?

[Nicoretti]

Hi @raayu83,

The default connection details for default Docker-DB are:

User: sys Password: exasol Port: 8563

That being said, according to your mapping, the settings you have been using look good from my point of view.

Are you using an Intel or an ARM-based Mac? AFIK, the Docker-DB image does not work on ARM-based systems, even when running in x86 VMs. When running on an ARM-based system, it will not function properly due to the lack of some SSE features. The container may be up, but the DB itself will not be functional. You can check this by obtaining a shell within the container and examining the DB logs (for more details checkout exasupport). You should see something like Shutting down due to missing SSSE3 support.

ARM-based Mac "Solutions" If you are using an ARM based Mac, currently the only known solution(s) from my side are:

1. Use a DB running on an other system
2. Use a Docker host running on another system (DOCKER_HOST). See some example usages here.

[raayu83]

Hi @Nicoretti ,

you are working at Exasol, correct? Could you imagine giving me free access to a single demo node in the cloud for the purpose of writing MRs such as the current one? I wouldn't use it for anything but contributions to pyexasol.

[Nicoretti]

Hi @raayu83, that's not something I can do myself, but I see where you are coming from. If you could send me this inquiry to nicola.coretti@exasol.com, ideally also formulated in a way that a third person can understand the context without reading through this ticket, then I can reach out to appropriate people.

[raayu83]

I've now confirmed the error raised is most likely a polars bug, since according to the docs any string or bytes IO should be supported. Waiting for response on https://github.com/pola-rs/polars/issues/17428 since loading the whole file into memory doesn't sound like a good option.

[littleK0i]

I suspect you do not need TextIOWrapper. Send binary stream directly to pollars.

Check my original commit for reference: https://github.com/exasol/pyexasol/compare/master...polars

[raayu83]

Hi,

thanks, I've tried that as well, but it doesn't work anymore. I've opened a bug at polars.