export_to_file running out of memory

[ghost]

Hello,

I'm currently experiencing export_to_file running OOM when exporting large amounts of data with limited memory. Since this isn't mentioned in the docs (unlike for the other export functions) I wonder if this is a bug?

[littleK0i]

It should not be the case. export_to_file() is a basic wrapper for shutil.copyfileobj: https://github.com/exasol/pyexasol/blob/master/pyexasol/callback.py#L45-L52

Could you post full error message and a bit more context?

Thank you.

[ghost]

Currently I don't have any further data on the error, will try to get some more information when the job runs again tomorrow.

[Nicoretti]

Hi @xyxz-web,

are there any news/updates from your side? We are happy to help, but without more details we can't really do anything.

Also feel free to close the issue and open/reopen an issue, once you have more details. I will add issue templates, to give issues creators more guidance in the future.

best Nico

[ghost]

Hi @Nicoretti ,

trying to nail down the exact spot where this happens... On my local machine the export to file doesn't seem to cause a memory spike.

I'll post again once I know more.

best regards

[ghost]

I've run the exact same code on Windows and a Kubernetes pod based on a Debian Bullseye image with the following results:

• On Windows the Job needs 63Mib RAM according to memory_profiler.
• On Kubernetes I currently don't have memory_profiler, but the Pod is killed with OOM at 8Gi RAM halfway through.

Do you have an idea what could cause this?

[Nicoretti]

Hi @xyxz-web

Thanks for the update, still this information is too vague to provide any sophisticated help.

In order to provide sophisticated help, we generally require the following things:

1. Detailed Description: Please give us a clear and detailed explanation of the issue, including any error messages and steps to reproduce it.

2. Version Info: Let us know which version of the library or software you're using.

3. Expected vs. Actual Behavior: Tell us what you expected to happen and what actually happened.

4. Reproducibility: If possible, provide steps to consistently reproduce the issue.

5. Logs/Screenshots: Include relevant logs or screenshots if available.

That said, as the library generally works well (also in your windows deployment), the issue also could be related to the Kubernetes deployment itself.

As alternative to this "public" issue tracking board, you also can reach out to the official Exasol-Support which isn't publicly disclosed (e.g. if the problem contains sensitive information).

best Nico

[ghost]

Hi Nico,

thanks for your answer! In that case we'll probably contact the support, though I need to discuss this internally beforehand.

Best regards

[Nicoretti]

@xyxz-web I'll, close this issue then. If you later want to come back to it with more details, feel free to reopen it.

best Nico