apache / doris-spark-connector

Spark Connector for Apache Doris
https://doris.apache.org/
Apache License 2.0
79 stars 92 forks source link

[Improvement]Put generateSerializedResult in try catch to avoid insufficient memory caused by excessive data size #202

Closed lxwcodemonkey closed 4 months ago

lxwcodemonkey commented 5 months ago

Proposed changes

Issue Number: close #xxx

Problem Summary:

When a batch of data is too large, it will cause OOM when generatingSerializedResult, but also it is not in trycatch, a batch of data cannot be divided. cf05a261a8f4d2338b9cfd2fca9f2af

Checklist(Required)

  1. Does it affect the original behavior: (Yes/No/I Don't know)
  2. Has unit tests been added: (Yes/No/No Need)
  3. Has document been added or modified: (Yes/No/No Need)
  4. Does it need to update dependencies: (Yes/No)
  5. Are there any changes that cannot be rolled back: (Yes/No)

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

lxwcodemonkey commented 5 months ago

@JNSimba Could you help me to have a look? thanks!