[BUG] : Exception of class ConcurrentDeleteDeleteException not raised by OPTIMIZE command

[Jigar191089]

Description

when 2 jobs run OPTIMIZE command on same delta table, one of them throws Py4JJavaError exception. I was expecting ConcurrentDeleteDeleteException exception.

Steps to reproduce

Run OPTIMIZE command concurrently on same delta table

Observed results

Exception of Py4JJavaError is raised

Expected results

Exception of ConcurrentDeleteDeleteException

Further details

Environment information

• DataBricks run time : 12.2 LTS (includes Apache Spark 3.3.2, Scala 2.12)
• Delta Lake version : 2.2.0
• Spark version: 3.3.2
• Scala version: 2.12

Willingness to contribute

The Delta Lake Community encourages bug fix contributions. Would you or another member of your organization be willing to contribute a fix for this bug to the Delta Lake code base?

• [ ] Yes. I can contribute a fix for this bug independently.
• [ ] Yes. I would be willing to contribute a fix for this bug with guidance from the Delta Lake community.
• [X] No. I cannot contribute a bug fix at this time.

[tlm365]

@Jigar191089 "Exception of Py4JJavaError is raised" it's not directly related to Delta, it comes from Spark (particular Py4j module). I'm not sure about why you wanna catch ConcurrentDeleteDeleteException, but perhaps you need accept current problem or using Delta lib instead of Spark (ref: oss docs / db docs). Hope it's useful to you.

[Jigar191089]

I am running OPTIMIZE command which is related to Delta. So I was expecting ConcurrentDeleteDeleteException Exception instead of Py4JJavaError