Error Handling for Corrupt Files in Chunk Processing

[toandaominh1997]

Is there an existing issue for this?

• [X] I have searched the existing issues

Current Behavior

I am using the spark-excel library to read Excel files in chunks from a directory. My process involves reading multiple Excel files and then performing operations on the data. However, I've encountered an issue where if any of the files are corrupt or cause an error during reading, the entire processing job fails and does not skip the problematic file.

Expected Behavior

I would expect the library to provide a mechanism to skip over files that cause errors during the read operation, allowing the process to continue with the next files in the list.

Steps To Reproduce

Set up a Spark session and configure it to read Excel files using spark-excel. Use a loop to read files in chunks from a specified directory. If an error occurs while reading a file (e.g., due to corruption or format issues), the process stops, and subsequent files are not processed.

Environment

- Spark version: 3.4.1
- Spark-Excel version: 0.20.3
- OS: Linux
- Cluster environment: Databricks

Anything else?

No response

[nightscape]

Could you post an excerpt of your code? This part sounds like you're looping over the files yourself:

Use a loop to read files in chunks from a specified directory.

If so, then it is under your control to catch any exceptions thrown during reading.