JohnSnowLabs / spark-nlp

State of the Art Natural Language Processing
https://sparknlp.org/
Apache License 2.0
3.88k stars 711 forks source link

[SPARKNLP-1052] Adding random suffix to avoid duplication in spark files #14340

Closed danilojsl closed 4 months ago

danilojsl commented 4 months ago

Description

This PR addresses the issue of file duplication by adding a random suffix when using sparkContext.addFile for ONNX and OpenVINO models. This enhancement ensures that each file added to Spark has a unique name, preventing conflicts and improving the robustness of file handling in distributed environments.

Motivation and Context

Check this jira issue

How Has This Been Tested?

Screenshots (if appropriate):

Types of changes

Checklist: