Dear all,
I am using spark-celery in my application.
I am calling my query-System (where I need to access broadcasted big files) from spark-celery. I want to broadcast (using spark-context) all the files from the spark-celery only one time, and keep it up in the memory. So that, for the each new request to the application, I don't have to broadcast the files each time. How can I do it from spark-celery? Unfortunately I am not able to find any solutions.
Could you please suggest some way forward in this situation?
Dear all, I am using spark-celery in my application. I am calling my query-System (where I need to access broadcasted big files) from spark-celery. I want to broadcast (using spark-context) all the files from the spark-celery only one time, and keep it up in the memory. So that, for the each new request to the application, I don't have to broadcast the files each time. How can I do it from spark-celery? Unfortunately I am not able to find any solutions. Could you please suggest some way forward in this situation?
Thank you very much.
Kind regards, Neel