Thanks @sourav-mazumder for providing this capability; it is a critical enabler in leveraging distributed nature of Spark when calling REST APIs such as the Watson services on IBM Cloud. One suggestion: Occasionally, there are timeout errors that come back from the REST APIs being called and currently, that kills the job completely. It would be good to add a flag to allow Spark to continue processing other records and for the ones that timeout, to return with a timeout message indicating such.
Thanks @sourav-mazumder for providing this capability; it is a critical enabler in leveraging distributed nature of Spark when calling REST APIs such as the Watson services on IBM Cloud. One suggestion: Occasionally, there are timeout errors that come back from the REST APIs being called and currently, that kills the job completely. It would be good to add a flag to allow Spark to continue processing other records and for the ones that timeout, to return with a timeout message indicating such.