civisanalytics / civis-python

Civis API Python Client
BSD 3-Clause "New" or "Revised" License
34 stars 26 forks source link

avoid duplicate parent job parameters when inferring joblib backend #363

Closed mheilman closed 4 years ago

mheilman commented 4 years ago

This addresses a bug where submitting jobs from a parallel backend from infer_joblib_backend failed when used in a child job where CIVIS_PARENT_JOB_ID and CIVIS_PARENT_RUN_ID were set. Those parameter names would be duplicated, which causes platform to reject the job.

I tweaked infer_joblib_backend because the issue pertains to use of that function. I decided to also replace rather than append parent job parameters in _ContainerShellExecutor.__init__ here because the behavior there also seemed problematic.

To do: