SGE ignores os.environ

[fsimkovic]

System information

• Environment (if applicable): SGE, possibly other job management platforms

Problem description

Environment variables defined in os.environ during a call are currently not received in the job execution on SGE clusters, regardless of qsub -V flag.

This only affects array submissions, i.e. single scripts directly submitted can see variables defined in os.environ.

Example

>>> import os, time
>>> from pyjob.platform import prep_array_script
>>> from pyjob.platform.sge import SunGridEngine
>>>
>>> def clear_env(key):
...     if key in os.environ:
...         os.environ.pop(key)
>>>
>>> def test():
...     key = "PYJOB_ENV"
...     clear_env(key)
...     os.environ[key] = "test_test_test"
...     jobs = [make_script(["echo $"+key], directory=os.getcwd()) for _ in range(2)]
...     array_script, array_jobs = prep_array_script(jobs, os.getcwd(), 
...                                                  SunGridEngine.TASK_ID)
...     jobid = SunGridEngine.sub(array_script, array=[1, 2], directory=os.getcwd(), 
...                               log=os.devnull)
...     while SunGridEngine.stat(jobid):
...         time.sleep(1)
>>>
>>> test()

[fsimkovic]

It appears that this issue was fixed by one of the most recent commits. Closing for now.