search

spotify / luigi

Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
Apache License 2.0
17.71k stars 2.39k forks source link

Change `subprocess.check_output` option for bytes/string mismatch #3285

Open marinelay opened 5 months ago

marinelay commented 5 months ago

https://github.com/spotify/luigi/blob/64d6c487c49548a5b97cc3ac6e0890f89d7dccd2/luigi/contrib/sge.py#L314-L322

In luigi.contrib.sge.py at line 321, text=True option have to be given in subprocess.check_output(['qstat']) because _parse_qstat_state function only allow string type. The current code always raise TypeError in _parse_qstat_state when running _track_job method.

If there is no reason qstat_out variable should be bytes type, I believe it is ideal to change subprocess.check_output(['qstat']) to subprocess.check_output(['qstat'], text=True).

marinelay commented 5 months ago

Wait, I found that this bug was already discussed and even given a pull request 6 years ago (#2480). I think it doesn't seem to be well maintained for some reason. I'll close the issue if it doesn't get fixed for a while.