galaxyproject / pulsar

Distributed job execution application built for Galaxy
https://pulsar.readthedocs.io
Apache License 2.0
37 stars 49 forks source link

Can't handle binary data in output streams #346

Closed natefoo closed 8 months ago

natefoo commented 8 months ago

e.g. someone ran a zip file through cutadapt, so the tool stderr contains:

Output will be written into the directory: /jetstream2/scratch/main/jobs/53509211/working/
File seems to be in SOLiD colorspace format which is not supported by Trim Galore (sequence is: '�Vn)}}%u`:�ᰱ��g/߶m
�)! Please use Cutadapt on colorspace files separately and check its documentation!                               o�<?v�yԩ(A��ФZ�5_hf/���/��η��mgz8;���6xv�M��=c7�z͛_Z�L������^���u]W���7|̍ߦ~��KF2-��b

This results in the following from Pulsar:

Exception in thread [manager=jetstream2]-[action=postprocess]-[job=53509211]:
  File "/usr/lib64/python3.9/threading.py", line 980, in _bootstrap_inner
    self.run()
  File "/srv/pulsar/main/venv/lib64/python3.9/site-packages/sentry_sdk/integrations/threading.py", line 72, in run
    reraise(*_capture_exception())
  File "/srv/pulsar/main/venv/lib64/python3.9/site-packages/sentry_sdk/_compat.py", line 115, in reraise
    raise value
  File "/srv/pulsar/main/venv/lib64/python3.9/site-packages/sentry_sdk/integrations/threading.py", line 70, in run
    return old_run_func(self, *a, **kw)
  File "/usr/lib64/python3.9/threading.py", line 917, in run
    self._target(*self._args, **self._kwargs)
  File "/srv/pulsar/main/venv/lib64/python3.9/site-packages/pulsar/managers/stateful.py", line 229, in do_postprocess
    self.__state_change_callback(final_status, job_id)
  File "/srv/pulsar/main/venv/lib64/python3.9/site-packages/pulsar/messaging/bind_amqp.py", line 73, in bind_on_status_change
    payload = manager_endpoint_util.full_status(manager, new_status, job_id)
  File "/srv/pulsar/main/venv/lib64/python3.9/site-packages/pulsar/manager_endpoint_util.py", line 26, in full_status
    full_status = __job_complete_dict(job_status, manager, job_id)
  File "/srv/pulsar/main/venv/lib64/python3.9/site-packages/pulsar/manager_endpoint_util.py", line 40, in __job_complete_dict
    stderr_contents = manager.stderr_contents(job_id).decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd1 in position 603: invalid continuation byte

This also results in a permanently non-terminal job.