Netflix / metaflow-service

:rocket: Metadata tracking and UI service for Metaflow!
http://www.metaflow.org
Apache License 2.0
193 stars 71 forks source link

async cache client fix: don't capture stderr since it contains logs #403

Open oavdeev opened 11 months ago

oavdeev commented 11 months ago

Cache subprocesses communicate with the main process via stdin/stdout pipe. However it turns out we were also redirecting stderr to stdout, which would unintentionally send a lot of log data there too. It would be typically cause JSONDecodeError on reader side on line 61:

        except JSONDecodeError as ex:
            if self.logger.isEnabledFor(logging.INFO):
                self.logger.info("Message: {}".format(line))

However in some cases that log output might be parseable as JSON which triggers other code paths. This PR fixes stderr redirect