alessandrothea
commented
3 years ago Hi @ron003, Thanks for looking into this. Indeed a python script seems the easiest way to achieve the behaviour we want and be independent by any system tool.
I quickly googled around how people re-implement tee in python, if they have to. This stackoverflow answer looks interesting:
class RunOutput():
def __init__(self, returncode, stdout, stderr):
self.returncode = returncode
self.stdout = stdout
self.stderr = stderr
async def _read_stream(stream, callback):
while True:
line = await stream.readline()
if line:
callback(line)
else:
break
async def _stream_subprocess(cmd, stdin=None, quiet=False, echo=False) -> RunOutput:
if isWindows():
platform_settings = {'env': os.environ}
else:
platform_settings = {'executable': '/bin/bash'}
if echo:
print(cmd)
p = await asyncio.create_subprocess_shell(cmd,
stdin=stdin,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE,
**platform_settings)
out = []
err = []
def tee(line, sink, pipe, label=""):
line = line.decode('utf-8').rstrip()
sink.append(line)
if not quiet:
print(label, line, file=pipe)
await asyncio.wait([
_read_stream(p.stdout, lambda l: tee(l, out, sys.stdout)),
_read_stream(p.stderr, lambda l: tee(l, err, sys.stderr, label="ERR:")),
])
return RunOutput(await p.wait(), out, err)
def run(cmd, stdin=None, quiet=False, echo=False) -> RunOutput:
loop = asyncio.get_event_loop()
result = loop.run_until_complete(
_stream_subprocess(cmd, stdin=stdin, quiet=quiet, echo=echo)
)
return result
ron003
Using
scriptis fundamentally flawed because it sends output to stdout and to a file so the output going to the file can only be filtered inside of the command line given in its command option. When filtering is done inside the command option, the command ahead of the filter (the CMake command in our case) will no longer consider its output to be atty(which would be the point of usingscript).Of the other options explored (
socatandunbuffer), onlyscriptreads from stdin (apparently assuming the command given to it wants stdin). An undesirable workaround would be to have the command be executed after dbt-build.sh in a paste buffer to be part of a pipeline or escaped command line:In the above
echo therewill be consumed byscriptinside ofdbt-build.sh.I understand that the other options are not optimal because the rpm (packages) that provides them may not be installed. To that end, I have created to small python scripts which could be added directly to
dbt-build.sh. For example,scriptcan be replaced by:and used:
With the above, the output will contain color and a "single progress line" (which will need filtering to turn into multi-line progress). There is a fundamental problem when trying to use
sedto accomplish this filtering.Sedis line (that isline feedand notcarriage return) oriented. NOTE: the currentsedincluding in... ${cmd} |& sed -e 's/\r/\n/g'is actually not doing anything becausecmake, being piped, stops sending\r. In other words, the current script implementation is basically equivalent to${cmd} | tee $build_log. Of the 3 major filter commands (sed, Perl, awk), only awk can do this because it supports "Record Separator" that can be multiple characters, i.e. bothcarriage returnandline feed. However, there is more to discuss here, but because this description is getting long, I'll save this for later. Just know, I believe I have a small python script (similar to the above) which I believe will do what is required.Speaking of requirements, here's a list, as I currently see things (and after some discussion with John and Alessandro) (not prioritized):
lesswith the -R option handles color)dbt-build.shshould not "eat" successive lines pasted; don't require a big pipeline.