Importing Obspy in ipython disables colored output on Windows CMD

[samuelchodur]

Avoid duplicates

• [X] I searched existing issues

Bug Summary

After importing obspy, my IPython terminal does not print colored output. Tried this in several terminals on Windows (Git Bash and Anaconda Powershell Prompt). I can import other packages and continue to see colored output. Tested with 1.2.2 as well and had same problem.

Code to Reproduce

ipython
print(dne) # variable dne doesn't exist and prints error in color
import obspy
print(dne) # variable dne doesn't exists and does not print error in color

Error Traceback

No response

ObsPy Version?

1.3.0, 1.2.2

Operating System?

Windows

Python Version?

3.10

Installation Method?

conda

[filefolder]

working as expected here on linux / python 3.9.5 / ipython 8.30 and osx python 3.9.9 / ipython 7.22

[megies]

Can't reproduce. Maybe try updating IPython. Please reopen if needed.

[samuelchodur]

Can't reproduce. Maybe try updating IPython. Please reopen if needed.

I am using IPython version 8.5.0. Are you trying to reproduce the problem on Windows? I don't have issues on Linux.

[megies]

Oh this is Windows.. whelp, can't say anything about Windows really.

[samuelchodur]

Oh this is Windows.. whelp, can't say anything about Windows really.

could this get set back to bug-unconfirmed then until someone else comes along with same issue as I am having?

[megies]

@samuelchodur what kind of prompt is that then exactly? Ipython started in an Anaconda shell or however that is called? If so what Anaconda version? Is that problem exclusive to doing "import obspy" as first import? What happens if you import some other package first (e.g. numpy)?

@ThomasLecocq ever had any issues like that?

[filefolder]

I just tried this on an windows laptop and I can reproduce with python 3.10.7, Ipython 8.5.0, obspy 1.3.0. This is when calling ipython from the dos terminal. When running from the IDLE shell, or via the non-ipython terminal, it works normally.

In ipython I even get the same "1;31m" errors just by attempting to import a non-existing obspy module, e.g. from obspy.geodetics.base import blahblah so it seems like the whole environment is being contaminated just by browsing. No idea what that's about or what 1;31m refers to.

[megies]

@filefolder can you confirm this exclusively happens for obspy import? No other modules like numpy etc?

[megies]

No idea what that's about or what 1;31m refers to.

That's fragments of ANSI color codes, although I only know about Linux and have no idea if Windows uses these

https://en.wikipedia.org/wiki/ANSI_escape_code#Colors

It looks like the escape character at the start of coloring commands gets garbled, that's why I'm puzzled how obspy would be responsible, since the coloring commands should be done by ipython.

[ThomasLecocq]

I confirm this happens on window's CMD console, not on my Cmder console...

so far tried to import numpy, scipy, requets, pkg_resources, matplotlib : no issue

[filefolder]

yeah I went down the line importing things related-to, seemed to be fine. weird one.

[ThomasLecocq]

I'm testing all imports sequentially, will report later :-)

[ThomasLecocq]

So, I've tested the import sequence of ``python -v -c "import obspy"`` :

>['_frozen_importlib', '_imp', '_thread', '_warnings', '_weakref', '_frozen_importlib_external', '_io', 'marshal', 'nt', '_thread', '_thread', '_weakref', '_weakref', 'winreg', '# installing ziphook\n', 'time', 'zipimport', '# installed ziphook\n', '_codecs', 'codecs', 'encodings.aliases', 'encodings', 'encodings.utf_8', 'encodings.cp1252', '_signal', 'encodings.latin_1', '_abc', 'abc', 'io', '_stat', 'stat', '_collections_abc', 'genericpath', 'ntpath', 'os', '_sitebuiltins', '_locale', '_bootlocale', 'types', 'warnings', 'importlib', 'importlib.machinery', 'importlib.abc', '_operator', 'operator', 'keyword', '_heapq', 'heapq', 'itertools', 'reprlib', '_collections', 'collections', '_functools', 'functools', 'contextlib', 'importlib.util', 'site', 'math', '_datetime', 'datetime', 'enum', '_sre', 'sre_constants', 'sre_parse', 'sre_compile', 'copyreg', 're', 'locale', 'calendar', 'numpy._globals', 'numpy.__config__', '_json', 'json.scanner', 'json.decoder', 'json.encoder', 'json', 'numpy._version', 'numpy._distributor_init', 'numpy.version', 'textwrap', 'numpy.core._multiarray_umath', 'numpy.compat._inspect', 'posixpath', 'fnmatch', 'errno', 'urllib', 'urllib.parse', 'pathlib', '_struct', 'struct', '_compat_pickle', '_pickle', 'pickle', 'numpy.compat.py3k', 'numpy.compat', 'numpy.core.overrides', 'numpy.core.multiarray', 'numpy.core.umath', 'numbers', 'numpy.core._string_helpers', 'numpy.core._dtype', 'numpy.core._type_aliases', 'numpy.core.numerictypes', 'numpy.core._exceptions', 'numpy.core._methods', 'numpy.core.fromnumeric', 'numpy.core.shape_base', 'collections.abc', 'numpy.core._ufunc_config', 'numpy.core.arrayprint', 'numpy.core._asarray', 'numpy.core.numeric', 'numpy.core.defchararray', 'numpy.core.records', 'numpy.core.memmap', 'numpy.core.function_base', 'numpy.core.machar', 'numpy.core.getlimits', 'numpy.core.einsumfunc', 'numpy.core._multiarray_tests', 'numpy.core._add_newdocs', 'platform', '_socket', 'select', 'selectors', 'socket', 'numpy.core._dtype_ctypes', '_ast', 'ast', '_ctypes', 'ctypes._endian', 'ctypes', 'numpy.core._internal', 'numpy._pytesttester', 'numpy.core', 'numpy.lib.mixins', 'numpy.lib.ufunclike', 'numpy.lib.type_check', 'numpy.lib.scimath', 'numpy.lib.stride_tricks', 'numpy.lib.twodim_base', 'numpy.linalg.lapack_lite', 'numpy.linalg._umath_linalg', 'numpy.linalg.linalg', 'numpy.linalg', 'numpy.matrixlib.defmatrix', 'numpy.matrixlib', 'numpy.lib.histograms', 'numpy.lib.function_base', 'numpy.lib.index_tricks', 'numpy.lib.nanfunctions', 'numpy.lib.shape_base', 'numpy.lib.polynomial', 'numpy.lib.utils', 'numpy.lib.arraysetops', '_weakrefset', 'weakref', 'numpy.lib.format', 'numpy.lib._datasource', 'numpy.lib._iotools', 'numpy.lib.npyio', 'numpy.lib.arrayterator', 'numpy.lib.arraypad', 'numpy.lib._version', 'numpy.lib', 'numpy.fft._pocketfft_internal', 'numpy.fft._pocketfft', 'numpy.fft.helper', 'numpy.fft', 'numpy.polynomial.polyutils', 'numpy.polynomial._polybase', 'numpy.polynomial.polynomial', 'numpy.polynomial.chebyshev', 'numpy.polynomial.legendre', 'numpy.polynomial.hermite', 'numpy.polynomial.hermite_e', 'numpy.polynomial.laguerre', 'numpy.polynomial', 'numpy.random._common', 'binascii', 'base64', '_hashlib', '_blake2', '_sha3', 'hashlib', 'hmac', '_bisect', 'bisect', '_sha512', '_random', 'random', 'secrets', 'threading', 'numpy.random.bit_generator', 'numpy.random._bounded_integers', 'numpy.random._mt19937', 'numpy.random.mtrand', 'numpy.random._philox', 'numpy.random._pcg64', 'numpy.random._sfc64', 'numpy.random._generator', 'numpy.random._pickle', 'numpy.random', 'numpy.ctypeslib', '_opcode', 'opcode', 'dis', 'token', 'tokenize', 'linecache', 'inspect', 'numpy.ma.core', 'numpy.ma.extras', 'numpy.ma', 'numpy', 'copy', 'obspy.core.compatibility', 'obspy.core.util.attribdict', 'glob', 'zlib', '_compression', '_bz2', 'bz2', '_lzma', 'lzma', 'shutil', 'tempfile', 'unicodedata', '__future__', 'zipfile', 'pkgutil', 'xml', 'xml.parsers', 'pyexpat', 'xml.parsers.expat', 'plistlib', 'email', 'email.errors', '_string', 'string', 'email.quoprimime', 'email.base64mime', 'quopri', 'email.encoders', 'email.charset', 'email.header', 'email._parseaddr', 'email.utils', 'email._policybase', 'email.feedparser', 'email.parser', 'pkg_resources.extern', 'pkg_resources._vendor', 'pkg_resources._vendor.six', 'win32api', '_win32sysloader', 'pywintypes', 'pythoncom', 'win32com', 'win32com.shell', 'pkg_resources._vendor.appdirs', 'pprint', 'traceback', 'sysconfig', 'pkg_resources', 'signal', 'msvcrt', '_winapi', 'subprocess', 'obspy.core.util.misc', 'obspy.core.util.base', 'obspy.core.util.obspy_types', 'obspy.core.util.version', 'obspy.core.util', 'obspy.core.utcdatetime', 'decorator', 'tarfile', 'unittest.util', 'unittest.result', 'concurrent', 'atexit', 'logging', 'concurrent.futures._base', 'concurrent.futures', '_ssl', 'ssl', 'asyncio.constants', 'asyncio.format_helpers', 'asyncio.base_futures', 'asyncio.log', 'asyncio.coroutines', '_contextvars', 'contextvars', 'asyncio.exceptions', 'asyncio.base_tasks', '_asyncio', 'asyncio.events', 'asyncio.futures', 'asyncio.protocols', 'asyncio.transports', 'asyncio.sslproto', 'typing', 'asyncio.locks', 'asyncio.tasks', 'asyncio.staggered', 'asyncio.trsock', 'asyncio.base_events', 'asyncio.runners', 'asyncio.queues', 'asyncio.streams', 'asyncio.subprocess', '_overlapped', 'asyncio.base_subprocess', 'asyncio.proactor_events', 'asyncio.selector_events', 'asyncio.windows_utils', 'asyncio.windows_events', 'asyncio', 'difflib', 'unittest.case', 'unittest.async_case', 'unittest.suite', 'unittest.loader', 'gettext', 'argparse', 'unittest.signals', 'unittest.runner', 'unittest.main', 'unittest', 'obspy.core.util.decorator', 'obspy.core.trace', 'obspy.core.stream', 'obspy.core', 'uuid', 'obspy.core.event.resourceid', 'obspy.core.event.header', 'obspy.core.event.base', 'obspy.imaging', 'obspy.core.event.source', 'obspy.imaging.scripts', 'obspy.imaging.scripts.mopad', 'obspy.imaging.beachball', 'obspy.imaging.mopad_wrapper', 'obspy.imaging.source', 'obspy.core.event.event', 'obspy.core.event.catalog', 'obspy.core.event.magnitude', 'obspy.core.event.origin', 'obspy.core.event', 'obspy.core.inventory.util', 'obspy.core.inventory.channel', 'obspy.geodetics.base', '_csv', 'csv', 'obspy.geodetics.flinnengdahl', 'obspy.geodetics', 'obspy.core.inventory.station', 'obspy.core.inventory.network', 'obspy.core.inventory.inventory', 'obspy.core.inventory.response', 'obspy.core.inventory', 'obspy']

and it seems that the first to break the ansi colors in the windows console when running ipython -c import obspy.core.util.misc;print(tom)

The reasoning seems to be linked to this old stuff https://github.com/ipython/ipython/issues/17, i.e. messing around with stdout!

So, removing some stdout rewrites from the file: https://github.com/obspy/obspy/blob/f6426d4fcd75932872ea2a56056d9e44ccec2e74/obspy/core/util/misc.py#L337-L339

makes the output look normally coloured !!

So, question is: do we really need def _py36_windowsconsoleio_workaround() ?

[megies]

So, question is: do we really need def _py36_windowsconsoleio_workaround() ?

What is that even and no idea. Probably just getting rid of it is fine. Can somebody on Windows check what happens without it on Python 3.7 (minimum for upcoming 1.3.1), 3.8 (minimum for upcoming 1.4.0), 3.9, 3.10?

In any case we don't support 3.6 no more

[filefolder]

commenting the _py36_windows... call on line 342 in core/util/misc.py fixes the ipython bug for python 3.7.9 and python 3.10.7. unlikely I will get around to testing 3.8 or 3.9 but seems safe to interpolate.

kudos to thomas!

[megies]

@filefolder @ThomasLecocq I just noticed.. this hack was for the SuppressOutput context manager, that is apparently what crashed. Can you try something like the following on Py 3.7 and maybe 3.10 with that hack commented out?

import os
import sys 
from obspy.core.util.misc import SuppressOutput
with SuppressOutput():
    print('Hello')
    print('Hello', file=sys.stdout)
    print('Hello', file=sys.stderr)
    os.system('echo "mystdout"')
    os.system('echo "mystderr" >&2')

[filefolder]

that crashes for me in 3.10 with commenting out _py36_windowsconsoleio_workaround() and works without commenting

[megies]

that crashes for me in 3.10 with commenting out _py36_windowsconsoleio_workaround() and works without commenting

Hmm.. so looks like the hack is still needed? Hrrrrmmmmm

[megies]

@filefolder @ThomasLecocq so I saw that workaround actually is copied from pytest, and it looks to me as if its still in there: https://github.com/pytest-dev/pytest/blob/1c31a7e6598ff0290012f56c4ee6ce48a31486d9/src/_pytest/capture.py#L128

So maybe it could be an option to do this workaround hack in that SuppressOutput context manager during __enter__ and then revert it during __exit__? That manager would have to be rewritten though, it's using this other yield setup right now. Would you like to look into that? I could, but then I couldnt try it out, lacking Windows, so that'd be kinda pointless.

[filefolder]

I'm always happy to test things out but it's via a field notebook we keep in our store room so it's not something I tend to use indoors. Windows in general is a bit of a mystery to me and I admit my desire to learn more about it is nil.

[megies]

@ThomasLecocq can you (or some other Win person) maybe try out the branch in #3171 on py 3.7 (or a later Python, I guess) and see if colored output works before and after running this in IPython:

import os
import sys 
from obspy.core.util.misc import SuppressOutput
with SuppressOutput():
    print('Hello')
    print('Hello', file=sys.stdout)
    print('Hello', file=sys.stderr)
    os.system('echo "mystdout"')
    os.system('echo "mystderr" >&2')

This is my last shot at this before bumping this to a later release, doesn't really make sense for me to work on this on a Linux machine.

[ThomasLecocq]

on it

[ThomasLecocq]

sadly not OK on the default CMD of windows or the Anaconda Prompt (maybe it's an old version), but OK on cmder ... Anyway, I think your PR makes sense as the "SuppressOutput" shouldn't be loaded by default , but only when needed, no ?

[megies]

Thanks for checking @ThomasLecocq.. hmm, so if I understand correctly that hacky workaround was added because Python was actually crashing on Py 3.6 at some point without it.

The new try in #3171 seems to not make anything crash and it at least makes it so IPython looks fine until SuppressOutput is first used, which shouldn't happen in user land at all pretty much, it's only used in some of our tests as far as I can see.

So, I'd be inclined to merge #3171

[megies]

should be fixed by #3171