search

yt-project / yt

Main yt repository
http://yt-project.org
Other
465 stars 276 forks source link

Matplotlib 3.3.0 Breaks _png #2752

Closed ax3l closed 4 years ago

ax3l commented 4 years ago

Bug report

Bug summary

Matplotlib 3.3.0 removed the internal _png module, which breaks https://github.com/yt-project/yt/blob/yt-3.6.0/yt/utilities/png_writer.py#L13

See the last mention in https://matplotlib.org/3.3.0/api/api_changes.html#matplotlib-now-uses-pillow-to-save-and-read-pngs

Code for reproduction

Just saw this on our CI: https://travis-ci.com/github/ECP-WarpX/WarpX/jobs/361956903

MPL 3.3.0 was release about 5hrs ago. https://github.com/matplotlib/matplotlib/releases/tag/v3.3.0

Actual outcome

File "/home/travis/.local/lib/python3.6/site-packages/yt/utilities/png_writer.py", line 13, in <module>
    import matplotlib._png as _png
ModuleNotFoundError: No module named 'matplotlib._png'

Expected outcome

:-)

Version Information

Installed via python -m pip install --upgrade cmake matplotlib mpi4py numpy scipy yt.

Work-Around

Downgrade matplotlib via python -m pip install --upgrade matplotlib==3.2.2.

Exact details:

welcome[bot] commented 4 years ago

Hi, and welcome to yt! Thanks for opening your first issue. We have an issue template that helps us to gather relevant information to help diagnosing and fixing the issue.

neutrinoceros commented 4 years ago

Thank you for reporting this ! If I'm not mistaken, the only function that needs to be updated is call_png_write_png since it calls matplotlib._png.write_png. breadcrumbs: here are some of the iterations matplotlib did itself during their transition to pillow

Furthermore it looks like they started using pillow as an optional dep since their 3.1.0 release (see release notes).

neutrinoceros commented 4 years ago

and this is the crucial commit of this transition: https://github.com/matplotlib/matplotlib/commit/370e9a2d5d9e637abc90b3270d368642c69f66c6

matthewturk commented 4 years ago

I thought for sure we had addressed this previously. Hmmmm. Thanks for looking at this, will test it out over here...

neutrinoceros commented 4 years ago

FWIW I tested it locally with the mpl 3.3 and it didn't crash, but changing the dpi seems to have no effect (I tried hardcoding a different value since it's not currently exposed). edit: I though you were answering my pull request instead of the issue itself. My bad.

ax3l commented 4 years ago

@neutrinoceros I have to admit that I just quickly saw this broke our CI, downgraded to mpl 3.2.2 and went on. The full call stack in the linked CI error above is:

  File "outscript.py", line X, in <module>
    import yt
  File "/home/travis/.local/lib/python3.6/site-packages/yt/__init__.py", line 63, in <module>
    from yt.fields.api import \
  File "/home/travis/.local/lib/python3.6/site-packages/yt/fields/api.py", line 19, in <module>
    from . import angular_momentum
  File "/home/travis/.local/lib/python3.6/site-packages/yt/fields/angular_momentum.py", line 26, in <module>
    from .vector_operations import \
  File "/home/travis/.local/lib/python3.6/site-packages/yt/fields/vector_operations.py", line 32, in <module>
    from yt.geometry.geometry_handler import \
  File "/home/travis/.local/lib/python3.6/site-packages/yt/geometry/geometry_handler.py", line 29, in <module>
    from yt.utilities.parallel_tools.parallel_analysis_interface import \
  File "/home/travis/.local/lib/python3.6/site-packages/yt/utilities/parallel_tools/parallel_analysis_interface.py", line 30, in <module>
    from yt.data_objects.image_array import ImageArray
  File "/home/travis/.local/lib/python3.6/site-packages/yt/data_objects/image_array.py", line 18, in <module>
    from yt.visualization.image_writer import write_bitmap, write_image
  File "/home/travis/.local/lib/python3.6/site-packages/yt/visualization/image_writer.py", line 29, in <module>
    import yt.utilities.png_writer as pw
  File "/home/travis/.local/lib/python3.6/site-packages/yt/utilities/png_writer.py", line 13, in <module>
    import matplotlib._png as _png
ModuleNotFoundError: No module named 'matplotlib._png'
neutrinoceros commented 4 years ago

@ax3l I'm sorry if my last message caused confusion, I was talking about #2754 . You are absolutely right to state that yt is currently not compatible with matplotlib 3.3. ! My aforementioned pull request should resolve the issue, so our next release will contain the fix. Currently, the next release we're aiming for is 4.0, and will contain a lot of new stuff but also some -minor- backward incompatible changes. We may consider doing a 3.6.1 bugfix release but I'm not in a position to make such a promise. Anyway thanks again for your early report !

ax3l commented 4 years ago

@neutrinoceros ah, got it now - thanks for the clarification and quick fix!

munkm commented 4 years ago

This should be fixed now on the master branch thanks to yt-project/yt#2754

ax3l commented 4 years ago

Thank you for the quick fix! Are you anticipating a potential 3.6.1 with this? Or maybe just an update pip release that markes mpl<3.3.0 in its meta-data?

2sn commented 4 years ago

I had the same issue. (Matplotlib 3.3.0)

As getting current version of yt from master is also broken 

----> 3 from yt.utilities.lib.misc_utilities import (
      4     obtain_position_vector,
      5     obtain_relative_velocity_vector,

ModuleNotFoundError: No module named 'yt.utilities.lib.misc_utilities'

I just copied the current version of of yt/utilities/png_writer.py from the current master branch and now the import of yt is no longer a roadblock...thanks. (below the file contents in case anyone else needs it)

from io import BytesIO

try:
    # matplotlib switched from an internal submodule _png to using pillow (PIL)
    # between v3.1.0 and v3.3.0
    # So PIL should be available on any system where matplotlib._png doesn't exist
    import matplotlib._png as _png
except ImportError:
    from PIL import Image

def call_png_write_png(buffer, fileobj, dpi):
    try:
        _png.write_png(buffer, fileobj, dpi)
    except NameError:
        Image.fromarray(buffer).save(fileobj, dpi=(dpi, dpi), format="png")

def write_png(buffer, filename, dpi=100):
    with open(filename, "wb") as fileobj:
        call_png_write_png(buffer, fileobj, dpi)

def write_png_to_string(buffer, dpi=100):
    fileobj = BytesIO()
    call_png_write_png(buffer, fileobj, dpi)
    png_str = fileobj.getvalue()
    fileobj.close()
    return png_str
neutrinoceros commented 4 years ago

As getting current version of yt from master is also broken

Hi @2sn , this error probably means you need to recompile yt (this is an expected side effect of how much the master branch evolved lately), so you should be able to fix this with pip install -e . from the top level of the repo !

2sn commented 4 years ago

I did install from scratch using pip3 install -U git+https://github.com/yt-project/yt.git after pip3 uninstall yt. No worries, I know it is an active development version and may have retrieved the snapshot at a bad time.

awsteiner commented 4 years ago

Hello. pip3 install -U git+https://github.com/yt-project/yt.git doesn't seem to work for me, as it just gives the error from above regarding misc_utilities. Is there a different branch with the right fix somewhere?

Type "help", "copyright", "credits" or "license" for more information.

import yt Traceback (most recent call last): File "", line 1, in File "/usr/local/lib/python3.8/site-packages/yt/init.py", line 63, in from yt.fields.api import ( File "/usr/local/lib/python3.8/site-packages/yt/fields/api.py", line 2, in from . import ( File "/usr/local/lib/python3.8/site-packages/yt/fields/angular_momentum.py", line 3, in from yt.utilities.lib.misc_utilities import ( ModuleNotFoundError: No module named 'yt.utilities.lib.misc_utilities'

awsteiner commented 4 years ago

Oh nvm, the following works...

pip uninstall yt git clone http://github.com/yt-project/yt.git cd yt pip install -e .