search

python / mypy

Optional static typing for Python
https://www.mypy-lang.org/
Other
18.36k stars 2.81k forks source link

Strange errors when modules are not in the same directory (they are in PYTHONPATH) #3027

Closed jcrmatos closed 2 years ago

jcrmatos commented 7 years ago

Hello,

My dev env is Win7P SP1, Py 3.5.2 32b, mypy 0.501.

If all the modules are in the same directory and I run mypy --strict-optional --warn-redundant-casts --disallow-untyped-defs --warn-unused-ignores --ignore-missing-imports --follow-imports skip --fast-parser everything is ok.

If I move the modules listed below to another directory and add that directory to PYTHONPATH mypy returns these strange errrors: data_processing.py:711: error: "module" has no attribute "HASHES_FN" data_processing.py:744: error: "module" has no attribute "create_hash_file"

dexp_config.py:48: error: Module 'crypto' has no attribute 'all_hashed' dexp_config.py:48: error: Module 'crypto' has no attribute 'my_hash' gui_ask_admin_pw.py:44: error: Module 'crypto' has no attribute 'my_hash' gui_change_config.py:55: error: Module 'crypto' has no attribute 'hash_list' gui_change_config.py:55: error: Module 'crypto' has no attribute 'my_hash' gui_confirm_export.py:46: error: Module 'crypto' has no attribute 'my_hash' gui_dexp.py:62: error: Module 'crypto' has no attribute 'data_file_hash_ok' receiver_main.py:57: error: Module 'crypto' has no attribute 'my_hash'

All those attributes (constant and functions) exist in their respective modules.

Even stranger are the first 2 because the module is identified in the code line, eg. data_processing.py line 711 is hashes_pn = EXPORTS_DIR + crypto.HASHES_FN # type: str

If I execute the program (which those modules above are just a part) it runs both ways (all modules in the same dir and in separate dirs) without any error.

Best regards,

JM

refi64 commented 7 years ago

Can we see your program? FWIW I think you're supposed to set MYPYPATH instead, but still doesn't sound right...

jcrmatos commented 7 years ago

Hello,

I assumed that mypy would use PYTHONPATH. I am wrong in assuming that?

Tried with MYPYPATH but gave the same results.

Yes, here is for example the start of crypto.py (the module imported by gui_ask_admin_pw.py):

#!/usr/bin/env python3
# -*- coding: utf-8 -*-

#
# Copyright (C) 2009-2017, João Carlos Roseta Matos
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
#
# * Redistributions of source code must retain the above copyright
#   notice, this list of conditions and the following disclaimer.
#
# * Redistributions in binary form must reproduce the above copyright
#   notice, this list of conditions and the following disclaimer in the
#   documentation
#   and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
# HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
#

"""Cryptography module."""

__all__ = ('HASHES_EXT', 'HASHES_FN', 'HASHES_PN', 'my_hash', 'hash_list',
           'create_hash_file', 'all_hashed', 'data_file_hash_ok',
           'check_hashes_in_file', 'lrc', 'lrc_using_xor',
           'fletcher_16_checksum')  # type: Tuple[str, ...]

from hashlib import sha256
from os import linesep, sep
from os.path import basename, dirname, normpath
from string import hexdigits
from typing import Any, List, Optional, Tuple, Union  # noqa: F401 # pylint: disable=unused-import

import base
from file_ops import load_bin, load_data, load_txt2lst, save_lst2txt
import localization as lcl

HASHES_EXT = '.digest.sha256'  # type: str
HASHES_FN = 'hashes' + HASHES_EXT  # type: str
HASHES_PN = base.APP_PATH + HASHES_FN  # type: str

_HEADER = 'SHA-256 Digest'  # type: str
_HASH_LEN = 64  # type: int
_N_HEADER_LINES = 2  # type: int

# this next item was changed to '' for security reasons
_SALT = ''  # type: str

def my_hash(data, salt=_SALT):
    # type: (Union[str, bytes], str) -> str
    """Hash text or file contents.

    :param data: data to be encrypted.
    :param salt: salt for encryption.
    :return: encrypted data.
    """
    assert ((isinstance(data, str) and data != '')
            or (isinstance(data, bytes) and data != b''))
    assert isinstance(salt, str) and salt != ''

    if isinstance(data, str):
        result = sha256(salt.encode() + data.encode()).hexdigest()  # type: str
    else:
        result = sha256(salt.encode() + data).hexdigest()
    return result

The error returned when checking gui_ask_admin_pw.py is gui_ask_admin_pw.py:44: error: Module 'crypto' has no attribute 'my_hash' if the crypto.py module is in a different directory (which is on PYTHONPATH and MYPYPATH).

If both are on the same directory there is no error running mypy.

The 44th line in gui_ask_admin_pw.py is from crypto import my_hash

If I execute the program (which those modules above are just a part) it runs both ways (all modules in the same dir and in separate dirs) without any error.

Best regards,

JM

jcrmatos commented 7 years ago

Hello,

Any idea of what the problem might be?

Best regards,

JM

gvanrossum commented 7 years ago

Maybe try asking on gitter? See the mypy README.md.

On Mar 29, 2017 3:09 AM, "Joao Matos" notifications@github.com wrote:

Hello,

Any idea of what the problem might be?

Best regards,

JM

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/python/mypy/issues/3027#issuecomment-290045228, or mute the thread https://github.com/notifications/unsubscribe-auth/ACwrMvJgTeJ2pPaL4kwiVKc5QBGKQIjmks5rqi3sgaJpZM4MhsVl .

jcrmatos commented 7 years ago

Hello,

I was asked in Gitter to run the mypy command with --verbose. I don't see anything that is helpful, but here it is:

(dexp) C:\Users\JMatos\MEOCloud\Python\dexp\dexp>mypy --strict-optional --warn-r
edundant-casts --disallow-untyped-defs --warn-unused-ignores --ignore-missing-im
ports --follow-imports silent --verbose --fast-parser gui_ask_admin_pw.py
LOG:  Mypy version 0.501
LOG:  Parsing gui_ask_admin_pw.py (gui_ask_admin_pw)
LOG:  Parsing C:\Users\JMatos\Envs\dexp\Lib\mypy\typeshed\stdlib\3\tkinter\__ini
t__.pyi (tkinter)
LOG:  Parsing C:\Users\JMatos\Envs\dexp\Lib\mypy\typeshed\stdlib\3\tkinter\ttk.p
yi (tkinter.ttk)
LOG:  Parsing C:\Users\JMatos\Envs\dexp\Lib\mypy\typeshed\stdlib\3\typing.pyi (t
yping)
LOG:  Silencing C:\Users\JMatos\MEOCloud\Python\dexp\dexp\base_dexp.py (base_dex
p)
LOG:  Parsing C:\Users\JMatos\MEOCloud\Python\dexp\dexp\base_dexp.py (base_dexp)

LOG:  Parsing C:\Users\JMatos\Envs\dexp\Lib\mypy\typeshed\third_party\2and3\cryp
to\__init__.pyi (crypto)
LOG:  Parsing C:\Users\JMatos\Envs\dexp\Lib\mypy\typeshed\stdlib\3\builtins.pyi
(builtins)
LOG:  Parsing C:\Users\JMatos\Envs\dexp\Lib\mypy\typeshed\stdlib\3\tkinter\const
ants.pyi (tkinter.constants)
LOG:  Parsing C:\Users\JMatos\Envs\dexp\Lib\mypy\typeshed\stdlib\3\sys.pyi (sys)

LOG:  Parsing C:\Users\JMatos\Envs\dexp\Lib\mypy\typeshed\stdlib\3\abc.pyi (abc)

LOG:  Parsing C:\Users\JMatos\Envs\dexp\Lib\mypy\typeshed\stdlib\3\os\__init__.p
yi (os)
LOG:  Silencing C:\Users\JMatos\MEOCloud\Python\dexp\dexp\base.py (base)
LOG:  Parsing C:\Users\JMatos\MEOCloud\Python\dexp\dexp\base.py (base)
LOG:  Parsing C:\Users\JMatos\Envs\dexp\Lib\mypy\typeshed\stdlib\3\types.pyi (ty
pes)
LOG:  Parsing C:\Users\JMatos\Envs\dexp\Lib\mypy\typeshed\third_party\2and3\mypy
_extensions.pyi (mypy_extensions)
LOG:  Parsing C:\Users\JMatos\Envs\dexp\Lib\mypy\typeshed\stdlib\3.4\pathlib.pyi
 (pathlib)
LOG:  Parsing C:\Users\JMatos\Envs\dexp\Lib\mypy\typeshed\stdlib\3\io.pyi (io)
LOG:  Parsing C:\Users\JMatos\Envs\dexp\Lib\mypy\typeshed\stdlib\3\os\path.pyi (
os.path)
LOG:  Parsing C:\Users\JMatos\Envs\dexp\Lib\mypy\typeshed\stdlib\3\imp.pyi (imp)

LOG:  Parsing C:\Users\JMatos\Envs\dexp\Lib\mypy\typeshed\stdlib\3\_importlib_mo
dulespec.pyi (_importlib_modulespec)
LOG:  Parsing C:\Users\JMatos\Envs\dexp\Lib\mypy\typeshed\stdlib\3\codecs.pyi (c
odecs)
LOG:  Loaded graph with 20 nodes
LOG:  Found 9 SCCs; largest has 12 nodes
LOG:  Processing SCC of size 12 (_importlib_modulespec pathlib mypy_extensions t
ypes abc typing codecs os.path io sys os builtins) as inherently stale
LOG:  Processing SCC singleton (imp) as inherently stale
LOG:  Processing SCC singleton (tkinter.constants) as inherently stale
LOG:  Processing SCC singleton (crypto) as inherently stale
LOG:  Processing SCC singleton (base) as inherently stale
LOG:  Processing SCC singleton (tkinter) as inherently stale
LOG:  Processing SCC singleton (base_dexp) as inherently stale
LOG:  Processing SCC singleton (tkinter.ttk) as inherently stale
LOG:  Processing SCC singleton (gui_ask_admin_pw) as inherently stale
LOG:  No fresh SCCs left in queue
LOG:  Build finished in 1.878 seconds with 20 modules, 4748 types, and 1 errors
gui_ask_admin_pw.py:44: error: Module 'crypto' has no attribute 'my_hash'

Best regards,

JM

jcrmatos commented 7 years ago

Hello,

I decided to run --verbose with the module crypto.py copied to the local dir and this is the result:

LOG:  Mypy version 0.501
LOG:  Parsing gui_ask_admin_pw.py (gui_ask_admin_pw)
LOG:  Parsing C:\Users\JMatos\Envs\dexp\Lib\mypy\typeshed\stdlib\3\tkinter\__ini
t__.pyi (tkinter)
LOG:  Parsing C:\Users\JMatos\Envs\dexp\Lib\mypy\typeshed\stdlib\3\tkinter\ttk.p
yi (tkinter.ttk)
LOG:  Parsing C:\Users\JMatos\Envs\dexp\Lib\mypy\typeshed\stdlib\3\typing.pyi (t
yping)
LOG:  Silencing C:\Users\JMatos\MEOCloud\Python\dexp\dexp\base_dexp.py (base_dex
p)
LOG:  Parsing C:\Users\JMatos\MEOCloud\Python\dexp\dexp\base_dexp.py (base_dexp)

LOG:  Silencing C:\Users\JMatos\MEOCloud\Python\dexp\dexp\crypto.py (crypto)
LOG:  Parsing C:\Users\JMatos\MEOCloud\Python\dexp\dexp\crypto.py (crypto)
LOG:  Parsing C:\Users\JMatos\Envs\dexp\Lib\mypy\typeshed\stdlib\3\builtins.pyi
(builtins)
LOG:  Parsing C:\Users\JMatos\Envs\dexp\Lib\mypy\typeshed\stdlib\3\tkinter\const
ants.pyi (tkinter.constants)
LOG:  Parsing C:\Users\JMatos\Envs\dexp\Lib\mypy\typeshed\stdlib\3\sys.pyi (sys)

LOG:  Parsing C:\Users\JMatos\Envs\dexp\Lib\mypy\typeshed\stdlib\3\abc.pyi (abc)

LOG:  Parsing C:\Users\JMatos\Envs\dexp\Lib\mypy\typeshed\stdlib\3\os\__init__.p
yi (os)
LOG:  Silencing C:\Users\JMatos\MEOCloud\Python\dexp\dexp\base.py (base)
LOG:  Parsing C:\Users\JMatos\MEOCloud\Python\dexp\dexp\base.py (base)
LOG:  Parsing C:\Users\JMatos\Envs\dexp\Lib\mypy\typeshed\stdlib\3\hashlib.pyi (
hashlib)
LOG:  Parsing C:\Users\JMatos\Envs\dexp\Lib\mypy\typeshed\stdlib\3\os\path.pyi (
os.path)
LOG:  Parsing C:\Users\JMatos\Envs\dexp\Lib\mypy\typeshed\stdlib\3\string.pyi (s
tring)
LOG:  Parsing C:\Users\JMatos\Envs\dexp\Lib\mypy\typeshed\stdlib\3\types.pyi (ty
pes)
LOG:  Parsing C:\Users\JMatos\Envs\dexp\Lib\mypy\typeshed\third_party\2and3\mypy
_extensions.pyi (mypy_extensions)
LOG:  Parsing C:\Users\JMatos\Envs\dexp\Lib\mypy\typeshed\stdlib\3.4\pathlib.pyi
 (pathlib)
LOG:  Parsing C:\Users\JMatos\Envs\dexp\Lib\mypy\typeshed\stdlib\3\io.pyi (io)
LOG:  Parsing C:\Users\JMatos\Envs\dexp\Lib\mypy\typeshed\stdlib\3\imp.pyi (imp)

LOG:  Parsing C:\Users\JMatos\Envs\dexp\Lib\mypy\typeshed\stdlib\3\_importlib_mo
dulespec.pyi (_importlib_modulespec)
LOG:  Parsing C:\Users\JMatos\Envs\dexp\Lib\mypy\typeshed\stdlib\3\codecs.pyi (c
odecs)
LOG:  Loaded graph with 22 nodes
LOG:  Found 11 SCCs; largest has 12 nodes
LOG:  Processing SCC of size 12 (_importlib_modulespec pathlib mypy_extensions t
ypes abc typing codecs io os.path sys os builtins) as inherently stale
LOG:  Processing SCC singleton (imp) as inherently stale
LOG:  Processing SCC singleton (string) as inherently stale
LOG:  Processing SCC singleton (hashlib) as inherently stale
LOG:  Processing SCC singleton (tkinter.constants) as inherently stale
LOG:  Processing SCC singleton (base) as inherently stale
LOG:  Processing SCC singleton (tkinter) as inherently stale
LOG:  Processing SCC singleton (crypto) as inherently stale
LOG:  Processing SCC singleton (base_dexp) as inherently stale
LOG:  Processing SCC singleton (tkinter.ttk) as inherently stale
LOG:  Processing SCC singleton (gui_ask_admin_pw) as inherently stale
LOG:  No fresh SCCs left in queue
LOG:  Build finished in 1.927 seconds with 22 modules, 5483 types, and 0 errors

The only difference I see is how mypy "treats" crypto.py.

If it is in a different dir it shows:

LOG:  Parsing C:\Users\JMatos\Envs\dexp\Lib\mypy\typeshed\third_party\2and3\cryp
to\__init__.pyi (crypto)

If it is in the same dir it shows:

LOG:  Silencing C:\Users\JMatos\MEOCloud\Python\dexp\dexp\crypto.py (crypto)
LOG:  Parsing C:\Users\JMatos\MEOCloud\Python\dexp\dexp\crypto.py (crypto)

Hope this helps.

Best regards,

JM

jcrmatos commented 7 years ago

Hello,

The bug in mypy was found with the help of the gitter community. The problem is that mypy is prepended to the path when resolving import names and because I had a module called crypro.py. It conflicted with crypto in typeshed. As soon as I changed the name of my module the problem went away. As instructed in gitter, I'm not closing this issue so that the prepending problem is resolved in mypy due to it's different behavior from Python.

Best regards,

JM

hauntsaninja commented 2 years ago

A lot has changed since 2017, feel free to open a new issue if you have a repro of bad behaviour with a recent mypy