dir() does not return the list of valid attributes for the object

[serhiy-storchaka]

BPO	40098
Nosy	@gvanrossum, @tim-one, @serhiy-storchaka, @vedgar

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = None closed_at = None created_at = labels = ['interpreter-core', 'type-bug'] title = 'dir() does not return the list of valid attributes for the object' updated_at = user = 'https://github.com/serhiy-storchaka' ``` bugs.python.org fields: ```python activity = actor = 'gvanrossum' assignee = 'none' closed = False closed_date = None closer = None components = ['Interpreter Core'] creation = creator = 'serhiy.storchaka' dependencies = [] files = [] hgrepos = [] issue_num = 40098 keywords = [] message_count = 11.0 messages = ['365227', '365271', '365272', '365411', '365416', '365417', '365419', '365420', '365423', '365426', '365428'] nosy_count = 4.0 nosy_names = ['gvanrossum', 'tim.peters', 'serhiy.storchaka', 'veky'] pr_nums = [] priority = 'normal' resolution = None stage = None status = 'open' superseder = None type = 'behavior' url = 'https://bugs.python.org/issue40098' versions = [] ```

[serhiy-storchaka]

Due to the difference in the code of __getattr and __dir for object and type dir() does not return the list of valid attributes for the object. It can return a name which is not a valid attribute and miss a name of the valid attribute.

1. It does not support metaclasses.

class M(type):
    x = 1

class A(metaclass=M):
    pass

assert hasattr(A, 'x') assert 'x' not in dir(A)

2. It does not use __mro, but uses __bases recursively.

class M(type):
    def mro(cls):
        if cls.__name__ == 'A':
            return cls, B
        return cls,

class B(metaclass=M):
    x = 1

class A(metaclass=M):
    pass

assert hasattr(A, 'x') assert 'x' not in dir(A)

3. It uses the __dict__ attribute instead of the instance dict (they can be different).

class A:
    @property
    def __dict__(self):
        return {'x': 1}

assert not hasattr(A(), 'x') assert 'x' in dir(A())

4. It uses the __class__ attribute instead of type().

class B:
    y = 2

class A:
    x = 1
    @property
    def __class__(self):
        return B

assert hasattr(A, 'x') assert not hasattr(A, 'y') assert hasattr(A(), 'x') assert not hasattr(A(), 'y') assert 'x' in dir(A) assert 'y' not in dir(A) assert 'x' not in dir(A()) assert 'y' in dir(A())

4.1. As a side effect dir() creates an instance dictionary if it was not initialized yet (for memory saving).

It is possible to make these implementations of __dir() returning exactly what the corresponding __getattr() accepts, not more and not less. The code will even be much simpler. But is it what we want?

[gvanrossum]

I think you're going too far for some of these.

1. metaclasses

This is reasonable.

2. __mro__

This is also reasonable. (I wonder if that part of the dir() implementation predates __mro__?) I'm not sure about honoring mro() in the metaclass, but it may be useful.

3. __dict__

I think this is probably a feature -- I can't think of a reason to override __dict as a property except when implementing some sort of proxy class, and then it's likely that there's a matching overrid of __getattr.

4. __class__

Again, I think this is a feature. For example, PEP-585 overrides __class__:

>>> t = list[int]
>>> type(t)
<class 'types.GenericAlias'>
>>> t.__class__
<class 'type'>

4.1 auto-creation of instance dict

This seems an accident of implementation, and if you can avoid it, that's better. (Though what use case of dir() currently suffers from memory overhead due to this?)

[gvanrossum]

FWIW it might be a good idea to look into how PEP-585 could benefit from the improvements to dir(). Currently, dir(list) and dir(list[int]) are quite different -- only the former shows list methods like append and insert.

See https://github.com/gvanrossum/cpython/tree/pep585

[fe5a23f9-4d47-49f8-9fb5-d6fbad5d9e38]

Guido wrote:

>> t = list[int]

A few years ago, I tried to explain to you that it's much more intuitive to use builtin types instead of their alter egos from typing (List etc.) for [] operator, you said it's not important. It's funny that you've implicitly acknowledged what I said through this typo. ;-)

[gvanrossum]

>>> t = list[int]

A few years ago, I tried to explain to you that it's much more intuitive to use builtin types instead of their alter egos from typing (List etc.) for [] operator, you said it's not important. It's funny that you've implicitly acknowledged what I said through this typo. ;-)

It wasn't a typo. See PEPE 585. Its time for this change *now. It wasn't *then.

[serhiy-storchaka]

5. dir(module) does not contain module type attributes (in contrary to dir() for regular object).

>>> import keyword
>>> dir(keyword)
['__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', 'iskeyword', 'kwlist']
>>> sorted(object.__dir__(keyword))
['__all__', '__builtins__', '__cached__', '__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__file__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__loader__', '__lt__', '__name__', '__ne__', '__new__', '__package__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__spec__', '__str__', '__subclasshook__', 'iskeyword', 'kwlist']

Seems dir() was initially designed for old-style objects and many changes in object model were passed its.

On one hand, it would look logical if dir() returned the list of all names that can be used as attributes. But my concerns are that it may add a "noise", names which can be legally used as attribute names, but which are rarely accessed as instance attribute instead of type attributes. This applies to 1 (metaclasses) and 5 (module type).

If left out 1 and 5 and implement all other options (and I remind that it simplifies the code), it will only break one test for unittest.mock. I'm exploring what's going on there.

[gvanrossum]

For me, one of the most annoying things about dir() is that it gives all the dunders. There are many dunders that are just always there (class, mostly __dict, __doc, __name__ etc.). I wish it would just not give dunders that are inherited from object or type. Though it's probably too late for that (people's code might break).

[fe5a23f9-4d47-49f8-9fb5-d6fbad5d9e38]

Guido:

First, wow!

Second, now it's even more glaring that list[str] has repr 'list[str]', while list doesn't have a repr 'list'. :-( Is it possible that the time for _that_ change will also come some day? :-)

Third, my problem with dir is not dunders, but dunders that really shouldn't be there. For example, __le__ when type doesn't support ordering. I understand how that happens, but it doesn't mean it isn't a bug.

[gvanrossum]

Vedran, please stay on topic for this issue.

FWIW I agree that it would be best if dir() showed only those dunders that are significant. E.g. __eq__ should only be shown if it is overridden by a subclass.

Serhiy did you see my feedback on 3 and 4?

[serhiy-storchaka]

If using __class and __dict attribuites is a feature, it does not work for __getattr(). I think it is confusing if __dir() and __getattr() are not consistent. I'll try to deal with unittes.mock and then will look how this will work with other classes that override __class or __dict. Also we need to check all classes with custom __getattr for the matter of custom __dir__.

I think that it might be better if dir() for instance excluded names which are not accessed as via instance, e.g. non-descriptor class attributes and class methods. But it may be not easy to do.

[gvanrossum]

At least for overriding __dict, I would presume there would be a matching override for __getattribute (not __getattr__).

Ditto for __class, e.g. look at GenericAlias in the PEP-585 implementation PR. (Here __class is not in the list of exceptions so it returns the attribute from list.)

[erlend-aasland]

See also:

• 66979