Traitlet API - Githubissues

SylvainCorlay commented 9 years ago

Since Matplotlib's folks are starting to write traitlet-based APIs, we probably need to think of a roadmap for the future of the library if it is to be more widely adopted by the Scipy community.

There are some improvements that we could easily make without splitting traitlets in two different repos:

1. Deprecate trait attribute declaration with TraitType types instead of TraitType instances

class Foo(HasTraits):
    bar = Int    # deprecated
    baz = Int()  # ok
    alpha = List(trait=Int)     # deprecated
    alpha = List(trait=Int())   # ok

(Implemented in #51 and #55 - merged)

2. Like in Atom, separate the metadata from the keyword arguments in TraitType's constructor.

x = Int(allow_none=True, sync=True)      # deprecated
x = Int(allow_none=True).tag(sync=True)  # ok

(Implemented in #53 - merged)

3. A replacement for the cumbersome on_trait_change in the future, with a more convenient signature and a simpler name.

We now use observe/ unobserve method instead of using a remove=True/False argument with on_trait_change.
observe takes 1 positional argument (the handler), and two keyword arguments, names and type.

The callbacks take a single change dictionary argument, containing

{
  'owner': the HasTraits instance,
  'old': the old trait attribute value,
  'new': the new trait attribute value,
  'name': the name of the changing attribute,
}

A new @observe decorator to register methods as trait change handlers.

(Implemented in #61 - merged)

4. Deprecate the custom cross-validation magic methods _*bar*_validate to the benefit of a @validate('bar') decorator.

(Implemented in #73 - merged)

5. Since the base trait type now inherits from BaseDescriptor and other descriptors are defined to work well with HasTraits, we could make the following changes:

rename MetaHasTraits into MetaHasDescriptors and deprecate the old name.
introduce a base class to HasTraits called HasDescriptors.

(Implemented in #70 - merged)

6. Deprecate the default-value initializer magic methods _*bar*_default to the benefit of a @default('bar') decorator.

(Implemented in #114 - merged)

7. What is the best place for a repository of extra trait types for common types in the scipy stack such as ` numpy arrays, pandas/xray data structures, and their (binary) serialization functions for widgets (or other clients of comms) and ipyparallel?

It would make sense to propose a reference implementation of those, otherwise we will see multiple competing implementation emerge in different projects.

Besides, it is unlikely that such things would be accepted in core Pandas and numpy as of now...

(Scipy Trait Types Incubator Proposal)

8. Would it make sense to have a once version of on_trait_change (now observe)?

(There seems to be mixed opinions on this. Deferred.)

9. A common pattern when observing an object is the following:

foo.observe(do_something) # register to future change notifications.
do_something()            # act on the current value right away.

maybe we could facilitate this by adding a boolean argument to observe, stating whether to also run the callback right-away or not.

This would especially be useful when registering observer with the decorator syntax.

10. One thing that we had in mind in the long-run for widgets is having finer-grained events for containers, such as List and Dict. The ability to create other descriptors than trait types, possibly outside of the traitlets repository could enable experiments in this directions, like an integration of @jdfreder 's eventful dicts and lists.

One idea that could be in the scope of traitlets though is to add an attribute to the change dictionary indicating the type of notification that is being sent.

{
    'owner': the HasTraits instance,
    'old': the old trait attribute value,
    'new': the new trait attribute value,
    'name': the name of the changing attribute,
    'type': the type of notification being sent,
}

The last attribute could be used to define notification types corresponding to operational transforms.

Then, the @observe decorator would then have a 'type' keyword argument, (defaulting to None), to filter by notification type.

class Foo(HasTraits):
    bar = Int()
    baz = EventfulList()

    @observe('bar')  # Filter to only observe trait changes
    def handler_bar(self, change):
        pass

    @observe('baz ', type='element_change')  # Register to element change notifications for `baz`
    def handler_baz(self, change):
        pass

    @observe('bar', 'baz', type=All)  # register to all notification types for `bar` and `baz` 
    def handler_all(self, change):
        pass

The only thing to do to enable this possibility would be to add a type item in the dictionary and have the current implementation of observe filter on the notification type.

(Implemented in #83 - merged)

minrk commented 9 years ago

@SylvainCorlay great! thanks for getting this started. Hopefully as matplotlib starts investigating traitlets we can figure out more things that would make it better.

ellisonbg commented 9 years ago

Thanks for posting this. I am very much in favor of having traitlets evolve in these ways.

On Sun, Jul 12, 2015 at 1:24 PM, Min RK notifications@github.com wrote:

@SylvainCorlay https://github.com/SylvainCorlay great! thanks for getting this started. Hopefully as matplotlib starts investigating traitlets we can figure out more things that would make it better.

— Reply to this email directly or view it on GitHub https://github.com/ipython/traitlets/issues/48#issuecomment-120759410.

Brian E. Granger Cal Poly State University, San Luis Obispo @ellisonbg on Twitter and GitHub bgranger@calpoly.edu and ellisonbg@gmail.com

ssanderson commented 9 years ago

On the subject of making traitlets more explicit and less magical, how would people feel about replacing the magic default methods with decorators? @llllllllll suggested the following to me today:

You current write a dynamic initializer as follows:

class Foo(HasTraits):

    dict_attr = Dict(...)

    def _dict_attr_default(self):
        return {}

This relies on the user knowing that HasTraits will magically look for methods with certain names. You could make this more explicit, and less prone to breakage from name changes by making changing the API to be

class Foo(HasTraits):
    dict_attr = Dict(...)

    @dict_attr.initializer
    def dict_init(self):
        return {}

This would work internally by having dict_attr.initializer be a method that registers the decorated method as an initializer. You could have a very similar API for notifiers.

llllllllll commented 9 years ago

In the example you posted, it would would probably be best if dict_init was called dict_attr and then dict_attr.initializer was a decorator that created a new traitlet. Prior art for this style would be property

ellisonbg commented 9 years ago

I like this idea - it would be interesting to explore other variations of this

@notify('dict_attr')
def do_this(self, old, new):
    ....

On Thu, Jul 16, 2015 at 7:25 PM, Scott Sanderson notifications@github.com wrote:

On the subject of making traitlets more explicit and less magical, how would people feel about replacing the magic default methods with decorators? @llllllllll https://github.com/llllllllll suggested the following to me today:

You current write a dynamic initializer as follows:

class Foo(HasTraits):
dict_attr = Dict(...)

def _dict_attr_default(self):
    return {}
This relies on the user knowing that HasTraits will magically look for methods with certain names. You could make this more explicit, and less prone to breakage from name changes by making changing the API to be

class Foo(HasTraits): dict_attr = Dict(...)
@dict_attr.initializer
def dict_init(self):
    return {}
This would work internally by having dict_attr.initializer be a method that registers the decorated method as an initializer. You could have a very similar API for notifiers.

— Reply to this email directly or view it on GitHub https://github.com/ipython/traitlets/issues/48#issuecomment-122151552.

Brian E. Granger Cal Poly State University, San Luis Obispo @ellisonbg on Twitter and GitHub bgranger@calpoly.edu and ellisonbg@gmail.com

tacaswell commented 9 years ago

:+1: for single shot notifications.

minrk commented 9 years ago

@ssanderson with the decorator, what would changing the initializer in a subclass look like? Right now, it's just overriding a method.

llllllllll commented 9 years ago

@minrk you could just decorate the new default, again, like with the builtin property

class C:
    a = Dict(..)

class D(C):
    @C.a.initializer
    def a(self):
        ...

jasongrout commented 9 years ago

I really like the idea of a decorator for notating initializers, change notifications, etc., if we can get it to work nicely.

minrk commented 9 years ago

@llllllllll thanks.

sccolbert commented 9 years ago

Seems like there's lots of desire for single shot notifications. Would anyone mind sharing their use case for that? I've yet to come across a case where I needed that.

blink1073 commented 9 years ago

Speaking of reducing magic, what if there were a python-only fallback mode for Atom? We have settled on an explicit signal mechanism where a class must expose a set of signals that can be observed, closer to the PyQt signal-slot mechanism. We could target a 1.0 release that supports Python 3 and has a python-only fallback mode if no suitable compiler is available. This would take the maintenance burden off of the Jupyter team. https://github.com/nucleic/atom

jasongrout commented 9 years ago

Does single-shot notifications mean the handler is only called once? Just checking...

sccolbert commented 9 years ago

@jasongrout yes

jasongrout commented 9 years ago

We've certainly been looking at how atom does things while thinking of how to make traits better. How hard would it be to make a python version of atom?

sccolbert commented 9 years ago

With the ideas Steven and I have been talking about for 1.0, it would be much easier than the current code. I'd like to limit the C code to optimized storage of attributes and a fast signal dispatching mechanism. Type checking would be done in Python, but with an optimized mode which bypasses type-checking for a production runs. This is a best-of-both-worlds approach I think, which makes it easy to write the validation code, but keeps production runs fast. Then, implementing a pure-python version of the C code should be simple.

Carreau commented 9 years ago

I'm in conf today, I'll try to catch up tomorrow.

sccolbert commented 9 years ago

I'm definitely leaning to a much more explicit approach with 1.0. I've seen an enormous amount of abuse of observers for control flow over the last few years, and I'd like to reduce that.

dwillmer commented 9 years ago

+1 for @blink1073,

Atom needs to be considered seriously for this, as it's already got a lot of the behaviour described above.

@ellisonbg : in atom you already have

@observe( 'my_property' )
def _some_method():
    ...

and you can have an arbitrary number of attribute names in the decorator.

For the initialisers, the majority use case is just setting simple defaults without having the objects created/assigned class-side, in which case a factory is the least verbose:

class Test( Atom ):
    my_attr = Int( factory= lambda: 5 )

which has the added advantage of allowing you to have a library of default functions, so you don't need extra methods on each class.

ellisonbg commented 9 years ago

Let's keep this thread focused on API changes to traitlets. If we want to consider adopting atom for everything, please bring that up on the jupyter list., as it is a very broad change that would affects lots of our own and third party projects. As an aside, I don't think we could adopt atom until it had a pure python mode...

On Fri, Jul 17, 2015 at 9:45 AM, Dave Willmer notifications@github.com wrote:

+1 for @blink1073 https://github.com/blink1073,

Atom needs to be considered seriously for this, as it's already got a lot of the behaviour described above.

@ellisonbg https://github.com/ellisonbg : in atom you already have

"@observe( 'my_property' ) def _some_method(): ...

and you can have an arbitrary number of attribute names in the decorator.

For the initialisers, the majority use case is just setting simple defaults without having the objects created/assigned class-side, in which case a factory is the least verbose:

class Test( Atom ): my_attr = Int( factory= lambda: 5 )

which has the added advantage of allowing you to have a library of default functions, so you don't need extra methods on each class.

— Reply to this email directly or view it on GitHub https://github.com/ipython/traitlets/issues/48#issuecomment-122336961.

Brian E. Granger Cal Poly State University, San Luis Obispo @ellisonbg on Twitter and GitHub bgranger@calpoly.edu and ellisonbg@gmail.com

SylvainCorlay commented 9 years ago

@ellisonbg I agree that we should only keep this on the subject of traitlets. The mention of Atom I made in the original post was merely meant to point to other implementations of this type of things, which may give some inspiration. Making APIs a bit more similar API is only a plus in that people who are familiar with one are not lost with the other, but I did not mean actual compatibility or replacement.
Regarding the reservations of @takluyver w.r.t. API changes, the two first items that I proposed in the original post are backward compatible and not very disruptive:
- deprecating the use of bare trait type types:
```
class Foo(HasTraits):
  bar = Int    # deprecated
  baz = Int()  # ok
```
It simplifies a lot the metaclass and will remove a lot of idiosyncrasies of the current implementation, especially for container trait types. Hence, I guess this is probably not the part you would disagree on.
- separating the metadata from keyword argument through a deprecation of the old behavior is also a rather minor change, which will clarify the API IMO.
- Default initializers are mostly useful when the construction of the default value is expensive and one does not want to pass it to the TraitType constructor. I am fine with the current mechanism of magic methods, but a decorator is nice as well.
Overall, these three items are not more disruptive than the other things that happened in traitlets lately such as not allowing None by default, changing the mechanism for default value initializaiton etc... I think that traitlets are already a bit cleaner now than a year ago.
- The more disruptive changes regarding having a nicer replacement for on_trait_change could clearly be more disruptive, and we should probably all agree before we move forward with the actual implementation.

SylvainCorlay commented 9 years ago

@ssunkara1

tacaswell commented 9 years ago

@sccolbert Responding very late, the use case I have is very specific.

I have been working with hardware (motors, detectors) controlled over the network (via EPICS). The command to move something is to basically do a put to a set point value and then the motor does it's thing at what ever rate it feels like. Some of the devices have hooks that will be triggered when when motion is done, which is where I wish there was a single shot call back (as I only want the call back to run this time the motor stops). Typically these call backs are hooked up to threading/asyncio Events so we can block some code flow until motion is complete.

This can of course be implemented by callbacks that remove them selves.

I also like the pattern of call backs that re-install them selves to be persistent, but am not sure of all of the consequences of that design.

SylvainCorlay commented 9 years ago

@tacaswell I am not quite sure if the use case that you are describing is really about observing state.

One thing that you can do is define other descriptors than the base TraitType, which inherit from BaseDescripor, in a non-intrusive way w.r.t traitlets. (For example, in https://github.com/ipython/ipywidgets/pull/46, we use other descriptors to implement qt-style signaling in IPython widgets.)

For your usecase, you could have HasTraits descriptors holding promises, that you could observe at the object level.

SylvainCorlay commented 9 years ago

I updated the description with the latest changes to the traitlets API and the other proposed changes by @rmorshea.

rmorshea commented 9 years ago

After meeting with some of the matplotlib devs about the current state of my refactor involving traitlets (matplotlib/matplotlib#4762), it's come to our attention that the stages of validation/notification and the order in which they occur aren't particularly clear or accessible. Thankfully these things are likely to be solved with a relative degree of ease now that #61 has come about, however @ellisonbg and I were hoping that we might be able to spark a conversation about how we might make these aspects of traitlets more coherent.

In working with matplotlib a couple of issues have arisen, the most controversial probably being notification "muting" as apposed to "holding" as proposed in #60, and perhaps a way of triggering them in a modular way (not in 60). With respect to that question, it might be relevant to compartmentalize validation and notification into more distinct stages so that someone "muting" notifications cannot accidentally obstruct cross-validation.

Within the topic of notification, since matplotlib has been transitioning from getters and setters into traitlets, there have been some questions about how the notifiers should be organized; in other words, whether we should be using notifiers that trigger on-set, on-change, on-get, or any combination thereof.

ellisonbg commented 9 years ago

Part of what i would like to see is a very clear conceptual model of setting, change, validation and getting that has well defined hooks for custom logic at each step, as well as fine grained muting.

On Fri, Aug 28, 2015 at 2:31 PM, Ryan Morshead notifications@github.com wrote:

After meeting with some of the matplotlib devs about the current state of my refactor involving traitlets (matplotlib/matplotlib#4762 https://github.com/matplotlib/matplotlib/pull/4762), it's come to our attention that the stages of validation/notification and the order in which they occur aren't particularly clear or accessible. Thankfully these things are likely to be solved with a relative degree of ease with the advent of

61 https://github.com/ipython/traitlets/pull/61, however @ellisonbg

https://github.com/ellisonbg and I were hoping that we might be able to spark a conversation about how we might make these aspects of traitlets more coherent.

In working with matplotlib a couple of issues have arisen, the most controversial probably being notification "muting" as apposed to "holding" as introduced in #60 https://github.com/ipython/traitlets/pull/60, and perhaps a way of triggering them in a modular way (not in 60). With respect to that question, it might be relevant to compartmentalize validation and notification into more distinct stages so that someone "muting" notifications cannot accidentally obstruct cross-validation.

Within the topic of notification, since matplotlib has been transitioning from getters and setters into traitlets, there have been some questions about how the notifiers should be organized; in other words, whether we should be using notifiers that trigger on-set, on-change, on-get, or any combination thereof.

— Reply to this email directly or view it on GitHub https://github.com/ipython/traitlets/issues/48#issuecomment-135893302.

Brian E. Granger Associate Professor of Physics and Data Science Cal Poly State University, San Luis Obispo @ellisonbg on Twitter and GitHub bgranger@calpoly.edu and ellisonbg@gmail.com

SylvainCorlay commented 9 years ago

How about we make the new trait types repo part of the jupyter incubator?

A least it would be a start for a centralized repo for pandas.Series, pandas.DataFrame and numpy.ndarray trait types.

I think that it is very unlikely that such trait types would be accepted in Pandas and Numpy, and the risk of not proposing an implementation is that we will probably see competing and incompatible implementations emerge in different projects relying on traitlets.

ellisonbg commented 9 years ago

Perfect usage of the incubator...

On Fri, Sep 4, 2015 at 4:22 PM, Sylvain Corlay notifications@github.com wrote:

How about we make the new trait types repo part of the jupyter incubator?

A least it would be a start for a centralized repo for pandas.Series, pandas.DataFrame and numpy.Array trait types.

I think that it is very unlikely that such trait types would be accepted in Pandas and Numpy, and the risk of not proposing an implementation is that we will probably see competing and incompatible implementations emerge in different projects relying on traitlets.

— Reply to this email directly or view it on GitHub https://github.com/ipython/traitlets/issues/48#issuecomment-137877128.

Brian E. Granger Associate Professor of Physics and Data Science Cal Poly State University, San Luis Obispo @ellisonbg on Twitter and GitHub bgranger@calpoly.edu and ellisonbg@gmail.com

NeilGirdhar commented 9 years ago

@SylvainCorlay Where can I find these numpy.ndarray trait types? (Various searches turned up side projects.)

SylvainCorlay commented 9 years ago

@NeilGirdhar I think that multiple people have done ad-hoc implementations. The proposal here is to create a new project with reference implementations of those trait types. I will make this issue point to it when we open the project in https://github.com/jupyter-incubator.

SylvainCorlay commented 9 years ago

I updated the issue and added another proposition.

Another point that @rmorshea brought up in the last discussion was whether we should change the observe handler signature from a single change dictionary to the **change, (and same with **proposal in the custo cross-validation).

rmorshea commented 9 years ago

It was mentioned that **change might be a problem if new content were ever added to the change dict. However I think that can be mitigated by tacking **kwargs on the end of change handler definitions to catch unused arguments. That way you can specify as many or as few arguments as might be needed without creating problems for future updates:

def change_handler(name, old, new, owner, **kwargs):
    # do something with name, old, new, and owner

```python
def change_handler(new, **kwargs):
    # do something just with new

ellisonbg commented 9 years ago

The current API doesn't use the **change syntax right? I think I prefer that...

On Tue, Sep 15, 2015 at 2:41 PM, Ryan Morshead notifications@github.com wrote:

It was mentioned that _change might be a problem if new content were ever added to the change dict. However I think that can be mitigated by tacking _kwargs on the end of change handler definitions to catch unused arguments. That way you can specify as many or as few arguments as might be needed without creating problems for future updates:

def change_handler(name, old, new, owner, **kwargs):

do something with name, old, new, and owner
    # do something just with new

—
Reply to this email directly or view it on GitHub
https://github.com/ipython/traitlets/issues/48#issuecomment-140555024.

Brian E. Granger Associate Professor of Physics and Data Science Cal Poly State University, San Luis Obispo @ellisonbg on Twitter and GitHub bgranger@calpoly.edu and ellisonbg@gmail.com

rmorshea commented 9 years ago

Yes, at the moment it doesn't use **change. Which one are you referring to by "that" though? The reason I brought it up is because it felt cumbersome to use change['name'] and change['new'] etc. in the case where the callback logic uses most of the provided information and isn't trivial.

SylvainCorlay commented 9 years ago

@ellisonbg the things I mentioned on gitter are in the issue description at the top.

jasongrout commented 9 years ago

for the type stuff, we should probably default in the observe decorator to 'all' rather than 'trait_change'.

jdfreder commented 9 years ago

I just talked to @SylvainCorlay about the last point he proposed. I'm +1 for this proposal. Observe is like on trait change - which fires when the state of a value changes. It just so happens that with things like evenful list and dict, the state is more complex and hence the change event can be more descriptive. The should share observe instead of implementing custom decorators because the underlying concept is the same.

I should be clear though, I would refrain from using observe of any purpose other than listening to the state change of traits. IOW, I still think it's appropriate to have other decorators, like validate.

SylvainCorlay commented 9 years ago

@jdfreder agreed.

rmorshea commented 9 years ago

Would it be more descriptive to use event rather than type?

SylvainCorlay commented 9 years ago

@rmorshea I agree. Changed the name from type to event in the description.

sccolbert commented 9 years ago

-1 on passing args via **kw - it uses a more expensive form of arg passing under the hood.

sccolbert commented 9 years ago

I also prefer type over event. This pattern is used lots of places: JS, Qt, etc... I think of the dict as simplified representation of an event object.

SylvainCorlay commented 9 years ago

type is used in Atom, so I guess it is a plus for type...

rmorshea commented 9 years ago

Suggested event just because it seems intuitive to say that one would "observe an event". When I see type I think, "the type of what?" If you wanted to be really explicit you could say event_type I guess?

sccolbert commented 9 years ago

def on_foo_changed(change):
    if change['type'] == 'create':
        pass
    else if change['type'] == 'update':
        pass
    else if change['type'] == 'delete':
        pass
    else if change['type'] == 'event':
        pass
    else:
        pass

contrasted against:

  handleEvent(event) {
    switch (event.type) {
    case 'mouseenter':
      this._evtMouseEnter(event);
      break;
    case 'mouseleave':
      this._evtMouseLeave(event);
      break;
    case 'mousedown':
      this._evtMouseDown(event);
      break;
    case 'mouseup':
      this._evtMouseUp(event);
      break;
    case 'contextmenu':
      this._evtContextMenu(event);
      break;
    case 'keydown':
      this._evtKeyDown(event;
      break;
    case 'keypress':
      this._evtKeyPress(event);
      break;
    }
  }

bool QWidget::event(QEvent *event)
{
    switch (event->type()) {
    case QEvent::MouseMove:
        mouseMoveEvent((QMouseEvent*)event);
        break;

    case QEvent::MouseButtonPress:
        mousePressEvent((QMouseEvent*)event);
        break;

    case QEvent::MouseButtonRelease:
        mouseReleaseEvent((QMouseEvent*)event);
        break;

    case QEvent::MouseButtonDblClick:
        mouseDoubleClickEvent((QMouseEvent*)event);
        break;
}

rmorshea commented 9 years ago

IMO event is more intuitive, but that's just me. If people are likely to use or have seen type in those contexts it makes more sense to do it that way.

minrk commented 9 years ago

+1 on type to keep the ball rolling.

NeilGirdhar commented 9 years ago

There were some comments here about avoiding passing **kwargs for performance reasons. I don't think that's an important concern for Python code. However, keyword arguments are the only way for cooperative multiple inheritance to pass arguments, and so it's a good habit to get into in my opinion.

sccolbert commented 9 years ago

I don't think that's an important concern for Python code.

It's an important perspective to keep when building core libraries. You yourself may not care about dispatch performance, but others may. And sure, this isolated example may not be big in the grand scheme, but perpetuation of that idea can quickly lead to a library which doesn't scale.

However, keyword arguments are the only way for cooperative multiple inheritance to pass arguments

I don't quite follow you here.

ipython / traitlets

Traitlet API #48

61 https://github.com/ipython/traitlets/pull/61, however @ellisonbg

do something with name, old, new, and owner