Panel Pydantic pane

[jbednar]

See https://github.com/holoviz/param/issues/486 for the proposal, which is basically to support Pydantic objects the same way we currently support Parameterized objects.

[MarcSkovMadsen]

Does this also include being able to panel.depends on Pydantic models?

[jbednar]

Depends (no pun intended) on whether Pydantic allows its users to watch or depend on attribute changes already, or else if it provides some hook in setattr that we could hook into to provide such watching ourselves. I can't find any such support by skimming the Pydantic docs, but I'm no Pydantic expert and could have missed it.

[MarcSkovMadsen]

A proof of concept for converting back and forth between Param and Pydantic classes can be found here https://discourse.holoviz.org/t/how-would-i-create-a-parameterized-class-programmatically/3149/2.

A user trying out Param inspired by Pydantic is here https://discourse.holoviz.org/t/creating-an-optional-datetime-parameter-using-param/3147/6.

[MarcSkovMadsen]

One way to support Pydantic would be to just "wrap" the pydantic instance into a Pydantic pane. That can be done with the above proof of concept pydantic_to_param_class.

parameterized_instance = Pydantic(object=my_pydantic_instance)

Another idea would to provide a Pyrameterized mixin that makes a Pydantic model watchable.

from pydantic import BaseModel
from param import Pyrameterized

class SomeWatchableModel(Pyrameterized, BaseModel):
    value: int = 10

This mixin can probably be implemented using the rootvalidator.

[MarcSkovMadsen]

[jbednar]

I'd be very excited to see solid Pydantic support in Panel along with easy interoperability between Pydantic and Param. I don't think there's any reason for Param to try to push out Pydantic in the applications where Pydantic already works well, but Panel adds reactive GUI programming features that would be useful for people who already have codebases using Pydantic. The mixin approach makes sense, though it would be great to somehow be able to monkeypatch what it does onto an existing class using Pydantic, so that people don't necessarily have to edit their Pydantic-based code to use it nicely with Panel.

[MarcSkovMadsen]

I believe technically I have it working. The difficult part would more be understanding when it makes sense to use Param or use Pydantic. And the right api for moving back and forth.

[jmosbacher]

I decided to stop waiting for someone to implement this and do it myself. If you arrived here like me looking for a solution please head over here and start contributing! I have made it an external package mostly because I want to use plum-dispatch and also monkey patching the synchronization from pydantic to panel depends on pydantic internals so it can break at any moment if pydantic changes their internals. If this package becomes stable for some time and gets interest i will be happy to make a PR to integrate it to panel.

@MarcSkovMadsen would especially appreciate your opinion on the approach, I tried a few different approaches including conversion to param and syncing the parameters but in the end this seems the most "natural" to me.

[MarcSkovMadsen]

I @jmosbacher . This sounds great and I will take a look.

My POC approach is to implement conversion from Pydantic model to Param model and back. From there I can start tapping into the Pydantic and Param related ecosystems. See https://github.com/MarcSkovMadsen/paithon/pull/1/files

[jbednar]

Looks great! @jmosbacher , if you're happy with how this works, it would be great if you could propose a PR on pydantic to add an explicit hook in its setattr for us to rely on, so that it will be more robust across future pydantic releases. Very promising in any case!

[jmosbacher]

@jbednar I think I can do that once things are stable and well tested. I'm supposed to be writing my thesis right now, so I cant really commit to a timeline.

Getting such a PR accepted though would probably be a challenge. My impression is that pydantic is strictly focused on data validation. Pydantic models are treated mostly as data containers that validate and coerce on creation. The whole validate_assignment (off by default, by the way) logic seems patched on and is essentially just recreating the entire class __dict__ from scratch every time you do an assignment. All validation is done on the class level, self is never passed to the validation methods, only cls.

To push them in the right direction I was thinking of maybe making a code-generator tool that automatically builds a panel-based UI for any FastAPI app, that can be a drop in replacement for the currently used swagger UI used as API docs. The compatibility with pyodide of all packages involved means you would get a superior user experience to swagger UI and still be able to serve it as a simple static page. I think this use case may convince them that integration with the panel package would be useful and then maybe they would be more susceptible to such a PR.

[rtbs-dev]

@jmosbacher @jbednar This is even more salient with the new pydantic v2 info drop we got, which puts top-lvl importance on pyodide/rust into a pydantic-core, and (as you hinted at) it will be doubling down on being a validation-only library.... more of a runtime typechecker with first-class jsonschema support.

That said, it's worth noting that pydantic will consequently move away from the BaseModel paradigm quite a bit, and allow for arbitrary object checking. So... it's possible we can stay with Param, or maybe patch-in via dataclasses, and show compatibility with pydantic v2 once that comes around? ntm the wasm (see No.2) functionality Sam is talking about fits so nicely with the Panelite idea.

[rtbs-dev]

(It occurs to me that the core shift here isn't actually Param vs. Pydantic vs. Dataclasses, etc., but rather that the newer libraries are relying nearly exclusively on typehints and Annotated types to achieve the kinds of attribute constraints you guys are doing with custom classes, and it's all being done at runtime now. See: beartype, which supports runtime O(1) typechecking of nested objects like dataclasses, typical, and GUI generation libraries like Opyrator, that are pretty opinionated toward a single stack.

Perhaps significant step here toward community interop would be to provide a way for Param to accept Annotated[] type hints (a la beartype or annotated-types, rather than/in addition to the param.<CustomType> syntax. There should be a way to have both, letting people's data modeling code get maximally reused! ^_^ )

[jmosbacher]

@tbsexton Im not sure I share the view that Param and Pydantic are competitors. Its true that at the most basic level they can both be used to validate data, but this is pretty much where the overlap ends. Just from periodically browsing through the Param code while using it i would guestimate that about 90-95% of the Param codebase is dedicated to dynamic data generation and synchronization and maybe a few percent is data validation. Where I would guesstimate Pydantic consists almost entirely of code related to data parsing/coercion, schema generation, and validation logic. Maybe im just biased because descriptors are my go-to for everything but I dont see descriptors being replaced by annotated types anytime soon, they are ugly and hard to read (at least to my eyes...). For me Pydantic is an excellent way to define interfaces with noisy sources (eg humans or other parts of my application) and Param is a great way to define implementations ie the actual logic of my application. Recently when I find myself writing user facing logic, i have a pydantic model sitting in front of the parameterized classes doing the data parsing and coercion as it arrives then once the data is parsed and validated the parameterized classes handle the actual logic of the application.

[rtbs-dev]

@jmosbacher right, I guess I didn't do a good job explaining. I totally agree.

isn't actually Param vs. Pydantic

I was trying to say I think that looking at this as "param" vs "pydantic" is a red herring. Trying to support Param+BaseModel+Dataclass+attr+pillow+.... gets too hard too fast.

Instead we could focus less on the data structure interface itself, and instead focus our interop efforts on the typehint vs. attribute divide. Param requires attributes with a list of custom types made in-house. Pydantic requires typehints with (again) a list of custom types made in-house, that from certain perspectives accomplish very similar things.

The key difference today is that the pydantic types all inherit from now-community-standard typing classes, while Param types are all subclasses of an isolated Parameter class.

After reading the param docs on "comparisons" again, I think the core assumption (that type hints were not a runtime-usable entity) has (IMO) shifted significantly for a lot of folks in the last few years. Pydantic, Typeguard, and Beartype are all well-loved examples of this. (see: FastAPI, Opyrator, the obsession of most major data science libraries with full-coverage typehints across the board: numpy, pydantic, pandas, jax(types), SpaCy, etc.) Not to mention that the introduction of LSP has changed the game for static typecheckers quite a bit, making the development workflow rely a lot on static type checkers now. I can't really comment on how much of an eyesore Annotated types are, but I personally love writing GUI logic in an Elm-lang style, with "type tetris", discriminated unions, currying, and single dispatching all over the place :sweat_smile:. And type hints have made that way more possible than in the past.

All this is to say, a compatibility layer between the parameter subclasses and annotated-types, etc. would be a likely much easier, modular, and reusable solution than continuously trying to add translators to and from all of these data schema libraries, since a huge chunk of them are now relying on type-hints+runtime checkers to function in the first place.

[jmosbacher]

@tbsexton sorry I misunderstood your point entirely :) I think you have convinced me on the point that support for annotated classes in Param would probably make it more appealing to a lot of people and I may be willing to put some time into it if you start a branch. Are you imagining something like what Pandera does? where they have the option to define a schema using pydantic models, buts its converted internally to their descriptor-based schema model? Can probably build on what @MarcSkovMadsen did, I think he was almost there. I guess you are more in favor of completely updating the parameter classes to inherit from python classes though

As a first step maybe just adding a __get_validators__ method to the Parameterized class so it can be used in pydantic models would already make mixing the two packages easier, albeit in the opposite direction of what you are proposing :)

[jmosbacher]

Just to give a concrete example. Is the end result of what you are proposing that we can do e.g.:

import param

class SomeModel(param.Parameterized):
    x: int = 1
    y: conint(le=10) = 4
    z: int

    @param.depends('x', 'y', watch=True)
    def result(self):
        self.z = self.x * self.y

If so, I would be interested in helping with this. Assuming you get the blessing from the maintainers of course...

[philippjfr]

Love seeing the discussion that has happened here since I stepped away. @jmosbacher Your pydantic-panel project looks excellent but with the recent changes of pydantic v2 in the pipeline it probably does make sense to keep iterating in your repo before migrating it to Panel or an official panel extension org. I would be very happy to start curating a list of extensions as part of the docs so that we can promote your package.

Being able to use type hints to declare parameterized objects as in your example above also sounds very interesting, but I would probably expect that to become part of param itself. Could we open an issue on that repo? Maybe let's also create yet another issue around aligning more closely with pydantic in param, e.g. by implementing __get_validators__.

[jbednar]

Yes, this discussion is very interesting, and I think there is a lot of opportunity here.

Note that Param intentionally has parameter types that are quite distinct from Python's typing; e.g. param.Number (probably the most widely used Parameter type in our code) is intentionally agnostic about whether it's an integer or float32 or float64 or a fraction or a rational or a decimal, as long as it is a scalar numeric type for which numeric operations make sense. I think Param is more true to Python's original focus on duck typing than to the machine-type annotations to which people seem to be moving nowadays. So there will definitely be some mismatch between what Param thinks of as a type and what Python does. Even so, I'm happy for Param to add Parameter classes that do align directly with Python types, to facilitate interoperability with other tools, along with other changes that can ease such interfacing.

As for the specific x: int = 1 syntax proposed above, it would be fine to accept it for compatibility purposes, but it's hard to see how Param could actually recommend using it, because (a) I don't see how to express numeric ranges allowed, e.g. to indicate the sort of numeric bounds that are crucial for Panel as well as for error checking, without a very clunky separate @validator, and (b) it doesn't specifically indicate that x is Param's responsibility to manage; are people ok with Param taking over all class attributes or at least all those that are annotated with types?

[jmosbacher]

@philippjfr Thanks! yeah, my feeling in any case is that its not yet at the standards of the Panel code base so I'm in no rush to merge. Once v2 of pydantic gets released i dont think it will take too long to adapt, their whole "shtick" is that they use standard python types so the bulk of my code that dispatches widgets by type should continue to work. I'm guessing the naming of the various constraints wont change much as they probably wont want to break people's existing model definitions. So my bet is it will be something trivial like having to change how i fetch the field definitions etc.

Really glad you guys are so receptive to @tbsexton s proposal for Param. After actually writing out an example, I think this way of defining a Parameterized class would be very appealing to a lot of people and could increase Params usage. It would also make it super easy to generate a pydantic class dynamically, which would allow for easy ingestion of data from FastAPI and others. Using Param to write the actual logic in a FastAPI app allows for much more complex and interesting apps without the mess, so making it easier to do would be great (for me at least).

@jbednar I think there is support for these special/constrained types in the world of type hinting world. e.g the numbers.Number, the pydantic conTYPE types (like the conint in my example which accepts lots of constraints including bounds) and of course as @tbsexton pointed out annotated-types. I personally would probably continue to use the descriptor-style syntax for these cases with more than one constraint, but would love to be able to mix in these simple annotations for cases where i have little or no constraints on the parameters (probably ~80-90% of the parameters i actually define). And according to my latest poll, 50% of users would want to use this style exclusively (sample size of 2, me and @tbsexton :stuck_out_tongue_winking_eye: )

For (b) I can think of two possible ways of doing it:

1. Have a class attribute called Config like pydantic does, where you can set these things and have different options. eg track all parameters/only parameters using descriptors/a list of names etc.
2. When generating the class, only auto convert attributes that are used in param.depends into params, otherwise only explicitly defined Parameter instances become params. Then at runtime if the you try to watch a non-parameter that has type annotations param can generate a Parameter for it and add it via the Parameters.add_parameter method

But I guess these are discussions we should probably have on the actual branch/issue in Param... @tbsexton so are you going to take the initiative and open a branch? I can help out but really cant lead the effort, my advisor would kill me if he finds out i took another side project while im supposed to be writing my thesis...

[MarcSkovMadsen]

Hi All

Great discussion. Learning a lot.

My main wishes are 1) better Param intellisense in VS Code and similarly other editors for productivity reasons 2) integrations between Param and Pydantic such that users of either ecosystem can tap into the other.

[jmosbacher]

@MarcSkovMadsen thats a really good point, when you use standard python type hinting you get all the cool new type-based IDE features that are popping up everywhere lately for free

[jbednar]

@jbednar I think there is support for these special/constrained types in the world of type hinting world. e.g the numbers.Number, the pydantic conTYPE types (like the conint in my example which accepts lots of constraints including bounds) and of course as @tbsexton pointed out annotated-types.

Cool! Those are all more in line with what I had in mind than Python's basic types, and it's both good to see them in use, and makes me tired to think that Param had all this so many years ago and people have had to do so much duplicative work since then! Annotated types are only Python 3.9+, so we'd have to be careful. But the rest is all good, apart from numbers.Number accepting complex values (which do support numeric operations, but are still wrong for all the Param applications I know of). The con types like confloat look like they cover most of what's needed, apart from missing the softbounds that are useful for GUI applications like Panel (where the hard bounds on a numeric parameter are often so wide to be useless for a slider). Anyway, thanks for those references; indeed, maybe one of those approaches could become the preferred syntax in the future, and make everyone happy!

[jmosbacher]

@jbednar right, I didnt mean we had to use those types specifically. I was just pointing out that from what I've seen in my very limited experience is that these things can and have been done using dedicated types. The constrained types in pydantic for example have an almost trivial implementation by just subclassing the native python type and adding some validation methods. Param could do the same thing just with the kwargs accepted by the relevant Parameter (eg soft bounds) instead of the ones pydantic uses and just add the _validate() method to them instead of the __get_validators__ pydantic uses.

And I do share the feeling like there is a huge amount of re-implementing and rediscovering things. But to be honest, the Enthought Traits and TraitsUI packages did all of this even earlier than Param (including the super-fast c-implemented type checking everyone praises pydantic for) already in python 2.5! and yet it was reinvented by traitlets+ipywidgets and Param+Panel and I am thankful for it!

[jbednar]

I think Traits and Param were contemporaries; they were both around in 2003, at least. Maybe Traits is older than that? In any case Traits was always a no-go for our projects due to the heavyweight baggage that it added on a small project, but yes, if it had been integrated into Python proper in 2004 the world would have been a much better place! Anyway, this discussion is giving me hope that there is a future where Param can happily accept syntax that makes @MarcSkovMadsen happy and works better with PyLance, VSCode, etc.. Cool!