django-polymorphic for building a custom model - is polymorphic the right tool?

[stfl]

I am writing this ticket to get some some feedback about my implementation and whether django-polymorphic is actually the right way to go. I get some massive performance issue in our prototype and when looking at the performance remarks in the docs I feel like my approach might be doomed and I should consider alternatives soon than later.

I wrote an app where (potentially) users can customize the fields of a model where all fields are stored in a polymorphic model. The aim of the project is to provide users with a model/form builder and not having to specify the fields in the code. Users can create a form structure with all the fields in the form and other users may create Projects and add their data according to the structure. I like the polymorphic approach as I can store the data in native format and profit from out of the box widgets and validations.

A simplified version:

class Project(models.Model):
    # some global fields here
    name = models.CharField()

class ProjectField(PolymorphicModel):
    project = models.ForeignKey(Project)
    slug = ...   # identify the field when searching

class IntegerProjectField(ProjectField):
    value = models.IntegerField()

class TextProjectField(ProjectField):
    value = models.TextField()
    label = models.CharField(...)

...

In reality there is a model for ProjectFields and another model for Structure which the user can configure and upon creation of a Project object, all ProjectField instances are created according to the slugs in Structure. But for simplicity I guess the above example should work.

The approach works so far and looks nice on an ER diagram :P

I get into performance and ergonomic issues when filtering for Projects with a particular value. I end up filtering for the concrete type like IntegerProejctField.objects.filter(slug="xyz", value=123) and retrieve the project from there. Chaining multiple filters together does not work anymore.

I can do the following to keep the view on the Project. (or probably use Q objects) This is not very convinient, kinda works but turns out to be slow.

Project.objects.filter(
    projectfield__integerprojectfield__value=123,
    projectfield__integerprojectfield__slug="xyz"
).filter(
    projectfield__textprojectfield__value="some text",
    projectfield__textprojectfield__slug="abc"
) 

The more pressing issue though is when using FormSets. Users shall be able to edit data in the Project so I need an inline FormSet. It works as expected but the performance is really bad. (Note that I also use additional models that need to be joined for each ProjectField so that also adds to the issue)

So to get to the point, I am not sure if django-polymorphic is the right solution for the job. I am not fully convinced anymore like I was at the beginning.

If django-polymorphic will be a dead end for me, are there any suggestions or libraries for building such a form builder solution?

[johncronan]

I wrote an app where (potentially) users can customize the fields of a model where all fields are stored in a polymorphic model. The aim of the project is to provide users with a model/form builder and not having to specify the fields in the code. Users can create a form structure with all the fields in the form and other users may create Projects and add their data according to the structure.

Are you still working on this? You're in for quite a ride, let me tell you! Take a look at this:

https://github.com/johncronan/formative

It's over 10k lines of code now, which is a lot more than I expected! I hope it can be helpful to you.

I like the polymorphic approach as I can store the data in native format and profit from out of the box widgets and validations.

This part worked out pretty well. I created a dynamic metamodel, which means that all the form stuff can plug into Django logic for fields, etc., using a modelform factory. And, yes, formsets are in there too - you'd want to look at the code relating to CollectionBlock.

As far as django-polymorphic, that's where I accrued some technical debt. Deserialization doesn't work. #517 has all the gory details, and there's been no response. Part of me wants to drop this library, but I think it would be disruptive so I'll probably fix the bug myself or try to hack around it.

If you're interested in testing out Formative, please get in touch! I'm at the point where nearly all my planned features are done, and beta testers would be good.