Problem integrating django-wm with Wagtail

[garrettc]

I can’t seem to get incoming web mentions working correctly on a Wagtail install, and I’m hoping you can offer some pointers.

I get the the Unable to find matching page on our server for url error when I try to process incoming mentions.

Background Info

I am using Wagtail as my CMS. I have a PostPage model for blog posts. It extends Wagtail’s Page model, and includes the mixins from wagtail-seo and django-wm.

The mentions app and middleware are enabled in my settings, and mentions.urls is in my urls.py.

I have the following in my settings:

DOMAIN_NAME = "polytechnic.co.uk"
WEBMENTIONS_USE_CELERY = False
WEBMENTIONS_AUTO_APPROVE = False
WEBMENTIONS_INCOMING_TARGET_MODEL_REQUIRED = False
WEBMENTIONS_ALLOW_SELF_MENTIONS = False

The PostPage model implements the all_text and get_absolute_url methods:

class PostPage(SeoMixin, Page, MentionableMixin):
    # field definitions

    def all_text(self):
            return f"{self.overview} {self.body}"

        def get_absolute_url(self):
            # Avoid circular imports
            from blog.templatetags.blogapp_tags import post_page_date_slug_url

            return post_page_date_slug_url(self, self.blog_page)

get_absolute_url() seems to be doing the right thing:

>>> pp = PostPage.objects.get(title="dConstruct 2022")
>>> pp.title
'dConstruct 2022'
>>> pp.get_absolute_url()
'/blog/2022/10/dconstruct-2022/'

Is there something else I’m missing?

[beatonma]

I'm not familiar with Wagtail but this seems like a good excuse to finally get acquainted!

It looks like it uses some custom urlpatterns logic so I suspect that has something to do with it. I'll dig around and let you know.

[garrettc]

Thank you.

If it helps your investigation, my blog landing page uses the RoutablePageMixin mixin and @route decorator. So I can capture patterns like /<year>/ and /<year>/<month>/ etc.

The specific code for a post in my case looks like:

class BlogPage(RoutablePageMixin, SeoMixin, Page):
    # field definitions and other bits    

    @route(r"^(\d{4})/(\d{2})/(.+)/$")
    def post_by_date_slug(self, request, year, month, slug, *args, **kwargs):
        # kwargs = {"model_name": "blog.PostPage"}  < this didn't work
        post_page = self.get_posts().filter(slug=slug).first()
        if not post_page:
            raise Http404
        # We're rendering another page, so call the serve method of that page instead
        return post_page.serve(request)

[beatonma]

Just an update: Need to test a few config variations and stuff but it's basically working.

I will include it with the upcoming 4.0 release as I refactored a lot of the URL resolution stuff for that already. I'll let you know when it's out - should be in the next week or so.

[beatonma]

^{Oops, looks like this week was a week longer than usual...} 4.0.0 is now available and works with Wagtail! There are some other changes too so please check the changelog and run makemigrations and migrate.

Making it work with the RoutablePageMixin decorators turned out to be a little more complicated than I realised. I ended up making new decorators which wrap the Wagtail ones and accept extra arguments to help resolve the correct page instance. I had already added new path/re_path helpers for vanilla Django urlpatterns which work a similar way so it ended up fitting together quite well in the end.

Anyway, your models should look something like this after updating:

from wagtail.templatetags.wagtailcore_tags import richtext

class PostPage(SeoMixin, Page, MentionableMixin):  
    # field definitions

    def get_content_html(self):
        # Method renamed from `all_text`
        return f"{richtext(self.overview)} {richtext(self.body)}"

    def get_absolute_url(self):
        # `return self.url` should work too, using the url generated by wagtail.
        # Either is fine
        return post_page_date_slug_url(self, self.blog_page)

    def should_process_webmentions(self) -> bool:  
        # Return True if this instance should process webmentions when saved. 
        return self.live

from mentions.helpers.thirdparty.wagtail import mentions_wagtail_re_path

class BlogPage(RoutablePageMixin, SeoMixin, Page):  
    # field definitions and other bits

    @mentions_wagtail_re_path(
        r"^(\d{4})/(\d{2})/(.+)/$",
        model_class=PostPage,
        model_filters=("date__year", "date__month", "slug")
    )
    def post_by_date_slug(self, request, year, month, slug):
        post_page = self.get_posts().filter(slug=slug).first()
        if not post_page:
            raise Http404
        # We're rendering another page, so call the serve method of that page instead
        return post_page.serve(request)

There are a few examples on the new wiki and more in the models used in the tests.

Hope that makes sense - let me know if you have any issues!

[garrettc]

I'm getting an error when trying to run makemigrations. I don't think it's me, but I've definitely said that before:

django-wm==4.0.0 wagtail==3.0.3

class BlogPage(RoutablePageMixin, SeoMixin, Page):
    # field definitions and other bits

    @mentions_wagtail_re_path(
        r"^(\d{4})/(\d{2})/(.+)/$",
        model_class="PostPage",
        model_filters=("date__year", "date__month", "slug"),
    )
    def post_by_date_slug(self, request, year, month, slug):
        post_page = self.get_posts().filter(slug=slug).first()
        if not post_page:
            raise Http404
        # We're rendering another page, so call the serve method of that page instead
        return post_page.serve(request) 

This is what I'm seeing:

(env) $ python manage.py makemigrations --dry-run
Traceback (most recent call last):
  File "/Users/garrettc/sandbox/polytechnic/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/Users/garrettc/sandbox/polytechnic/env/lib/python3.9/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/Users/garrettc/sandbox/polytechnic/env/lib/python3.9/site-packages/django/core/management/__init__.py", line 395, in execute
    django.setup()
  File "/Users/garrettc/sandbox/polytechnic/env/lib/python3.9/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/Users/garrettc/sandbox/polytechnic/env/lib/python3.9/site-packages/django/apps/registry.py", line 114, in populate
    app_config.import_models()
  File "/Users/garrettc/sandbox/polytechnic/env/lib/python3.9/site-packages/django/apps/config.py", line 301, in import_models
    self.models_module = import_module(models_module_name)
  File "/opt/homebrew/Cellar/python@3.9/3.9.15/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/Users/garrettc/sandbox/polytechnic/blog/models.py", line 28, in <module>
    class BlogPage(RoutablePageMixin, SeoMixin, Page):
  File "/Users/garrettc/sandbox/polytechnic/blog/models.py", line 80, in BlogPage
    def post_by_date_slug(self, request, year, month, slug):
  File "/Users/garrettc/sandbox/polytechnic/env/lib/python3.9/site-packages/mentions/helpers/thirdparty/wagtail.py", line 100, in decorator
    wagtail_path = wagtail_path_func(pattern, name=name)
  File "/Users/garrettc/sandbox/polytechnic/env/lib/python3.9/site-packages/mentions/helpers/thirdparty/wagtail.py", line 33, in <lambda>
    wagtail_re_path = lambda *args, **kwargs: config_error(
TypeError: config_error() got multiple values for argument 'name'

Have I triggered a weird edge-case?

[beatonma]

I will check it out properly tomorrow but at first glance I think it might be because you're passing the model class as a string (model_class="PostPage") which I agree should work but I overlooked that possibility. Should be a quick fix.

[beatonma]

The string name was an issue but the actual cause of your error turned out to be the older version of Wagtail. I've run out of time to work on it today but I'll get a patch out asap.

[garrettc]

Ah okay. I’m not quite ready to upgrade Wagtail yet, so that would be great. Thank you.

[beatonma]

There is now a pre-release of 4.0.1 available if you want to try it - still trying to resolve another issue before the proper release but the Wagtail stuff should be fine for wagtail>=3.0.3.

You can install it with pip install django-wm --upgrade --pre.

[garrettc]

I've installed the pre-release, and it's running, but I'm not sure if I'm misunderstanding how to retrieve things, or if it's not quite working correctly.

I have this post which has some Mastodon webmentions via Brid.gy that I can see in the admin area, but nothing is displaying for the "Target Object".

And I can confirm that through the shell:

>>> pp = PostPage.objects.get(title="Bye bye Twitter")
>>> pp.title
'Bye bye Twitter'
>>> pp.id
12209
>>> pp.get_mentions()
[]
>>> Webmention.objects.filter(object_id=pp.id)
<QuerySet []>

But, if I use a post that has older webmentions that I imported into Wagtail during the migration I can retrieve them with a filter, but not with get_mentions():

>>> pp2 = PostPage.objects.get(id="12153")
>>> pp2.title
'IndieWebCamp Oxford Day 2'
>>> pp.get_mentions()
[]
>>> Webmention.objects.filter(object_id=pp2.id)
<QuerySet [<Webmention: https://twitter.com/BaronVonLeskis/status/1045018944338350080 -> /blog/2018/09/indiewebcamp-oxford-day-2/ [validated=True, approved=True,content_type=blog | post page, id=12153]>, 
<Webmention: https://twitter.com/garrettc/status/1044569595456245760#favorited-by-373232559 -> /blog/2018/09/indiewebcamp-oxford-day-2/ [validated=True, approved=True,content_type=blog | post page, id=12153]>, 
<Webmention: https://twitter.com/garrettc/status/1044569595456245760#favorited-by-5752732 -> /blog/2018/09/indiewebcamp-oxford-day-2/ [validated=True, approved=True,content_type=blog | post page, id=12153]>, 
<Webmention: https://twitter.com/henrahmagix/status/1044646017285664770 -> /blog/2018/09/indiewebcamp-oxford-day-2/ [validated=True, approved=True,content_type=blog | post page, id=12153]>, 
<Webmention: https://twitter.com/garrettc/status/1044569595456245760#favorited-by-129810881 -> /blog/2018/09/indiewebcamp-oxford-day-2/ [validated=True, approved=True,content_type=blog | post page, id=12153]>, 
<Webmention: https://twitter.com/garrettc/status/1044569595456245760#favorited-by-430130910 -> /blog/2018/09/indiewebcamp-oxford-day-2/ [validated=True, approved=True,content_type=blog | post page, id=12153]>]>

There weren't any errors thrown during the run of pending_mentions, that all seemed to go fine.

(It could be that I imported the older ones incorrectly, happy to go down that route if that's the case)

[beatonma]

Hey, sorry for the delay.

It looks like it's not able to resolve the target_url to a page instance. Can you please try the following in the shell and see what it says?

from mentions.resolution import get_model_for_url

pp = PostPage.objects.get(title="Bye bye Twitter")
get_model_for_url(pp.url)

# Also
get_model_for_url("https://polytechnic.co.uk/blog/2022/11/bye-bye-twitter/")

Both should resolve the PostPage instance but it looks like yours will throw TargetDoesNotExist.

[garrettc]

No worries, it's that time of year 🙂

Getting the model via the object property worked:

>>> from mentions.resolution import get_model_for_url
>>> pp = PostPage.objects.get(title="Bye bye Twitter")
>>> get_model_for_url(pp.url)
<PostPage: Bye bye Twitter>

But getting it via the full URL bailed:

>>> get_model_for_url("https://polytechnic.co.uk/blog/2022/11/bye-bye-twitter/")
Traceback (most recent call last):
  File "/Users/garrettc/sandbox/polytechnic/env/lib/python3.9/site-packages/mentions/resolution.py", line 85, in get_model_for_url
    model_name = urlpattern_kwargs.pop(contract.URLPATTERNS_MODEL_NAME)
KeyError: 'model_name'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/Users/garrettc/sandbox/polytechnic/env/lib/python3.9/site-packages/mentions/resolution.py", line 89, in get_model_for_url
    return get_model_for_url_by_wagtail(match)
  File "/Users/garrettc/sandbox/polytechnic/env/lib/python3.9/site-packages/mentions/helpers/thirdparty/wagtail/resolution.py", line 63, in get_model_for_url_by_wagtail
    return get_model_for_url_by_helper(model_class, view_args, view_kwargs)
  File "/Users/garrettc/sandbox/polytechnic/env/lib/python3.9/site-packages/mentions/helpers/resolution.py", line 38, in get_model_for_url_by_helper
    model_filter_map: ModelFilterMap = urlpattern_kwargs.pop(
KeyError: 'model_filter_map'

[edited to add more info after tinkering]

I have custom URLs for my blog posts, of the form /blog/<YYYY>/<MM>/<title>/, but the url model attribute returns the internal URL /blog/<title>/. Could this be the source of the issue?

[beatonma]

That should be fine as long as the @mentions_wagtail_re_path decorator is configured for each custom route. From the error you got it appears that the data which should be attached by the decorator is missing... although the one that's posted above looks like it should do the job.

I can reproduce the error if the wagtail path|re_path|route decorator is used instead of the mentions one. Can you confirm that each route uses the mentions decorator?

---

I've put up another preview - can you please update (pip install django-wm==4.0.1.dev4) and do get_model_for_url("https://polytechnic.co.uk/blog/2022/11/bye-bye-twitter/") again?

It should show log messages of the form:

• pattern: {...} | model_class: {...} | lookup: {...} for each decorated route when you start the shell session.
• f"Wagtail match is not mentionable: {...} | args={...} | kwargs={...}" when the error is encountered.

Hopefully that will help us to make sure the config is right.

(The update also includes a new management command so once we get get_model_for_url working for the full URL then you can reprocess those mentions with python manage.py mentions_reverify "target_url=https://polytechnic.co.uk/blog/2022/11/bye-bye-twitter/".)

[garrettc]

I am an idiot.

I was on a branch that didn't have the decorator on the route, so that was causing issues. Sorry.

With 4.0.1.dev4 installed get_model_for_url() works with both forms of the URL:

In [1]: from mentions.resolution import get_model_for_url

In [2]: pp = PostPage.objects.get(title="Bye bye Twitter")

In [3]: get_model_for_url(pp.url)
Out[3]: <PostPage: Bye bye Twitter>

In [4]: get_model_for_url('/blog/2022/11/bye-bye-twitter/')
get_model_for_url_by_helper(model_class=<class 'blog.models.PostPage'>, urlpattern_args=('2022', '11', 'bye-bye-twitter') urlpattern_kwargs={'model_name': 'PostPage', 'model_filter_map': set(), 'model_filters': ('post_date__year', 'post_date__month', 'slug')})
Out[4]: <PostPage: Bye bye Twitter>

As does get_mentions():

In [7]: pp2 = PostPage.objects.get(id="12153")

In [8]: pp2.title
Out[8]: 'IndieWebCamp Oxford Day 2'

In [9]: pp2.get_mentions()
Out[9]:
[<Webmention: https://twitter.com/BaronVonLeskis/status/1045018944338350080 -> /blog/2018/09/indiewebcamp-oxford-day-2/ [validated=True, approved=True,content_type=blog | post page, id=12153]>,
 <Webmention: https://twitter.com/garrettc/status/1044569595456245760#favorited-by-373232559 -> /blog/2018/09/indiewebcamp-oxford-day-2/ [validated=True, approved=True,content_type=blog | post page, id=12153]>,
 <Webmention: https://twitter.com/garrettc/status/1044569595456245760#favorited-by-5752732 -> /blog/2018/09/indiewebcamp-oxford-day-2/ [validated=True, approved=True,content_type=blog | post page, id=12153]>,
 <Webmention: https://twitter.com/henrahmagix/status/1044646017285664770 -> /blog/2018/09/indiewebcamp-oxford-day-2/ [validated=True, approved=True,content_type=blog | post page, id=12153]>,
 <Webmention: https://twitter.com/garrettc/status/1044569595456245760#favorited-by-129810881 -> /blog/2018/09/indiewebcamp-oxford-day-2/ [validated=True, approved=True,content_type=blog | post page, id=12153]>,
 <Webmention: https://twitter.com/garrettc/status/1044569595456245760#favorited-by-430130910 -> /blog/2018/09/indiewebcamp-oxford-day-2/ [validated=True, approved=True,content_type=blog | post page, id=12153]>]

[beatonma]

Great! :D

Did you try the mentions_reverify command to reprocess the mentions with the missing target object? Did they update correctly?

[garrettc]

I'm not able to test that part fully as I'm running this locally and haven't deployed it to my live server yet, but the error message looks like it's going to work okay:

(env) $ python manage.py mentions_reverify "target_url=https://polytechnic.co.uk/blog/2022/11/bye-bye-twitter/"
pattern: ^(\d{4})/(\d{2})/(.+)/$ | model_class: PostPage | lookup: {'model_filter_map': set(), 'model_filters': ('post_date__year', 'post_date__month', 'slug')}
Received webmention does not target our domain: https://brid.gy/like/mastodon/@garrettc@mastodon.org.uk/109410407121650265/109395755670057760 -> https://polytechnic.co.uk/blog/2022/11/bye-bye-twitter/

I'm guessing that "Received webmention does not target our domain" is because I'm running on localhost?

[beatonma]

OK cool. It means the domain isn't in settings.ALLOWED_HOSTS so yeah that's probably it.

[beatonma]

(4.0.1 proper is now available with those extra log messages removed btw.)

[garrettc]

My fault again, manage.py wasn't pointing to my production settings by default. It's all good.

I finally got some time to deploy this to live, but the mentions_reverify command still isn't working. I'm getting the same Received webmention does not target our domain error as I was running it locally.

[garrettc]

And now that it's all styled up, it's working brilliantly. Thank you for all your hard work on this.

[beatonma]

Excellent! You're welcome - I'll close the issue now but let me know if you hit any more problems.