JSONDecodeError when trying to enter YouTube URL to wagtail admin

[noobmaster19]

Found a bug? Please fill out the sections below. 👍

Issue Summary

Im having issues embedding the YouTube URL . It returns the error : JSONDecodeError Expecting value: line 1 column 1 (char 0) The version i am on is wagtail 2.9.3

Here is the full error trace back from my console:

  File "C:\Users\dream\Desktop\NoboCMS\backend\cms_env\lib\site-packages\django\core\handlers\exception.py", line 34, in inner
    response = get_response(request)
  File "C:\Users\dream\Desktop\NoboCMS\backend\cms_env\lib\site-packages\django\core\handlers\base.py", line 115, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "C:\Users\dream\Desktop\NoboCMS\backend\cms_env\lib\site-packages\django\core\handlers\base.py", line 113, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "C:\Users\dream\Desktop\NoboCMS\backend\cms_env\lib\site-packages\django\views\decorators\cache.py", line 44, in _wrapped_view_func
    response = view_func(request, *args, **kwargs)
  File "C:\Users\dream\Desktop\NoboCMS\backend\cms_env\lib\site-packages\wagtail\admin\urls\__init__.py", line 109, in wrapper
    return view_func(request, *args, **kwargs)
  File "C:\Users\dream\Desktop\NoboCMS\backend\cms_env\lib\site-packages\wagtail\admin\auth.py", line 188, in decorated_view
    return view_func(request, *args, **kwargs)
  File "C:\Users\dream\Desktop\NoboCMS\backend\cms_env\lib\site-packages\wagtail\admin\views\pages.py", line 380, in edit
    if form.is_valid() and not page_perms.page_locked():
  File "C:\Users\dream\Desktop\NoboCMS\backend\cms_env\lib\site-packages\modelcluster\forms.py", line 315, in is_valid
    form_is_valid = super(ClusterForm, self).is_valid()
  File "C:\Users\dream\Desktop\NoboCMS\backend\cms_env\lib\site-packages\django\forms\forms.py", line 180, in is_valid
    return self.is_bound and not self.errors
  File "C:\Users\dream\Desktop\NoboCMS\backend\cms_env\lib\site-packages\django\forms\forms.py", line 175, in errors
    self.full_clean()
  File "C:\Users\dream\Desktop\NoboCMS\backend\cms_env\lib\site-packages\django\forms\forms.py", line 376, in full_clean
    self._clean_fields()
  File "C:\Users\dream\Desktop\NoboCMS\backend\cms_env\lib\site-packages\django\forms\forms.py", line 394, in _clean_fields
    value = field.clean(value)
  File "C:\Users\dream\Desktop\NoboCMS\backend\cms_env\lib\site-packages\wagtail\core\blocks\base.py", line 543, in clean
    return self.block.clean(value)
  File "C:\Users\dream\Desktop\NoboCMS\backend\cms_env\lib\site-packages\wagtail\core\blocks\stream_block.py", line 198, in clean
    (child.block.name, child.block.clean(child.value), child.id)
  File "C:\Users\dream\Desktop\NoboCMS\backend\cms_env\lib\site-packages\wagtail\core\blocks\stream_block.py", line 198, in clean
    (child.block.name, child.block.clean(child.value), child.id)
  File "C:\Users\dream\Desktop\NoboCMS\backend\cms_env\lib\site-packages\wagtail\core\blocks\struct_block.py", line 129, in clean
    result.append((name, self.child_blocks[name].clean(val)))
  File "C:\Users\dream\Desktop\NoboCMS\backend\cms_env\lib\site-packages\wagtail\embeds\blocks.py", line 69, in clean
    if isinstance(value, EmbedValue) and not value.html:
  File "C:\Users\dream\Desktop\NoboCMS\backend\cms_env\lib\site-packages\django\utils\functional.py", line 48, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "C:\Users\dream\Desktop\NoboCMS\backend\cms_env\lib\site-packages\wagtail\embeds\blocks.py", line 22, in html
    return embed_to_frontend_html(self.url)
  File "C:\Users\dream\Desktop\NoboCMS\backend\cms_env\lib\site-packages\wagtail\embeds\format.py", line 9, in embed_to_frontend_html
    embed = embeds.get_embed(url)
  File "C:\Users\dream\Desktop\NoboCMS\backend\cms_env\lib\site-packages\wagtail\embeds\embeds.py", line 24, in get_embed
    embed_dict = finder(url, max_width)
  File "C:\Users\dream\Desktop\NoboCMS\backend\cms_env\lib\site-packages\wagtail\embeds\embeds.py", line 20, in finder
    return finder.find_embed(url, max_width=max_width)
  File "C:\Users\dream\Desktop\NoboCMS\backend\cms_env\lib\site-packages\wagtail\embeds\finders\oembed.py", line 64, in find_embed
    oembed = json.loads(r.read().decode('utf-8'))
  File "C:\Python38\lib\json\__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "C:\Python38\lib\json\decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "C:\Python38\lib\json\decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
[04/Aug/2020 01:33:54] "POST /admin/pages/3/edit/ HTTP/1.1" 500 207887

Steps to Reproduce

My blocks.py

class VideoBlock(blocks.StreamBlock):
    '''rich text'''
    content = blocks.StructBlock(
            [
            ("title",blocks.CharBlock(required=True, help_text="Add your Title")),
            ("video", EmbedBlock())
            ]
    )

    class Meta:
        icon = "edit"
        label = "Video Block"

which is then used in my homepage model as

class HomePage(Page):
    '''home page model'''
    content = StreamField(
        [
            ("video",block.VideoBlock()),
        ] , blank=True
    )
    max_count = 1
    content_panels = [
        StreamFieldPanel('content'),
    ] + Page.content_panels 
    api_fields= [
        APIField('content'),
    ]

    class Meta:
        verbose_name = "Home Page"
        verbose_name_plural = "Home Pages"

To get the YouTube URL , i right clicked on the YouTube video url and clicked the copy video url option . The url is https://youtu.be/bbLlvoGbbEc . Upon clicking publish , the error mentioned above appeared

• I have confirmed that this issue can be reproduced as described on a fresh Wagtail project: (yes / no) yes

Technical details

• Python version: 3.8.2
• Django version: 3.0.5
• Wagtail version: 2.9.3
• Browser version: Chrome 84

[gasman]

Hi @neowenshun - thanks for the report. This is working fine when I try with the same setup and URL - are you seeing it fail on other Youtube URLs too, or just this specific one?

In the past we've seen failures with embeds happening due to broken Python installations (#5381), although we haven't seen this specific error. Please could you try running the following code at the ./manage.py shell command line, and let me know the results?

from wagtail.embeds.embeds import get_embed
get_embed('https://youtu.be/bbLlvoGbbEc').html

Also, what do you get if you visit https://www.youtube.com/oembed?url=https://youtu.be/bbLlvoGbbEc in your browser? (It's possible that Youtube has region-specific restrictions preventing you from embedding the video.)

[noobmaster19]

Thanks for the quick reply! Im getting this error when running from python shell.

Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "C:\Users\dream\Desktop\NoboCMS\backend\cms_env\lib\site-packages\wagtail\embeds\embeds.py", line 24, in get_embed
    embed_dict = finder(url, max_width)
  File "C:\Users\dream\Desktop\NoboCMS\backend\cms_env\lib\site-packages\wagtail\embeds\embeds.py", line 20, in finder
    return finder.find_embed(url, max_width=max_width)
  File "C:\Users\dream\Desktop\NoboCMS\backend\cms_env\lib\site-packages\wagtail\embeds\finders\oembed.py", line 64, in find_embed
    oembed = json.loads(r.read().decode('utf-8'))
  File "C:\Python38\lib\json\__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "C:\Python38\lib\json\decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "C:\Python38\lib\json\decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

When i visit the url u have provided , im getting

{"thumbnail_height":360,"version":"1.0","provider_url":"https:\/\/www.youtube.com\/","type":"video","author_url":"https:\/\/www.youtube.com\/user\/Demonisios","height":270,"html":"\u003ciframe width=\"480\" height=\"270\" src=\"https:\/\/www.youtube.com\/embed\/bbLlvoGbbEc?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen\u003e\u003c\/iframe\u003e","title":"#\u6296\u97f3\u795e\u66f22020#\u6296\u97f350\u9996\u5fc5\u807d\u65b0\u6b4c ||\u963f\u5197 - \u4f60\u7684\u7b54\u6848 , \u8aaa\u597d\u4e0d\u54ed Won't Cry , \u8292\u7a2e , \u56c2\u5f35 , \u4f60\u7684\u9152\u9928\u5c0d\u6211\u6253\u4e86\u70ca , \u7da0\u8272 , \u63a5\u500b\u543b\uff0c\u958b\u4e00\u69cd , \u771f\u7684\u50bb , \u904e\u5ba2","author_name":"KKBOX \u83ef\u8a9e\u65b0\u6b4c 2019","width":480,"provider_name":"YouTube","thumbnail_url":"https:\/\/i.ytimg.com\/vi\/bbLlvoGbbEc\/hqdefault.jpg","thumbnail_width":480}

[noobmaster19]

The weird thing is that its working on my other project which is currently deployed on a linux server . The differences between the 2 setup is that the one running on my local host (the one with the problem) is running on windows and also has the v2 api enabled

[gasman]

Please can you retry the code above on a new project created with wagtail start, just to confirm that this isn't something specific to that project's configuration?

Also, another test to try (on the ./manage.py shell command line again):

from urllib import request
from urllib.request import Request
import json
rq = Request('https://www.youtube.com/oembed?url=https://youtu.be/bbLlvoGbbEc&format=json')
rq.add_header('User-agent', 'Mozilla/5.0')
response = request.urlopen(rq)
response_text = response.read().decode('utf-8')
print(response_text)
print(json.loads(response_text))

[kaedroho]

Are you behind a corporate proxy that blocks requests to Youtube?

[noobmaster19]

Please can you retry the code above on a new project created with wagtail start, just to confirm that this isn't something specific to that project's configuration?

Also, another test to try (on the ./manage.py shell command line again):

from urllib import request
from urllib.request import Request
import json
rq = Request('https://www.youtube.com/oembed?url=https://youtu.be/bbLlvoGbbEc&format=json')
rq.add_header('User-agent', 'Mozilla/5.0')
response = request.urlopen(rq)
response_text = response.read().decode('utf-8')
print(response_text)
print(json.loads(response_text))

Hey , i've tried running this code

from urllib import request
from urllib.request import Request
import json
rq = Request('https://www.youtube.com/oembed?url=https://youtu.be/bbLlvoGbbEc&format=json')
rq.add_header('User-agent', 'Mozilla/5.0')
response = request.urlopen(rq)
response_text = response.read().decode('utf-8')
print(response_text)
print(json.loads(response_text))

Edit Seems that this piece of code runs on both my old and the fresh wagtail project

[noobmaster19]

Are you behind a corporate proxy that blocks requests to Youtube?

Hello! I don't think so , im working from a home lan , and it seems that the new code snippet that @gasman has given me is working on the old wagtail repo

[gasman]

Does the original code snippet also work on the new Wagtail project?

from wagtail.embeds.embeds import get_embed
get_embed('https://youtu.be/bbLlvoGbbEc').html

[noobmaster19]

Does the original code snippet also work on the new Wagtail project?

from wagtail.embeds.embeds import get_embed
get_embed('https://youtu.be/bbLlvoGbbEc').html

Nope this does not work , even on the new project

Heres the error traceback:

Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "C:\Users\dream\Desktop\wagtailtest\wagtailtest_env\lib\site-packages\wagtail\embeds\embeds.py", line 24, in get_embed
    embed_dict = finder(url, max_width)
  File "C:\Users\dream\Desktop\wagtailtest\wagtailtest_env\lib\site-packages\wagtail\embeds\embeds.py", line 20, in finder
    return finder.find_embed(url, max_width=max_width)
  File "C:\Users\dream\Desktop\wagtailtest\wagtailtest_env\lib\site-packages\wagtail\embeds\finders\oembed.py", line 64, in find_embed
    oembed = json.loads(r.read().decode('utf-8'))
  File "C:\Python38\lib\json\__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "C:\Python38\lib\json\decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "C:\Python38\lib\json\decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None

[chosak]

This works for me on a vanilla Wagtail 2.9.3, on MacOS, Python 3.6.10:

>>> from wagtail.embeds.embeds import get_embed
>>> get_embed('https://youtu.be/bbLlvoGbbEc').html
'<iframe width="480" height="270" src="https://www.youtube.com/embed/bbLlvoGbbEc?feature=oembed" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

[gasman]

OK, I'm clutching at straws a bit, but one more thing to try, please...

from urllib import request
from urllib.request import Request
import json
rq = Request('http://www.youtube.com/oembed?url=https://youtu.be/bbLlvoGbbEc&format=json')
rq.add_header('User-agent', 'Mozilla/5.0')
response = request.urlopen(rq)
response_text = response.read().decode('utf-8')
print(response_text)
print(json.loads(response_text))

(in other words: the same code as before, but the first https in the URL is now http)

[noobmaster19]

Hello gasman , it seems to work for response_text heres the log :

<html>
<head>
<title>Home Internet Filter</title>
</head>
<body >
<div style='font-size:12.0pt;line-height:115%;font-family:"Arial","sans-serif"'>
<p><img src="assets/singtel_logo.png" alt="Singtel Logo"></p>
<p>&nbsp;</p>
<p>The page you are trying to access has been blocked by the Home Internet Filter service.</p>
<p>If you wish to continue viewing this page, you may log into <a href="https://www.singtel.com/hifportal">Home Internet Filter</a> to adjust
your settings.</p>
<p>For more information about this service, please refer to the "Home Internet Filter" section in the
  <a href="https://www.singtel.com/personal/products-services/broadband/add-ons">Fibre Broadband Add-ons page</a>.</p>
</div>
</body>
</html>

It seems that http was blocked by my home filter .

print(json.loads(response_text)) returns the following error log :

Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "C:\Python38\lib\json\__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "C:\Python38\lib\json\decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "C:\Python38\lib\json\decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

[noobmaster19]

Please can you retry the code above on a new project created with wagtail start, just to confirm that this isn't something specific to that project's configuration? Also, another test to try (on the ./manage.py shell command line again):

from urllib import request
from urllib.request import Request
import json
rq = Request('https://www.youtube.com/oembed?url=https://youtu.be/bbLlvoGbbEc&format=json')
rq.add_header('User-agent', 'Mozilla/5.0')
response = request.urlopen(rq)
response_text = response.read().decode('utf-8')
print(response_text)
print(json.loads(response_text))

Hey , i've tried running this code

from urllib import request
from urllib.request import Request
import json
rq = Request('https://www.youtube.com/oembed?url=https://youtu.be/bbLlvoGbbEc&format=json')
rq.add_header('User-agent', 'Mozilla/5.0')
response = request.urlopen(rq)
response_text = response.read().decode('utf-8')
print(response_text)
print(json.loads(response_text))

Edit Seems that this piece of code runs on both my old and the fresh wagtail project

The previous code which used https worked for both the fresh and old wagtail repo though

[gasman]

OK, now this makes sense! Wagtail is (perhaps wrongly...) using http instead of https for the endpoint that fetches the embed:

https://github.com/wagtail/wagtail/blob/master/wagtail/embeds/oembed_providers.py#L17

Normally this wouldn't be a problem, because Youtube redirects the http URL to https, and Python's urllib follows redirects automatically - but on your network the redirect isn't happening, and it ends up trying to parse your home filter's error message instead...

We should fix the URLs in oembed_providers.py to use https. In the meantime, you should be able to work around this by adding the following to your settings file:

from wagtail.embeds.oembed_providers import all_providers

youtube_fixed = {
    "endpoint": "https://www.youtube.com/oembed",
    "urls": [
        r'^http(?:s)?://(?:[-\w]+\.)?youtube\.com/watch.+$',
        r'^http(?:s)?://(?:[-\w]+\.)?youtube\.com/v/.+$',
        r'^http(?:s)?://youtu\.be/.+$',
        r'^http(?:s)?://(?:[-\w]+\.)?youtube\.com/user/.+$',
        r'^http(?:s)?://(?:[-\w]+\.)?youtube\.com/[^#?/]+#[^#?/]+/.+$',
        r'^http(?:s)?://m\.youtube\.com/index.+$',
        r'^http(?:s)?://(?:[-\w]+\.)?youtube\.com/profile.+$',
        r'^http(?:s)?://(?:[-\w]+\.)?youtube\.com/view_play_list.+$',
        r'^http(?:s)?://(?:[-\w]+\.)?youtube\.com/playlist.+$',
    ],
}

WAGTAILEMBEDS_FINDERS = [
    {
        'class': 'wagtail.embeds.finders.oembed',
        'providers': [youtube_fixed] + all_providers,
    }
]

[noobmaster19]

Ah ! Okay works now , thanks a whole lot for the help !

[noobmaster19]

Hello , apologies for asking a new question here , this is related to the issue that i raised. It seems that even though i was able to save a video into the admin console , i was not able to display it . Here is the error that im facing in my frontend

Refused to display 'https://www.youtube.com/watch?v=IP8vBL5Q8Ac&feature=youtu.be' in a frame because it set 'X-Frame-Options' to 'sameorigin'.

Upon further reading up it seems that i should be using /embed instead of watch , however strangely enough , the video url automatically changed to contain the /embed when i uploaded the video using a rich text stream . This however was not the case with the EmbedBlock , im wondering if its because of the changes above that led to this situation

[gasman]

I don't think that's related to the issue here. For support queries, please use the channels listed at https://docs.wagtail.io/en/stable/support.html.