Certain parts of the embed plugin not working on development and production server, but ok locally depending on the YouTube url format

[terryupton]

Firstly, I appreciate this might not be something you can assist with and might not be a problem with the plugin per-say, however, I am a bit at a loss as to where else to go from here. So I am hoping if you or anyone might be able to shed some light, help me resolve this issue? I need some fresh eyes/mind...

So previously (perhaps a week ago) everything was working fine locally, staging and production at the time I was using a different plugin called VideoEmbedder. I then noticed that both staging and production (same server) all the videos and thumbnails had stopped working entirely. At first, I thought it might have been a rate-limit, but after 24 hours and re-cheking nothing had changed. I then came across this thread https://github.com/mikestecker/craft-videoembedder/issues/33 and as the plugin appears to be inactive. I decided to switch to this OEmbed. Partly, as I have used it on another site I have recently launched, so i know it works and is still supported.

So, I have gone about swapping all the code over from VideoEmbedder to OEmbed. Doing this locally, initially and everything working as expected. I then deployed the changes to the staging server, and unfortunately, I have the same issue. No videos load on either back-end or front-end. And no thumbnails load, and several other media.handle details don't fire either. (more on this below).

I have been trying all sorts to test and try and diagnose the problem, but the more I test the more confusing things seem. I have tried my best to summarise my findings below.

So everything I cover below, all works absolutely fine in my local (valet) environment. So if I paste in a URL like so https://youtu.be/sz5JDPkG0zI - I get the following errors in the back-end.

Inspecting the chrome dev tools I see this: Refused to display '<URL>' in a frame because it set 'X-Frame-Options' to 'sameorigin'.

After doing some reading and research, I think this might be down to the format of the URL? Could you confirm this?

If I swap the URL to https://youtube.com/embed//sz5JDPkG0zI then the video loads ok in the back-end like so:

But to illustrate how this works in local it looks like so (which is a better presentation too):

With the above example on the front-end. I get the same issue and error in the console when I use the first url format https://youtube.com/embed//sz5JDPkG0zI

If I use the second url format https://youtube.com/embed//sz5JDPkG0zI Then the video embeds and loads ok, but some of the other data elements do not. See as follows:

If I do things manually....

If I hard-code or embed code directly into the template, I can get the actual thumbnails and videos. So it isn't like a block on the URL or IP. As an example:

Working ok on other sites on the same server

I would also like to add, that the same plugin and craft version are working absolutely fine (with any url format) on another site on the same server and the setup/headers/php/apache config etc is identical to this website.

This leads me to believe it is an issue with possible the domain (perhaps a block) but I cannot find any information to support this or how I would go about finding out if it is blocked and getting it unblocked.

Recap

So on my local environment (Valet) all is working regardless of the video url format and all fields/data are being rendered correctly etc. I can embed the youtube code directly (as in copy/paste from YouTube embed code) and I can load the images directly as well.

I have looked through the logs and errors and I cannot find anything to help. I have inspected the headers and I cannot see any core differences between local and the staging/live website. I am running the following version on the staging/live server:

Craft Pro: 3.5.17.1. oEmbed: 1.3.6 PHP version: 7.3.26. OS version: Linux 3.10.0-1160.11.1.el7.x86_64 MySQL 5.5.5.

I could drop the plugin and just code the embed myself. But the video preview in the back-end is a nice touch and the client was used to it. It also annoys me why I cannot figure out the cause and resolve this issue on the server. I want to get it working...

I wondered, if you have any suggestions, ideas or fixes/solutions I can try to either diagnose the real cause or try and resolve this issue. Any help or advise would be massively appreciated, as I am at a dead-end now.

Thanks in advance.

[reganlawton]

I'll look into this and see what I can dig up. I know there's been weird things happening with oembed protocol all the time it's normally just troubleshooting till I find it.

Lots of information to digest but from my skim over it lots quite detailed and should help my debugging so thanks that's quite helpful.

[terryupton]

Thanks @reganlawton I think having spent a bit longer, but yet to prove, there may be two factors at play here. One being the URL format and the other possibly being rate limits.

I will do some further checks and tests myself tomorrow and I will follow up with any more information I can share.

Thanks again.

[reganlawton]

Rate limiting is a know pain with this protocol but let's see if I can help out

[terryupton]

Hey @reganlawton I have done some further trials and testing this morning and I am not sure if this helps shed any more light, but here are my findings.

So yesterday after I posted the original thread, I changed one of the urls to this format https://youtube.com/watch/sz5JDPkG0zI (notice the /watch/). Upon doing this, all the data parameters started to populate and work as expected, the thumbnails loaded and everything seemed fine. At this point I wondered if it was 2 issues, one being the URL format and x-headers issue and a separate issue being perhaps rate limits, that had somehow reset in the period of testing.

I should also add that the presentation in the back-end when using this scheme is much preferred as the video presents as follows:

Coming back to it this morning, I can confirm that the handful of videos I updated yesterday still function this morning using the URL scheme with the /watch/id. However, if I use the following scheme /embed/id e.g (https://youtube.com/embed/HvK37CVYRtY) then I have the issues as experienced yesterday, whereby the embed of the video works but several of the data parameters do not.

This occurs across multiple videos I have tested. The old urls (e.g. https://youtu.be/sz5JDPkG0zI) still continue to cause a problem wit the 'x-headers' error on the staging/live server.

As mentioned previously all variations of the YouTube url work fine locally on Valet e.g. https://youtu.be/sz5JDPkG0zI https://youtube.com/embed/sz5JDPkG0zI https://youtube.com/watch/sz5JDPkG0zI As a side note, all 3 versions of the url locally present the video in the nicer format (full video) in the back-end.

So I am not sure why/what the discrepancy is on the staging/live server or if it is just YouTube causing the X-Frame-Options' to 'sameorigin' issue because it is realising it is from a 'proper' domain rather than locally where the tld is a '.loc'

I am hoping you can test and replicate some of these issues, otherwise I appreciate it is extremely difficult to hunt down and figure out. I am more than happy to share any further information, screen-share, chart on Discord or anything if it might help with this issue.

I intend to enable the cache on the live site, but I don't want to do this until the client has updated all the URLS to the working scheme, but I am not asking them to do this just yet, in case there is a better fix/solution.

Thanks in advance and apologies for another long thread.

[reganlawton]

Find my details for Discord on my GitHub Profile but I believe this is a réponse header on your side "X-Frame-Options" is used to stop iframe s and such loading on your page and prevent clickjacking. You might need to check your web server configuration as it seems Valet isn't using the same websrver it's not a header sent from CraftCMS so I take it will be a config setting on your system. NGINX or Apache? Hit me on discord to chat

[tomkiss]

Hmm, just launching a website today and noticing this issue too. Same issue where local is fine, public staging is also fine (this is confusing to me) but production is not OK. Only just investigating now, the website is on servd.host so its not quite the same access to adjust things, but it's NGINX for sure.

[tomkiss]

This person seems to be claiming that the /embed endpoint needs to be used: https://stackoverflow.com/questions/25661182/embed-youtube-video-refused-to-display-in-a-frame-because-it-set-x-frame-opti

This then ensures that the youtube URL does not contain the 'X-Frame-Options' with 'SAMEORIGIN' value.

So, the outputted URL which is currently:

<iframe src="https://www.youtube.com/watch?autopause=1&amp;loop=0&amp;mute=0&amp;rel=0&amp;autoplay=0&amp;v=RClJ_DAotaA" width="100%" frameborder="0" scrolling="no" title="YouTube"></iframe>

Should be:

<iframe src="https://www.youtube.com/embed/RClJ_DAotaA?autopause=1&amp;loop=0&amp;mute=0&amp;rel=0&amp;autoplay=0" width="100%" frameborder="0" scrolling="no" title="YouTube"></iframe>

This certainly appears to solve the problem I'm seeing (just trying to fix the front end right now in terms of videos not displaying, not sure about the CP side of things).

[reganlawton]

Great temp fix in @terryupton case we jumped on Discord And his issue is that his being rate limited from YouTube servers. It's seems that they might be getting very aggressive with their rate limiting but I do recommend using caching for production sites due to the rate limiting issue.

[tomkiss]

I've now gone down the hole enough to work out the rate limit issue. My rushed fix didn't work for too long, as I didn't have caching on, and subsequent entry queries started to not return the URL, so it eventually failed. I've deleted my post above incase it helps, then fails to help, anyone else, heh.

My logs, incidentally were:

2021-01-22 01:41:28 [-][-][-][error][Exception] Embed\Exceptions\InvalidUrlException: Invalid url 'https://www.youtube.com/watch?v=6xpFUgCp900' (Status code 429) in /var/www/html/vendor/embed/embed/src/Embed.php:142
...
Next Exception: Serialization of 'class@anonymous' is not allowed in /var/www/html/vendor/yiisoft/yii2/caching/Cache.php:249

So yeah, too many requests, which explains all the different environments. 🤦‍♀️

My only suggestion here is that it might be nice if the plugin could feed the 429 response code back to the CP and display it in place of the "Please check your URL." error?

[terryupton]

Just an update on my side. Everything was down to rate limiting by YouTube which I think was a result of a previously installed plugin, that I replaced for this. After managing to stop further requests, after a few days it all settled down and I was able to enable the cache and please to confirm all formats of the YouTube are working again.

I had to get the client in the interim to change all formats to the /watch/youtubeID (https://youtube.com/watch/sz5JDPkG0zI) but this wasn't a wasted exercise, as it made sure they were consistent across two sites, so should anything like this occur again, it would be much easier to manipulate the URL.

One thing I would add @reganlawton is if possible, to maybe feedback some more details on the errors etc. If that is even possible, as I am not sure what you get back from YouTube.

Thanks again for the time spent assisting with this.