feat: make the open graph tags dynamic based on the ``base64`` and ``url`` query parameters

helios2003 commented 4 months ago

This PR addresses the issue mentioned in #224. I have added the following features in this PR.

The studio version I am testing on is hosted here: https://studio-studio-next.vercel.app. (I will switch this to Netlify deployment once we are able to resolve #1118)
The Image Generator service in charge of creating the dynamic image is hosted here: https://ogp-studio.netlify.app
The code for the image Generator service can be found here: https://github.com/helios2003/ogp-studio

Features Added

[x] Add a middleware that captures Open Graph crawler requests when they hit the website and rewrites the URL to the route api/crawler. This is done to serve minimal HTML when the crawler hits the page, reducing the chance of timeout.
[x] Parse the URL and send it to the Image Generator service to add the dynamic image.
[x] Show the default og:image if there is an invalid base64 string provided.
[x] Fall back on the usual behavior added in #1106 if there isn't a base64 string.

Todos

[ ] Address unusual behavior observed like Slack showing the default image even when the document is valid.
- For example: if we go to https://studio-studio-next.vercel.app and use MQTT template, then the default behavior is observed instead of dynamic tags.
- Sometimes, Slack crawlers arent able to detect the open graph image.
[ ] Figuring out why sometimes the dynamic og:image does not appear on site like Whatsapp.
[ ] Add additonal crawlers to the list.

cc : @smoya

changeset-bot[bot] commented 4 months ago

⚠️ No Changeset found

Latest commit: 35c4a72388a1d8495a7beba4cd10a70612843745

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

netlify[bot] commented 4 months ago

Deploy Preview for modest-rosalind-098b67 ready!

Name	Link
Latest commit	35c4a72388a1d8495a7beba4cd10a70612843745
Latest deploy log	https://app.netlify.com/sites/modest-rosalind-098b67/deploys/6734d5b982fb020008a9961f
Deploy Preview	https://deploy-preview-1122--modest-rosalind-098b67.netlify.app
Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

netlify[bot] commented 4 months ago

Deploy Preview for asyncapi-studio-design-system ready!

Name	Link
Latest commit	35c4a72388a1d8495a7beba4cd10a70612843745
Latest deploy log	https://app.netlify.com/sites/asyncapi-studio-design-system/deploys/6734d5b97770160008d6da62
Deploy Preview	https://deploy-preview-1122--asyncapi-studio-design-system.netlify.app
Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

netlify[bot] commented 4 months ago

Deploy Preview for studio-next ready!

Name	Link
Latest commit	19858da79f29a313365b33999df2c5a02978d21a
Latest deploy log	https://app.netlify.com/sites/studio-next/deploys/66e98efe0094a20008517e8b
Deploy Preview	https://deploy-preview-1122--studio-next.netlify.app
Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

helios2003 commented 4 months ago

Thank you for the review @KhudaDad414. I will add the suggested changes.

helios2003 commented 4 months ago

Also, I had a few questions:

Is it okay if I hardcode the popular social media site crawlers as I have done now or is it better if we use a library like https://github.com/monperrus/crawler-user-agents as suggested to me by @smoya. The main problem I could think of is that by including all the potential sets of crawlers as mentioned in the repository we could be causing rewrites for nonopen graph crawlers as well which is not required. But @smoya mentioned that he does not expect other bots to use the query parameters like that of open graph crawlers.
Also, could you please let me know your thoughts on https://github.com/helios2003/ogp-studio and its code?

smoya commented 4 months ago

Is it okay if I hardcode the popular social media site crawlers as I have done now or is it better if we use a library like https://github.com/monperrus/crawler-user-agents as suggested to me by @smoya. The main problem I could think of is that by including all the potential sets of crawlers as mentioned in the repository we could be causing rewrites for nonopen graph crawlers as well which is not required.

If you don't use a complete list of crawlers, you will end up facing issues like the one you mentioned:

Figuring out why sometimes the dynamic og:image does not appear on site like Whatsapp.

This is happening most probably because you are not considering Whatsapp User-Agent(s).

But @smoya mentioned that he does not expect other bots to use the query parameters like that of open graph crawlers.

AFAIK, indexer robots such as Google only follow links, meaning that unless there is a published link somewhere to a link such as https://studio.asyncapi.com/?base64=<base64_doc> or the variant with url query param, the parsing won't happen. In fact, it does make sense to do the parsing and return the proper title, description, etc to the results of such indexer bot request so it appears in the results (like google results).

smoya commented 4 months ago

Also, could you please let me know your thoughts on https://github.com/helios2003/ogp-studio and its code?

Will that work in a Netlify setup? I mentioned an alternative in case it doesn't in https://github.com/asyncapi/studio/issues/224.

Additionally, I think we can include more statistics in the rendered image like # of operations

helios2003 commented 4 months ago

Will that work in a Netlify setup? I mentioned an alternative in case it doesn't in #224.

Yes, it works. The service is currently hosted on Netlify: https://ogp-studio.netlify.app.

Additionally, I think we can include more statistics in the rendered image like # of operations

Okay, I will add it.

helios2003 commented 3 months ago

Added modifications:

Using a package for detecting crawlers.
Using sharp to optimize the image generation as mentioned here: https://nextjs.org/docs/messages/install-sharp.
Add the dynamic behavior for url query param as well.

helios2003 commented 3 months ago

The PR is ready for review @KhudaDad414, @smoya.

helios2003 commented 3 months ago

Also, I'm unsure if the middleware code I wrote is the best way to achieve this functionality. Any suggestions or improvements would be appreciated.

smoya commented 2 months ago

@helios2003 Are you planning to move the OG Image generator service (Netlify Edge Function) into this PR?

helios2003 commented 2 months ago

Maintainers, can you please review the PR? I have addressed all the comments above and included the unit tests. Additionally, I had to change the version of @asyncapi/converter after this commit: https://github.com/asyncapi/converter-js/commit/b3592ef21a288231871080945b0b7c53b70adb32. I did not include these changes in the studio repository since we are soon switching to studio-next as the default version for studio based on this issue: https://github.com/asyncapi/studio/issues/1134