Gutenberg support

[GasimGasimzada]

I am opening this issue; so, we can discuss all Gutenberg development here if it is okay with you guys. I am willing to implement this feature but I need some feedback on this regarding how to proceed. Here is the problem that I am facing with this. Plugins such as WPML create separate posts for each language instead of using one post and doing string interpolation. That's why, implementing Gutenberg on that is much easier since every language post is separately stored in the database.

When checking Gutenberg database record, it looks something like this:

<!-- wp:paragraph -->
<p>Some paragraph</p>
<!-- /wp:paragraph -->

On the other hand, qTranslate-X adds the following syntax for database records. [:en]...[:de]....

Is there a way to know if there is a chance for these boxes to clash? I can do a big database change to separate these texts into a separate table with columns such as:

post_id, language, title, content, date_published

As a result, this will also increase page performance because we will be able to select much less data from the database. Writing a migration tool will also be easy but the whole thing is going to take some time. If everyone agrees and approves this idea, I can work on it and get it running in a week or two with my limited availability.

Moreover, there are two problems that arise from qTranslate implementation:

1. qTranslate uses tinymceInit function and if we use Gutenberg, the function does not exists, meaning we get an error and nothing is saved. I will investigate this and try to fix and make qTranslate not dependent on tinymce when using gutenberg.

2. Gutenberg editor takes up the entire space for the content editor and as far as I can tell, there is no way to add content above the editor. So, my suggestion is that we add language selectors to the "top bar" (the black bar that is always on top in the admin page). This can also give a better and cleaner UI that will look nice with WP design language.

[dschmidtke]

Thanks for making a first step to Gutenberg support. I must admit that I did not look into this as thorough as you already did.

However, I think a design principle of qTranslate was to put everything into the wp_posts table instead of opening additional tables for posts by language. I would prefer to stick to the old principle and I don't see a reason why the language tags of qTranslate might not play nice with the Gutenberg comments.

As I see it, the retrieval of a post goes like that: 1. Pull the post from the database; 2. Extract the content for the requested language; 3. Feed that to Gutenberg and let it do its block magic.

[herrvigg]

I've only started to think about it and check a few things.

The main idea behind qTranslate (and all its variants) is to store all the multilingual data in a single post. It has pros and cons. If we continue to stick to this, Gutenberg should probably be embedded at a lower level:

[:en]
<!-- wp:paragraph -->
<p>An english paragraph</p>
<!-- /wp:paragraph -->
[:de]
<!-- wp:paragraph -->
<p>Ein deutscher Absatz</p>
<!-- /wp:paragraph -->
[:]

Doing the other way around could be interesting, ie handling the translation in the Gutenberg blocks themselves. It may have some advantages for example if you have some common blocks and only a few ones to be translated, not the whole post. But it would lead to multiplying many qTranslate blocks so this is probably not the best in the general case. Also, we should think about the compatibility of the DB between the two editors. Doing the first way should keep everything compatible. Relying on the Gutenberg blocks will break the retro-compatibility.

So, to handle Gutenberg with the usual approach (qTranslate on top, with Gutenberg embedded), it's not necessarily very complicated (sic). qTranslate doesn't need to care much what Gutenberg does during the edition. The icons to switch language should not be a big deal, there should always be a solution to hack something in the GUI in the worst case. The main problem is the load/save operations. We need some triggers to:

• intercept the post being loaded before it is sent to Gutenberg. Store the whole post somewhere but only send the q-block of the current language to the editor.
• handle the language switches (not the most urgent task for now)
• intercept the post being saved for the current language and recompose the whole post before it is written in DB.

Technically i don't know how to do this yet. One should look at the hooks related to Gutenberg. If there is no hook we should look at the code and try to find a hack. If i understand well Gutenberg will rely a lot on the REST API so this is definitely something to look at. I don't know if it can work without REST. Relying on REST could even have an interest with the classic editor but that's another story.

[GasimGasimzada]

Okay, I have figured out completely how Gutenberg works. Even the initial load is loaded through WP REST API. I am going to play with it for a little while and see if I can make REST requests to be translated. This way, translated content will at least show properly. Then, I will check on how to save the data.

[herrvigg]

Actually the support for the REST API should not even be dependent on the editor being used and it could be a nice feature for many other things. At least there are already some tickets asking to fix a bug on REST redirects. Once we have clarified the protocol used by Gutenberg we should take all the REST related stuff and fix it.

After that we'll need to change the JS part to clearly separate all the editor-dependent parts. Ideally we should extract the parts related to TinyMCE so that both Gutenberg and TinyMCE use a common structure as much as possible, i guess both editors should live together for a while.

[herrvigg]

I started investigating a bit more and from what i see the API REST used by Gutenberg is working fine. When you use ?context=edit it returns the post with the content in two versions, raw and rendered:

• the rendered version is the translated one after calling all the hooks such as the_content and the_title.
• the raw version is the original content from the DB (with all blocks from qTranslate and Gutenberg)

If we want to reproduce a behavior similar to the classic editor we need obviously to work on the raw content... client-side. So most of the work will actually be in JS.

Gutenberg provides a lot of packages related to Wordpress. The one that interests us most should be the editor: https://wordpress.org/gutenberg/handbook/packages/packages-editor/

Basically for the first part, the loading, we need to intercept the moment when it calls wp.blocks.parse( post.content.raw ) and replace the raw content with the selected language! For the save it's the other way around in wp.blocks.serialize( blocks ). I'm not sure it's the best way to work with Gutenberg but this should be the most similar to the current qTranslate approach.

[GasimGasimzada]

The problem here is that it is really hard to modify the editor itself like we could do with old editor. The entire editor is one big React component and there is no way to modify the sections (e.g toolbar). The only API available is for adding custom blocks.

This creates a serious problem when you want to parse the incoming raw data because it is essentially impossible.

We can create a custom block that supports both languages but that will mean that all blocks will have qtranslate-X alternatives (just like in ACF where there is Image field and qTranslate Image field if you install the proper plugin.

I personally cant get find a modify any data coming to the editor because the entire logic of parsing text for database is dependent on a per block basis.

There is also one more alternative that I have but I am not sure if this is possible.

Create a container type of block (e.g div or box type of blocks) and allow other blocks to live in it. Maybe we can add a tabular interface to it (all blocks are React components after all). But the user will require to add the block before working, which is inconvenient.

If dynamically changing languages is not necessary, we can update the raw data to only show the correct language and modify saving of posts. But we won’t be able to click a tab and without page load change the language. Every language will change the page (e.g post.php?id=...&lang=de)

[herrvigg]

As said the toolbars and the switches are not a problem. The generic metabox already shows up if you remove the TinyMCE errors. This is with Gutenberg:

Why would React be a problem? I don't know about the API but in the worst case it should be possible to create the buttons through the DOM so we don't really care. How could React prevent this?

I think the loading part should be feasible because you should always be able to patch the data before Gutenberg takes it. For the save operation hopefully there's a trigger but in the worst case you can also hack a listener on the save button through the DOM. Finally we must find how to switch language and this can be a bit tricky depending on the API but i hope it's feasible.

[herrvigg]

Mmm so the API doesn't seem very straightforward to use, part because it's React on top of Wordpress packages, part because the whole thing is not very clearly documented. Not even the wordpress API is clear. The Gutenberg docs only focus on the blocks part but we don't really care about this at the moment.

When the page loads most of the work seems to be done with wp.apiFetch.use( wp.apiFetch.createPreloadingMiddleware...). The idea would be to reinitialise dynamically the editor but i don't know how to do it on the fly. There is a reinitializeEditor method here in https://github.com/WordPress/gutenberg/blob/5632f6880fff49ce8866836db9bba0d1f67cb2d3/edit-post/index.js. But i'm not able to use it yet. Another concern is about all the hidden states that could be set when loading the page.

So for now maybe we should forget a bit about the dynamic change client-side and deal only with one language at a time. It would be more constraining (save for each language and full reload on switch) but at least it could be a good baseline! Also i find it easier to change stuff server-side using REST. So here is the plan:

• edit the raw content for one language by intercepting the REST request server-side,
• let Gutenberg client-side do all its stuff with the blocks without any change on the content,
• on save, done by REST, the full content should be re-assembled server-side taking only the changes for the language being edited,
• handle the language switches "statically" by reloading the whole page separately for each language.

I am currently able to interact with the REST API to feed Gutenberg with the content for a given language and re-assemble stuff on the server-side!

What i lack is a way to transmit the current language when the post is submitted. It's done in AJAX through a PUT call on wp-json/wp/v2/posts/<post_id>. But i need to change the POST data so that a qTranslate context is transmitted together with the JSON stuff sent by Gutenberg. For example {status: "publish", content: "gutenberg blocks...", id: <post_id>} but here i want to send a new field telling about the qTranslate context. I haven't found yet how to do that. But i hope it's possible and then we should be able to get a baseline working.

[GasimGasimzada]

One thing I also want to mention is that React manages DOM in a different fashion. All the DOM changes a React application does on a node is initially saved into virtual DOM, which is essentially a JS object. Then, React DOM reconciler updates the real DOM by diffing previous and current state changes. So, if we are going to to update the real DOM that React works on, it will just result in very weird behavior (if we are lucky).

[GasimGasimzada]

What i lack is a way to transmit the current language when the post is submitted. It's done in AJAX through a PUT call on wp-json/wp/v2/posts/. But i need to change the POST data so that a qTranslate context is transmitted together with the JSON stuff sent by Gutenberg. For example {status: "publish", content: "gutenberg blocks...", id: } but here i want to send a new field telling about the qTranslate context. I haven't found yet how to do that. But i hope it's possible and then we should be able to get a baseline working.

Here is one thing I had in mind. If we are implementing every language being a separate page, we can do everything in a database level. Firstly, we can make "raw" property in the REST JSON output to only give us the current language. Secondly, because we are in admin page, we already have all the cookie parameters to access the API. Even the non-raw version gives the correct language based on admin cookies. So, all we need to do is to intercept REST update/insert endpoints, parse the language content into the proper place in the raw data and save it. There should be a filter for that. I will check it tomorrow and let you know.

[herrvigg]

The REST hooks i use are rest_prepare_{$post_type} (for each post type) and rest_request_before_callbacks, i've made some experiments and that works fine.

I'm not sure it's a good idea to use the admin cookie, first i don't like to rely on cookies directly (who wants to read cookies in 2018?) and it's not necessarily the language you want for the edition. Isn't the cookie more for the non-admin parts? Anyway there should be a way to add some hooks with these new wp.hooks API (the equivalent of PHP in JS) but as said it's poorly documented, it lacks some concrete examples.

Just to give you an overview you can check this video: https://www.youtube.com/watch?v=uqoGG9l2BBs

[herrvigg]

Maybe the solution is here: https://github.com/WordPress/gutenberg/issues/4674

But i haven't tried this yet.

[herrvigg]

ok so i'm beginning to understand how it works with Gutenberg. It uses extensively Redux.js so together with React.js this is a prerequiste before being able to do anything smart in there. https://redux.js.org

[herrvigg]

I tried different things. I have a kinda hacky solution with the meta fields but it's not so great. I asked directly on the Gutenberg repo, hoping they can help: https://github.com/WordPress/gutenberg/issues/10078

[herrvigg]

Mmm i think i have a decent solution, it seems to be working at least as a baseline :)

[vvasilev-]

Hi @herrvigg,

I saw your issue in Gutenberg's repository and found that I'm in the same situation as you(I'm working on custom fields library). Could you share some thoughts on the approach that you're currently testing? Thanks in advance. :)

[herrvigg]

hello @vvasilev-

the main idea is to add a new middleware to the React.js stack through wp.apiFetch.use so i can send custom data to the server. Then i can process stuff differently at every update. It works for create, update and auto-save so that looks good.

I would like to do more client-side but it's quite complicated so the main part is in on the server at the moment. I customize the API REST behavior using rest_prepare_{$post_type} and rest_request_before_callbacks.

The entrance ticket with Gutenberg is much higher than it used to be with the old framework. Many developers will not be able to follow. It's very poorly documented. It's quite a risky move for Wordpress if they don't document properly all their stuff and write a bunch of tutorials. There's a huge way to go before it can be used by many.

[vvasilev-]

Hi @herrvigg,

Thanks for sharing! I really appreciate it. I also have hard times with the Gutenberg's documentation, so I end up with storing my custom data directly in the block's comment. On a side note, how do you track the content on the client side? Are you using a custom data store since you're adding the data via middleware?

[herrvigg]

I've been quite busy, i'll come back soon for new updates :)

[herrvigg]

@vvasilev- I've created an experimental branch called exp/gutenberg: https://github.com/qtranslate/qtranslate-xt/tree/exp/gutenberg

It's still very experimental but here is the code for the custom middleware (with some hard-coded values): https://github.com/qtranslate/qtranslate-xt/blob/exp/gutenberg/admin/js/src/editor-gutenberg.js

As gutenberg is in React/ES2015 i'm also using the same coding syntax. This should be transpiled for compatibility with old browsers. I created a minimal npm config just for babel and CLI usage. Note the transpiled versions are not in the repo.

[herrvigg]

All right, great news! It's beginning to work with Gutenberg, woo-hoo!! 🏆

If you are adventurous you can start playing with my experimental Gutenberg branch: https://github.com/qtranslate/qtranslate-xt/tree/exp/gutenberg

Disclaimer: you shouldn't use that in production, only on a development site! Checkout the branch, install the node.js stuff with npm install and build the ES6 JS script running npm run build. You can use the main Gutenberg plugin, no need to use Gutenberg in dev version.

For now it's very basic, there are many limitations but it's a proof of concept. By default the language being edited in Gutenberg will be the current admin language. But you can enforce the language for the post in Gutenberg with a qtx_lang=code param. It's different from the lang param as this is for the admin UI. For example: http://localhost/wp-admin/post.php?post=13571&action=edit&qtx_lang=fr

Current big issues:

• I've spotted some issues after changing the front language (saved in the client cookies). I suspect the troubles are caused by some nasty REST redirections already mentioned before in other tickets. But it seems to works fine if you select the main language on the client-side. If you change to visualize the translated results, be sure to move back when editing the post as admin.
• the whole loop with the server handling is quite complicated because of the REST protocol involving 3 separate steps. I think the multi-lang re-composition has to be done client-side to make it viable!

Note i've disabled the autosaves for now as they are triggering way too often and i would need to test more with that. So if you want to test, better you also install this plugin: https://wordpress.org/plugins/disable-gutenberg-autosave/

[Kano3D]

Now Wordpress 5.0 is released the new Gutenberg editor is enabled by default, and q-translate is not working. How's the progress? Now you can test with the stable version of Gutenberg.

[herrvigg]

Ouch it doesn't work with the new Classic Editor in WP 5.0... that's bad. We'll need a hotfix, for now better wait with the upgrade if you absolutely need qTranslate!

The support for Gutenberg is not ready yet, i didn't do any change since the last news posted here. All is available for experiments and PRs. But it's not like i got much help from you guys ^^'

I mainly worked on other changes to handle the add-ons (WooCommerce and others) and also there's an important fix to validate for the REST API. This is absolutely crucial for Gutenberg so we need this anyway before anything new with Gutenberg. So please help me to validate this: https://github.com/qtranslate/qtranslate-xt/pull/621

On my side i'll try to find a fix for the new Classic Editor plugin.

[Kano3D]

I will help testing

[Kano3D]

Classic editor is working for me Correction: It seems to work, but text from some custom posts doesn't appear on website.

[herrvigg]

Did you install the plugin called "Classic Editor" v1.2 by Wordpress Contributors? It should be the official one. I got this error in the console: tinyMCEPreInit is not defined. The order of the scripts loading has changed and the global variables for the MCE editor now comes after qtranslate. So i don't see how it could work for any post without a fix.

[Kano3D]

Yes. I use the official Classic Editor. I use the following custom post: This is before updating to wp 5.0 (and old q-translate x): http://www.tridimensional.info/products/men-in-black-3/ (note the 3D section below) This is with wp 5.0 and replaced the old q-translate with the xt): http://www.tridimensional.info/products/ant-man-and-the-wasp/ (note the upper section works, but none of the content of 3D appears -it should appear a text and a youtube embed as the first link- so as last resort I copied the content of disappeared section into the upper section, it is not nice, but I need to show the youtube to visitors) UPDATE: I can modify existing posts and work, but it doesn't work on new ones.

[herrvigg]

@Kano3D i have just pushed a candidate fix to master, can you check it (you'll need github though)?

What's behind this:

• a few weeks ago, i changed the ready handler to remove some deprecate warnings. I wrote a wrapper to the jQuery variable to create a closure for $. However it assumed the script could be fully loaded in the footer.
• with the classic editor, i haven't found a way to load these scripts after this plugin. This fix ensures the relevant qTranslate functions are loaded later on, when the DOM object is fully loaded. Imo it's the easiest solution for now.

I tried on my platform and it works. Please let me know if this works for you.

[Kano3D]

I need to uninstall first? It say folder already exist. I will try again tomorrow, it's late

[herrvigg]

You don't need to uninstall. If you had a release (possibly updated with Github updater) you can simply rename the current folder and git clone this repo as long as it ends up in plugins/qtranslate-xt. If you already had a cloned repo then just do git pull. In theory you should deactivate/activate but here there's no need for these changes. I will prepare a release soon but it would be good if some if you can verify the last changes.

[reddo]

Looks like there are filters for default blocks in Gutenberg now: https://wordpress.org/gutenberg/handbook/designers-developers/developers/filters/block-filters/#editor-blocklistblock

[Kano3D]

You don't need to uninstall. If you had a release (possibly updated with Github updater) you can simply rename the current folder and git clone this repo as long as it ends up in plugins/qtranslate-xt. If you already had a cloned repo then just do git pull. In theory you should deactivate/activate but here there's no need for these changes. I will prepare a release soon but it would be good if some if you can verify the last changes.

I can't install github udpater (my provider still didn't updated to the required php version), so I install the zip manually (On plugins - new - upload plugin). But I get the "folder already exist" error. I tried to deactivate plugin but same error, and after that the plugins section didn't load! Luckily I was able to activate again going back with the browser.

If I rename qt-translate-xt master folder before installing plugin I get a full screen error and I can't use wordpress

Update: Ok, I deactivated, renamed, and installed the new updated version. I will test

[reddo]

@Kano3D I usually just delete the plugin folder from the server and upload the new version with the same name as the one I deleted.

[Kano3D]

@Kano3D I usually just delete the plugin folder from the server and upload the new version with the same name as the one I deleted.

I have fear if doing that my site will break

[herrvigg]

Looks like there are filters for default blocks in Gutenberg now: https://wordpress.org/gutenberg/handbook/designers-developers/developers/filters/block-filters/#editor-blocklistblock

The problem with Gutenberg is not the internal block editor itself but the whole API, it has nothing to do with the previous one. But we might find a solution, it's tedious because of the lack of documentation and they clearly didn't care about the previous client-side functionalities, in complete contradiction with what they use to do on the server-side... but we may find a solution eventually, i still have some hope something can be done.

I have fear if doing that my site will break

You will hardly break your site by removing the qTranslate folder, but if you want to be completely safe in production with a manual update, you should first of all put your site in maintenance mode so you don't have any client request, then disable the plugin, do your operations on your files (rename / delete and install) and finally enable the new one.

[JulianCataldo]

I'm using ACF Blocks. You can get translation INSIDE blocks in frontend (but not in backend of course, with all this jquery / react soup). I'm not sure it's easy to do this with traditional way of doing blocks for Gutenberg, but maybe I'm wrong (not really tested).

[herrvigg]

New topic for Gutenberg in #723.