Refactor General Indexing Wordpress/Woocommerce plugin with doofeeds

[albertodoofinder]

Adapt the woocommerce plugin as it is more up to date.

The proposal is: Unlink from doofAPI in the plugin using indexing via the feed we generated in wordpress. Modify the configuration of the search_engine so that it is not API type. With this we unload the plugin from the indexing responsibility and we get closer to the model we have in Magento, Prestashop, etc.

We keep in the plugin the login/signup and the saving in the DB of the configuration.

We would have to leave some endpoints in doomanager ready for the update on save.

The multi-language indexing is maintained in the model of 1 search_engine per language. *We need to investigate the generation of the feed per language.

Document code. Create / update functional tests. Create Indexation mermaid diagrams in doomanager repo.

blocks https://github.com/doofinder/doofinder-wordpress/issues/54 https://github.com/doofinder/doofinder-wordpress/issues/43

[eduardogomez97]

We have to find a way to improve how we relate multilanguage to indexing because right now it's a bit chaotic. Apart from that we have WPML and supposedly Polylang support (although I think this one needs to be checked more thoroughly).

Right now the indexing is a monolith of code, we can simplify it and try to clean up the code as much as possible. Taking into account functions that have been deprecated without being deleted and especially parts of the code that generate errors (Many nested ifs and exception catches):

There is a lot of repetitive code that can be simplified into a generic function.

For example the issue of saving the temporary index in database. Maybe the Post_id field doesn't make sense either, since saving the last indexed one is to avoid reindexing everything. However, it is possible to make a change that is left out. It also does not make sense because we reset that field, so in my opinion it does not make sense.

The indexing status part was made so that indexing could be paused and resumed (that's why the post_id is also needed).

But is this functionality worth it? Isn't it better if for the indexing to be relaunched?

[eduardogomez97]

After meeting @albertodoofinder and @pedromcp90 and discussing the best options we have come to the conclusion that it is better to adapt the woocommerce plugin as it is more up to date.

The proposal is: Unlink from doofAPI in the plugin using indexing via the feed we generated in wordpress. Modify the configuration of the search_engine so that it is not API type. With this we unload the plugin from the indexing responsibility and we get closer to the model we have in Magento, Prestashop, etc.

We keep in the plugin the login/signup and the saving in the DB of the configuration.

We would have to leave some endpoints in doomanager ready for the update on save.

The multi-language indexing is maintained in the model of 1 search_engine per language. *We need to investigate the generation of the feed per language.

[eduardogomez97]

One functionality that would be good to keep is to choose the post_type we want to index. This translates to create an index per selected post_type.

This option would be lost with the refactor, unless we add the option to choose the post_types we want to index during the installation to create the payload with that data. Also leave a section in the plugin control panel where you can select the "indexes" you want to create and a button that launches the call to create those that do not exist.

However, it would be necessary to investigate if wordpress has enabled the API to obtain the data of the different types.

You would also have to generate a feed for each type selected if indexing is done via feed_url.

[eduardogomez97]

We have to investigate if the change of model would affect multi-language stuff since we usually translate the posts with a plugin called WPML so maybe through the API we can't get the data. What is more feasible is that a feed can be created with the data in the corresponding language.

[eduardogomez97]

When indexing the feed can be done in 3 ways:

• Through the wordpress API and woocommerce API.
• Through the creation of the different feeds (such as prestashop)
• Through doofAPI with the BULK system that we are already using.

WORDPRESS & WOOCOMMERCE API To get the feed you can use the wordpress API and if you activate woocommerce with its API we can also get the products. The problem of using the API's is that in the case of Wordpress, although by default it is accessible, the clients usually have it capped with a user/password system.

In the woocommerce API we have the problem that the customer has to activate it (by default it is not activated) and create a KEY by hand.

This means that the client has to intervene in the installation and it will complicate our work if the client has problems with permissions, etc.

WORDPRESS API https://developer.wordpress.org/rest-api/reference/ WOOCOMMERCE API: https://woocommerce.github.io/woocommerce-rest-api-docs/?shell#introduction

CREATION OF FEEDS We can create a feed per language and index via url in doomanager. This seems optimal. The only problem with this is that right now we use a system that allows you to decide which post_types you can choose. However the feeds would only be created once. So we can create the most used feeds (posts and pages) plus products if you have woocommerce installed and enable a section that is to create new feeds. For example if you have a post type that is projects to create the feed.

In the case of this option we would remove a lot of code from the plugin because everything would be launched from doomanager.

KEEP THE BULK This option would keep all the functionality in the plugin. Instead of removing the code monolith we would have to do a major refactor and add the woocommerce part to the wordpress part. Which adds a lot of complexity, but would keep all the options as they are now.

[eduardogomez97]

After talking it over with @agutierrezrodriguez and explaining the options, we think it's a good idea to go for the wordpres & woocommerce API's way.

In this case warn in the requirements that you need to have the API's active and well configured. It would be indicated with a message in the plugin installation.

This would have the disadvantage that I have commented above. In case of having to use woocommerce the client has to activate and generate the key manually adding it in the installation process or later in the configuration.

It would slow down the process a bit, however it would save us a lot of code from the plugin.

I will discuss this with @CristobalDolz to see the impact at the product level.

[CristobalDolz]

@albertodoofinder @eduardogomez97 @agutierrezrodriguez From what Eduardo has told me regarding the most optimal option (1), that option is not viable. It changes the installation process and makes it worse. That would imply many things: documentation changes, sales callflow changes, worse sales pitch, etc etc.

Also, when a client is new and is installing the plugin for the first time, they have not entered doofinder yet, they do not have a store created, therefore they do not have an API KEY to copy to finish the installation process that you are proposing.

[eduardogomez97]

After the meeting the other day it was decided to opt for the REST API version of wordpress, since in the most recent versions it is integrated in the core and also includes the use of the woocommerce API.

I have been investigating and I have found the way to create that password by means of code, thanks to the fact that it is part of the core.

You would have to use the WP_Application_Passwords class.

Also implement the call to the api you must indicate in the header the user and the password.

It is necessary to associate this password to a user, which is the admin who installs the plugin. After discussing this with Antonio this does not seem to be a problem in principle.

[eduardogomez97]

The indexing part would be done through the wordpress API. So I am creating the process. Doomanager > Doofeeds > Items_transformation

We are basing this on the process currently being used for Shopify. When installing the plugin in the payload the different types of indexes are indicated and in the datasource.options the feed_type is stored. This data is key when indexing to be able to build the feed with the necessary fields according to the type because a product does not need the same attributes as a blog post.

Once the task has been launched, depending on the data in store.options, the processing of the data in doofeeds is done in one context or another. To distinguish those of wordpress in the payload we will have added the two key data for authentication. "api_user" and "api_password".

We process the feed from item_transformation according to the type to build the api call which would be the shop_url plus the path of the API call.

For example if we detect that the feed to process is of type product the path would be generated as follows:

shop_url <> /wp-json/wc/v3/products

http://687955d2-a742-4170-8388-0f573d0dc89d.clouding.host/wp-json/wc/v3/products

Once the data is obtained, it is constructed.

Example of the data obtained with the API of a product: { "id": 29, "name": "Single Edición 3", "slug": "single", "permalink": "https://687955d2-a742-4170-8388-0f573d0dc89d.clouding.host/producto/single/", "date_created": "2022-02-02T09:54:51", "date_created_gmt": "2022-02-02T09:54:51", "date_modified": "2022-10-27T08:04:14", "date_modified_gmt": "2022-10-27T08:04:14", "type": "simple", "status": "publish", "featured": false, "catalog_visibility": "visible", "description": "<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vestibulum sagittis orci ac odio dictum tincidunt. Donec ut metus leo. Class aptent taciti sociosqu ad litora torquent per conubia nostra, per inceptos himenaeos. Sed luctus, dui eu sagittis sodales, nulla nibh sagittis augue, vel porttitor diam enim non metus. Vestibulum aliquam augue neque. Phasellus tincidunt odio eget ullamcorper efficitur. Cras placerat ut turpis pellentesque vulputate. Nam sed consequat tortor. Curabitur finibus sapien dolor. Ut eleifend tellus nec erat pulvinar dignissim. Nam non arcu purus. Vivamus et massa massa.</p>\n", "short_description": "<p>This is a simple, virtual product.</p>\n", "sku": "woo-single", "price": "2", "regular_price": "3", "sale_price": "2", "date_on_sale_from": null, "date_on_sale_from_gmt": null, "date_on_sale_to": null, "date_on_sale_to_gmt": null, "on_sale": true, "purchasable": true, "total_sales": 0, "virtual": true, "downloadable": true, "downloads": [ { "id": "c7778f5d-853d-4a93-9ecf-383ac4b9b58f", "name": "Single", "file": "https://demo.woothemes.com/woocommerce/wp-content/uploads/sites/56/2017/08/single.jpg" } ], "download_limit": 1, "download_expiry": 1, "external_url": "", "button_text": "", "tax_status": "taxable", "tax_class": "", "manage_stock": false, "stock_quantity": null, "backorders": "no", "backorders_allowed": false, "backordered": false, "low_stock_amount": null, "sold_individually": false, "weight": "", "dimensions": { "length": "", "width": "", "height": "" }, "shipping_required": false, "shipping_taxable": false, "shipping_class": "", "shipping_class_id": 0, "reviews_allowed": true, "average_rating": "0.00", "rating_count": 0, "upsell_ids": [], "cross_sell_ids": [], "parent_id": 0, "purchase_note": "", "categories": [ { "id": 22, "name": "Music", "slug": "music" } ], "tags": [], "images": [ { "id": 73, "date_created": "2022-02-02T09:55:04", "date_created_gmt": "2022-02-02T09:55:04", "date_modified": "2022-02-02T09:55:04", "date_modified_gmt": "2022-02-02T09:55:04", "src": "https://687955d2-a742-4170-8388-0f573d0dc89d.clouding.host/wp-content/uploads/2022/02/single-1.jpg", "name": "single-1.jpg", "alt": "" } ], "attributes": [], "default_attributes": [], "variations": [], "grouped_products": [], "menu_order": 0, "price_html": "<del aria-hidden=\"true\"><span class=\"woocommerce-Price-amount amount\"><bdi>3,00&nbsp;<span class=\"woocommerce-Price-currencySymbol\">&euro;</span></bdi></span></del> <ins><span class=\"woocommerce-Price-amount amount\"><bdi>2,00&nbsp;<span class=\"woocommerce-Price-currencySymbol\">&euro;</span></bdi></span></ins>", "related_ids": [ 28, 107 ], "meta_data": [ { "id": 284, "key": "_original_id", "value": "75" }, { "id": 658, "key": "_wpml_media_duplicate", "value": "1" }, { "id": 664, "key": "_wpcom_is_markdown", "value": "1" }, { "id": 936, "key": "_wpml_media_featured", "value": "0" } ], "stock_status": "instock", "has_options": false, "_links": { "self": [ { "href": "https://687955d2-a742-4170-8388-0f573d0dc89d.clouding.host/wp-json/wc/v3/products/29" } ], "collection": [ { "href": "https://687955d2-a742-4170-8388-0f573d0dc89d.clouding.host/wp-json/wc/v3/products" } ] } },