Implement Cron Job for periodic property data updates

[fredericalpers]

Implement Cron Job for periodic property and address data updates

Introduction

The current architecture of the plugin requires API requests to generate content each time a page is called. This process starts by finding the page and then identifying the shortcode. The settings for this shortcode are then read from the WordPress database. Corresponding API requests are then sent to onOffice enterprise to retrieve the required data. Once the data has been received, it is inserted into the corresponding template. Finally, the template is executed and the dynamic content is displayed on the website.

User Story

As a real estate agent, it is important to me to provide a quality website that offers my clients a smooth and efficient search experience. I have found that current load times can be very slow for page views with property or address data. In order to improve website performance and optimize the user experience for my clients, I would like to implement a regular cron job that updates the property or address data in the background.

Implementing the cron job to update property or address data in the background will improve website performance and optimize the user experience, which will help increase my clients' level of trust and engagement on the website.

Requirements / Tasks

• [ ] Analyze the current plugin architecture and identify the performance issues caused by the repeated API requests.
• [ ] Development of a cron job for background updating of property and address data to reduce server load and optimize page load times.
• [ ] Configure the cron job to run at regular intervals and update the database with current property data.
• [ ] Thorough testing of the cron job to ensure proper function and reliability.

[yeneastgate]

@fredericalpers After investigation, I want to confirm with you this task's requirements help me suggest a suitable solution.

• Analyze the current plugin architecture and identify the performance issues caused by the repeated API requests.

With the current plugin architecture, based on the "clear cache" feature and the "oo_plugin_cache" table in the database, I check that:

• An API has been sent "multiple times with different parameters" to onOffice enterprise to retrieve the required data. Ex: When loading "estate list" page fields(2 times), searchCriteriaFields(one time), idsfromrelation(one time), estatepictures(one time), regions(one time), estateCategories(one time), estate(three-time)
• In my opinion, sending "repeated API requests" to onOffice enterprise => increased Server Load(onOffice enterprise load), degraded server performance, or even server downtime during peak usage periods. => slower response API times for clients => performance issues(load times can be very slow for page views with property or address data).

• Development of a cron job for background updating of property and address data to reduce server load and optimize page load times.

Development of a cron job for background updating of properties and address data As I understand it:

• Do you want to create a cronjob with the same "oo_cache_cleanup" hook action and schedule a hook using the "wp_schedule_event" method?
• Do you only want to update properties retrieved by using the "real estate" API Send a request to onOffice enterprise API(https://apidoc.onoffice.de/actions/datensatz-lesen/objekte/)
• Do you only want to update address data which is retrieved by using the "read address" API sent request to onOffice enterprise API(https://apidoc.onoffice.de/actions/datensatz-lesen/adressen/))

• Configure the cron job to run at regular intervals and update the database with current property data.

• How long do you want to set for the "$recurrence" parameter in the "wp_schedule_event" method?
• Do you want to implement the "update the property and address data" duration setting in the admin backend onOffice plugin, the same feature for the "current cache" feature(https://github.com/onOffice-Web-Org/oo-wp-plugin/issues/720)
• "current property data" in this phrase "update the database with current property data" means: "current property data" retrieves the latest data in onOffice enterprise at the time of page load and within the time set for the "$recurrence" variable above.

[fredericalpers]

@yeneastgate At the moment all data is retrieved on the first page request. Our aim is to outsource this to a cron job. This way, no more data has to be retrieved when the page is called, as this will be done by the cron job. If the process then runs in the cron-job the performance issues are not noticable to the end user/front end user.

We would like to keep the requests but outsource all data retrieving into the cron job, as mentioned in your comment below. Also we would like to adjust this to oo_cache_renew instead of oo_cache_cleanup.

An API has been sent "multiple times with different parameters" to onOffice enterprise to retrieve the required data. Ex: When loading "estate list" page fields(2 times), searchCriteriaFields(one time), idsfromrelation(one time), estatepictures(one time), regions(one time), estateCategories(one time), estate(three-time) In my opinion, sending "repeated API requests" to onOffice enterprise => increased Server Load(onOffice enterprise load), degraded server performance, or even server downtime during peak usage periods. => slower response API times for clients => performance issues(load times can be very slow for page views with property or address data).

Do you want to implement the "update the property and address data" duration setting in the admin backend onOffice plugin, the same feature for the "current cache" feature

The settings for the cron job recurrence should be as they were initially thought of in #720

[yeneastgate]

@fredericalpers

At the moment all data is retrieved on the first page request. Our aim is to outsource this to a cron job

• When the page is loaded for the first time and the cache is not yet available in the database, we cannot determine the necessary parameters to accurately call the API, so it cannot be executed.

• Each time the cron job runs, we will only update all the data, including both new data (which has been recently updated on the office system) and old data.

• On the first page request and there is a cache record in the database, I have implemented a cron job to handle this case. Here are the steps to implement it:

Step 1: First, set the timing for the cron job, for example, every 10 minutes. Step 2: Then, retrieve the latest data on the enterprise to save the cache into the database Step 3: According to the cron job settings, after 10 minutes, the system will update the database with the latest data from the onOffice enterprise. => This ensures that users accessing the data for the first time will receive this latest information directly from the API without needing to call the API again, resulting in faster loading times.

You can review the code and test it in this branch 46528-implement-Cron-Job-for-periodic-property-data-updates. This is a demo video:

https://github.com/onOffice-Web-Org/oo-wp-plugin/assets/104892108/0d33de8e-cdc4-4416-9ca0-be5630ba27c4

[fredericalpers]

@yeneastgate thank you, we will review this and give you feedback as soon as possible :)

[yeneastgate]

@fredericalpers This is the website performance result when running a cron job with big data(485) on the first page request and having cache record in the database This is video test with big data:

https://github.com/onOffice-Web-Org/oo-wp-plugin/assets/104892108/7f664f2c-5d09-47f4-afe1-4d9fecb984d3

Note: While implementing this feature, when testing big data, calling repeated API requests on the "custom label feature" of fields in those screens related to estates (estate list view, detail view, similar estate) has reduced website performance This is the video recording of this issue: https://files.fm/u/n64gt6j98z#/view/zjvpgwgu2p

We have fixed it, please watch the demo video: https://files.fm/u/f3x5z97jc3#/view/2r8cvftut7

[fredericalpers]

@yeneastgate thank you for the update :)

[yeneastgate]

@fredericalpers Here is a summary of what we have implemented in this task, so you can have an overview of the feasibility of this issue:

Analyze the current plugin architecture and identify the performance issues caused by the repeated API requests.

• We have fixed an issue related to repeated API requests. This is the evidence for this issue.

https://github.com/onOffice-Web-Org/oo-wp-plugin/assets/104892108/39a4b994-0092-4c40-b86e-c995b12181a5

=> The loading time of the Estate List has been decreased from 25.65s ( the "master" branch) to 5.31s ( "46528-implement-Cron-Job-for-periodic-property-data-updates" branch) .

Development of a cron job for background updating of property and address data to reduce server load and optimize page load times.

• We implemented the cron job and tested with big data (ex: 485 data records) on the first page request and there is a cache record in the database.

Master branch

https://github.com/onOffice-Web-Org/oo-wp-plugin/assets/104892108/be9bef91-aeab-4711-ba57-35541997bf05

46528-implement-Cron-Job-for-periodic-property-data-updates branch

https://files.fm/u/h5c5skreez

=> The loading time of the Estate List has been decreased from 1.9m ( the "master" branch) to 25.30s ( "46528-implement-Cron-Job-for-periodic-property-data-updates" branch).

Configure the cron job to run at regular intervals and update the database with current property data.

• We have implemented scheduled tasks to run the cron job. The setting time is presented as below

We will come back to this issue and continue to implement it when you review and give feedback for our solutions.