Performance and scaling the number of plantings per plan

[mstenta]

We did an experiment where we imported 250 plantings into a single crop plan, to see how well the current code would perform. The plan UI still worked, but there were noticeable performance issues.

I would break the performance issues I saw down into three categories (based on cause):

Problems

1. Initial load time

Loading the main crop plan page (/farm/plan/%) with 250 plantings takes a long time. In my local dev environment it was taking around 10 seconds to load. This would probably be better on a production server, but maybe not by much.

This is most likely due to the fact that farm_asset_load() and log_load() are being run for every asset (250 assets) and every log (potentially 500 logs, if each planting has both a seeding and transplanting).

It is also due to the fact that we are building a planting form (with fields for crop/variety, seeding date/location/quantity/status, transplanting date/location/quantity/status, harvest details, etc) for EACH planting. This leads to a total page size of around 7 megabytes.

So not only is there time spent on the server loading/building the page, but there is also time spent sending the HTML over the wire from the server to the client.

2. Browser memory consumption

Once the page is loaded, it performs very poorly. I experienced noticeable lag while doing simple things like scrolling, opening planting fieldsets, etc.

In Firefox's Task Manager, the crop plan tab (with 250 plantings) was reportedly using ~256 mb of RAM. This is probably just due to the sheer size of the HTML/DOM, and the JS behaviors that are attached to many form elements.

3. Ajax load time

When performing tasks that trigger Ajax calculations (eg: calculating dates based on "Days to harvest" or "Days to transplant"), it took around 5 seconds for the Ajax request to complete and update the form.

This is probably related to "1. Initial load time" described above, because the form is rebuilt on each Ajax request. But it also may be related to the JavaScript settings that are sent back with each response. I was seeing a huge payload of JS settings related to the date_popup module. It's worth digging more into that to see if it can be improved.

Potential solutions

There are a few things I think we can do to address these issues:

1. Backend improvements

1a: Use Views to load asset/log details

Currently, we are manually running a database query to get all plantings in the plan, and then running farm_asset_load() on each of them, as well as log_load() on each log. This results in a LOT of database queries, as well as lots of other unnecessary logic running (eg: all implementations of hook_entity_load() etc).

A much simpler approach would be to convert the whole page to a View, which would allow us to load all the information we need in a single database query, without any calls to farm_asset_load() or log_load(). This is probably the most impactful change we can make right now to help with server-side performance.

1b: Examine the Ajax response payloads

As noted above, I was seeing a lot of date_popup related content included in the Ajax responses coming back from the server. It would be worth reviewing that to see how much of an impact it has, and if there's any way we can avoid it.

This could also be solved via the "Frontend improvements" described below, by reducing the number of date popup fields on the page. So it might be worth looking into those first.

2. Frontend improvements

2a: Only allow editing one planting at a time

Instead of building a planting form fieldset for every planting, we could consider just building a single one, and loading planting details into it one at a time to edit them. This would have a few tricky aspects to figure out, but would greatly reduce the size of the HTML/DOM we are building.

The biggest disadvantage of this is that you would only be able to edit one planting at a time. Maybe this is OK? It's nice that you can open multiple fieldsets at once right now, but perhaps that doesn't justify the performance hit.

2b: Use Vue

We have been talking about eventually moving towards converting the timeline/form UI to a Vue.js app. This could greatly improve the frontend performance, but it is also a lot of work. That said, it is where we will probably go with this module regardless, so we can also consider punting some of these improvements until then, and just recommend that users make multiple plans to limit the number of plantings in each.

[mstenta]

Given all of the above, I think we could fix all of the issues with these two changes:

1. Use Views to load asset/log details
2. Only allow editing one planting at a time

[mstenta]

Use Views to load asset/log details

I created a new issue for this specifically: https://github.com/mstenta/farm_crop_plan/issues/14

Only allow editing one planting at a time

I'd like to get some other thoughts on this before we decide to move forward with it.

Is it OK to remove this ability, and assume that we will get it back when we move to using Vue?

Is the effort required to load planting details into a single fieldset worth it? Or should we just wait for Vue?

[mstenta]

We are in the process of porting this module to farmOS v3, and moving it to https://www.drupal.org/project/farm_crop_plan, along with any open issues that are still relevant.

The new module is taking different approaches to everything described here, but will probably have it's own set of performance and scaling questions. I'm going to close this as "not planned" but I expect we'll open an issue in the new queue to address specific things.