Several problems with Manifest that have a large number

[egbot]

Several issues include:

• Manifests with large numbers of records are slow to load and display. It is not uncommon for the browser to ask if you want to abort the task. Sortable table functionality compounds the issue.
• Records greater than 10 records fail to process due to PHP server configuration settings

Possible solutions:

• Increase post_max_size PHP variable. Currently set at 10K. Shipment #2032 has 14,158 sample records, thus requiring variable to be increased to 15K. Setting it to 0 would disable this limitation, but not recommended.
• Break large manifests into 2-3 separate manifests of no more than 5,000 records. However, this would require workers to process records in batches.
• Add pagination to sample display table. Displaying only 5,000 records at a time would improve speed of display and solve post_max_size issue. This would require workers to process records in batches. Sortable table functions would be limited only to records loaded within table, unless that function was expanded. Using the table sort JS library used within occurrence editor might also resolve this issue.

[sunray1]

^^ yes, it's quite easy to add pagination with the current library. Can be reduced to however many number of records we'd like. That being said since it's js, I think all those records would still be loaded up front and im not sure if that would speed up loading times.

[egbot]

We discussed this issue within the 2024-08-1 biorepo IT meeting and decided that adding pagination or creating multiple sub-manifests would create restrictions and issues with samples check-in process. It was decided that load tests were need to determine which processes of loading manifests were most restrictive.

[egbot]

Load testing update:

When loading manifest with 5,716 records:

• Server-side PHP process: fraction of a second
• Loading data: 5s
• Rendering: 24s
• Scripting: 51s

Load tests with the JS table sort libraries deactivated rendered results 3-14 times faster. Manifest with > 10,000 records often timed out with JS library activated, but rendered result within 15-20 seconds when deactivated. Specific test results displayed below.

Shipment #888

• 5,716 records
• Without sorting: 5-24s
• With sorting: 35-85s, but more often around 50s

Shipment #899

• 7,875 records
• Without sorting: 7-9s
• With sorting: 63-65s

Shipment #2392

• 12,311 records
• Without sorting: 11-12s
• With sorting: 148-241s, but often times out

Shipment #2032

• 14,158 records
• Without sorting: 14-15s
• With sorting: typically, times out

[egbot]

Load profile viewing manifest #2032 (>14K records) with table sort feature turned off.

[egbot]

I propose that we add a checkbox or radio buttons that turns on/off the table sort feature. The sort option can be turned on by default, but if the manifest has > 3-5K records, the sort option should be turned off on first load, with the user having the option to reload the manifest with the sort option selected.

An other option should be to refactor the sort option to take place server-side. The downside would be that the page would have to be reloaded anytime a column was sorted, but the pages would load fast.

[egbot]

Issue partially resolved with: https://github.com/BioKIC/NEON-Biorepository/commit/8bbad22c5a984c4bc3cf7933bded8144d9a875c0

[egbot]

Working with Greg to increase POST variable limit within PHP config file