search

fisharebest / webtrees

Online genealogy
https://webtrees.net
GNU General Public License v3.0
478 stars 301 forks source link

Allowed memory size exhausted #4839

Open hartenthaler opened 1 year ago

hartenthaler commented 1 year ago

I'm searching for a solution to a problem of a friend of mine. She has a GEDCOM with about 70,000 people. The value of memory_limit is set to 512M (maximum of that hoster package). webtrees 2.1.16 is used.

Problem 1: Selecting a source that is related to about 10,000 persons and 3,000 families. All other sources (with fewer persons+families) can be selected.

Allowed memory size of 536870912 bytes exhausted (tried to allocate 60571648 bytes) /xxx/ahnen/webtrees/resources/views/record-page-links.phtml: 125

Problem 2: Exporting this GEDCOM tree as member is not possible. Exporting as admin is possible.

Allowed memory size of 536870912 bytes exhausted (tried to allocate 23068912 bytes) /xxx/ahnen/webtrees/app/Services/GedcomExportService.php: 267

What could be the problem behind those two similar error messages?

hartenthaler commented 1 year ago

This is maybe related to #3656 because there are many RESN tags used in this GEDCOM file.

fisharebest commented 1 year ago

1) You are trying to load and format 13000 records. For any amount of memory, there will always be a maximum number of records that can be displayed.

2) Exporting as admin does not need to do any privacy filtering. We simply copy data from the database to the GEDCOM file. This design allows you to export very large trees with very little memory. But when you apply privacy filtering (export as member), then you must load each record into memory and calculate the privacy. This records memory for each record. Again, for any amount of memory, there will always be a maximum number of records that can be processed.

Do you have any suggestions for changes that will reduce the memory requirements?

hartenthaler commented 1 year ago

You are right, there is always a limit.

But when you apply privacy filtering (export as member), then you must load each record into memory and calculate the privacy.

She told me, that she is still able to export the same tree as member when using webtrees 1.7. Was there a significant change when calculating the privacy from 1.7 to 2? Maybe this can give a hint of how to reduce the memory requirements. At the moment I suggested a workaround: export in webtrees as admin. Use another program to eliminate confidential information. Then transfer the result to the final (public) database. This is working (ok, one additional step every 4 weeks).