mkroeders commented 11 months ago

When running psalm on microsoft/microsoft-graph it crashes due to spawnining multiple, ~10, PHP processes. Each process continuously increases it's memory footprint, beginning from 0,5GB to over 5GB each. Which effectively makes makes it run out of memory;

Composer.json;

{
  "require": {
    "microsoft/microsoft-graph": "^2.1"
  },
  "require-dev": {
    "vimeo/psalm": "^5.0.0"
  },
  "config": {
    "allow-plugins": {
      "php-http/discovery": false
    }
  }
}

psalm.xml

<?xml version="1.0"?>
<psalm
        xsi:schemaLocation="https://getpsalm.org/schema/config vendor/vimeo/psalm/config.xsd"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xmlns="https://getpsalm.org/schema/config"
        errorLevel="1"
>
    <projectFiles>
        <ignoreFiles>
            <directory name="vendor"/>
        </ignoreFiles>
    </projectFiles>
    <issueHandlers>
        <MixedAssignment errorLevel="suppress"/>
    </issueHandlers>
</psalm>

test.php

<?php
declare(strict_types=1);

require __DIR__ . '/vendor/autoload.php';

use Microsoft\Graph\GraphServiceClient;

$graphServiceClient = new GraphServiceClient($tokenContext, $scopes);

First thought the issue was with PHPstorm so reported it there, bug report.

psalm-github-bot[bot] commented 11 months ago

Hey @mkroeders, can you reproduce the issue on https://psalm.dev ?

orklah commented 11 months ago

Well, a known issue is that Psalm threads will follow the php.ini directive (or use the memory-limit option if provided) for each thread.

While each thread individually will probably consume less memory than analyzing everything in the same thread, it will inevitably have an overlap.

Psalm can also be pretty resource consuming on big codebases. Both of these combined can eat up a laptop memory pretty fast. Have you tried running in a single thread?

mkroeders commented 11 months ago

If I use the option --threads=1 I get a memory error;

PHP Fatal error:  Allowed memory size of 8589934592 bytes exhausted (tried to allocate 20480 bytes) in /vendor/vimeo/psalm/src/Psalm/Internal/Cache.php on line 76
Fatal error: Allowed memory size of 8589934592 bytes exhausted (tried to allocate 20480 bytes) in /vendor/vimeo/psalm/src/Psalm/Internal/Cache.php on line 76

However when I do php -i | grep memory_limit the result is memory_limit => 128M => 128M. Which to my understanding means there is a memory limit of 128MB, so how it even reaches 8GB is a mystery to me

Nikolay-Ivanchenko commented 11 months ago

I have the same problem

If you delete the directory with the cache, it starts 1 time. Then it falls into error again

orklah commented 11 months ago

Can you try with --no-cache and --threads=1 ?

how it even reaches 8GB is a mystery to me

I forgot that Psalm does that on its own, unless you tell it explicitly to follow the ini directive: https://github.com/vimeo/psalm/blob/afc2df424fbf6552eacd8db80d36ff4b33a2646a/src/Psalm/Internal/CliUtils.php#L466

mkroeders commented 11 months ago

Tested with --no-cache and --threads=1 got me the following total command

./vendor/bin/psalm --output-format=checkstyle -c ./psalm.xml --show-info=true --find-unused-code --no-cache --find-unused-psalm-suppress --monochrome --threads=1 src/test.php

Which resulted in finishing the check so that works.

Eye balling the activity monitor for memory usage I saw it spike to 2.47 GB.

mkroeders commented 11 months ago

Not knowing the exact in and outs. But looking at how threads is detected. I assume that the amount of threads is based upon the cpu count - 1. So is it not also possible to detect the max amount of memory of the system, so instead of 8GB it can be a more educated guess?

Which got me thinking perhaps the max memory should be based upon the max amount of threads in combination with max memory. So the total combination of threads * max memory does not exceed the max system memory

sj-i commented 11 months ago

Apart from that, I think I have found a cause and workaround for the unnatural increase in memory consumption when using parallel processing and caching. I will send a report and PR later.

sj-i commented 11 months ago

Psalm uses \serialize() and \unserialize() for both caching and communications with workers. Due to the current behavior of the PHP VM, each unserialized object is always having unnecessary memory overhead of dynamic properties table. Psalm unserializes large number of objects representing syntactic constructs or types, so the cumulated overhead becomes huge.

A possible workaround for now is to define __unserialize() on each unserialized class having many instances, to bypass the creation of dynamic porperties tables.

I'm here from the GitHub search to test our memory profiler (Reli 0.11.2). "Allowed memory size of" is my personal trending word these days. I've done some investigation about this issue, so I share it.

Preconditions

Additional Test script

analyzer.php

<?php
register_shutdown_function(
    function (): void {
        $error = error_get_last();
        if (is_null($error)) {
            return;
        }
        if (strpos($error['message'], 'Allowed memory size of') !== 0) {
            return;
        }
        $pid = getmypid();
        $file_opt = '--memory-limit-error-file=' . escapeshellarg($error['file']);
        $line_opt = '--memory-limit-error-line=' . escapeshellarg($error['line']);
        system("docker run --entrypoint=php -it --security-opt=\"apparmor=unconfined\" --cap-add=SYS_PTRACE --pid=host reliforp/reli-prof -dmemory_limit=16G ./reli i:m -p {$pid} --no-stop-process {$file_opt} {$line_opt} >{$pid}.memory_analyzed.json");
    }
);

This code snippet kicks Reli on memory limit violations via docker, to analyze the memory usage of the dying process itself.

Note that as the memory analyzing code of Reli itself isn't very memory efficient currently, I use 16GB memory_limit for the process of Reli on this test.

The PHP VM used to test

PHP version

php --version

PHP 8.2.13 (cli) (built: Jan  7 2024 07:32:17) (NTS)
Copyright (c) The PHP Group
Zend Engine v4.2.13, Copyright (c) Zend Technologies
    with Zend OPcache v8.2.13, Copyright (c), by Zend Technologies

php-memprof

php -m | grep memprof

memprof

This time I use php-memprof in combination with Reli. Reli is used to analyze where the most memory is used, and php-memprof is used to find out when the memory space is allocated.

Run with 512M memory limit

To avoid Reli running out of memory, this time I reserve only 512M for psalm.

I run psalm twice: the first run is used to analyse the effects of parallel processing only, while the second run is used to analyse the effects of the cache together.

The 1st run

MEMPROF_PROFILE=dump_on_limit php -dauto_prepend_file=analyzer.php ./vendor/bin/psalm --memory-limit=512M --threads=2 test.php

Target PHP version: 8.2 (inferred from current PHP version).
Scanning files...
PHP Fatal error:  Allowed memory size of 536870912 bytes exhausted (tried to allocate 20480 bytes) (memprof dumped to /tmp/memprof.callgrind.1787474019042074) in /home/sji/work/oss/tmp/psalm_memory_test/vendor/vimeo/psalm/src/Psalm/Internal/Fork/Pool.php on line 358
Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 20480 bytes) (memprof dumped to /tmp/memprof.callgrind.1787474019042074) in /home/sji/work/oss/tmp/psalm_memory_test/vendor/vimeo/psalm/src/Psalm/Internal/Fork/Pool.php on line 358

The output of memprof

kcachegrind /tmp/memprof.callgrind.1787474019042074

first-run

As you can see, the amount of memory allocated during \unserialize() seems to be large: Psalm uses serialisation to communicate with workers, so it seems natural that the more data exchanged, the more area is allocated by \unserialize().

However, is this really the case?

The output from Reli

memory_analyzed_reli=14258.memory_analyzed.json

Extracting the summary

cat $memory_analyzed_reli | jq .summary

[
  {
    "zend_mm_heap_total": 536285184,
    "zend_mm_heap_usage": 400364587,
    "zend_mm_chunk_total": 283115520,
    "zend_mm_chunk_usage": 261628896,
    "zend_mm_huge_total": 253169664,
    "zend_mm_huge_usage": 138735691,
    "vm_stack_total": 262144,
    "vm_stack_usage": 10512,
    "compiler_arena_total": 1638400,
    "compiler_arena_usage": 1550992,
    "possible_allocation_overhead_total": 17044713,
    "possible_array_overhead_total": 82895216,
    "memory_get_usage": 533571712,
    "memory_get_real_usage": 536285184,
    "cached_chunks_size": 0,
    "heap_memory_analyzed_percentage": 75.03482249823618,
    "php_version": "v82",
    "analyzer": "reli 0.11.2"
  }
]

Reli can analyze 75% of the memory usage this time. Not too bad.

Top 20 most memory-consuming types of memory locations

cat $memory_analyzed_reli | jq .location_types_summary | jq -r '(["location_type", "count", "memory_usage"] | (., map(length*"="))),(to_entries|.[:20]|.[]|[.key,.value.count,.value.memory_usage])|@tsv' | column -t -o ' | '

location_type                           | count  | memory_usage
=============                           | =====  | ============
ZendStringMemoryLocation                | 255012 | 159411898
ZendArrayTableOverheadMemoryLocation    | 134612 | 82807856
ZendArrayTableMemoryLocation            | 135803 | 77237144
ZendObjectMemoryLocation                | 127516 | 47748656
ZendArrayMemoryLocation                 | 139649 | 7820344
ZendOpArrayBodyMemoryLocation           | 3484   | 4899952
ObjectsStoreMemoryLocation              | 1      | 1048576
ZendOpArrayHeaderMemoryLocation         | 3748   | 929504
RuntimeCacheMemoryLocation              | 1500   | 308744
ZendClassEntryMemoryLocation            | 479    | 241416
ZendArgInfosMemoryLocation              | 3402   | 217312
ZendPropertyInfoMemoryLocation          | 1423   | 79688
ZendReferenceMemoryLocation             | 2436   | 77952
LocalVariableNameTableMemoryLocation    | 2606   | 75272
DefaultPropertiesTableMemoryLocation    | 306    | 31840
ZendClassConstantMemoryLocation         | 612    | 24480
CallFrameVariableTableMemoryLocation    | 11     | 9632
DynamicFuncDefsTableMemoryLocation      | 103    | 5872
DefaultStaticMembersTableMemoryLocation | 52     | 1984
StaticMembersTableMemoryLocation        | 41     | 1616

Strings occupies the most memory space, while the combination of array elements and unused areas of arrays consumes as much memory.

Top 10 largest strings

cat $memory_analyzed_reli | jq '. as $root | path(..|objects|select(."#type"=="StringContext"))| . as $path | $root|getpath($path) as $string | {path: $path|join("."), size: $string."#locations"[0].size, value: $string."#locations"[0].value, node_id:$string."#node_id"}' | jq -rs '(["size", "node_id" ,"path", "value"] | (., map(length*"="))),(sort_by(.size,.value) | .[-10:] | reverse | .[] | [.size, .node_id, .path, .value])|@tsv' | column -t -o ' | ' -s$'\t'

size     | node_id | path                                                                                                                                                                                                                                                      | value
====     | ======= | ====                                                                                                                                                                                                                                                      | =====
93450252 | 14396   | context.call_frames.2.local_variables.serialized_messages.referenced.array_elements.0.value                                                                                                                                                               | Tzo0MjoiUHNhbG1cSW50ZXJuYWxcRm9ya1xGb3JrUHJvY2Vzc0RvbmVNZXNzYWdlIjoxOntzOjQ6ImRhdGEiO2E6MTI6e3M6MTU6ImNsYXNzbGlrZXNfZGF0YSI7YTo5OntpOjA7YToxMTM3OntzOjE2OiJpbnRlcm5hbGl0ZXJhdG9yIjtiOjE7czo5OiJleGNlcHRpb24iO2I6MTtzOjE0OiJlcnJvcmV4Y2VwdGlvbiI7YjoxO3M6NToiZXJy
45285439 | 14377   | context.call_frames.2.local_variables.content.array_elements.0.value                                                                                                                                                                                      | Tzo0MjoiUHNhbG1cSW50ZXJuYWxcRm9ya1xGb3JrUHJvY2Vzc0RvbmVNZXNzYWdlIjoxOntzOjQ6ImRhdGEiO2E6MTI6e3M6MTU6ImNsYXNzbGlrZXNfZGF0YSI7YTo5OntpOjA7YToxMTM4OntzOjE2OiJpbnRlcm5hbGl0ZXJhdG9yIjtiOjE7czo5OiJleGNlcHRpb24iO2I6MTtzOjE0OiJlcnJvcmV4Y2VwdGlvbiI7YjoxO3M6NToiZXJy
168289   | 59466   | context.class_table.psalm\\internal\\provider\\fileprovider.static_properties.open_files.array_elements./home/sji/work/oss/tmp/psalm_memory_test/vendor/microsoft/microsoft-graph/src/Generated/Models/User.php.value                                     | <?php\n\nnamespace Microsoft\\Graph\\Generated\\Models;\n\nuse DateTime;\nuse Microsoft\\Kiota\\Abstractions\\Serialization\\Parsable;\nuse Microsoft\\Kiota\\Abstractions\\Serialization\\ParseNode;\nuse Microsoft\\Kiota\\Abstractions\\Serialization\\SerializationWriter;\nuse Mic
72415    | 59264   | context.class_table.psalm\\internal\\provider\\fileprovider.static_properties.open_files.array_elements./home/sji/work/oss/tmp/psalm_memory_test/vendor/microsoft/microsoft-graph/src/Generated/Reports/ReportsRequestBuilder.php.value                   | <?php\n\nnamespace Microsoft\\Graph\\Generated\\Reports;\n\nuse DateTime;\nuse Exception;\nuse Http\\Promise\\Promise;\nuse Microsoft\\Graph\\Generated\\Models\\ODataErrors\\ODataError;\nuse Microsoft\\Graph\\Generated\\Models\\ReportRoot;\nuse Microsoft\\Graph\\Generated\\Reports\\
37639    | 59200   | context.class_table.psalm\\internal\\provider\\fileprovider.static_properties.open_files.array_elements./home/sji/work/oss/tmp/psalm_memory_test/vendor/microsoft/microsoft-graph/src/Generated/DeviceManagement/DeviceManagementRequestBuilder.php.value | <?php\n\nnamespace Microsoft\\Graph\\Generated\\DeviceManagement;\n\nuse Exception;\nuse Http\\Promise\\Promise;\nuse Microsoft\\Graph\\Generated\\DeviceManagement\\ApplePushNotificationCertificate\\ApplePushNotificationCertificateRequestBuilder;\nuse Microsoft\\Graph\\Gener
34919    | 59160   | context.class_table.psalm\\internal\\provider\\fileprovider.static_properties.open_files.array_elements./home/sji/work/oss/tmp/psalm_memory_test/vendor/microsoft/microsoft-graph/src/Generated/Users/Item/UserItemRequestBuilder.php.value               | <?php\n\nnamespace Microsoft\\Graph\\Generated\\Users\\Item;\n\nuse Exception;\nuse Http\\Promise\\Promise;\nuse Microsoft\\Graph\\Generated\\Models\\ODataErrors\\ODataError;\nuse Microsoft\\Graph\\Generated\\Models\\User;\nuse Microsoft\\Graph\\Generated\\Users\\Item\\Activities\\Act
32192    | 58802   | context.class_table.psalm\\internal\\provider\\fileprovider.static_properties.open_files.array_elements./home/sji/work/oss/tmp/psalm_memory_test/vendor/symfony/polyfill-mbstring/Mbstring.php.value                                                      | <?php\n\n/*\n * This file is part of the Symfony package.\n *\n * (c) Fabien Potencier <fabien@symfony.com>\n *\n * For the full copyright and license information, please view the LICENSE\n * file that was distributed with this source code.\n */\n\nnamespace Symfony\\
26572    | 59044   | context.class_table.psalm\\internal\\provider\\fileprovider.static_properties.open_files.array_elements./home/sji/work/oss/tmp/psalm_memory_test/vendor/amphp/amp/lib/Loop/Driver.php.value                                                               | <?php\n\nnamespace Amp\\Loop;\n\nuse Amp\\Coroutine;\nuse Amp\\Promise;\nuse React\\Promise\\PromiseInterface as ReactPromise;\nuse function Amp\\Promise\\rethrow;\n\n/**\n * Event loop driver which implements all basic operations to allow interoperability.\n *\n * Watchers 
26459    | 59154   | context.class_table.psalm\\internal\\provider\\fileprovider.static_properties.open_files.array_elements./home/sji/work/oss/tmp/psalm_memory_test/vendor/microsoft/microsoft-graph/src/Generated/BaseGraphClient.php.value                                 | <?php\n\nnamespace Microsoft\\Graph\\Generated;\n\nuse Microsoft\\Graph\\Generated\\Admin\\AdminRequestBuilder;\nuse Microsoft\\Graph\\Generated\\AgreementAcceptances\\AgreementAcceptancesRequestBuilder;\nuse Microsoft\\Graph\\Generated\\Agreements\\AgreementsRequestBuilder;\n
26336    | 58754   | context.class_table.psalm\\internal\\provider\\fileprovider.static_properties.open_files.array_elements./home/sji/work/oss/tmp/psalm_memory_test/vendor/amphp/amp/lib/functions.php.value                                                                 | <?php\n\nnamespace Amp\n{\n\n    use React\\Promise\\PromiseInterface as ReactPromise;\n\n    /**\n     * Returns a new function that wraps $callback in a promise/coroutine-aware function that automatically runs\n     * Generators as coroutines. The returned function

The value is truncated to 256 bytes on Reli. As you can see, the base64-encoded serialised data for IPC occupies the majority of the area for strings.

Call trace at the timing of the death

cat $memory_analyzed_reli | jq -r '(["frame_no", "function", "line"] | (., map(length*"="))),(path(.context.call_frames[]|objects) as $path | [$path[2], getpath($path).function_name, getpath($path).lineno])|@tsv' | column -t

frame_no  function                                                               line
========  ========                                                               ====
0         system                                                                 -1
1         {closure}(/home/sji/work/oss/tmp/psalm_memory_test/analyzer.php:3-15)  14
2         Psalm\\Internal\\Fork\\Pool::readResultsFromChildren                   358
3         Psalm\\Internal\\Fork\\Pool::wait                                      413
4         Psalm\\Internal\\Codebase\\Scanner::scanFilePaths                      384
5         Psalm\\Internal\\Codebase\\Scanner::scanFiles                          280
6         Psalm\\Codebase::scanFiles                                             505
7         Psalm\\Internal\\Analyzer\\ProjectAnalyzer::checkPaths                 1064
8         Psalm\\Internal\\Cli\\Psalm::run                                       375
9         <main>                                                                 9
10        <main>                                                                 120

This process died on reading the results from the children.

Top 20 most memory-consuming classes of instances

cat $memory_analyzed_reli | jq .class_objects_summary | jq -r '(["class_name", "count", "memory_usage"] | (., map(length*"="))),(to_entries|.[:20]|.[]|[.key,.value.count,.value.memory_usage])|@tsv' | column -t -o ' | '

class_name                                | count | memory_usage
==========                                | ===== | ============
Psalm\\Type\\Union                        | 28182 | 14654640
Psalm\\CodeLocation                       | 30306 | 12364848
Psalm\\Storage\\MethodStorage             | 6219  | 6716520
Psalm\\CodeLocation\\DocblockTypeLocation | 8367  | 3413736
Psalm\\Storage\\FunctionLikeParameter     | 6242  | 2346992
Psalm\\Type\\Atomic\\TNamedObject         | 11425 | 2285000
Psalm\\Storage\\ClassLikeStorage          | 799   | 1105816
Psalm\\Type\\Atomic\\TNull                | 8491  | 1018920
Psalm\\Type\\Atomic\\TString              | 4695  | 563400
Psalm\\Internal\\MethodIdentifier         | 7058  | 508176
Psalm\\Type\\Atomic\\TArray               | 2309  | 350968
Psalm\\Type\\Atomic\\TMixed               | 1903  | 258808
Psalm\\Aliases                            | 1176  | 235200
Closure                                   | 666   | 223776
Psalm\\Type\\Atomic\\TArrayKey            | 1684  | 202080
Psalm\\Storage\\FileStorage               | 425   | 166600
Psalm\\Type\\Atomic\\TGenericObject       | 693   | 160776
Psalm\\Storage\\ClassConstantStorage      | 627   | 155496
Psalm\\Type\\Atomic\\TCallable            | 834   | 153456
Psalm\\Type\\Atomic\\TLiteralString       | 1087  | 147832

Psalm instantiates many objects for representing language constructs in the analyzed codebase, especially for types and code locations.

The references of top 100 largest arrays

cat $memory_analyzed_reli | jq '. as $root | path(..|objects|select(."#type"=="ArrayHeaderContext"))| . as $path | $root|getpath($path) as $header | $header.array_elements as $elements | {path: $path|join("."), size: $elements."#locations"[0].size, count: $elements."#count", node_id:$header."#node_id"}' | jq -rs '(["size", "count", "node_id" ,"path"] | (., map(length*"="))),(sort_by(.size) | .[-100:] | reverse | .[] | [.size, .count, .node_id, .path])|@tsv' | column -t -o ' | '

size   | count | node_id | path
====   | ===== | ======= | ====
635168 | 15752 | 1513693 | context.class_table.psalm\\internal\\codebase\\internalcallmaphandler.static_properties.call_map
493440 | 10701 | 1795258 | context.interned_strings
121760 | 2781  | 2329310 | context.objects_store.75083.object_properties.data.array_elements.scanner_data.value.array_elements.4.value
117920 | 2661  | 124     | context.call_frames.2.this.object_properties.config.object_properties.predefined_constants
112640 | 2495  | 1715683 | context.class_table.psalm\\internal\\type\\typetokenizer.static_properties.memoized_tokens
70784  | 1700  | 8098    | context.call_frames.2.this.object_properties.config.object_properties.predefined_functions
63008  | 1457  | 27515   | context.call_frames.4.this.object_properties.store_scan_failure
61184  | 1400  | 2337468 | context.objects_store.75083.object_properties.data.array_elements.scanner_data.value.array_elements.5.value
61184  | 1400  | 22355   | context.call_frames.4.this.object_properties.classlike_files
59200  | 1338  | 2325305 | context.objects_store.75083.object_properties.data.array_elements.scanner_data.value.array_elements.2.value
52768  | 1137  | 2307591 | context.objects_store.75083.object_properties.data.array_elements.classlikes_data.value.array_elements.0.value
52384  | 1125  | 2310872 | context.objects_store.75083.object_properties.data.array_elements.classlikes_data.value.array_elements.1.value
40480  | 1009  | 32048   | context.call_frames.4.local_variables.files_to_scan
40032  | 995   | 2322394 | context.objects_store.75083.object_properties.data.array_elements.classlikes_data.value.array_elements.8.value
39072  | 965   | 2319329 | context.objects_store.75083.object_properties.data.array_elements.classlikes_data.value.array_elements.6.value
38688  | 953   | 2313910 | context.objects_store.75083.object_properties.data.array_elements.classlikes_data.value.array_elements.2.value
38432  | 945   | 2316631 | context.objects_store.75083.object_properties.data.array_elements.classlikes_data.value.array_elements.4.value
37920  | 929   | 2341567 | context.objects_store.75083.object_properties.data.array_elements.scanner_data.value.array_elements.7.value
28448  | 633   | 14494   | context.call_frames.4.this.object_properties.codebase.object_properties.classlikes.object_properties.existing_classlikes_lc
28128  | 623   | 1685838 | context.class_table.psalm\\internal\\codebase\\internalcallmaphandler.static_properties.call_map_callables
28064  | 621   | 16396   | context.call_frames.4.this.object_properties.codebase.object_properties.classlikes.object_properties.existing_classes_lc
19904  | 494   | 17654   | context.call_frames.4.this.object_properties.codebase.object_properties.classlikes.object_properties.existing_classes
19864  | 1241  | 1479956 | context.class_table.psalm\\internal\\provider\\statementsprovider.static_properties.parser.object_properties.actionCheck
19864  | 1241  | 1477471 | context.class_table.psalm\\internal\\provider\\statementsprovider.static_properties.parser.object_properties.action
18848  | 461   | 19130   | context.call_frames.4.this.object_properties.codebase.object_properties.classlikes.object_properties.existing_interfaces_lc
18728  | 1170  | 1482441 | context.class_table.psalm\\internal\\provider\\statementsprovider.static_properties.parser.object_properties.actionBase
18464  | 449   | 1472130 | context.class_table.psalm\\internal\\provider\\classlikestorageprovider.static_properties.new_storage
18464  | 449   | 140616  | context.class_table.psalm\\internal\\provider\\classlikestorageprovider.static_properties.storage
18464  | 449   | 20496   | context.call_frames.4.this.object_properties.codebase.object_properties.classlikes.object_properties.existing_traits_lc
18208  | 441   | 21424   | context.call_frames.4.this.object_properties.codebase.object_properties.classlikes.object_properties.existing_enums_lc
17696  | 425   | 140108  | context.class_table.psalm\\internal\\provider\\filestorageprovider.static_properties.new_storage
17696  | 425   | 60149   | context.class_table.psalm\\internal\\provider\\filestorageprovider.static_properties.storage
17696  | 425   | 58618   | context.class_table.psalm\\internal\\provider\\fileprovider.static_properties.open_files
17696  | 425   | 26662   | context.call_frames.4.this.object_properties.scanned_files
16608  | 391   | 1498355 | context.class_table.psalm\\internal\\provider\\filereferenceprovider.static_properties.classlike_files
15296  | 350   | 2344464 | context.objects_store.75083.object_properties.data.array_elements.classlike_storage.value
13728  | 301   | 663199  | context.class_table.psalm\\internal\\provider\\classlikestorageprovider.static_properties.storage.array_elements.symfony\\component\\string\\abstractunicodestring.value.object_properties.constants.array_elements.TRANSLIT_FROM.value.object_properties.type.object_properties.types.array_elements.array.value.object_properties.properties.referenced.array_elements.0.value.object_properties.literal_string_types
13728  | 301   | 660170  | context.class_table.psalm\\internal\\provider\\classlikestorageprovider.static_properties.storage.array_elements.symfony\\component\\string\\abstractunicodestring.value.object_properties.constants.array_elements.TRANSLIT_FROM.value.object_properties.type.object_properties.types.array_elements.array.value.object_properties.properties.referenced.array_elements.0.value.object_properties.types
11784  | 736   | 1484784 | context.class_table.psalm\\internal\\provider\\statementsprovider.static_properties.parser.object_properties.actionDefault
10112  | 252   | 1462907 | context.class_table.psalm\\internal\\provider\\classlikestorageprovider.static_properties.storage.array_elements.microsoft\\graph\\generated\\models\\user.value.object_properties.inheritable_method_ids
10112  | 252   | 1462652 | context.class_table.psalm\\internal\\provider\\classlikestorageprovider.static_properties.storage.array_elements.microsoft\\graph\\generated\\models\\user.value.object_properties.overridden_method_ids
10112  | 252   | 1462397 | context.class_table.psalm\\internal\\provider\\classlikestorageprovider.static_properties.storage.array_elements.microsoft\\graph\\generated\\models\\user.value.object_properties.appearing_method_ids
10112  | 252   | 1461638 | context.class_table.psalm\\internal\\provider\\classlikestorageprovider.static_properties.storage.array_elements.microsoft\\graph\\generated\\models\\user.value.object_properties.declaring_method_ids
10112  | 252   | 1376210 | context.class_table.psalm\\internal\\provider\\classlikestorageprovider.static_properties.storage.array_elements.microsoft\\graph\\generated\\models\\user.value.object_properties.methods
10072  | 629   | 1487520 | context.class_table.psalm\\internal\\provider\\statementsprovider.static_properties.parser.object_properties.gotoCheck
10072  | 629   | 1486259 | context.class_table.psalm\\internal\\provider\\statementsprovider.static_properties.parser.object_properties.goto
9848   | 615   | 1491990 | context.class_table.psalm\\internal\\provider\\statementsprovider.static_properties.parser.object_properties.reduceCallbacks
9848   | 615   | 1490756 | context.class_table.psalm\\internal\\provider\\statementsprovider.static_properties.parser.object_properties.ruleToLength
9848   | 615   | 1489523 | context.class_table.psalm\\internal\\provider\\statementsprovider.static_properties.parser.object_properties.ruleToNonTerminal
8088   | 505   | 33064   | context.call_frames.4.local_variables.process_file_paths.array_elements.0.value
8072   | 504   | 33573   | context.call_frames.4.local_variables.process_file_paths.array_elements.1.value
7384   | 461   | 1472679 | context.class_table.psalm\\internal\\provider\\statementsprovider.static_properties.lexer.referenced.object_properties.tokens
6560   | 141   | 1475862 | context.class_table.psalm\\internal\\provider\\statementsprovider.static_properties.lexer.referenced.object_properties.tokenMap
6344   | 396   | 1476676 | context.class_table.psalm\\internal\\provider\\statementsprovider.static_properties.parser.object_properties.tokenToSymbol
6336   | 134   | 2322047 | context.objects_store.75083.object_properties.data.array_elements.classlikes_data.value.array_elements.7.value
6240   | 131   | 20100   | context.call_frames.4.this.object_properties.codebase.object_properties.classlikes.object_properties.existing_interfaces
4960   | 123   | 664826  | context.class_table.psalm\\internal\\provider\\classlikestorageprovider.static_properties.storage.array_elements.symfony\\component\\string\\abstractunicodestring.value.object_properties.constants.array_elements.TRANSLIT_TO.value.object_properties.type.object_properties.types.array_elements.array.value.object_properties.properties.referenced.array_elements.0.value.object_properties.literal_string_types
4960   | 123   | 663616  | context.class_table.psalm\\internal\\provider\\classlikestorageprovider.static_properties.storage.array_elements.symfony\\component\\string\\abstractunicodestring.value.object_properties.constants.array_elements.TRANSLIT_TO.value.object_properties.type.object_properties.types.array_elements.array.value.object_properties.properties.referenced.array_elements.0.value.object_properties.types
4768   | 117   | 126894  | context.class_table.psalm\\internal\\provider\\filestorageprovider.static_properties.storage.array_elements./home/sji/work/oss/tmp/psalm_memory_test/vendor/microsoft/microsoft-graph/src/generated/reports/reportsrequestbuilder.php.value.object_properties.namespace_aliases.array_elements.7.value.object_properties.uses_flipped
4768   | 117   | 126641  | context.class_table.psalm\\internal\\provider\\filestorageprovider.static_properties.storage.array_elements./home/sji/work/oss/tmp/psalm_memory_test/vendor/microsoft/microsoft-graph/src/generated/reports/reportsrequestbuilder.php.value.object_properties.namespace_aliases.array_elements.7.value.object_properties.uses
4768   | 117   | 126276  | context.class_table.psalm\\internal\\provider\\filestorageprovider.static_properties.storage.array_elements./home/sji/work/oss/tmp/psalm_memory_test/vendor/microsoft/microsoft-graph/src/generated/reports/reportsrequestbuilder.php.value.object_properties.referenced_classlikes
4736   | 84    | 4771100 | context.objects_store.75083.object_properties.data.array_elements.classlike_storage.value.array_elements.microsoft\\graph\\generated\\users\\item\\authentication\\authenticationrequestbuilderdeleterequestconfiguration.value.dynamic_properties
4736   | 84    | 4769803 | context.objects_store.75083.object_properties.data.array_elements.classlike_storage.value.array_elements.microsoft\\graph\\generated\\users\\item\\authentication\\temporaryaccesspassmethods\\temporaryaccesspassmethodsrequestbuilder.value.dynamic_properties
4736   | 84    | 4760448 | context.objects_store.75083.object_properties.data.array_elements.classlike_storage.value.array_elements.microsoft\\graph\\generated\\users\\item\\authentication\\phonemethods\\phonemethodsrequestbuilder.value.dynamic_properties
4736   | 84    | 4751093 | context.objects_store.75083.object_properties.data.array_elements.classlike_storage.value.array_elements.microsoft\\graph\\generated\\users\\item\\authentication\\operations\\operationsrequestbuilder.value.dynamic_properties
4736   | 84    | 4741738 | context.objects_store.75083.object_properties.data.array_elements.classlike_storage.value.array_elements.microsoft\\graph\\generated\\users\\item\\authentication\\methods\\methodsrequestbuilder.value.dynamic_properties
4736   | 84    | 4732383 | context.objects_store.75083.object_properties.data.array_elements.classlike_storage.value.array_elements.microsoft\\graph\\generated\\users\\item\\authentication\\emailmethods\\emailmethodsrequestbuilder.value.dynamic_properties
4736   | 84    | 4723026 | context.objects_store.75083.object_properties.data.array_elements.classlike_storage.value.array_elements.microsoft\\graph\\generated\\users\\item\\assignlicense\\assignlicensepostrequestbody.value.dynamic_properties
4736   | 84    | 4710988 | context.objects_store.75083.object_properties.data.array_elements.classlike_storage.value.array_elements.microsoft\\graph\\generated\\models\\approleassignment.value.dynamic_properties
4736   | 84    | 4697733 | context.objects_store.75083.object_properties.data.array_elements.classlike_storage.value.array_elements.microsoft\\graph\\generated\\users\\item\\approleassignments\\approleassignmentsrequestbuildergetrequestconfiguration.value.dynamic_properties
4736   | 84    | 4688061 | context.objects_store.75083.object_properties.data.array_elements.classlike_storage.value.array_elements.microsoft\\graph\\generated\\users\\item\\approleassignments\\count\\countrequestbuilder.value.dynamic_properties
4736   | 84    | 4683255 | context.objects_store.75083.object_properties.data.array_elements.classlike_storage.value.array_elements.microsoft\\graph\\generated\\users\\item\\agreementacceptances\\item\\agreementacceptanceitemrequestbuilder.value.dynamic_properties
4736   | 84    | 4678431 | context.objects_store.75083.object_properties.data.array_elements.classlike_storage.value.array_elements.microsoft\\graph\\generated\\users\\item\\activities\\activitiesrequestbuilderpostrequestconfiguration.value.dynamic_properties
4736   | 84    | 4675715 | context.objects_store.75083.object_properties.data.array_elements.classlike_storage.value.array_elements.microsoft\\graph\\generated\\models\\useractivitycollectionresponse.value.dynamic_properties
4736   | 84    | 4669902 | context.objects_store.75083.object_properties.data.array_elements.classlike_storage.value.array_elements.microsoft\\graph\\generated\\users\\item\\activities\\item\\useractivityitemrequestbuilder.value.dynamic_properties
4736   | 84    | 4659144 | context.objects_store.75083.object_properties.data.array_elements.classlike_storage.value.array_elements.microsoft\\graph\\generated\\users\\item\\activities\\count\\countrequestbuilder.value.dynamic_properties
4736   | 84    | 4654337 | context.objects_store.75083.object_properties.data.array_elements.classlike_storage.value.array_elements.microsoft\\kiota\\abstractions\\serialization\\serializationwriter.value.dynamic_properties
4736   | 84    | 4626397 | context.objects_store.75083.object_properties.data.array_elements.classlike_storage.value.array_elements.doctrine\\common\\annotations\\annotationreader.value.dynamic_properties
4736   | 84    | 4609664 | context.objects_store.75083.object_properties.data.array_elements.classlike_storage.value.array_elements.microsoft\\kiota\\abstractions\\requestheaders.value.dynamic_properties
4736   | 84    | 4595334 | context.objects_store.75083.object_properties.data.array_elements.classlike_storage.value.array_elements.league\\oauth2\\client\\token\\accesstokeninterface.value.dynamic_properties
4736   | 84    | 4591670 | context.objects_store.75083.object_properties.data.array_elements.classlike_storage.value.array_elements.microsoft\\graph\\generated\\serviceprincipalswithappid\\serviceprincipalswithappidrequestbuildergetrequestconfiguration.value.dynamic_properties
4736   | 84    | 4585527 | context.objects_store.75083.object_properties.data.array_elements.classlike_storage.value.array_elements.microsoft\\graph\\generated\\directoryroleswithroletemplateid\\directoryroleswithroletemplateidrequestbuilderpatchrequestconfiguration.value.dynamic_properties
4736   | 84    | 4582812 | context.objects_store.75083.object_properties.data.array_elements.classlike_storage.value.array_elements.microsoft\\graph\\generated\\directoryroleswithroletemplateid\\directoryroleswithroletemplateidrequestbuilderdeleterequestconfiguration.value.dynamic_properties
4736   | 84    | 4580097 | context.objects_store.75083.object_properties.data.array_elements.classlike_storage.value.array_elements.microsoft\\graph\\generated\\deviceswithdeviceid\\deviceswithdeviceidrequestbuildergetrequestconfiguration.value.dynamic_properties
4736   | 84    | 4573952 | context.objects_store.75083.object_properties.data.array_elements.classlike_storage.value.array_elements.microsoft\\kiota\\abstractions\\store\\backingstore.value.dynamic_properties
4736   | 84    | 4564178 | context.objects_store.75083.object_properties.data.array_elements.classlike_storage.value.array_elements.microsoft\\graph\\generated\\applicationswithappid\\applicationswithappidrequestbuildergetrequestconfiguration.value.dynamic_properties
4736   | 84    | 4558035 | context.objects_store.75083.object_properties.data.array_elements.classlike_storage.value.array_elements.microsoft\\graph\\generated\\users\\usersrequestbuilderpostrequestconfiguration.value.dynamic_properties
4736   | 84    | 4555320 | context.objects_store.75083.object_properties.data.array_elements.classlike_storage.value.array_elements.microsoft\\graph\\generated\\users\\usersrequestbuildergetrequestconfiguration.value.dynamic_properties
4736   | 84    | 4546167 | context.objects_store.75083.object_properties.data.array_elements.classlike_storage.value.array_elements.microsoft\\graph\\generated\\users\\getbyids\\getbyidsrequestbuilder.value.dynamic_properties
4736   | 84    | 4540540 | context.objects_store.75083.object_properties.data.array_elements.classlike_storage.value.array_elements.microsoft\\graph\\generated\\users\\delta\\deltarequestbuilder.value.dynamic_properties
4736   | 84    | 4535722 | context.objects_store.75083.object_properties.data.array_elements.classlike_storage.value.array_elements.microsoft\\graph\\generated\\tenantrelationships\\tenantrelationshipsrequestbuilderpatchrequestconfiguration.value.dynamic_properties
4736   | 84    | 4533007 | context.objects_store.75083.object_properties.data.array_elements.classlike_storage.value.array_elements.microsoft\\graph\\generated\\tenantrelationships\\tenantrelationshipsrequestbuildergetrequestconfiguration.value.dynamic_properties
4736   | 84    | 4526864 | context.objects_store.75083.object_properties.data.array_elements.classlike_storage.value.array_elements.microsoft\\graph\\generated\\tenantrelationships\\findtenantinformationbydomainnamewithdomainname\\findtenantinformationbydomainnamewithdomainnamerequestbuilder.value.dynamic_properties
4736   | 84    | 4521521 | context.objects_store.75083.object_properties.data.array_elements.classlike_storage.value.array_elements.microsoft\\graph\\generated\\tenantrelationships\\delegatedadmincustomers\\delegatedadmincustomersrequestbuilder.value.dynamic_properties
4736   | 84    | 4512165 | context.objects_store.75083.object_properties.data.array_elements.classlike_storage.value.array_elements.microsoft\\graph\\generated\\models\\teamwork.value.dynamic_properties
4736   | 84    | 4502682 | context.objects_store.75083.object_properties.data.array_elements.classlike_storage.value.array_elements.microsoft\\graph\\generated\\teamwork\\workforceintegrations\\workforceintegrationsrequestbuilder.value.dynamic_properties
4736   | 84    | 4493327 | context.objects_store.75083.object_properties.data.array_elements.classlike_storage.value.array_elements.microsoft\\graph\\generated\\teamwork\\sendactivitynotificationtorecipients\\sendactivitynotificationtorecipientsrequestbuilder.value.dynamic_properties
4736   | 84    | 4487733 | context.objects_store.75083.object_properties.data.array_elements.classlike_storage.value.array_elements.microsoft\\graph\\generated\\teamstemplates\\teamstemplatesrequestbuilderpostrequestconfiguration.value.dynamic_properties
4736   | 84    | 4485017 | context.objects_store.75083.object_properties.data.array_elements.classlike_storage.value.array_elements.microsoft\\graph\\generated\\models\\teamstemplatecollectionresponse.value.dynamic_properties
4736   | 84    | 4479204 | context.objects_store.75083.object_properties.data.array_elements.classlike_storage.value.array_elements.microsoft\\graph\\generated\\teamstemplates\\item\\teamstemplateitemrequestbuilder.value.dynamic_properties

Unlike the case of strings, for arrays there are no extremely large arrays on their own, so there must be a large number of relatively small arrays. For example, there are many dynamic_properties later in the list. Oh wait, dynamic properties? Really?

Digging into dynamic properties

As described in this article, PHP objects may have a hash table for dynamic properties internally. Normally this hash table is not allocated unless it is needed, e.g. for the use of dynamic properties.

Let's extract all the dynamic properties tables and get the total size of them.

cat $memory_analyzed_reli | jq '. as $root | path(..|objects|select(."#type"=="ArrayHeaderContext"))| select(.[-1] == "dynamic_properties") | . as $path | $root|getpath($path) as $header | $header.array_elements as $elements | $header.possible_unused_area as $unused_area | {path: $path|join("."), size: $elements."#locations"[0].size, unused_size: $unused_area."#locations"[0].size, count: $elements."#count", node_id:$header."#node_id"}' | jq -rs '. as $all | {count: $all|length , total_used_size: [$all[].size]|add, total_unused_size: [$all[].unused_size]|add}'

{
  "count": 57590,
  "total_used_size": 65160032,
  "total_unused_size": 68518368
}

The dynamic properties tables take up about 130MB of memory in total.

Then take a look at one of the dynamic properties tables. What keys are there?

cat $memory_analyzed_reli | jq '.context.objects_store."75083".object_properties.data.array_elements.classlike_storage.value.array_elements."microsoft\\graph\\generated\\teamstemplates\\item\\teamstemplateitemrequestbuilder".value.dynamic_properties.array_elements|keys'

[
  "#count",
  "#locations",
  "#node_id",
  "#type",
  "abstract",
  "aliases",
  "appearing_method_ids",
  "appearing_property_ids",
  "attributes",
  "class_implements",
  "constants",
  "custom_metadata",
  "declaring_method_ids",
  "declaring_property_ids",
  "declaring_pseudo_method_ids",
  "declaring_yield_fqcn",
  "dependent_classlikes",
  "deprecated",
  "description",
  "direct_class_interfaces",
  "direct_interface_parents",
  "docblock_issues",
  "documenting_method_ids",
  "enforce_template_inheritance",
  "enum_cases",
  "enum_type",
  "extension_requirement",
  "external_mutation_free",
  "final",
  "final_from_docblock",
  "has_visitor_issues",
  "hash",
  "implementation_requirements",
  "inheritable_method_ids",
  "inheritable_property_ids",
  "inheritors",
  "initialized_properties",
  "internal",
  "invalid_dependencies",
  "is_enum",
  "is_interface",
  "is_trait",
  "location",
  "methods",
  "mixin_declaring_fqcln",
  "mutation_free",
  "name",
  "namedMixins",
  "namespace_name_location",
  "overridden_method_ids",
  "overridden_property_ids",
  "override_method_visibility",
  "override_property_visibility",
  "parent_class",
  "parent_classes",
  "parent_interfaces",
  "populated",
  "potential_declaring_method_ids",
  "preserve_constructor_signature",
  "properties",
  "pseudo_methods",
  "pseudo_property_get_types",
  "pseudo_property_set_types",
  "pseudo_static_methods",
  "public_api",
  "readonly",
  "sealed_methods",
  "sealed_properties",
  "specialize_instance",
  "stmt_location",
  "stubbed",
  "suppressed_issues",
  "template_covariants",
  "template_extended_offsets",
  "template_extended_params",
  "template_type_extends_count",
  "template_type_implements_count",
  "template_type_uses_count",
  "template_types",
  "templatedMixins",
  "trait_alias_map",
  "trait_alias_map_cased",
  "trait_final_map",
  "trait_visibility_map",
  "type_aliases",
  "used_traits",
  "user_defined",
  "yield"
]

Keys beginning with # holds informational values added by Reli. Other 84 keys represent actual property names in this dynamic properties table.

The object having this table is an instance of \Psalm\Storage\ClassLikeStorage. This class declares 84 properties in total. They have exactly the same names as the keys in the dynamic properties table.

This means that there are actually no dynamic properties in this case. For some reason, dynamic properties tables have been unnecessarily constructed. And the reason is in the implementation of \unserialize(), as can be seen from the fact that memprof reports \unserialize() as a major part of the increased memory consumption. Once this table has been constructed, there is no way to release this area from an user-land script except deleting the instance itself by setting its reference count to 0.

This issue is already reported in php-src (https://github.com/php/php-src/issues/10126).

The 2nd run

MEMPROF_PROFILE=dump_on_limit php -dauto_prepend_file=analyzer.php ./vendor/bin/psalm --memory-limit=512M --threads=2 test.php

Target PHP version: 8.2 (inferred from current PHP version).
Scanning files...
PHP Fatal error:  Allowed memory size of 536870912 bytes exhausted (tried to allocate 20480 bytes) (memprof dumped to /tmp/memprof.callgrind.1787474184950721) in /home/sji/work/oss/tmp/psalm_memory_test/vendor/vimeo/psalm/src/Psalm/Internal/Fork/Pool.php on line 358
Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 20480 bytes) (memprof dumped to /tmp/memprof.callgrind.1787474184950721) in /home/sji/work/oss/tmp/psalm_memory_test/vendor/vimeo/psalm/src/Psalm/Internal/Fork/Pool.php on line 358

The output of php-memprof

kcachegrind /tmp/memprof.callgrind.1787474184950721

second-run

In this second run, \unserialize() is also called from the code for caching.

I omit the output of Reli here, as it was no different from the first run in that the serialized data for IPC and the dynamic_properties table occupy a large amount of space.

A possible workaround

The default behavior of the PHP VM allocates dynamic properties table unnecessarily.

https://3v4l.org/nRUnW

Funnily, defining __unserialize() in the serialized class magically avoids the construction of the table.

https://3v4l.org/nhKJu

So if we define __unserialize() in the classes that are serialized and fill the properties ourselves, we can avoid the needless construction of dynamic properties tables. Also, contrary to intuition, this does not slow down the speed.

I'll send a PR of this.

About igbinary

Using igbinary reduces the size of the serialized data for IPC, but the dynamic properties tables are still unnecessarily constructed even with igbinary. Worse, the workaround via __unserialize() does not work with igbinary currently...

I will also look into the implementation of igbinary when I have time.

vimeo / psalm

Psalm validation crashes laptop due to memory issues #10522

Preconditions

Additional Test script

analyzer.php

The PHP VM used to test

PHP version

php-memprof

Run with 512M memory limit

The 1st run

The output of memprof

The output from Reli

Extracting the summary

Top 20 most memory-consuming types of memory locations

Top 10 largest strings

Call trace at the timing of the death

Top 20 most memory-consuming classes of instances

The references of top 100 largest arrays

Digging into dynamic properties

The 2nd run

The output of php-memprof

A possible workaround

About igbinary