No caching files generated after entrification

[gddotorg]

Due to performance optimisation, we have decided to entrify the global sets. All conversions worked perfectly and we have given the adjustments to the test system. Unfortunately, the console command (./craft blitz/cache/generate --queue) creates a queue job, but it ends after 300 seconds without any progress and without having built up a cache. In the logs I find the following:

Queue Log: 2023-10-04 08:50:52 [queue.ERROR] [blitz] The request was not processed and can be safely retried [https://example.com/path?token=qJHiujfo2R1mm0Aw8Qk7DpUUvek5OtFR]

I see the same message in the Blitz logs. It seems to be an issue with the HTTP request. We are using the HTTP Generator. Is it worth to give the Local Generator a shot? Is there maybe any other settings we missed?

Our blitz.php config file

<?php

return [
    // blitz settings see https://github.com/putyourlightson/craft-blitz/blob/v4/src/config.php
    '*' => [
        'cachingEnabled' => true,
        'trackElementQueries' => false,
        'trackElements' => false,
        'includedQueryStringParams' => [
            [
                'siteId' => '',
                'queryStringParam' => '.*',
            ],
        ],
        // Whether the cache should automatically be refreshed after a global set is updated.
        'refreshCacheAutomaticallyForGlobals' => false,

        // The integrations to initialise.
        'integrations' => [
            'putyourlightson\blitz\drivers\integrations\FeedMeIntegration',
            'putyourlightson\blitz\drivers\integrations\SeomaticIntegration',
        ],

        // dont cache wishlist items
        // 1. they dont have own twig pages which could be cached
        // 2. each product page visit adds last viewed items which triggers refresh blitzcache jobs
        'nonCacheableElementTypes' => [
            'verbb\wishlist\elements\ListElement',
            'verbb\wishlist\elements\Item',
            'craft\elements\User',
            'craft\elements\Asset',
            'craft\elements\Category',
            'craft\elements\Tag'
        ],
        'queryStringCaching' => 0,
        'refreshCacheJobPriority' => 110
    ],
    'local' => [
        'cachingEnabled' => false,
        'refreshCacheAutomaticallyForGlobals' => false
    ]
];

I also tried some other settings here but without any success. Also worth to be mentioned: ENVIRONMENT=staging

general.php preloadSingles => true

PHP version | 8.1.24
MySQL 5.7.23
GD 8.1.24
Yii version 2.0.48.1
Twig version v3.4.3
Guzzle version 7.8.0

Craft Pro | 4.5.6.1
Asset Rev | 7.0.0
Blitz | 4.5.5
CP Field Inspect | 1.4.4
Craft Commerce | 4.3.0
Digital Products | 3.2.3
Formie | 2.0.37
ImageOptimize | 4.0.5
Imager X | 4.2.3
Sprig | 2.7.1
Super Table | 3.0.9
Wishlist | 2.0.6

[bencroker]

Can you please test it with devMode enabled and then show me the full stack trace of the error that appears in the logs?

[gddotorg]

First of all, sorry for my late reply. I have now done some more tests on my own. First of all, the question: If I use the HTTP Generator, is it possible that the HTTP requests do not go through on a website that is secured by basic auth? Because I see every single page to be cached as a log entry in blitz.log with the message "The request was not processed and can be safely retried". Even with devMode switched on and debug (blitz) set to true, I can't get any better logs out.

I then also tried the Local Generator. I noticed immediate progress and caches were created. However, the process stops after about 1000 entries and I get the following error message: Loop exceptionally stopped without resolving the promise. You have reached the limits of stream_select(). It has a FD_SETSIZE of 1024, but you have file descriptors numbered at least as high as 1024. You can install one of the extensions listed on https://amphp.org/amp/event-loop/#implementations to support a higher number of concurrent file descriptors. If a large number of open file descriptors is unexpected, you might be leaking file descriptors that aren't closed correctly.

Also I got message from our hoster, that we have around 990 defunct processes still running. I don't have any further information on this, but I wanted to mention this

[bencroker]

If I use the HTTP Generator, is it possible that the HTTP requests do not go through on a website that is secured by basic auth?

Yes, those requests will almost certainly be blocked, so using the Local Generator is probably your best bet. What is the generator concurrency set to? This should define the largest number of processes that can be created at any one time.

[gddotorg]

I have tried both concurrency, the default value of 3 and the value of 1. The result was always the same. After around 1000 created cached pages, the process stops and I got the loop exception I posted before.

Even if it worked with the value 1, we have over 50k entries and products that we have to cache and that would take quite a long time with only one process running in parallel.

[bencroker]

Can you please check the composer.lock file for the version number of the amphp/http-client package? It should ideally be at version 4.6.x.

            "name": "amphp/http-client",
            "version": "v4.6.3",

[gddotorg]

Version v4.6.3 is installed as fare as I can see.

[bencroker]

I’ll have to investigate why that is happening. Issues with the local generator can be tough to troubleshoot since they tend to be environment-specific.

Is your site only temporarily protected by basic auth? if that’s the case then things should begin to work once it is made publicly available. I realise this is not ideal but it should be possible to test your Blitz setup locally.

[bencroker]

Any update on this @gddotorg?

[bencroker]

Closing due to inactivity.

[gddotorg]

@bencroker Sorry I haven't replied more. We have now published it to production with the HttpGenerator and the caching is generated as before with Craft 3. It seems the Basic Auth was the issue here. I have not pursued the solution of using the LocalGenerator any further. In general, however, I still have performance questions. But I'll open a new issue for this.