CI4 performance regression

[joanhey]

PHP Version

8.1

CodeIgniter4 Version

4.2.10

CodeIgniter4 Installation Method

Composer (as dependency to an existing project)

Which operating systems have you tested for this bug?

Linux

Which server did you use?

fpm-fcgi

Database

MySQL 8

What happened?

After update to CI4 in the TechEmpower bench.

CI3 PHP 8.1 vs CI4 PHP8.1 (with JIT) Req/s Framework	JSON	1-query	20-query	Fortunes	Updates	Plaintext
CI3	26,313	25,189	12,049	24,358	--	25,646
CI4	5,746	5,314	4,244	5,320	2,086	5,791

Please, could anybody review the code in the bench.

If after review the code, still get this performance, exist another problem.

Thank you

Steps to Reproduce

https://tfb-status.techempower.com/

I can help, only ask me.

Expected Output

Similar or higher numbers than CI3. Not 5 times slower.

Anything else?

PR that update to CI4: https://github.com/TechEmpower/FrameworkBenchmarks/pull/7686

Final code to review: https://github.com/TechEmpower/FrameworkBenchmarks/tree/master/frameworks/PHP/codeigniter

Previous code with CI3: https://github.com/TechEmpower/FrameworkBenchmarks/tree/e062401d0dbc660ed2e7c969ddbcc84e66ae5824/frameworks/PHP/codeigniter

When finish the run, I'll send the logs.

[kenjis]

How do we run this CI4 app? Where is the database schema?

[joanhey]

Clone https://github.com/TechEmpower/FrameworkBenchmarks in your computer. It's using docker.

In the main directory:

Verify tests: ./tfb --mode verify --test codeigniter

Bench in local: ./tfb --mode benchmark --test codeigniter

Debug mode, for watch in your browser that is running OK: ./tfb --mode debug --test codeigniter

[joanhey]

You can visualize your local bench results in: https://www.techempower.com/benchmarks/#section=test

or

https://tfb-status.techempower.com/share

The results are generated in the dir results/

You also can run other fw bench to compare: cakephp, laravel, ... Also go before in the git repo, and test it with CI3.

[joanhey]

The results from CI3 https://www.techempower.com/benchmarks/#section=test&runid=8ed7e635-3068-4000-8fd2-e418231c7ba0&test=fortune

The logs: https://tfb-status.techempower.com/unzip/results.2022-11-19-10-45-41-715.zip/results/20221113183242/codeigniter

The actual running with CI4 : https://www.techempower.com/benchmarks/#section=test&runid=03363299-b1b5-42e6-8ca5-6e0665be82c2&test=fortune

When finish, I'll send the logs. Only show the logs at the end of the run.

[joanhey]

Here you can see the plaintext test numbers for CI3: https://tfb-status.techempower.com/unzip/results.2022-11-19-10-45-41-715.zip/results/20221113183242/codeigniter/plaintext/raw.txt

Remember than in local will be lower.

[joanhey]

Some days before I updated Slim, with good results.

You can check it here: https://github.com/TechEmpower/FrameworkBenchmarks/pull/7660#issuecomment-1312533420

[kenjis]

CI3's code does not work. https://github.com/TechEmpower/FrameworkBenchmarks/tree/e062401d0dbc660ed2e7c969ddbcc84e66ae5824/frameworks/PHP/codeigniter

$ ./tfb --mode verify --test codeigniter
...
techempower/mysql:latest:  ---> Running in f296bd6da0f6
techempower/mysql:latest: W: GPG error: http://repo.mysql.com/apt/ubuntu bionic InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 467B942D3A79BD29
techempower/mysql:latest: E: The repository 'http://repo.mysql.com/apt/ubuntu bionic InRelease' is not signed.
techempower/mysql:latest: Removing intermediate container f296bd6da0f6
techempower/mysql:latest: Docker build failed; terminating
techempower/mysql:latest: Traceback (most recent call last):
techempower/mysql:latest:   File "/FrameworkBenchmarks/toolset/utils/docker_helper.py", line 55, in __build
techempower/mysql:latest:     raise Exception(token['errorDetail']['message'])
techempower/mysql:latest: Exception: The command '/bin/sh -c apt-get update > /dev/null' returned a non-zero code: 100
techempower/mysql:latest: Build time: 2m 39s
A fatal error has occurred
Traceback (most recent call last):
  File "/FrameworkBenchmarks/toolset/run-tests.py", line 244, in main
    any_failed = benchmarker.run()
  File "/FrameworkBenchmarks/toolset/benchmark/benchmarker.py", line 58, in run
    self.docker_helper.build_databases()
  File "/FrameworkBenchmarks/toolset/utils/docker_helper.py", line 315, in build_databases
    tag="techempower/%s" % db)
  File "/FrameworkBenchmarks/toolset/utils/docker_helper.py", line 55, in __build
    raise Exception(token['errorDetail']['message'])
Exception: The command '/bin/sh -c apt-get update > /dev/null' returned a non-zero code: 100
Shutting down (may take a moment)

[joanhey]

Failed the apt-get update, this happen some times. Exception: The command '/bin/sh -c apt-get update > /dev/null' returned a non-zero code: 100

In some minutes I'll try in my local.

PD: Do you remember me ?? I remember you from your benchmark.

https://github.com/kenjis/php-framework-benchmark/commits?author=joanhey https://github.com/kenjis/php-framework-benchmark/issues/62

[kenjis]

I copied CI3 code into the latest repository and ran the benchmarks on my macOS.

https://www.techempower.com/benchmarks/#section=test&shareid=4e82ab70-1a3a-4e64-a9cf-010c67062d6a&test=json https://www.techempower.com/benchmarks/#section=test&shareid=ea90a962-cbb4-4d02-b866-b9df123a73a5&test=json

[kenjis]

I found the database schema. https://github.com/TechEmpower/TFBDatabases/blob/master/MySQL/create.sql

[joanhey]

The regression is global, so you can test it with the json and plaintext tests.

Also, after checking the numbers and code from CI3 and CI4, I check an architectural problem. Plaintext, Json and 1-query have similar req/s. Because always connect to the database, even when it isn't necessary.

https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/PHP/codeigniter/app/Controllers/BaseController.php#L51

[joanhey]

Perhaps using the ORM will be fixed.

[joanhey]

Perhaps the regression is only in the database connection:
Config\Database::connect();

But for the above line included always, we see the regression in all tests.

And it isn't recommended to use it so, as will be very bad for use with any persistent platform using workers and childs. Like Workerman, Swoole, ...

[joanhey]

I can make some changes, and send a new PR to the bench.

[joanhey]

Added a fast change, later you can change it. https://github.com/TechEmpower/FrameworkBenchmarks/pull/7723

But we don't have a reference from CI3, because always was loading the bd also.

[joanhey]

Still with the last change is slow:

Before: Now:

Only gain in json and plaintext 100-200 req/s

CI3 json test:

[kenjis]

Try preload.php: https://github.com/codeigniter4/appstarter/blob/master/preload.php

[joanhey]

All the frameworks using preload gain ~1-2%.

[kenjis]

Try this change: https://github.com/kenjis/ci4-app-template/commit/df8226e3165e6b9281babb9d536304dd10a9fc1d

[patches]

Hello,

We recently moved to CI 4.2 in one of our project. Before that, we were using CI 4.1. As soon as we updated, we started seeing an increase in CPU usage on our servers. We have hundreds of routes, we might be able to optimize some things on our side, but I thought pitching in what we found could help in finding this regression.

We ran on 4.2 for two days. During that time, we were trying to figure out what was causing the spike. I marked that period in red in the screenshot below.

We ran a few tests locally and used xDebug's profiler to see if there was any bottleneck in our application. We noticed an increase in execution time in RouteCollection.php.

We reverted the following changes from #6644.

More specifically: https://github.com/codeigniter4/CodeIgniter4/pull/6644/files#diff-b28efb4cd802e8a3ead515befe9f46254b6cc9d17ab1beeb8a42a16cff69d283L1247-R1258

As soon as we did, the CPU usage of our servers went back to normal (purple line in the screenshot).

Like I said, it might not be what you're looking for, but hopefully that helps.

[kenjis]

@patches Thank you for your investigation. Indeed dot_array_search() seems heavy.

For reference, how many routes are defined? E.g.

$ php spark routes | grep :: | wc -l

[patches]

@kenjis We're talking about 1571 routes. Which is quite a lot.

[kenjis]

I have confirmed that the time to register a lot of routes has increased significantly since v4.2.8. See #7175

[melroyvandenberg]

See also my post on the forum: https://forum.codeigniter.com/showthread.php?tid=88083

The performance degradation is partly solved I notice. But still not as it used to be I think.

You can also try to run the benchmarks of this person: https://github.com/myaaghubi/PHP-Frameworks-Bench. Maybe try running it on CI3 vs CI4 on the same PHP version, without OPCache and maybe also with OPCache+JIT?

[kenjis]

CI4 performance has not improved at all. I believe this is due to the fact that CI4 has considerably more functionality than CI3.

Also, there does not seem to be a particular need for performance tuning and no one is doing it. In fact, CI4 is probably fast enough.

The only way to improve performance is to benchmark (profile) and find bottlenecks, and eliminate them.

[kenjis]

The Hello World benchmark only measures the minimal overhead of the framework. Other processes are much larger in the actual application. So even if the Hello World benchmark is faster, like Symfony, it is not the same as if the actual app is faster.

In any case, benchmarks are meaningless unless you measure them in your own environment. The numbers will change depending on various conditions.

[kenjis]

That said, I would like CI4 to be faster and would welcome performance tuning work and PRs for it.

[joanhey]

A hello world benchmark is not the best, but say a lot about the framework.

A framework can have more functionality, but for that is the autoload. I can understand a 10-20% regression for the extra functionality, but not 5 times slower than CI3.

PD: Also it is using a lot of memory !!! Check the new benchmark, copied from your benchmark: https://github.com/myaaghubi/PHP-Frameworks-Bench

[kenjis]

I got the Hello World benchmark with Apache on Ubuntu 22.02. Yes, CI4 is very slow, 6.9 times slower.

$ wrk -t10 -d5s http://localhost/ci3.1.13/
Running 5s test @ http://localhost/ci3.1.13/
  10 threads and 10 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.13ms    1.25ms  20.01ms   96.12%
    Req/Sec     1.04k   229.18     1.54k    57.40%
  52703 requests in 5.10s, 8.10MB read
Requests/sec:  10334.21
Transfer/sec:      1.59MB

$ wrk -t10 -d5s http://localhost/ci4.3.7/
Running 5s test @ http://localhost/ci4.3.7/
  10 threads and 10 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     6.81ms    3.24ms  46.62ms   82.29%
    Req/Sec   151.34     35.28   340.00     62.30%
  7609 requests in 5.10s, 1.50MB read
Requests/sec:   1492.09
Transfer/sec:    301.88KB

$ php -v
PHP 8.1.2-1ubuntu2.13 (cli) (built: Jun 28 2023 14:01:49) (NTS)
Copyright (c) The PHP Group
Zend Engine v4.1.2, Copyright (c) Zend Technologies
    with Zend OPcache v8.1.2-1ubuntu2.13, Copyright (c), by Zend Technologies

$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.3 LTS"

[kenjis]

By doing the following, I got about 4240 rps. It is 2.4 times slower than CI3.

• Change the parent class of Config\Services to CodeIgniter\Config\Services
• Comment out the process of overwriting Config values with environment variables
• Enable preload

$ wrk -t10 -d5s http://localhost/ci4.3.7/
Running 5s test @ http://localhost/ci4.3.7/
  10 threads and 10 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     2.50ms    1.59ms  23.41ms   91.58%
    Req/Sec   428.97     91.32   710.00     63.24%
  21625 requests in 5.10s, 4.27MB read
Requests/sec:   4240.65
Transfer/sec:    858.00KB

[kenjis]

The Config performance can be improved by #7696. See https://github.com/codeigniter4/CodeIgniter4/pull/7696#issuecomment-1668836549

[kenjis]

See https://github.com/codeigniter4/CodeIgniter4/pull/8017#issuecomment-1751987525

[melroyvandenberg]

Would you mind to share if we need to change any specific configuration options for these kind of caching? I now try to go to all the file changed in these PRs, however this doesn't help me much/isn't efficient.

Bottom-line: Is there something I need to be aware of, as an user? Or it is only internal stuff :).?

[kenjis]

See the "Configuration" in the comment.

[kenjis]

Profiling 4.5 with

• Config caching
• FileLocator caching
• Change the parent class of Services

```diff diff --git a/app/Config/Services.php b/app/Config/Services.php index df7c8ad086..8289cbf614 100644 --- a/app/Config/Services.php +++ b/app/Config/Services.php @@ -2,6 +2,8 @@ namespace Config; +use CodeIgniter\Autoloader\FileLocator; +use CodeIgniter\Autoloader\FileLocatorCached; use CodeIgniter\Config\BaseService; /** @@ -17,16 +19,18 @@ use CodeIgniter\Config\BaseService; * method format you should use for your service methods. For more examples, * see the core Services file at system/Config/Services.php. */ -class Services extends BaseService +class Services extends \CodeIgniter\Config\Services { - /* - * public static function example($getShared = true) - * { - * if ($getShared) { - * return static::getSharedInstance('example'); - * } - * - * return new \CodeIgniter\Example(); - * } - */ + public static function locator(bool $getShared = true) + { + if ($getShared) { + if (! isset(static::$instances['locator'])) { + static::$instances['locator'] = new FileLocatorCached(new FileLocator(static::autoloader())); + } + + return static::$mocks['locator'] ?? static::$instances['locator']; + } + + return new FileLocator(static::autoloader()); + } } diff --git a/public/index.php b/public/index.php index 8cf5ce347c..e5557cd706 100644 --- a/public/index.php +++ b/public/index.php @@ -49,8 +49,8 @@ if (! defined('ENVIRONMENT')) { } // Load Config Cache -// $factoriesCache = new \CodeIgniter\Cache\FactoriesCache(); -// $factoriesCache->load('config'); +$factoriesCache = new \CodeIgniter\Cache\FactoriesCache(); +$factoriesCache->load('config'); // ^^^ Uncomment these lines if you want to use Config Caching. /* @@ -79,7 +79,7 @@ $app->setContext($context); $app->run(); // Save Config Cache -// $factoriesCache->save('config'); +$factoriesCache->save('config'); // ^^^ Uncomment this line if you want to use Config Caching. // Exits the application, setting the exit code for CLI-based applications ```

[kenjis]

Profiling plain 4.5 with Xdebug and PHP 8.1 on Ubuntu 22.04.

[kenjis]

With branch 4.5, the welcome page benchmark has improved by about 40%.

Plain develop (after 4.4.6) with Apache on Ubuntu 22.02 (PHP 8.1.2-1ubuntu2.14).

$ wrk -t10 -d5s http://localhost/CodeIgniter4/
Running 5s test @ http://localhost/CodeIgniter4/
  10 threads and 10 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     7.00ms    3.26ms  34.53ms   82.66%
    Req/Sec   147.05     35.81   290.00     61.39%
  7410 requests in 5.10s, 123.37MB read
Requests/sec:   1452.83
Transfer/sec:     24.19MB

Plain 4.5:

$ wrk -t10 -d5s http://localhost/CodeIgniter4/
Running 5s test @ http://localhost/CodeIgniter4/
  10 threads and 10 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     4.91ms    2.32ms  26.29ms   81.95%
    Req/Sec   209.90     49.85   530.00     51.50%
  10482 requests in 5.10s, 174.80MB read
Requests/sec:   2055.98
Transfer/sec:     34.29MB

[kenjis]

Bottom-line: Is there something I need to be aware of, as an user? Or it is only internal stuff :).?

If you use the caching features, the hello world benchmark will be probably improved by 2 times. But the caches never expire, so you need to delete them when you deploy new version. See the User Guide for details.

Also, it is just showing "hello world" or something static. In the real world, there is more to do besides that. And the slowest part of the code in the whole thing should be improved.

[kenjis]

Property access is 6 times faster than Services.

Test            Time    Memory
$this->request      0.0653  0 Bytes
services::request()     0.4168  0 Bytes
service('request')  0.4345  0 Bytes

```php add('$this->request', static function () use ($that) { $path = $that->request->getPath(); }); $iterator->add('Services::request()', static function () { $path = Services::request()->getPath(); }); $iterator->add('service(\'request\')', static function () { $path = service('request')->getPath(); }); return $iterator->run(300000); } } ``` ``` $ php -v PHP 8.2.16 (cli) (built: Feb 16 2024 05:30:16) (NTS) Copyright (c) The PHP Group Zend Engine v4.2.16, Copyright (c) Zend Technologies with Zend OPcache v8.2.16, Copyright (c), by Zend Technologies $ sw_vers ProductName: macOS ProductVersion: 12.7.3 BuildVersion: 21H1015 ```

[lonnieezell]

Property access is 6 times faster than Services.

This doesn't account for the actual class instantiation, though, does it? Once you're accessing $this->request it has already been instantiated, whereas the first call to services() or Service::x() will include that cost. Not sure how much difference that makes over 300,000 reps, but it would be there.

[kenjis]

No. That is code in a controller. Services already instantiated the Request object.

[kenjis]

FYI, if we change the parent class of Config\Services, Services would be faster more than 2 times.

Test            Time    Memory
$this->request      0.0617  0 Bytes
services::request()     0.1693  0 Bytes
service('request')  0.1828  0 Bytes

--- a/app/Config/Services.php
+++ b/app/Config/Services.php
@@ -17,7 +17,7 @@ use CodeIgniter\Config\BaseService;
  * method format you should use for your service methods. For more examples,
  * see the core Services file at system/Config/Services.php.
  */
-class Services extends BaseService
+class Services extends \CodeIgniter\Config\Services
 {
     /*
      * public static function example($getShared = true)

[kenjis]

Measured including the time to inject into the constructor, using Services is about four times slower.

Test            Time    Memory
constructor injection   0.1153  0 Bytes
services        0.4307  0 Bytes

```php add('constructor injection', static function () use ($typography) { $obj1 = new DiClass($typography, new \stdClass()); $obj1->foo(); }); $iterator->add('Services', static function () use ($typography) { $obj2 = new SlClass(new \stdClass()); $obj2->foo(); }); return $iterator->run(300000); } } class DiClass { public function __construct(private $typography, private $config) { } public function foo() { $this->typography; } } class SlClass { public function __construct(private $config) { } public function foo() { Services::typography(); } } ```

[kenjis]

Retrieving objects from Factories is also quite slow due to too much functionality (options) and use of the magic method.

Test        Time    Memory
config()    0.4859  0 Bytes
container   0.0704  0 Bytes

```php set('App', $config); $iterator->add("config()", static function () { $config = config('App'); }); $iterator->add('Container', static function () use ($container) { $config = $container->get('App'); }); return $iterator->run(300000); } } class Container { private array $instances; public function set(string $name, mixed $val): void { $this->instances[$name] = $val; } public function get(string $name): mixed { if (! isset($this->instances[$name])) { throw new InvalidArgumentException('No such entry.'); } return $this->instances[$name]; } } ```

If we use a global array variable.

Test        Time    Memory
config()    0.4865  0 Bytes
container   0.0481  0 Bytes

```diff diff --git a/app/Controllers/Home.php b/app/Controllers/Home.php index 5934333309..cda6c0375a 100644 --- a/app/Controllers/Home.php +++ b/app/Controllers/Home.php @@ -6,6 +6,21 @@ class Home extends BaseController { public function index(): string { - return view('welcome_message'); + $iterator = new \CodeIgniter\Debug\Iterator(); + + $config = config('App'); + global $container; + $container['App'] = $config; + + $iterator->add("config()", static function () { + $config = config('App'); + }); + + $iterator->add('Container', static function () { + global $container; + $config = $container['App']; + }); + + return $iterator->run(300000); } } diff --git a/public/index.php b/public/index.php index 8cf5ce347c..f3166d38be 100644 --- a/public/index.php +++ b/public/index.php @@ -68,6 +68,9 @@ $app->initialize(); $context = is_cli() ? 'php-cli' : 'web'; $app->setContext($context); +// Global container. +$container = []; + /* *--------------------------------------------------------------- * LAUNCH THE APPLICATION ```

[melroyvandenberg]

See the User Guide for details.

I won't want to bother you too much, it could be me, but I'm unable find the FileLocator caching (where the idea is to change /app/Config/Services.php manually yourself?). I'm unable to find FileLocatorCached example in the official user-guide:

https://codeigniter4.github.io/userguide/search.html?q=FileLocatorCached&check_keywords=yes&area=default

I did however found the Config factory section in the user-guide: https://codeigniter4.github.io/userguide/concepts/factories.html#id21. Which also explain how to enable that feature: https://codeigniter4.github.io/userguide/concepts/factories.html#how-to-enable-config-caching.

But again maybe I'm just stupid, but I couldn't find the FileLocator cache stuff. The only mention of FileLocator was here:

The following is a list of the core system classes that are invoked every time CodeIgniter runs:

CodeIgniter\Autoloader\Autoloader
CodeIgniter\Autoloader\FileLocator

[kenjis]

We have two git branches for development. Now it is develop and 4.5. See https://github.com/codeigniter4/CodeIgniter4/blob/develop/contributing/pull_request.md#branching

I meant it was the in-progress user guide on the 4.5 branch. https://github.com/codeigniter4/CodeIgniter4/tree/4.5/user_guide_src There is no HTML user guide for it. You need to build the docs. https://github.com/codeigniter4/CodeIgniter4/tree/4.5/user_guide_src#setup-instructions

In 4.5, we have the "Deployment" page.

[melroyvandenberg]

ow I see. Looking forward for the official 4.5 release 🥳 !

[melroy89]

Small feedback on: https://github.com/codeigniter4/CodeIgniter4/blob/4.5/user_guide_src/source/installation/deployment.rst#adding-htaccess

This is talking about .htaccess and if you might already know, this only works on Apache (Nginx is focused on performance). It would be nice to a section for Nginx as well, with an example snippet of location {} to achieve the same under Nginx server. Currently I now use (for info):

server {
    ///....

    root /var/www/somedomain.com/html/public;

    location / {
        try_files $uri $uri/ /index.php$is_args$args;
    }

    location ~ \.php(?:$|/) {
        include snippets/fastcgi-php.conf;
    }

    location = /robots.txt {
        allow all;
        log_not_found off;
        access_log    off;
    }

     location = /favicon.ico {
        log_not_found off;
        access_log off;
     }

     location ~ /\. {
         deny all;
         return 404;
         access_log off;
    }
}

Ps. my fastcgi-php.conf which I reuse for other projects as well, looks like:

# regex to split $uri to $fastcgi_script_name and $fastcgi_path
fastcgi_split_path_info ^(.+?\.php)(/.*)$;
set $path_info $fastcgi_path_info;

# Check that the PHP script exists before passing it
try_files $fastcgi_script_name =404;

include fastcgi.conf;

## fastcgi settings
# Set PHP handler
fastcgi_pass unix:/run/php/php8.2-fpm.sock;
fastcgi_index index.php;
# Increase timeout
fastcgi_read_timeout 600;
fastcgi_intercept_errors on;
# Disable buffering (for uploading files)
fastcgi_request_buffering off;

# Increase memory size of the buffer segments (used for the payload of the response)
# 32 x 16k
fastcgi_buffers 32 16K;
# Increase memory buffer for HTTP response header
fastcgi_buffer_size 32k;

# fastcgi params
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_param PATH_INFO $path_info;
fastcgi_param HTTPS on;
fastcgi_param modHeadersAvailable true;         # Avoid sending the security headers twice
fastcgi_param front_controller_active true;     # Enable pretty 

But you can argue that only accepting index.php is more secure instead of location ~ \.php(?:$|/) {.

[kenjis]

@melroy89 Thank you for the feedback. But it is off topic. This issue is on CI4 performance. And we already have the page for nginx configuration. https://codeigniter4.github.io/CodeIgniter4/installation/running.html#hosting-with-nginx If you want to improve it, feel free to send PRs!

[melroy89]

But it is off topic.

Sorry about that. I know. I just noticed it, because it was on the same "Deployment" page. Thanks I will look into that other page and create PRs if needed.