otel will increase CPU usage

[runtu666]

I did a stress test. Under the condition of 20 concurrency per second, the CPU usage data after turning on tracing is as follows:

scene	concurrent	average time	TPS	CPU
0%	20	115ms	172	0.98U
10%	20	110ms	179	1.21U
50%	20	124ms	158	1.89U
100%	20	136ms	146	2.53U

[brettmc]

Please check whether you are using the native or extension version of protobuf, since that's the most obvious cause of extra CPU usage. If that is not the issue, then please provide more detail about your setup, and ideally a docker-based reproduction of the issue.

[runtu666]

https://github.com/opentelemetry-php/example-auto-yii/tree/main/fpm

I tested it with this image. I saw that there was no protobuf expansion inside. Is it caused by this?

[brettmc]

Yes, using the native protobuf library is known to be poor for resource usage, see https://opentelemetry.io/docs/instrumentation/php/#ext-protobuf

Please try again with the extension installed and report back :)

[runtu666]

machine	sences	Threads	average	TPS	CPU	memory
C4M4	Do not import OTEL package	20	65	305.9	0.86C	124M
C4M4	import OTEL package && off switch	20	65	306.7	0.84C	133M
C4M4	1% ratio	20	73	272.8	1.65C	130M
C4M4	10% ratio	20	72	275.8	1.75C	158M
C4M4	50% ratio	20	75	263.0	1.81C	128M
C4M4	100% ratio	20	79	251.6	1.89C	128M

After this test, I found that as long as OTEL is turned on, the CPU usage will double. @brettmc

here is my dockerfile:

ARG PHP_VERSION=8.2 FROM yiisoftware/yii2-php:${PHP_VERSION}-fpm

RUN apt upgrade

RUN apt install $PHPIZE_DEPS

RUN pecl install opentelemetry protobuf RUN docker-php-ext-enable opentelemetry protobuf RUN docker-php-source delete

RUN cp /usr/local/etc/php/php.ini-development /usr/local/etc/php/php.ini

WORKDIR /var/www/html

RUN composer create-project --prefer-dist yiisoft/yii2-app-basic . RUN composer require \ open-telemetry/opentelemetry-auto-yii \ open-telemetry/sdk \ open-telemetry/exporter-otlp \ php-http/guzzle7-adapter

[brettmc]

Thanks, that is interesting. The one I'm most interested in is the 1%, where CPU usage appears to almost double, but it's presumably not doing very much exporting. Can you profile a couple of the requests (eg with xdebug) to get an idea of where the expense may be coming from? (eg one that was traced and one that was not, to compare)

I'd also like to know:

• which exporter (otlp+protobuf, otlp+json, ?)
• simple or batch span processor?

[runtu666]

I didn't do anything special, just set the following environment variables my config is

• name: OTEL_PHP_AUTOLOAD_ENABLED value: 'true'
• name: OTEL_TRACES_EXPORTER value: 'otlp'
• name: OTEL_EXPORTER_OTLP_PROTOCOL value: 'http/protobuf'
• name: OTEL_EXPORTER_OTLP_ENDPOINT value: 'http://tracing-otelcol-collector.opentelemetry-operator-system:4318'
• name: OTEL_SERVICE_NAME value: 'otel-demo-php-agent'
• name: OTEL_TRACES_SAMPLER value: 'traceidratio'
• name: OTEL_TRACES_SAMPLER_ARG value: '0.5'

@brettmc

[brettmc]

@runtu666 that's all pretty normal config, I agree. Could you use a profiler (eg xdebug's) against a sampled and not-sampled request (eg with traceidratio=100 and 0) to try to see where the resource utilisation comes from?

[runtu666]

I'm not a professional phper, I don't know xdebug @brettmc

[brettmc]

I'm not a professional phper, I don't know xdebug

I've profiled a few scenarios using yii + autoinstrumentation + otlp/console exporting. I didn't think anything really stood out as expensive in terms of #calls or time spent (which is all I get from xdebug). I've attached a couple of output files and you can look at them in a tool such as kcachegrind described in the xdebug docs

I thought resource detection looked like the highest time spent amongst the opentelemetry calls - you could test turning that off entirely with OTEL_PHP_DETECTORS=none to see if that makes a difference.

I think the most useful profile to look at is cachegrind.out.helloworld-console.gz below - it just starts and stops a span, and prints "hello world" - literally the simplest application with tracing that there could be (and no export delays etc to worry about) - that's where I see some time spent on finding resource attributes.

cachegrind.out.yii-console.gz cachegrind.out.helloworld-console.gz

Also, keep in mind that you're load testing the simplest Yii app possible, it doesn't do very much work. I think if you were to add in some i/o (database calls, for example) then the relative expense of otel would be less?

edit: it's also clear from looking at profiles that we go through http client discovery + resource detection even if we don't end up recording a span. That's a little wasteful for a shared-nothing setup like apache or nginx, and maybe we could do something about lazy-loading...

[brettmc]

@runtu666 some potential performance improvements have been made around resource creation. Can you update to the latest versions and re-run your tests?

[runtu666]

Extended version? Or is it the SDK version? @brettmc

[brettmc]

SDK, and otlp exporter.

[runtu666]

OK，Which version is the latest?

[brettmc]

composer update should update all packages to their latest versions.

[runtu666]

ok

[brettmc]

@runtu666 were you able to re-test this? If so, can you post the results?

[runtu666]

@brettmc yes ， After the Spring Festival

[runtu666]

machine	sences	Threads	average	TPS	CPU	memory
C4M4	do not improt	20	187	106.1	1.43	502
C4M4	off	20	189	105.2	1.44	486
C4M4	1%	20	214	92.6	1.49	517
C4M4	10%	20	195	102.0	1.79	505
C4M4	50%	20	239	83.1	1.76	504
C4M4	100%	20	256	70.1	1.34	503

I tested it with real business, and it didn’t cause too much CPU usage. @brettmc

[runtu666]

BTW，is here a package for auto-redis?

[brettmc]

Thanks for the updated test results. It sounds like the overhead is not too bad for non-trivial applications, which is good news! I'll close this issue.

BTW，is here a package for auto-redis?

No. We're always happy to accept contributions if you'd like to try creating one, though!