Amp v3 with ext-fiber support

[dbalabka]

The author and ext-async extension disappeared for unknown reasons https://github.com/concurrent-php/ext-async, which is very sad.

Thanks to others, the fork is available: https://github.com/dreamsxin/ext-async

@kelunik is there still a plan to implement the 3rd version with ext-async support? Here are some mentions about 3rd version plans:

1. https://github.com/amphp/amp/issues/253#issuecomment-449875440
2. https://github.com/amphp/amp/issues/229#issuecomment-448705155

Thanks!

[kelunik]

The plan still exists, but not sure when that will happen.

[xpader]

The coroutinue implement by third-party extension is not a good choice for amphp I think. There's a coroutinue extension for php is swoole, and pure php implement is amphp. One of the biggest advantages for amphp is no extension required. I like pure php implements, I've been development a coroutinue framework by Workerman+Amphp, but unfortunately amphp v3 use an extension "ext-fiber". If depend an extension is amphp's future, I will choice another framework Hyperf...

[kelunik]

@xpader We plan to make ext-fiber part of php-src.

[xpader]

@kelunik Wow!!! Does it mean that v3 can be released depends on whether ext-fiber can enter php-src? But PHP 8.0 is RC now..

[kelunik]

It's not going to be part of PHP 8, but maybe PHP 8.1.

[dbalabka]

@kelunik is there anything that might help you to bring such a great plan to live?

[trowski]

The coroutinue implement by third-party extension is not a good choice for amphp I think. There's a coroutinue extension for php is swoole, and pure php implement is amphp. One of the biggest advantages for amphp is no extension required. I like pure php implements, I've been development a coroutinue framework by Workerman+Amphp, but unfortunately amphp v3 use an extension "ext-fiber". If depend an extension is amphp's future, I will choice another framework Hyperf...

@xpader ext-fiber is not just a coroutine implementation, it implements fibers (or green-threads), allowing the entire execution stack to be swapped, not just the PHP VM stack, but the entire C stack.

Swoole's coroutines are interesting, but the interface is clunky and the entire library strikes me as poorly thought-out and bloated. ext-fiber works in the background, requiring few changes to existing code. Existing sync interfaces could be implemented as async versions.

With ext-fiber I intend to continue with Amp's tradition implementing most components in PHP and putting only that which is necessary in extension code. Hopefully we will be able to integrate ext-fiber into PHP 8.1 as @kelunik suggested.

@kelunik is there anything that might help you to bring such a great plan to live?

@dbalabka We could certainly use help in further testing and code review of ext-fiber (I'm still learning PHP internals and I probably did something wrong somewhere 😄). We could also use help with proposing the ext for integrating into PHP. The API is a bit unlike existing fiber implementations, but we think it makes sense for integrating with an event loop. I'd prefer feedback at this stage, rather than after proposing it to internals.

[dbalabka]

@trowski I'm ready to help with testing. I’m not C developer but can run this extension on my code.

[enumag]

Is there a repo where I can see the progress of ext-fiber? It would be a big step for async php.

I'm no C programmer but if there is any other way to help, I'd be happy to.

[dbalabka]

@trowski I've found that there is another async/await PHP support development happening:

1. https://github.com/concurrent-php/demo
2. https://github.com/concurrent-php/php-src/pull/1

[trowski]

@trowski I've found that there is another async/await PHP support development happening:

1. https://github.com/concurrent-php/demo
2. concurrent-php/php-src#1

Interesting, I think he had posted something similar some time ago. I'm not sure what his intention is. He originally authored ext-async, but eventually dropped it. Doesn't change my plans to keep moving forward with ext-fiber, the approach is a bit different.

Is there a repo where I can see the progress of ext-fiber? It would be a big step for async php.

I'm no C programmer but if there is any other way to help, I'd be happy to.

@trowski I'm ready to help with testing. I’m not C developer but can run this extension on my code.

The code is at https://github.com/amphp/ext-fiber. There's tests in the repo and PRs for more tests would be appreciated.

There are ext-fiber compatible branches of most of the Amp libraries as a vX branch, (e.g. v3 in amphp/amp, v2 in amphp/postgres) so you can require them with vX-dev in composer.

Take a look at the various examples in the Amp repos to see how boilerplate code may need to be changed/eliminated. If you have questions, let me know here or in chat and I can take a look. Promises have largely been eliminated from public APIs, so in general you'll need swap to returning proper types from functions instead of promises, probably removing a lot of calls to Amp\call() and dropping yield in front of calls previously returning promises.

[enumag]

Can you improve the readme of the extension with some info how to compile it and run the tests? What kind of tests would you like?

[trowski]

Sure, I can add that. It's like many PHP extensions:

phpize
./configure
make
make install

#run tests with:
make test

# add "extension=fiber.so" to your php.ini for running any script:
echo "extension=fiber.so" >> "$(php -r 'echo php_ini_loaded_file();')"

It must be compiled against PHP 8.0-RC1 or later.

[kelunik]

Note that you no longer need the .so suffix, it'll be added automatically.

[enumag]

@trowski cloned and successfully compiled the extension, tests are passing... what's the current status of amphp/http-client? the last commit is wip, does that mean it's not yet expected to work with fiber?

[trowski]

Yes, the WIP commit was me just pushing what I had when I went to bed. I force-pushed a new commit to that branch, http-client should work with ext-fiber now.

[enumag]

With ext-fiber, do we still need an extension for loop (uv/event/ev) and filesystem (uv/eio/parallel)? So far none of these are compatible with PHP 8 but I'm unsure if it's relevant with fiber.

[kelunik]

Yes, nothing changes there.

[enumag]

Reading your RFC I have a bit of a problem understanding your async() function and keyword. I understand that await() will replace current yields but what is the use-case for async()? Where in actual code would I need to use it? Can you provide some example of user-land code using let's say amphp/http-client, amphp/postgres or amphp/file to demonstrate when I need to use the async() function? I think I mainly need to see the same code written with amphp v2 and amphp v3 to understand this.

[trowski]

async() is used when you want to start a new fiber (a.k.a. green-thread or coroutine) and await the result at a later time. It is essentially the v3 equivalent of Amp\call().

A good example of this is concurrent fetches using http-client: Amp v2 vs. Amp v3.

Similarly, defer() in v3 is analogous to asyncCall() in v2, for when you want to create a fiber that doesn't produce a usable result, but carries out a separate task that should run concurrently.

[kelunik]

I wouldn't say async is the equivalent of call. Call is basically no longer necessary. Async is needed where you want to do things concurrently.

[trowski]

Equivalent was a poor word choice there, since we used call() more generally to return promises from functions/methods.

@enumag async(function () { … }) is more of a replacement of new Coroutine((function () { … })());, but the Coroutine constructor was never used like that, instead call(function () { … }) was used. Keep in mind that functions/methods in v3 generally won't return promises anymore, so usage of async() will be much more rare compared to call(), only being used when explicit concurrency is required.

[enumag]

Tbh you got me worried for a while there with "async() is ... essentially the v3 equivalent of Amp\call()." :-D Because my understanding was that call() will be unnecessary. But your other comment and the http-client example explain it fine I think. Thanks!

I really hope the RFC will pass. Thanks for your hard work on that!

[trowski]

Right, the boilerplate needed to use yield in functions/methods and return a promise won't be necessary, but there were use-cases for call() outside of that. Those cases, like the linked example in http-client, still make sense in v3 and that's where async() will be used instead. Sounds like the example demonstrated that well. Sorry to have caused confusion, I'm good at that it seems!

[enumag]

Hmm... another question maybe... Since most code won't directly work with promises and return them, does it mean that amphp v3-based code will use less await() calls than amphp v2-based code used yields? In v2 it was a good practice to return a Promise and not a Generator - at least in the public API of any given class. This caused everything returning a Promise and in turn all calls using yields. But my understanding of v3 so far is that only IO and similar operations (those that would be blocking in sync code) will actually return a Promise / Awaitable, but this will no longer be the case in the rest of the code. So if I understand it correctly await() should also appear less often than yield did in v2, although it willn't be as significant reduction as with call() / async()?

[trowski]

await() will rarely be used – most typically in cases where async() is used to later await the promise returned.

I/O operations in v3 do not return promises, they return the "proper" type (e.g., InputStream::read() returns a string and the query methods in the sql libraries return Result objects).

Promises will sometimes be used internally for async I/O, such as this method in amphp/postgres, where await() is used to return a Result object instead of the promise from the deferred.

Returning promises will be generally discouraged in v3. Functions should use await() internally to return actual values instead. This isn't quite analogous to the v2 rule that public API should never return generators, only promises. In v3 there will still situations where a promise in a public API may make sense, but they will be rare.

So the short answer to your question is yes, await() will appear far less often than yield.

[enumag]

Oooh, okay, that's even better than I thought. 👍 Do you have some rule or example where returning a promise in public API will make sense?

[trowski]

No concrete rules yet, this is all still pretty new. 😃

The only example I can think of in v3-compatible code is the Gateway interface in websocket-server for the methods that communicate to many clients. Those methods still return promises since awaiting them is not likely to be desired, but the promise still provides to ability to do so.

So as an initial "rule", I would suggest that returning a promise makes sense when awaiting the operation is usually not desired.

[Sarke]

Is the plan that v3 requires ext-fiber or it's integration into php-src?

How far away is an alpha release for v3?

[enumag]

@Sarke All of that most likely depends on what the PHP devs say to the Fibers RFC, whether or not it gets accepted and when that decision is made. Currently the discussion is rather quiet - presumably the PHP devs didn't have time to look into it yet due to christmas vacations. At this point it's very likely impossible to answer your questions because it doesn't depend on just the Amphp team.

[Sarke]

So can I therefore infer that the current plan is to have the RFC accepted and then complete amphp v3 with those fibers as a requirement (and min PHP version 8.1)?

[kelunik]

Yes, though the extension can also be installed on PHP 8.0.

[enumag]

@trowski Reacting to your last message in the Fibers RFC discussion. (Sorry for doing it here, I have no idea how to send something there.)

Profilers that manage their own stack of frames will have to be modified to account for fibers.

I think this is worth mentioning in the Are fibers compatible with extensions section of the RFC (or a new separate section). Not the details ofc, just the fact that it does have an impact on profilers.

Or possibly in "Why add this to PHP core?" since the profilers will likely not work correctly with the ext-fiber extension.

[trowski]

There already is the following note in the "Are fibers compatible with extensions" section.

As noted in “Why add this to PHP core?”, extensions that profile code, create backtraces, provide execution times, etc. will need to be updated to account for switching between fibers to provide correct data.

As well as a mention in "Why add this to PHP core."

Extensions that profile code need to account for switching fibers when creating backtraces and calculating execution times. This needs to be provided as a core internal API so any profiler could support fibers. The internal API that would be provided is out of scope of this RFC as it would not affect user code.

Profiling extensions are definitely mentioned as being affected by fibers, however, I can make it clearer that existing profilers will need to be modified to provide correct information when fibers are used.

[Sarke]

Do we think this RFC will pass without the extension being stable in the wild for a bit?

I'm available for testing if you need help with that. We manage about 200 sites and a few webapps, and I would really like to use the benefits of this once it is more stable. Let me know.