zendframework / maintainers

Contributors and maintainers information for Zend Framework.
BSD 3-Clause "New" or "Revised" License
50 stars 26 forks source link

[RFC] Apigility on Expressive #11

Open weierophinney opened 7 years ago

weierophinney commented 7 years ago

The following is a working draft of what we plan to build for an Expressive-based Apigility.

Updates will occur

We will be periodically updating the issue summary with feedback.

Table of Contents

Middleware

These are in order of operation.

Generating responses in middleware

Routed API middleware

Configuring Apigility pipeline middleware

Consider the following pipeline and routing:

// In config/pipeline.php:

$app->pipeRoutingMiddleware();
// additional middleware, such as implicit options, implicit head, etc.
$app->pipe(ApigilityDispatchMiddleware::class);
$app->pipe(NotFoundHandler::class);
// In config/route.php:

$app->get('/api/user', [
    ProblemDetailsMiddleware::class,
    AcceptNegotiationMiddleware::class,
    UserCollectionMiddleware::class,
], 'api.user.collection');

$app->post('/api/user', [
    ProblemDetailsMiddleware::class,
    ContentTypeNegotiationMiddleware::class,
    AuthenticationMiddleware:class,
    AuthorizationMiddleware:class,
    AcceptNegotiationMiddleware::class,
    ContentValidationMiddleware::class,
    UserCollectionMiddleware::class,
]);

$app->get('/api/user/{user_id}', [
    ProblemDetailsMiddleware::class,
    AcceptNegotiationMiddleware::class,
    UserEntityMiddleware::class,
], 'api.user.entity');

$app->route(
    '/api/user/{user_id}',
    [
        ProblemDetailsMiddleware::class,
        ContentTypeNegotiationMiddleware::class,
        AuthenticationMiddleware:class,
        AuthorizationMiddleware:class,
        AcceptNegotiationMiddleware::class,
        ContentValidationMiddleware::class,
        UserEntityMiddleware::class,
    ],
    ['PATCH', 'DELETE']
);

While this is readable and gives the developer an overview in a glance of what will trigger for any given matched route, the problem is that we need to configure each of the pipeline middleware based on the middleware requested; these particular middleware need to be stateful for the given pipeline.

Below are some ideas we've brainstormed.

Fixed schema

One approach is to mimic what we've done in Apigility, and use a fixed workflow for every routed request. For example:

$app->pipe(ProblemDetailsMiddleware::class);
$app->pipe(ContentTypeNegotiationMiddleware::class);
$app->pipe(AuthenticationMiddleware:class);
$app->pipe(AuthorizationMiddleware:class);
$app->pipe(AcceptNegotiationMiddleware::class);
$app->pipe(ContentValidationMiddleware::class);

Each middleware would identify the currently matched middleware (based on either the route name, or, if necessary, by adding functionality to fetch the middleware name within the route match) against configuration in order to retrieve the configuration specific to its context.

Configuration-driven

Alternately, we could create route-specific pipelines, but still map the results of routing to the appropriate configuration. In this particular case, we know for certain at the time this middleware is dispatched:

This would allow such middleware to pull things such as the authenticator, authorization service, accept header whitelist, content-type whitelist, and input filter when dispatched.

The downside to that approach is that these middleware then need access to the container, and we then need to do a lot of logic for testing existence of configuration and/or service keys within this middleware.

Abstract factories using string context

Another approach would be to use an abstract factory that would use a common prefix (e.g., the middleware name) plus specifics (the requested middleware name, and/or request method, and/or a service name — such as the input filter name):

function (ContainerInterface $container, string $name, array $options = null)
{
    $context = str_replace(self::BASE_NAME . '::', '', $name);
    return new ${self::BASE_NAME}($container->get($context));
}

This has the benefit of being cleaner, and letting us fail earlier if a service is unavailable (canCreate() could, for instance, return false if the $context is not available).

The downside is that abstract factories are slower. This could be mitigated somewhat by creating factory entries for the virtual service that point to the abstract factory, however.

Decorator middleware

Another approach would be to use some sort of wrapper:

new ConfigurableMiddleware(
    ContentValidationMiddleware::class, // middleware name
    [UserEntityFilter::class]           // list of services to inject during instantiation
)

This adds some overhead during application initialization (additional objects) and during runtime (proxying). Additionally, we'd need some way for the container to be available for this "ConfigurableMiddleware".

Taking it to another extreme, we could make this into a string:

sprintf(
    '%s(%s, [%s])',
    ConfigurableMiddleware::class,
    ContentValidationMiddleware::class,
    UserEntityFilter::class
)

An abstract factory could then intercept these. That would provide access to the container, and keep some of the benefits of lazy-loading; as noted before, we can mitigate some performance issues by mapping factories, though the names will become quite long.

Custom factories

Another approach, for those really into performance: they can create custom factories for each of these middleware in order to inject exactly the configuration desired. That, however, leads to an explosion of factories.

Pipeline factories

Since the plan is to continue to have an admin API and UI, and thus code generation, one possibility is to generate pipeline factories.

As an example:

function (ContainerInterface $container)
{
    $config = $container->get('config');
    $pipeline = new Application($container->get(RouterInterface::class), $container);

    $pipeline->pipe(ProblemDetailsMiddleware::class));
    $pipeline->pipe(new ContentTypeNegotiationMiddleware([
        'application/json',
        'application/vnd.user.v1+json
    ]));
    $pipeline->pipe(new AuthenticationMiddleware($config['zf-apigility']['authentication']['oauth2']));
    $pipeline->pipe(new AuthorizationMiddleware($container->get(UserAcl::class));
    $pipeline->pipe(new AcceptNegotiationMiddleware([
        'application/json',
        'application/vnd.user.v1+json
    ]));
    $pipeline->pipe(new ContentValidationMiddleware($container->get(UserFilter::class));
    $pipeline->pipe(new UserEntityMiddleware(
        $container->get(UserTable::class),
        $container->get(HalResponseGenerator::class)
    );

    return $pipeline;
}

The above could also be part of a delegator factory instead:

function (ContainerInterface $container, $name, callable $callback)
{
    $config = $container->get('config');
    $pipeline = new Application($container->get(RouterInterface::class), $container);

    $pipeline->pipe(ProblemDetailsMiddleware::class));
    $pipeline->pipe(new ContentTypeNegotiationMiddleware([
        'application/json',
        'application/vnd.user.v1+json
    ]));
    $pipeline->pipe(new AuthenticationMiddleware($config['zf-apigility']['authentication']['oauth2']));
    $pipeline->pipe(new AuthorizationMiddleware($container->get(UserAcl::class));
    $pipeline->pipe(new AcceptNegotiationMiddleware([
        'application/json',
        'application/vnd.user.v1+json
    ]));
    $pipeline->pipe(new ContentValidationMiddleware($container->get(UserFilter::class));

    // Inject originally requested middleware:
    $pipeline->pipe($callback());

    return $pipeline;
}

In either case, the pipeline would not be provided as an array, but simply a service name:

$app->route('/api/user/{user_id}', UserEntityMiddleware::class, ['PATCH', 'DELETE']);

This would simplify the config/routes.php file.

The examples above provide a mix of service-based middleware, hard-coded dependencies, and middleware composing other services. While it's possible some of the middleware will never be executed (e.g., if authentication fails, none of the following four items would execute), most dependencies are such that no extra work happens unless they are invoked, meaning minimal overhead. (Certainly less overhead than we had in the zend-mvc-based Apigility itself!) Users who can prove need to further streamline performance could always wrap these in anonymous class decorators.

Using a delegator factory makes re-use easier, which could also mean separate middleware per method:

$app->get('/api/user', UserListMiddleware::class, 'api.user.collection');
$app->post('/api/user', UserCreateMiddleware::class);
$app->get('/api/user/{user_id}', UserDisplayMiddleware::class, 'api.user.entity');
$app->patch('/api/user/{user_id}', UserUpdateMiddleware::class);
$app->delete('/api/user/{user_id}', UserDeleteMiddleware::class);

Each of the second, fourth, and fifth entries above would compose the same delegator factory detailed above, giving them the exact same workflow. (Technically, a delete operation would likely not need the ContentValidationMiddleware, however.)

The problem with the approach is code generation. Generating configuration is easy. We'd likely need to re-generate these delegator factories whenever something is manipulated in the admin API/UI, meaning they would also need to be necessarily marked as "DO NOT EDIT". (We could have a rule that if certain artifacts are missing from the file when we prepare to regenerate, we raise a warning in the UI instead.)

Recommendation

Configuration-driven is likely the easiest approach, though it hides many details in configuration instead of code. It would be the easiest from a migration standpoint.

The last approach, creating the pipeline in a dedicated delegator factory, appeals as it retains explicitness and provides re-use, while retaining simplicity in routing.

UI Ideas

Other Questions

Initial roadmap

We propose the following:

We can continue discussing architecture for other aspects while the above are worked on.

harikt commented 7 years ago

I will vote for moving away from zend-mvc and full support via zend-expressive. Easy to read and understand. I do agree it is a bit of work for people to migrate. But in the long run, if using psr-7 means people can plug to other psr-7 based frameworks easily.

harikt commented 7 years ago

@weierophinney can you update the post to keep the link to RestDispatchTrait . Also link to other places when pointing to ideas / implementation will be helpful.

weierophinney commented 7 years ago

I will vote for moving away from zend-mvc and full support via zend-expressive.

@harikt : Is this a vote for a separate project, or making Apigility v2 based on Expressive?

harikt commented 7 years ago

@weierophinney I would not vote for a separate project . I will vote for making Apigility v2 based on Expressive .

Why not a separate project ?

In the long run it will be hard to track both projects. Maintainers have lots now in the bucket :-) .

weierophinney commented 7 years ago

Why not a separate project ?

In the long run it will be hard to track both projects.

While I agree, it's also a strange situation: most of the code cannot be directly re-used, and, in many cases, there will not be 1:1 correlations between existing Apigility modules and their middleware replacements (e.g., versioning; authentication and authorization would be split; etc.).

The other factor would be v1 users coming to the project, and seeing, by default, code that does not resemble what they have installed, and wondering how and where to submit patches.

Essentially, a lot of factors to consider here. But thanks for clarifying your vote!

harikt commented 7 years ago

@weierophinney where can we track the progress for apigility v2 ? Will there be an update regarding the choice as a separate project or based on expressive ?

I am interested to look in case I get some time.

weierophinney commented 7 years ago

where can we track the progress for apigility v2 ? Will there be an update regarding the choice as a separate project or based on expressive ?

For now, here! We haven't started on any of the various middleware yet; once we do, we'll update the summary to indicate where that development is happening. Additionally, when we decide on whether to continue as the same or a new project, we'll note that as well, by crossing out the question and noting the decision.

gsomoza commented 7 years ago

If from an architectural point of view making Apigility middleware-driven is "the way forward" (and I think it is) then the decision of making it "Apigility 2" vs a "new project" will probably come down mostly to branding: after all, Apigility v2 would still mean you'd have two code-bases to maintain, especially if the upgrade path is not simple and people take their time to do it.

I think most people using Apigility would (or should) understand a thing or two about versioning anyways, and would therefore understand that Apigility v2 has a very different architectural approach than it's predecessor. The branding aspect would then remain intact: "Apigility" will still be a platform where you can quickly release and manage APIs without having to worry about tons of boilerplate. Therefore I'd vote for making this Apigility v2.

With more time, I'll see if I can contribute my two cents to some of the other items.

gsomoza commented 7 years ago

I would also like to add that keeping this mostly a configuration-driven project is almost a "must" from my point of view.

And the Admin UI is of secondary importance to me: I barely even use it.

weierophinney commented 7 years ago

I would also like to add that keeping this mostly a configuration-driven project is almost a "must"

Can you elaborate on why you feel this way, please? (Genuinely curious, and would like to hear your perspective.)

gsomoza commented 7 years ago

Well, I often work building solutions for medium/large clients (e.g. enterprise) that sometimes have very weird requirements or constraints. Often that means they don't always follow standards perfectly, or not for all endpoints, etc.. So I prefer to use tools that allow for the most flexibility in their configuration to avoid me the need to overwrite much in the underlying framework. Convention-driven tools in my experience are harder to bend to a client's unique - and probably very unconventional - needs.

More generally speaking, I think Apigility's main value is that it provides the boilerplate needed to create and maintain "beautiful" APIs (that definition may vary from person to person). With good guidance, a developer can be productive in an Apigility project without understanding much about the inner workings of Apigility itself. But on a more convention-driven architecture there's a bigger cognitive entry-barrier (depending on how far you go with the conventions) because in order to be productive you'd have to learn some of those conventions. For Apigility contributors it might also make it a bit harder to make changes to the framework over time, cause it's not just code you're managing, but also the conventions. So IMO a good balance between convention and configuration would be nice, but I'd much prefer to err on the side of configuration than convention.

Having said that, if by convention-driven you mean "sensible defaults", then I'm all for that. Apigility already does a good job at that.

PowerKiKi commented 7 years ago

This is certainly far fetched, but has there been any consideration for GraphQL ? Would some kind of configuration make it possible to switch between REST and GraphQL ? or drop REST entirely in favor GraphQL ? or is there another project in Zend ecosystem that would better address that ?

nomaan-alkurn commented 7 years ago

Just a little suggestion about Apigility Admin UI. We can use latest Angular 4 in Apigility v2.

weierophinney commented 7 years ago

@nomaan-alkurn We will definitely be updating the UI to use modern libraries. The question will be whether that will be Angular 4, or something else; we will likely poll our contributors to see what they are most familiar and experienced with before making a decision.

weierophinney commented 7 years ago

@PowerKiKi GraphQL and REST are not equivalent; in many cases, they are orthoganal. Phil Sturgeon has written some excellent articles on this subject that show why.

I do think we should likely look at GraphQL and figure out if there's something we could do with it in Apigility; I think that can be a phase 2 story, however.

wshafer commented 7 years ago

As a professional user of Apigility, ZF2+3, and Expressive, I would vote that this be Apigility 2.0 not a separate project and move to Expressive. After having worked with all the frameworks, it seems to me that Expressive is more geared for this kind of project then ZF3+.

As a side note, one of the major problems I have in Apigility + Doctrine is that in the current configuration Entities can only define one hydrator. It would be nice if in 2.0 if Apigility allowed different versions to specify the same target entities, but allow for different versioning hydrators. Since in Doctrine our entities are the source of truth to the DB schema, versioning these leads to errors, would be much better in this world to have versioned hydrators for the entities then versioning the entities themselves.

On that same note, it would also be nice to load each versions config file as needed (which would also solve the above problem). So if a client asks for version 2 we only load the configuration for version 2, instead of the configs for all versions.

Other then that, I like where this thread is going.

snapshotpl commented 7 years ago

Expressive it's simple, powerful and elastic so it's great tool for Apigility!

weierophinney commented 7 years ago

@wshafer —

one of the major problems I have in Apigility + Doctrine is that in the current configuration Entities can only define one hydrator

This is something to raise in the zf-apigility-doctrine module, as it can likely be addressed now.

it would also be nice to load each versions config file as needed (which would also solve the above problem). So if a client asks for version 2 we only load the configuration for version 2, instead of the configs for all versions

There's actually a reason for this: we inherit configuration from previous versions. As such, it means that the configuration for, say, version 3, may only have a few changes to those over v2, and v2 to v1. By using inheritance, if changes are made to earlier versions of the configuration, we don't need to worry about whether those changes are propagated to later versions.

wshafer commented 7 years ago

@weierophinney - Don't know why I was thinking about this today, but I wonder... What if Apigility V2 is not an all or nothing approach?

What if you combine the ideas? Apigility v2 adds all the middleware layers and uses the psr7 bridge and middleware layer in ZF with a fallback to the MVC stack as is? This gives us a migration path to Apigility v3 where we replace the MVC stack for expressive?

weierophinney commented 7 years ago

@wshafer — You could certainly compose an Expressive application itself as the middleware in the zend-mvc MiddlewareListener. The bigger problem is that a lot of the configuration and runtime aspects of the current Apigility codebase are very much tied to the zend-mvc workflow. Most items are exposed as event listeners (typically on the route event), and consume or produce zend-http messages. Additionally, much of the configuration uses controller class names for lookups, which while they can map to middleware, don't do so cleanly.

In looking through most of the Apigility modules, there's no clean way to make them work under both the zend-mvc and middleware paradigms. There's also a logistical issue: we get a lot of pushback about having optional dependencies; if we support both, we'd have to make both the various zend-mvc components and PSR-7/PSR-15/Expressive components optional, meaning the component cannot work out-of-the-box unless you first choose the components that fit the architecture you're using.

Additionally, if we take this route, the endgame would be to support only middleware, and that fact poses new problems. The support experience will be difficult for those on v1 if the default branch becomes v2 and has a completely different architecture; reporting bugs is harder, as is creating patches. For those maintaining, merging and creating releases becomes quite a lot more difficult. Having separate repositories makes these stories easier.

With all those points in mind, I'd argue the migration path from the current Apigility to an Expressive-based one will be the same as a ZF2/3 application to Expressive: either gradually migrating to middleware services in your MVC until you can switch over entirely, or composing a middleware as a fallback in your Expressive stack that executes the zend-mvc application.

What we may be able to do is write tools that convert configuration from the current Apigility to whatever the new version is. We would not necessarily be able to convert existing classes, however, as those are heavily tied to the zend-mvc workflow (even the zend-rest Resource classes use the eventmanager and expect zend-mvc events!). However, this would at least provide some initial working routes for users, so that they can start migrating their code into a working, if empty, application.

corentin-larose commented 6 years ago

Hello, catching up with all your excellent work. I will start working on the migration of zf-http-cache to another middleware, which sounds about perfect for this module, as soon as I read all the codebase in order to "take the mood" of the project. I think moving toward an Expressive powered Apigility is the final step to make Apigility a de facto standard in PHP APIs management solutions.

tylkomat commented 6 years ago

I also vote for Apigility 2, while in the end it doesn't matter. In the long run, the Old will disappear. In any way there should be a documentation how to migrate from the Old to the New. New projects will likely be started with the New, because developers like to try out new things. The New is always "hyped". If it is a smaller project people will like to migrate, since the New feels better (and also I expect it to be better and easier to learn).

Generally building on Expressive is the right choice. The Pipeline approach is much more clean than the event architecture where most things happen "in the background" and can't be followed without deep knowledge of the framework.

mrVrAlex commented 6 years ago

And so ... after year of decisions which option is chosen? Use old zfcampus/zf-apigility repo? Any news on apigility "next"? Expressive already 3.0. :)

geerteltink commented 6 years ago

@shandyDev The most recent answer I could find: https://discourse.zendframework.com/t/create-api-with-expressive/411/9