[Proposal] Extension block

[mcandeia]

Extension Blocks

Author: Marcos Candeia (@mcandeia) State: Discussion

Overview

When using loaders for fetching data from APIs, it is common to need to add (or change) new fields to existing returns. This can be a challenging task when working with imported loaders, such as the ones that returns Product from schema.org. One feasible solution would be forking the loader source code and apply the necessary modifications, the drawback is that now you need to give up from getting automatic updates from the loader's creators. Another solution would be just import the loader and add your new fields, which makes you to be aware and replicate it for every new loader that is implemented, so let's say you have 10 loaders that returns Products, so now you have to import/export all of them. Let's take the as an example a real-world use case of adding the reviews of a product (the number of "stars" of a given product).

This task requires changing the aggregateRating property of Product types, please notice the following conditions;

1. It can't be enable by default because it requires configuration (e.g secrets that will be used to fetch such information)
2. For this scenario we want to extend an existing type without modifying the source code, this should be possible especially because the developer who's responsible for adding the ratings feature can have no access to the source code repository that own the loader source code.
3. Not every product needs such information and sometimes it is tied to the platform, so the way you get product ratings can vary across loaders at the same site.

One solution to this problem is the use of extension blocks, which allow developers to add new fields to existing types without modifying the source code. Extension blocks provide a way to "extend" types with additional functionality, without having to modify the original source of data.

Background

There are multiple challenges when extending existing types, including;

Codebase fragmentation

When a new property needs to be added or modified, the codebase may become fragmented as different parts of the application may be affected. This can lead to increased complexity and difficulty in maintaining the codebase.

Dependency management

If the loaders are dependent on other sites, a change to the property may require updating those dependencies as well. This can lead to conflicts with other parts of the application that depend on different versions of the same site.

Testing and validation

Any changes to the property require testing and validation to ensure that they do not introduce bugs or unintended behavior. This can be time-consuming and expensive, especially if the changes affect critical parts of the application.

Documentation and communication

When a new property is added, it is important to update the documentation and communicate the changes to other developers who may be affected. This can be challenging if there are multiple loaders or if the changes are complex.

Maintainability and backward compatibility

Finally, any changes to the property need to be done in a way that maintains backward compatibility and does not break existing code. This can be difficult if the property is deeply integrated into the codebase or if there are many dependent modules.

Detailed design

Extension blocks are implemented using a simple and effective design pattern. The basic idea is to provide a modular way to extend existing code without modifying the source code itself. The implementation is quite straightforward, and it involves a few simple steps.

Creating the Product extension

First, the developer creates a function that takes the original type and returns an extended type. This function is referred to as an extension block. The extension block can be used to add new properties or methods to the original type.

The following code is the code that would be used to add the aggregateRatings into an existing product instance.

import { Product } from "deco-sites/std/commerce/types.ts";
import {
  ConfigYourViews,
  RatingFetcher,
} from "deco-sites/std/commerce/yourViews/client.ts";
import { ExtensionOf } from "https://denopkg.com/deco-cx/live@3c5ca2344ff1d8168085a3d5685c57100e6bdedb/blocks/extension.ts";
import { createClient } from "../commerce/yourViews/client.ts";

export type Props = ConfigYourViews;

const aggregateRatingFor =
  (fetcher: RatingFetcher) => async ({ isVariantOf }: Product) => {
    const productId = isVariantOf!.productGroupID;
    const rating = await fetcher(productId);

    return rating
      ? {
        "@type": "AggregateRating" as const,
        ratingCount: rating.TotalRatings,
        ratingValue: rating.Rating,
      }
      : undefined;
  };

export default function AddYourViews(config: Props): ExtensionOf<Product> {
  const client = createClient(config);
  const aggregateRating = aggregateRatingFor(client.rating.bind(client));

  return {
    aggregateRating,
  };
}

This code should live within the extensions/ folder with an arbitrary name. The format of an extension is basically the same fields as the product that we want to extend by instead of returning the values directly developers can fetch data from APIs for every field that needs to be modified/added. Also, the extension function is a function that has the following signature

export type ExtFunc<
  T,
  TBase,
  IsParentOptional,
  PropIsOptional = IsParentOptional,
> = (
  arg: TBase,
  current: IsParentOptional extends true ? T | undefined : T,
) => PromiseOrValue<
  PropIsOptional extends false ? DeepPartial<T> : DeepPartial<T> | undefined
>;

Where:

1. T is the current property value, for instance in the case of aggregateRating is gonna be the current aggregateRating value.
2. TBase is the entire target object, in this case the entire Product
3. The return is a DeepPartial<T> which means that the result will be merged with the original object.

Optionally, when dealing with collections that should be changed as a whole (a new property should be added or changed on each element) a simple property name _forEach is allowed to provide a function that will be used for each element.

The example below show how to add +10 on every price inside the offers array (yes, the type Product has offers.offers proerty (the latter is an array and the former an object)).

export default function Add10Price(): ExtensionOf<Product> {
  return {
    offers: {
      offers: {
        _forEach: {
          price: (p: Product, curr: number) => curr + 10,
        },
      },
    },
  };

The WithExtensions loader

A new loader is being added alongside the extensions block, the WithExtensions loader, which has basically a single task: get data (products in this case) from loaders and apply the configured extensions transformations. This is a simple loader that can be used on any field that accepts a loader, and it has basically two properties: The data and the extension, the WithExtensions loader is used as a middle-man to get data from loaders and apply the transformations in parallel, merging them together.

This is the proposed implementation for such loader.

export interface Props<T> {
  data: T;
  extension: Extension<T>;
}

export default async function withExtensions(
  _req: Request,
  ctx: LoaderContext<Props>,
) {
  const extended = await ctx.state.$live.extension?.(ctx.state.$live.data);
  return extended?.merged(); // this return the extension applied to the target object
}

Composite extension

As you can see in the previous example the loader contains only one extension property and not an array of them. This is only for simplicity to avoid code duplication when dealing with multiple extensions, for that, I propose to have a Composite extension that receive an array of extensions and compose them together as a single one, which makes really easy to allow extensions on other blocks in the future. You can see the proposed code below

import { Extended, Extension } from "$live/blocks/extension.ts";
import { notUndefined } from "$live/engine/core/utils.ts";
import { deepMergeArr } from "$live/utils/object.ts";
import { DeepPartial } from "https://esm.sh/v114/utility-types";

export interface Props {
  extensions: Extension[];
}
const apply = <T, R>(param: T) => (f: (arg: T) => Promise<R>) => f(param);

export default function composite({ extensions }: Props) {
  return async <TData>(data: TData) => {
    const applied = (await Promise.all(
      extensions?.filter(notUndefined).map(
        apply(data),
      ),
    )) as Extended<TData>[];
    return applied.reduce(
      (finalObj, extended) =>
        deepMergeArr<DeepPartial<TData>>(
          finalObj,
          extended.value,
        ),
      {},
    );
  };
}

One key advantage of this approach is that it allows for composability of extensions. Since each extension block creates a separate instance of the extended type, multiple extension blocks can be combined to create even more complex objects. This makes it easy to add new functionality to existing code without modifying the original source.

Overall, extension blocks are a powerful tool for developers looking to extend existing code in a modular and composable way. By allowing for easy extension of existing types and objects, extension blocks help to improve code maintainability and reduce the need for code duplication.

Important to mention that only one task for each persona is required,

For developers who want to extend existing types:

1. A new extension needs to be added, thus a new extension module inside extensions/ folder.

For business users:

1. Configure the new extension on existing loaders.

Expectations and alternatives

Completion checklist

[ ] Create the extension block [ ] Update documentation

[mcandeia]

It's worth noting that the following items were intentionally left out of this discussion in order to focus exclusively on the extension feature:

1. Saving a loader as a partial
2. Copying from an existing partial save
3. Global loaders

[mcandeia]

WIP https://github.com/deco-cx/live.ts/pull/163/files

[lucis]

Looks good, @mcandeia!

I think that this pushes us even forward to work as an "API orchestrator" since this is a very common use case in user-facing applications: get data from a primary source (database) and complement that with data from some other API (ex: blogs from Wordpress, comments from https://commento.io/).

This gets more powerful if we can augment schema.org's Schema and provide complete and extensible (e.g additionalProperty) + Universal Components for them, so it's even more attractive to create an integration in our platform.

I think sooner rather than later it'll be good to provide batching capabilities as well, but we're almost there.

[mcandeia]

Closed this as it was achieved by returning a function using loaders.