Simplified syntax / higher-level abstractions for making HTTP Requests with schema decoding and retries

[ridler]

What is the problem this feature would solve?

In @effect/platform version 0.63, HTTP requests could be retried and decoded with this syntax:

async function listTodos(headers: Headers, searchParams: URLSearchParams) {
  return Effect.runPromise(
    HttpClientRequest.get(`${url}?${searchParams.toString()}`, { headers }).pipe(
      HttpClient.fetchOk,
      HttpClientResponse.json,
      Effect.retry(Schedule.exponential(1000).pipe(Schedule.compose(Schedule.recurs(3)))),
      Effect.andThen((data) => Schema.decodeUnknownSync(todoListSchema)(data)),
    )
  )
}

The above code strikes me as something that an average TypeScript application developer could understand and maintain. It is also consumable by code that doesn't have to know about Effect.

Now, in version 0.66, we would have to write generator functions and classes to achieve the same thing:

https://github.com/Effect-TS/effect/blob/main/packages/platform/README.md#integration-with-schema

import {
  FetchHttpClient,
  HttpClient,
  HttpClientRequest
} from "@effect/platform"
import { Schema } from "@effect/schema"
import { Effect } from "effect"

const program = Effect.gen(function* () {
  const client = yield* HttpClient.HttpClient
  const addTodo = HttpClient.schemaFunction(
    client,
    Schema.Struct({
      title: Schema.String,
      body: Schema.String,
      userId: Schema.Number
    })
  )(HttpClientRequest.post("https://jsonplaceholder.typicode.com/posts"))

  const response = yield* addTodo({
    title: "foo",
    body: "bar",
    userId: 1
  })

  const json = yield* response.json

  console.log(json)
}).pipe(Effect.scoped, Effect.provide(FetchHttpClient.layer))

Effect.runPromise(program)
/*
Output:
{ title: 'foo', body: 'bar', userId: 1, id: 101 }
*/

and from the effect.website home page, retries:

import { FetchHttpClient, HttpClient } from "@effect/platform"
import { Console, Effect, Layer } from "effect"

const makeUsers = Effect.gen(function*() {
  const client = (yield* HttpClient.HttpClient).pipe(
    HttpClient.filterStatusOk
  )

  const findById = (id: number) =>
    client.get(`/users/${id}`).pipe(
      Effect.andThen((response) => response.json),
      Effect.scoped,
      Effect.retry({ times: 3 })
    )

  return { findById } as const
})

class Users extends Effect.Tag("Users")<Users, Effect.Effect.Success<typeof makeUsers>>() {
  static Live = Layer.effect(Users, makeUsers).pipe(
    Layer.provide(FetchHttpClient.layer)
  )
}

const main = Users.findById(1).pipe(
  Effect.andThen((user) => Console.log("Got user", user))
)

The new documented syntax is not something I would feel comfortable advocating that a real team of average TypeScript application developers adopt, since it is significantly more complex at the application logic layer. The new documented syntax is also more complex than achieving the same thing without any dependencies, as demonstrated on the website's home page:

I was super excited about the Effect project's way of solving the request / decode / retry class of problems, but I'm now feeling very nervous about introducing it into my project's code, and am honestly considering abandoning it as a dependency over performing the upgrade and refactor indicated by the example code above due to the new bits of complexity team members would have to think about and maintain:

• clients
• layers
• scopes
• generator functions / classes (While these are reasonable things for lower-level users of Effect to understand, requiring them for retry-able HTTP requests at the application logic layer feels like a major barrier in approachability / adoptability of the library)

What is the feature you are proposing to solve the problem?

I'd love to have higher level abstractions over HTTP Client modules, similar to what was in version @effect/platform 0.63 (but not necessarily identical). I am happy to refactor in order to upgrade, and do not expect stability of these v0 APIs, but the increased complexity from application developers' point of view with the recent changes to:

import { Http* } from '@effect/platform';

...has been very hard to digest.

But thank you so much for making this open source library! I understand that it's hard to make APIs that please everyone, and really appreciate all the work that has gone into every version, even the one I'm hoping will change 💕

What alternatives have you considered?

No response

[mikearnaldi]

The issue with the old API is lack of composability having a mix of default instances and context-accessed apis, the current version is standard in effect, the default way to expose functionality is via services and accessing services is first-class supported in effect

[mikearnaldi]

that said my feeling is that the HttpClient apis need complete reset, we are trying way too hard to abstract and the current API is not usable by the average developer (including myself), it's not a doc problem it's an API problem.

[tim-smart]

Some changes have been made to bring the same service methods (httpClient.*) to the module as well.

This makes it easier to use a client without using the service approach. Here are some usage patterns that are available now:

import { FetchHttpClient, HttpClient, HttpClientResponse } from "@effect/platform"
import { BrowserHttpClient } from "@effect/platform-browser"
import { NodeHttpClient } from "@effect/platform-node"
import { Schema } from "@effect/schema"
import { Effect, ManagedRuntime, Schedule } from "effect"

const todoListSchema = Schema.Array(Schema.Unknown)

// you could provide a client directly:

async function listTodos(url: string, headers: Headers, searchParams: URLSearchParams) {
  return Effect.runPromise(
    HttpClient.get(`${url}?${searchParams.toString()}`, { headers }).pipe(
      Effect.flatMap(HttpClientResponse.filterStatusOk),
      Effect.flatMap(HttpClientResponse.schemaBodyJson(todoListSchema)),
      Effect.scoped,
      Effect.retry({
        times: 3,
        schedule: Schedule.exponential(1000)
      }),
      Effect.provide(FetchHttpClient.layer)
    )
  )
}

// or use a ManagedRuntime:

const runtime = ManagedRuntime.make(
  FetchHttpClient.layer
  // or one of these:
  //
  // NodeHttpClient.layer,
  // NodeHttpClient.layerUndici,
  // BrowserHttpClient.layerXMLHttpRequest
)

async function listTodos(url: string, headers: Headers, searchParams: URLSearchParams) {
  return runtime.runPromise(
    HttpClient.get(`${url}?${searchParams.toString()}`, { headers }).pipe(
      Effect.flatMap(HttpClientResponse.filterStatusOk),
      Effect.flatMap(HttpClientResponse.schemaBodyJson(todoListSchema)),
      Effect.scoped,
      Effect.retry({
        times: 3,
        schedule: Schedule.exponential(1000)
      })
    )
  )
}

// or the recommended service approach:

export class Todos extends Effect.Service<Todos>()("Todos", {
  effect: Effect.gen(function*() {
    const client = HttpClient.filterStatusOk(yield* HttpClient.HttpClient)

    const list = (url: string, headers: Headers, searchParams: URLSearchParams) =>
      client.get(url, {
        headers,
        urlParams: searchParams
      }).pipe(
        Effect.flatMap(HttpClientResponse.schemaBodyJson(todoListSchema)),
        Effect.scoped,
        Effect.retry({
          times: 3,
          schedule: Schedule.exponential(1000)
        })
      )

    return { list } as const
  }),
  dependencies: [NodeHttpClient.layerUndici]
}) {}