Feature request: A combinator that would work like D.array but would always succeed, only discarding the failed items.

[kmbt]

🚀 Feature request

Current Behavior

There is only one way to parse arrays - D.array which fails completely if any item fails.

Desired Behavior

A combinator that would work like D.array but would always succeed, only discarding the failed items.

Suggested Solution

I have written a prototype combinator failsafeArray for this, that works in a two-step manner: Firstly, the collapse combinator recovers a failed decoder and returns E.Right(undefined). After this, undefined items ale filtered out and the original type is recovered. I guess the use of undefined is not particularly elegant here and better solution could be achieved with internal Decoder features which are not documented.

import * as D from 'io-ts/Decoder'
import * as E from 'fp-ts/Either'

const collapse = <I, A>(d: D.Decoder<I, A>) => ({
  decode: (i: I) => pipe(
    i,
    d.decode,
    E.fold<D.DecodeError, A, E.Either<D.DecodeError, A | undefined>>(
      (e) => E.right(undefined),
      (a) => E.right(a)
    )
  )
})

const failsafeArray = <A>(d: D.Decoder<unknown, A>) => pipe(
  d,
  collapse,
  D.array,
  D.parse(xs => E.right(A.filter((x): x is A => x !== undefined)(xs)))
)

Who does this impact? Who is this for?

Personally, I found that it is useful when parsing an array of items returned from an API, when I only care about the ones which match my decoder.

Discarding the items which do not parse also enables parsing certain attributes into unions of literals instead of strings while making the parser more future-proof and liberal.

Consider the following type with represents a hypothetical array of product attributes from an API:

{
   id: number,
   slug: string,
   value: string,
}[]

I might however like to parse it into:

{
   id: number,
   slug: "width" | "height" | "color",
   value: string,
}[]

while discarding the items which do not match.

This might be useful for anybody parsing collections of items to which certain identifiers might be added in the future or the shape of only some items might change.

Now, let's suppose the objects in the array are of various shapes, but the shape actually depends on the slug property. If we put it this way, the items actually represent a tagged-union type which is open to extension by the API. In that manner, parsing only the array items of known slugs is dual to what struct combinator does now in terms of objects (which is allowing for extension of objects by adding new keys while retaining compatibility).

Describe alternatives you've considered

One alternative is to just use the D.array but this does not satisfy my need fully.

I've also considered writing the above collapse function as a plain decoder instead of combinator:

const collapseD = {
  decode: (i: unknown) => E.right(undefined)
}

and using it in union with a decoder for item that could fail thus recovering the failure with value as undefined - like this:

D.array(D.union(failingDecoder, collapseD)

, and then filtering out undefined as in the first code snippet. However this approach relies on the order of processing union which I guess should be commutative.

Your environment

Software	Version(s)
io-ts	2.2.16
fp-ts	2.11.7
TypeScript	4.5.4

[kmbt]

After giving it some additional thought, I've managed to rewrite my solution in a much cleaner manner by simply relying on the Option type instead of using undefined.

The optional combinator replaces the above collapse and may be found useful by itself - it turns a decoder into one that always succeeds (Right) but returns its result as an Option - Some<A> for success and None for failure. This might be useful in general for parts of a decoder that are expected to fail sometimes but should be prevented from making the whole decoder fail - an optional part of a pattern so to speak.

import * as A from 'fp-ts/Array'
import * as D from 'io-ts/Decoder'
import * as O from 'fp-ts/Option'
import { flow } from 'fp-ts/lib/function';

const optional = <I, A>(d: D.Decoder<I, A>): D.Decoder<I, O.Option<A>> => ({
  decode: flow(
    d.decode,
    O.fromEither,
    D.success
  )
})

const filteredArray: <A>(d: D.Decoder<unknown, A>) => D.Decoder<unknown, Array<A>> = flow(
  optional,
  D.array,
  D.parse(flow(
    A.filterMap(identity),
    D.success
  )))

[florianbepunkt]

Can your requirement also be expressed as a computation that can partially fail? If so, another approach would be to use a These instead of an Either, where you would treat Both as a warning.

This way the decoding errors of failed array elements would not vanish into thin air. Also it would allow you, if needed, to distinguish between exceptions and warnings during decoding.

E. g.

Array where all elements can be decoded > E.right(as)
Exception while decoding array elements > E.left(e)
Array that can be partially decoded > TH.both(e, as)

[bmeverett]

I am also looking for this feature as well. Is there a similar way this could be accomplished using the stable api instead of using Decoder?

[bmeverett]

I took a stab at implementing this. This seems to work, however, I want to log the failures and if I have a union type it'll log failures for the unions it wasn't able to decode, but the final object passes as is the correct type.

export function validItemsOnlyArray<C extends t.Mixed>(theType: C) {
  return withValidate(t.array(theType), (itemToDecode, context) => {
    return pipe(
      // validate that the value is at least an array
      // if it passes, then chain and validate each item
      t.UnknownArray.validate(itemToDecode, context),
      E.chain((validArrayObject) => {
        const decoded = pipe(
          validArrayObject,
          A.map((arrayItem) => {
            const decodeResult = theType.decode(arrayItem)
            //TODO: figure out how to log errors, currently logging for each failed union type
            return isRight(decodeResult)
              ? t.success(decodeResult.right)
              : t.failure(arrayItem, context)
          }),
          A.separate,
        )

        return t.success(decoded.right)
      }),
    )

  })
}