Proposal for the async action problem

[ashaffer]

There have been a number of issues surrounding the problem of asynchronous (really just impure generally) actions. But I think the variety of things to be done by these actions are actually relatively small in number, with the overwhelming majority being simple api requests.

Api requests can be trivially described by data: {method, url, headers, body}. So why are we imperatively triggering these requests when we can produce descriptions that can be executed by middleware? We can create action creators for these declarative descriptions as well, even with apis identical to, say, the real fetch api.

Except of course that the native fetch api returns a promise. But what is a promise really, if not a tree of the form: {then: [{success, failure, then}]}, where success/failure too can be declarative, pure descriptions of what should happen in each condition, or curried pure functions that return declarative descriptions.

To state the proposal succinctly: Contain all true side-effects in middleware by producing only declarative descriptions of those side-effects in your action creators.

The benefits of this over redux-thunk or redux-promise are that you no longer have to make mocks, ever, because action creators themselves no longer imperatively trigger side-effects, you can simply ignore requests for pieces of data your tests don't care about. And probably more importantly, unlike promises and thunks which are completely opaque, these descriptions would be transparent to middleware. The middleware can interpret (and re-interpret) these descriptions, substantially increasing the power and flexibility of middleware.

Ultimately I think this would allow a pretty interesting effect-middleware ecosystem to evolve and I think the community would be able to come up with some really interesting and powerful applications that wouldn't otherwise be possible.

@gaearon Would love to hear your thoughts on this, and i'd be happy to make some demo middleware if people are interested / don't think this is a terrible idea.

EDIT: I should note that I didn't make a PR for this (as was requested in the closing of the other threads) because this doesn't require any changes to core, it'd just be a change of convention.

EDIT2: As a motivating example, one obvious use-case for this would be a middlware that automatically memoizes API requests for data that already exists on the client. Or similarly, grouping api requests in 50ms buckets and sending them to be executed as a batch, etc. There are all kinds of crazy, powerful things you could do with this.

EDIT3: I just discovered the redux-requests package, and it is doing basically what i'm suggesting, except that it hasn't taken the final step and had the middleware actually execute the effect, and in so doing it IMO loses most of the potential benefits of this approach.

[ashaffer]

I have created some super super basic prototypes if people want to mess around with them:

• redux-fetch - redux middleware that executes declarative-fetch actions.
• declarative-fetch - fetch action creator, that behaves like the native fetch api but produces objects instead of side-effects.

Example:

function signupUser (user) {
  return post('/user/', {
    method: 'post',
    body: JSON.stringify(user),
    headers: {
      Accept: 'application/json',
      'Content-Type': 'application/json'
    }
  })
  .then(userDidLogin)
  .toJSON()
}

const userDidLogin = createAction('USER_DID_LOGIN')

The toJSON() at the end there is just to turn the declarative-promise back into a plain object. The userDidLogin action will then get dispatched with the result of the request as its payload (though you can of course chain another declarative fetch here if you so desire).

Also, I made a bunch of design decisions really quickly here prototyping this, so don't read too much into the format of the actions or the API interface. Its very possible that 'fetch' shouldn't be an action type at all, or that I shouldn't be trying so hard to mimic the original APIs or whatever, I certainly haven't thought it all through yet, this is just a PoC for now.

It is also totally untested except for the fact that it has all worked together so far a couple times on my local project - so don't be surprised if something is broken.

[gaearon]

Relevant: https://github.com/evancz/elm-effects

[agraboso]

A glaring omission of this approach — as far as I understand — is that there is no signal to the UI that the request has been made. I'm also not sure what a failure of the API call would result in.

I've been working on a variation of the approach in the real-world example in the redux repository — now available as redux-api-middleware in npm. It takes care of API calls with the {method, url, headers, body} pattern you mention above.

I'm using this right now and works as I expected it to. I have to write tests for it though, but I don't want to do that until people say it looks like something they might use.

[ghost]

I'm surely not understanding the full discourse here so please take my question with that in mind: What do these approaches provide above and beyond a simple 'agent' or 'client' or 'driver' that allow record/replay of request/response?

[ashaffer]

@agraboso Ya, it looks like you did the exact same thing I did with redux-fetch.

I'm not sure what you mean though that there is no signal to the UI that the request has been made. Why can't you simply dispatch multiple actions?

function createUser (user) {
  return (dispatch) => {
    dispatch(REQUEST_START)
    dispatch(fetch('/user', {method: 'post', body: user})
  }
}

Or have the middleware do it, or whatever. I don't think the loading state here is any more complex than with any other approach really.

[ashaffer]

@danmartinez101 It offers two primary benefits, as I see it:

• App-level code is side-effect free. At the very least, this makes it easier to write tests, and easier to write isomorphic code.
• API requests become transparent data, that can be reasoned about by middleware. The most obvious use-case is automatic caching, but there are many others.

It allows you to decouple the specification of the side-effects from their actual imperative execution. What this means is that the equivalent of your driver/client is just another redux middleware. And say, request de-duplication is another middleware (that isn't necessarily coupled to the execution middleware).

[speedskater]

@ashaffer @agraboso I think separating api design from your redux actions might be the favorable approach. So my envisoned approach is

• to design a reusable framework independent application specific api based on the whatwg fetch api. Although i really hope that redux/react will be feasable solution for a very long time I don't think it is a good idea to couple api and redux middleware too tightly together.
• dispatch you api calls by redux-promise
• use a thunk to signal the start of a request/api call/multiple api calls
• create higher order functions like submitRequest, submitRequests which combines the above techniques.

What do I miss out?

[ashaffer]

@speedskater Can you elaborate a little more on what you mean in point 1?

[speedskater]

@ashaffer The idea of point one is: You have some kind of server side API. As pointed out in my comment a RESTful API. You create some API functions based on your REST-Interface. This API functions return promises. For example you will generate an API function like filterTodos('myspecialfilter', SortOrder.ASC)

Based on this basic API functions you can construct simple action creators which return FSA actions (https://github.com/acdlite/flux-standard-action) with the corresponding promise (result of your api call)

In case you have more complex api calls where an action creator triggers multiple api calls (imho, it is important that they solely dependent on the parameters of the action creator) there are two possiblities:

• If you are only interested in the final result: Create a compound API function which returns a single Promise and use it in the action creator.
• If you are interested in intermediate states like begin, number successful operation, number failed operations or overall finishd things become a bit more difficult, but I think this could be a possible solution:
  • Use the redux-thunk API to dispatch your actions/promises...
  • Convert your reducer into a state machine which allows you to decoratively declare what should happen when.

Regarding the last point as far as i know there is currently no project making this an easy task. But i envison something like the following example modeling a session reducer which holds the current logged in state and the corresponding data. States are LOGGED_OUT, LOGGED_IN_WITHOUT_SESSION_CONTEXT, LOGGED_IN, LOGGED_IN_WITH_EXPIRED_PASSWORD. and state transitions can happen on ACTIONS (successul/failed). Errors are narrowed by their Exception classes (in the example InvalidPermisssions) and filter functions on the exception (eg. hasInvalidCredentials, isPasswordExpired). The same filter mechanism could be used on the actual payload in the successful case.

Corresponding actions to this reducer would be login(username, password) which tries to login first and load the session context. changepassword(username, oldpw, newpw, newpw2) which tries to change the password and load the session context.

To formulate such a reducer I would envison something like the following code snippet:

LoginStateMachine = {
    //possible state transitions in the LOGGED_OUT state
    LOGGED_OUT: {
        [ succesful(SESSION_CREATE) ]: (previousStateContent, { sessionId }) => {
            return [ LOGGED_IN_WITHOUT_SESSION_CONTEXT, { sessionId } ];
        },
        [ failed(SESSION_CREATE).with(InvalidPermissions).and(hasInvalidCredentials)
]: (previousStateContent, error) => {
            return [ LOGGED_OUT, { error } ];
        },
        [ failed(SESSION_CREATE).with(InvalidPermissions).and(isPasswordExpired) ]:
(previousStateContent, error ) => {
            return [ PASSWORD_EXPIRED, { error } ];
        },
        [ failed() ]: (previousStateContent, error) => {
            return [ LOGGED_OUT, { error } ];
        }
    },
    //possible state transitions after the inital session was created but the load of the session context is still in progress
    LOGGED_IN_WITHOUT_SESSION_CONTEXT: {
        [ succesful(SESSION_CONTEXT_LOAD) ]: (previousStateContent, { sessionContext }) => {
            return [ LOGGED_IN, Object.assign{ previousStateContent, { sessionContext } ];
        }
    },
    //possible state transitions in case the logged in user has an expired password
    PASSWORD_EXPIRED: {
        [ succesful(RENEW_PASSWORD) ] : (previousStateContent, { sessionId }) => {
            return [ LOGGED_IN_WITHOUT_SESSION_CONTEXT ];
        },
        [ failed(RENEW_PASSWORD).with(InvalidPermission).and(hasInvalidCredentials)
]: (previousStateContent, error) => {
            return [ PASSWORD_EXPIRED, { error }]
        }
    },
    //possible state transitions ins case of logged in state
    LOGGED_IN: .....
}

I am planning to create something maybe calling it redux-fsm. But it could take some more time till I have enought time to implement it. If this is interesting for someone I would really like to join forces.

@ashaffer @gaearon Does the proposed solutions make any sense or is it too over engineered?

[ashaffer]

@speedskater I think your concept of having state machines for dealing with these things is definitely interesting, and could conceivably alleviate some of the boilerplate associated with handling these sorts of complicated state transitions with many intermediates. However, I think it's mostly orthogonal to the purpose of this issue which is to think about declaratively specifying side-effects in middleware, so maybe it'd be best to discuss in a separate issue?

[ashaffer]

An interesting consequence of this kind of request de-duplication is that for idempotent requests loading is no longer a state, it is a materialized view. This is because anytime the data is not yet here, from the perspective of your ui code, you can just make the request again.

Consider a / route with a different view if you're an authenticated user:

router({
  '/': function (state) {
    if (!state.currentUserResolved) {
      dispatch(getCurrentUser())
      return (<Spinner />)
    }

    return state.loggedIn ? <App /> : <HomePage />
  }
})

We don't need to know whether a request is in-flight. The only state to consider is whether or not the data is here yet.

[aaronjensen]

We implemented this middleware to handle api actions via axios:

/*
 * Redux middleware that handles making API requests using axios.
 *
 * Actions should be of the form:
 * {
 *   api: {
 *     url: '/url/to/request',
 *     method: 'GET|POST|PUT|DELETE',
 *     data: { ... }
 *   },
 *   pending: String|Object|Function,
 *   success: String|Object|Function,
 *   failure: String|Object|Function
 * }
 *
 * "Callbacks" can be one of three types and are optional.
 * - String: dispatch action with { type: String, payload: ? }
 *   payload will be undefined for pending,
 *   the server data response for success,
 *   or the err for failure
 * - Object: A flux action to dispatch. This could be another API call if you like
 *   as it goes through the middleware.
 * - Function: Will be called with these parameters:
 *   - dispatch: Redux store's dispatch
 *   - getState: function to get the store's state
 *   - payload: See above for information on payload
 */
module.exports = function createApiMiddleware(
  axios = require('axios'),
  Raven = require('raven-js')
) {
  return ({ dispatch, getState }) => {
    function maybeDispatch(action, payload) {
      if (!action) {
        return;
      }

      if (typeof action === 'function') {
        action(dispatch, getState, payload);
        return;
      }

      if (typeof action === 'object') {
        dispatch(action);
        return;
      }

      action = { type: action };

      if (typeof payload !== 'undefined') {
        action.payload = payload;
      }

      dispatch(action);
    }

    return next => action => {
      if (!action.api) {
        return next(action);
      }
      maybeDispatch(action.pending);

      return axios(action.api)
      .then((data) => {
        maybeDispatch(action.success, data);
      })
      .catch((err) => {
        if (err instanceof Error) {
          console.error(err);
          Raven.captureException(err);
        } else {
          console.error('API Error: ', err);
          Raven.captureMessage('API Error', { extra: err });
        }

        try {
          maybeDispatch(action.failure, err);
        } catch(dispatchErr) {
          console.error(dispatchErr);
          Raven.captureException(dispatchErr);
        }
      });
    };
  };
};

const createApiMiddleware = require('app/redux/create_api_middleware');

describe('Api Middleware', () => {
  let axios, apiMiddleware, next, dispatch, getState, Raven;
  const api = {
    url: '/url',
    method: 'GET'
  };

  beforeEach(() => {
    axios = sinon.stub().returnsPromise();
    next = sinon.stub().returns('result');
    dispatch = sinon.stub();
    getState = sinon.stub();
    Raven = {
      captureException: sinon.stub(),
      captureMessage: sinon.stub()
    };

    apiMiddleware = createApiMiddleware(axios, Raven)({ dispatch, getState });
  });

  it('invokes and returns next if the action is not an api action', () => {
    const action = { type: 'FOO' };
    const result = apiMiddleware(next)(action);

    expect(result).to.equal('result');
    expect(next).to.have.been.calledWith(action);
  });

  it('makes the api call if one is specified', () => {
    const action = { api };

    apiMiddleware(next)(action);

    expect(axios).to.have.been.calledWith(api);
  });

  it('dispatches the pending action', () => {
    const pending = 'PENDING';

    const action = {
      api,
      pending
    };

    apiMiddleware(next)(action);

    expect(dispatch).to.have.been.calledWith({ type: pending });
  });

  it('can dispatch already made actions', () => {
    const pending = { type: 'PENDING', payload: 3 };
    const action = {
      api,
      pending
    };

    apiMiddleware(next)(action);

    expect(dispatch).to.have.been.calledWith(pending);
  });

  it('dispatches the success action when axios resolves', () => {
    const success = 'SUCCESS';
    const action = {
      api,
      success
    };
    const payload = 'payload';

    axios.resolves(payload);
    apiMiddleware(next)(action);

    expect(dispatch).to.have.been.calledWith({ type: success, payload });
  });

  it('dispatches the failure action when axios rejects', () => {
    const failure = 'FAILURE';
    const action = {
      api,
      failure
    };
    const err = 'err';

    axios.rejects(err);
    apiMiddleware(next)(action);

    expect(dispatch).to.have.been.calledWith({ type: failure, payload: err });
  });

  it('calls functions when passed as actions', () => {
    const action = {
      api,
      pending: sinon.stub(),
      success: sinon.stub(),
      failure: sinon.stub()
    };

    const payload = 'payload';
    const err = 'err';
    axios.resolves(payload);
    apiMiddleware(next)(action);

    expect(action.pending).to.have.been.calledWith(dispatch, getState);
    expect(action.success).to.have.been.calledWith(dispatch, getState, payload);

    axios.rejects(err);
    apiMiddleware(next)(action);

    expect(action.failure).to.have.been.calledWith(dispatch, getState, err);
  });

  it('sends api errors to raven', () => {
    const failure = 'FAILURE';
    const action = {
      api,
      failure
    };
    const err = 'err';

    axios.rejects(err);
    apiMiddleware(next)(action);

    expect(Raven.captureMessage).to.have.been.calledWith('API Error', { extra: err });
  });

  it('sends exceptions to raven', () => {
    const failure = 'FAILURE';
    const action = {
      api,
      failure
    };
    const err = new Error();

    axios.rejects(err);
    apiMiddleware(next)(action);

    expect(Raven.captureException).to.have.been.calledWith(err);
  });

  it('sends exceptions to raven if failure throws', () => {
    const failureErr = new Error('failure');
    const failure = () => { throw failureErr; };
    const action = {
      api,
      failure
    };
    const err = new Error();

    axios.rejects(err);
    apiMiddleware(next)(action);

    expect(Raven.captureException).to.have.been.calledWith(err);
    expect(Raven.captureException).to.have.been.calledWith(failureErr);
  });
});

[speedskater]

@ashaffer You are right. The state machine should be in a separate issue. But the basic question stays where to put the side effect without bundling the application code to tightly to redux. Imho a special middleware for requests is not an ideal solution, because its only synthactic sugar on top of thunk middleware.

[matystl]

alt data sources for inspiration http://techblog.trunkclub.com/alt/react/flux/babel/2015/08/13/using-alts-data-sources.html

[ashaffer]

@matystl @gaearon @jlongster (if you're still interested)

Alright guys. I've been working on this thing for a little while now, and I think its complete enough to establish the concept:

redux-effects

Its aim is to be a sub-middleware system of redux that handles all side-effects and all sources of non-determinism (e.g. Math.random) via object descriptors. By using this with redux and a virtual-dom library of some kind, it should be possible to write an entire front-end application without writing a single impure, non-deterministic function anywhere. Perhaps a good way of thinking about it is virtual-dom for effects.

Because everything is just transparent data, function composition can be used to solve a much greater variety of problems.

The core abstraction of this library is declarative-promise, which allows all of your effects to compose indefinitely. The return value of your .step is dispatched back into redux, triggering either more effects or ultimately actions that will preserve their results in state.

This all works particularly well with redux-multi, which just allows you to dispatch an array of actions simultaneously (e.g. for loading states):

const userDidLogin = createAction('USER_DID_LOGIN')
const userCreateFailed = createAction('USER_CREATE_FAILED')
const creatingUser = createAction('CREATING_USER')

function createUser (userData) {
  return [
    api.user.create(userData).step(userDidLogin, userCreateFailed),
    creatingUser()
  ]
}

Here's an example action creator file from an application i'm currently working on that uses this pattern. Hopefully i'll have some more real demos up soon (lib/api is using declarative-fetch):

/**
 * Imports
 */

import {createAction} from 'redux-actions'
import {handleOnce} from 'declarative-events'
import cookie from 'declarative-cookie'
import {bindUrl} from 'declarative-location'
import api from 'lib/api'

/**
 * Action Creators
 */

function initializeApp () {
  return handleOnce('domready', () => [
    appDidLoad(),
    bindUrl(setUrl),
    initializeUser()
  ])
}

function initializeUser () {
  return cookie('authToken')
    .step(token => token ? api.user.getCurrentUser(token) : null)
    .step(userDidResolve)
}

function signupUser (user) {
  return api.user
    .createUser(user)
    .step(userDidLogin)
}

const userDidResolve = createAction('USER_DID_RESOLVE')
const userDidLogout = createAction('USER_DID_LOGOUT')
const userDidLogin = createAction('USER_DID_LOGIN')

const setUrl = createAction('SET_URL')

const appDidLoad = createAction('APP_DID_LOAD')
const appDidRender = createAction('APP_DID_RENDER')

function loginUser (credentials) {
  return api.user
    .loginUser(credentials)
    .step(user => [
      userDidLogin(user),
      cookie('authToken', user.token)
    ])
}

/**
 * Exports
 */

export default {
  initializeApp,
  appDidLoad,
  appDidRender,
  signupUser,
  loginUser
}

All of these functions are pure. They just return data - they don't getState and they don't imperatively dispatch (the return value of your .step functions are dispatched internally by redux-effects). DOM initialization, url watching, etc..can all be handled in a pure way using this strategy. This makes all of this completely testable, instrumentable, and isomorphic, without really sacrificing (IMO) any developer ergonomics, aside from having to install the effect middleware.

I've been working on this for a little while now and I feel pretty confident that it's the right approach for me at least, but if other people are into it i'd love to get some feedback and start writing more docs and creating some demos so other people can start using it if people are interested.

EDIT: Updated examples to use .step() rather than .then() per https://github.com/redux-effects/declarative-promise/issues/1

[sjmueller]

this is pretty seksi @ashaffer. well done.

[ashaffer]

I've made some updates. redux-effects is no longer its own middleware stack, that package is now solely responsible for effect composition, and can be replaced with other effect composition libraries if you want. The composition strategy is completely orthogonal to the actual effect execution now.

I've also added a non-mutative and simpler composition helper, bind-effect. It doesn't mutate any data (unlike declarative-promise) and it returns a plain object. Also, all of the action creators are no longer opinionated about composition (i.e. they don't wrap their return values in a declarative-promise anymore).

Now the only requirement for something being an effect middleware is that it return a promise. If it does that, then redux-effects will let you compose other pure functions around its result.

A cool thing that is enables is stuff like orthogonal normalization middleware:

function normalize () {
  return next => action =>
    isGetRequest(action)
      ? next(action).then(normalizr)
      : next(action)
}

Request caching is similarly trivial:

function httpCache () {
  const {get, set, check} = cache()

  return next => action =>
   !isGetRequest(action) 
      ? next(action)
      : check(action.payload.url) 
        ? Promise.resolve(get(action.payload.url)) 
        : next(action).then(set(action.payload.url))
}

And these things are fully orthogonal both to each other and to the implementation details of fetch. All they need to do is agree on the spec for the object descriptor.

[joshrtay]

I've been working on a generator based approach to this called redux-gen. Would love to know what people think.

The control flow in the previous examples seems to be mimicking a generator. So if we say that actions return generators or are GeneratorFunctions then we can abstract this form of writing async action creators even further.

import { createStore, applyMiddleware } from 'redux'
import gen from '@weo-edu/redux-gen'
import rootReducer from './reducers/index'
import fetch from 'isomorphic-fetch'

// create a store that has redux-thunk middleware enabled
const createStoreWithMiddleware = applyMiddleware(
  gen()
  fetch
)(createStore);

const store = createStoreWithMiddleware(rootReducer);

// returns [
//  {username: "josh", id: 1},
//  {username: "tio", id: 2},
//  {username: "shasta", id: 3}
// ]
store.dispatch(getUsers())

// Side Effects Middleware

function fetch ({dispatch, getState}) {
  return next => action =>
    action.type === 'FETCH'
      ? fetch(action.payload.url, action.payload.params).then(res => res.json())
      : next(action)
}

// Actions

function getUsers *() {
  var userIds = yield {url: '/users', method: 'GET', type: 'FETCH'}
  return yield userIds.map(userId => {
    return {url: '/user/' + userId, method: 'GET', type: 'FETCH'}
  })
}

In order to avoid writing action creators as actual GeneratorFunction, we can use libraries like yields, which returns functions that return generators. Yields parallels the control flow that can be achieved with GeneratorFunctions, but enforces that the functions are pure. Since the action creators have no side effects they can easily be tested by iterating over the generator returned by an action.

In the previous example, using this action would be equivalent.

import yields from '@weo-edu/yield'
var getUsers = yields(function () {
  return {url: '/users', method: 'GET', type: 'FETCH'}
}).yields(function (userIds) {
  return userIds.map(userId => {
    return {url: '/user/' + userId, method: 'GET', type: 'FETCH'}
  })
})

[ashaffer]

@joshrtay The above is really just an alternate composition strategy for redux-effects's middleware ecosystem. It is fully compatible with all the same middleware. You'd just need to put gen() in place of effects in your stack.

The primary disadvantage of using generators is that they hold their own state in an opaque way, and they encourage an impure programming style. But on the other hand they offer significant syntactic benefits.

What's cool about this approach is that you can use either of our effect composition libraries (or both, if you wanted) without changing anything about the effect middleware or transformation middleware in your stack. Each feature is responsible for itself and precisely nothing more.

[gaearon]

Closing as redux-effects seems to fill the niche for people who are interested in describing effects explicitly.