Automattic / wp-calypso

The JavaScript and API powered WordPress.com
https://developer.wordpress.com
GNU General Public License v2.0
12.4k stars 1.98k forks source link

Framework: Async Load State Pieces #19829

Closed apeatling closed 4 years ago

apeatling commented 6 years ago

Figure out a way to async-load the pieces of our state engine—reducers/selectors/data-layer—in a way that doesn’t detract from the experience of a global, centralized, and readily available data store. Alternatively, avoid pulling large dependencies on reducers by modularizing further or separating concerns as there is no immediate reason for our state reducers to be a significant drag on code weight.

timdream commented 6 years ago

So I spend some time figuring out how Calypso is built and how it asks Webpack split the chunks. Thankfully there is npm run analyze-bundles to visualize the chunk size and content.

Essentially, this issue asks for the ability to reduce this part of the build.js bundle:

screenshot-2017-11-25 webpack bundle analyzer

There are three targets — reducers, data-layers, and selectors. I decided to tackle reducers given prior art exists (1 and 2).

To simplify the experiment, I've selected state/timezones/reducer as the target.

I've come up with two approaches: the first one touches the bundler and has it load the reducers the async bundles asks.

diff --git a/client/devdocs/index.js b/client/devdocs/index.js
index ef26b5bcbe..3245f90f16 100644
--- a/client/devdocs/index.js
+++ b/client/devdocs/index.js
@@ -13,5 +13,9 @@ import config from 'config';
 import controller from './controller';

-export default function() {
+import { combineReducers } from 'state/utils';
+import { reducers } from 'state';
+import timezones from 'state/timezones/reducer';
+
+export default function sectionEntry() {
    if ( config.isEnabled( 'devdocs' ) ) {
        page( '/devdocs', controller.sidebar, controller.devdocs );
@@ -35,2 +39,4 @@ export default function() {
    }
 }
+
+sectionEntry.reducer = combineReducers( Object.assign( {}, reducers, { timezones } ) );
diff --git a/client/state/index.js b/client/state/index.js
index d23743d11f..686d1b79db 100644
--- a/client/state/index.js
+++ b/client/state/index.js
@@ -73,5 +73,5 @@ import storedCards from './stored-cards/reducer';
 import support from './support/reducer';
 import terms from './terms/reducer';
-import timezones from './timezones/reducer';
+//import timezones from './timezones/reducer';
 import themes from './themes/reducer';
 import ui from './ui/reducer';
@@ -89,5 +89,5 @@ import config from 'config';
 const extensions = combineReducers( extensionsModule.reducers() );

-const reducers = {
+export const reducers = {
    analyticsTracking,
    accountRecovery,
@@ -147,5 +147,5 @@ const reducers = {
    support,
    terms,
-   timezones,
+   //timezones,
    themes,
    ui,
diff --git a/server/bundler/loader.js b/server/bundler/loader.js
index 4d4ad2a5de..9e117bc341 100644
--- a/server/bundler/loader.js
+++ b/server/bundler/loader.js
@@ -119,5 +119,7 @@ function splitTemplate( path, section ) {
        '       controller.setSection( ' + sectionString + ' )( context );',
        '       if ( ! _loadedSections[ ' + moduleString + ' ] ) {',
-       '           require( ' + moduleString + ' )( controller.clientRouter );',
+       '           var module = require( ' + moduleString + ' );',
+       '           module( controller.clientRouter );',
+       '           module.reducer && context.store.replaceReducer( module.reducer );',
        '           _loadedSections[ ' + moduleString + ' ] = true;',
        '       }',

This is a fragile setup because we are asking everyone annotate the top level section (i.e. devdocs) to ensure their React componments work correctly. So I decided to implement something saner instead. This diff implements a globally-available injectReducers and use it in QueryTimezones:

diff --git a/client/components/data/query-timezones/index.jsx b/client/components/data/query-timezones/index.jsx
index b3cf286440..0a8ccdae1a 100644
--- a/client/components/data/query-timezones/index.jsx
+++ b/client/components/data/query-timezones/index.jsx
@@ -13,4 +13,8 @@ import { connect } from 'react-redux';
  */
 import { requestTimezones } from 'state/timezones/actions';
+import timezonesReducer from 'state/timezones/reducer';
+import { injectReducers } from 'state/reducer-inject';
+
+injectReducers( { timezones: timezonesReducer } );

 export class QueryTimezones extends Component {
diff --git a/client/state/index.js b/client/state/index.js
index d23743d11f..07c9846a70 100644
--- a/client/state/index.js
+++ b/client/state/index.js
@@ -62,4 +62,5 @@ import purchases from './purchases/reducer';
 import reader from './reader/reducer';
 import receipts from './receipts/reducer';
+import { setInitialReducers, reducerInjecter } from './reducer-inject';
 import sharing from './sharing/reducer';
 import shortcodes from './shortcodes/reducer';
@@ -73,5 +74,5 @@ import storedCards from './stored-cards/reducer';
 import support from './support/reducer';
 import terms from './terms/reducer';
-import timezones from './timezones/reducer';
+//import timezones from './timezones/reducer';
 import themes from './themes/reducer';
 import ui from './ui/reducer';
@@ -89,4 +90,6 @@ import config from 'config';
 const extensions = combineReducers( extensionsModule.reducers() );

+const placeholderReducer = ( state = {} ) => state;
+
 const reducers = {
    analyticsTracking,
@@ -147,5 +150,10 @@ const reducers = {
    support,
    terms,
-   timezones,
+   // XXX Improve this without referencing the reducer in the build chunk.
+   timezones: combineReducers( {
+       rawOffsets: placeholderReducer,
+       labels: placeholderReducer,
+       byContinents: placeholderReducer,
+   } ),
    themes,
    ui,
@@ -157,4 +165,5 @@ const reducers = {

 export const reducer = combineReducers( reducers );
+setInitialReducers( reducers );

 /**
@@ -201,4 +210,5 @@ export function createReduxStore( initialState = {} ) {
    const enhancers = [
        isBrowser && window.app && window.app.isDebug && consoleDispatcher,
+       isBrowser && config.isEnabled( 'code-splitting' ) && reducerInjecter,
        applyMiddleware( ...middlewares ),
        isBrowser && sitesSync,
diff --git a/client/state/reducer-inject/index.js b/client/state/reducer-inject/index.js
new file mode 100644
index 0000000000..920ed00589
--- /dev/null
+++ b/client/state/reducer-inject/index.js
@@ -0,0 +1,44 @@
+/**
+ * Reducer injecter Redux store enhancer
+ *
+ * `reducerInjecter` injects into the `createStore` enhancer chain
+ * in order to provide a globally available method to update the reducer
+ * from newly loaded chunks.
+ */
+
+/**
+ * External dependencies
+ */
+import { forIn } from 'lodash';
+
+/**
+ * Internal dependencies
+ */
+import { combineReducers } from 'state/utils';
+
+let store;
+let currentReducers;
+
+export const setInitialReducers = ( reducers ) => {
+   currentReducers = reducers;
+};
+
+export const injectReducers = ( reducers ) => {
+   let updated = false;
+
+   forIn( reducers, ( reducer, key ) => {
+       if ( currentReducers[ key ] && currentReducers[ key ] === reducer ) {
+           return
+       }
+
+       currentReducers[ key ] = reducer;
+       updated = true;
+   } );
+
+   updated && store.replaceReducer( combineReducers( currentReducers ) );
+};
+
+export const reducerInjecter = next => ( reducer, initialState ) => {
+   store = next( reducer, initialState );
+   return store;
+};

state/timezone is being removed nicely; see if you could spot the difference :)

screenshot-2017-11-25 webpack bundle analyzer 1

There are some improvements to be made here before this can be pushed as a pull request:

@ehg @apeatling Let me know what you think, especially on the proposed approach and the scope. Thanks.

timdream commented 6 years ago

Surprising, selectors can easily be removed by making sure transform-imports works. A change is proposed; that cross out one item out of three easily here :) .

It does open some possibility on whether or not we could leverage transform-imports for data-layers and/or reducers. Further investigation is needed.

ehg commented 6 years ago

One point of concern could be data dependencies, do we need some reducers before others?

It would be tedious to manually move every reducer

Maybe a codemod could make this easier? https://github.com/Automattic/wp-calypso/tree/master/bin/codemods

ehg commented 6 years ago

Looping @coderkevin, @lamosty and @jsnajdr in :)

timdream commented 6 years ago

It does open some possibility on whether or not we could leverage transform-imports for data-layers and/or reducers. Further investigation is needed.

I've given more thought to my previous comment: Even with tools like codemods, It is indeed tedious to have more async loader calls in the components, so perhaps we can do better by containing these complexities within client/state.

Leveraging the unidirectional data flow, given that reducers are responsible for reducing the actions dispatched, as long as the subset of the reducers and data-layer is loaded for the specific action, the action should be dealt correctly, i.e. reducing to the correct set of state changes and trigger the correct set of data-layer actions.

Locate the right reducers or data-layers can be as easy as locating the files that reference a specific action type constant. Or not.

I wonder if it's possible to reuse transform-imports in some way to dissemble action types/data-layer/ reducers into bundle chunks. I should spend time experimenting on this.

(Sorry if I over-communicated and thinking out loud here. I want to make sure I don't hit any walls that have been tried already.)

jsnajdr commented 6 years ago

Hello @timdream! Thank you for your work on this important issue. It looks awesome so far!

I have a few ideas and suggestions that could be interesting:

It's helpful to think about Calypso not as one big web app, but as a conglomerate of many small apps that are fairly indepenent from each other. There is signup flow for new users and sites at /start, Reader app at /, plugins management at /plugins, domain management at /domains and many others. A full list of these "sections" is in client/sections.js. That file is transformed by Webpack loader at server/bundler/loader.js into code that async loads the code for that section and executes it.

The major current weakness of that system is that it async loads mostly just React UI components for a particular section. The data part -- the reducers and data-layer effect handlers -- is still a part of the global build.js chunk. If I navigate to root path of Calypso, I use only the Reader app, but the state tree contains everything, including extensions:

big-redux-state

I think the most promising approach for your task is to find reducers that are used only by one particular section. I.e., they are specific to a single Calypso sub-app and don't implement any cross-section functionality. There should be plenty of them. Then load these reducers asynchronously in the section-loading code generated by the server/bundler/loader loader.

Extensions should be the easiest target for this. They are explicitly designed to be independent mini-apps that are as independent from the rest of Calypso as possible. You could start with the wp-job-manager extension for example. Note that every extension has a "manifest" file called package.json and that file is loaded in sections.js. Try to load its reducer asynchronously. I don't want to see extensions.wpJobManager in my state tree until I actually go to the /extensions/wp-job-manager route.

The point is that we might not need a generic solution that works with any import statement at any place. Maybe everything we'll ever need is being able to async load state pieces at a "section" level. Adding that functionality only to the section-loading code might be much easier and more maintainable.

timdream commented 6 years ago

@jsnajdr Thanks for the comment!

I spent some time looking into how extensions are being loaded; it does look like an easier target — assuming nowhere else in Calypso access the state of a specific extension. The other thing is that it's data-layer and selectors have already been moved into its section chunk. data-layers are loaded dynamically with state/data-layer/extensions-middleware.js and selectors only gets imported by the extension components. It would be pretty straightforward for me to complete an "extension reducer loader" in the bundler. I will share my work when it's ready.

I would still love to investigate where it leads though, I mean, whether or not this can become a generic solution to other sections. But I will aim to achieve the small win first.

Thank you again!

coderkevin commented 6 years ago

I once wrote some code to dynamically add and remove reducers for extensions. Several felt that Calypso wasn't ready for it at the time, but perhaps that is changing now?

One question we have to ask ourselves about things like this is, if we can dynamically add/load things, can we also dynamically remove them as well? We can't really dynamically unload code, as far as I know, but there could be performance gains to removing things like reducers when we're not using them.

Also, specifically for reducers and redux state, should we consider creating a separate redux state tree for each extension? That would make things like removing/unloading easier.

timdream commented 6 years ago

Allow me to summarize where I am going currently: so with #20542 there is going to be a generalized, special combineReducersAndAddLater() method (obviously naming can be improved) that accepts adding more reducer function after-the-fact. This will be much useful than managing extension reducers. With that, by only import the reducer at the places where it will be eventually utilized (i.e. action creators and selectors), we can tweak our dependency graph and move away the reducer from the build bundle.

Here is how we could do it, with #20542, again, using timezones as example:

diff --git a/client/state/index.js b/client/state/index.js
index 5dd130953e..19532228cd 100644
--- a/client/state/index.js
+++ b/client/state/index.js
@@ -12,5 +12,4 @@ import { reducer as form } from 'redux-form';
  * Internal dependencies
  */
-import { combineReducers } from 'state/utils';
 import { combineReducersAndAddLater, reducerRegistryEnhancer } from './reducer-registry';
 import activityLog from './activity-log/reducer';
@@ -74,5 +73,4 @@ import storedCards from './stored-cards/reducer';
 import support from './support/reducer';
 import terms from './terms/reducer';
-import timezones from './timezones/reducer';
 import themes from './themes/reducer';
 import ui from './ui/reducer';
@@ -148,5 +146,4 @@ const reducers = {
    support,
    terms,
-   timezones,
    themes,
    ui,
@@ -158,5 +155,5 @@ const reducers = {
 };

-export const reducer = combineReducers( reducers );
+export const reducer = combineReducersAndAddLater( reducers );

 /**
diff --git a/client/state/selectors/get-raw-offsets.js b/client/state/selectors/get-raw-offsets.js
index 6aa0193b30..04f83e3957 100644
--- a/client/state/selectors/get-raw-offsets.js
+++ b/client/state/selectors/get-raw-offsets.js
@@ -7,4 +7,12 @@
 import { get } from 'lodash';

+/**
+ * Internal dependencies
+ */
+import { addReducers } from 'state/reducer-registry';
+import timezones from 'state/timezones/reducer';
+
+addReducers( { timezones } );
+
 /**
  * Return manual utc offsets data
diff --git a/client/state/selectors/get-timezones-by-continent.js b/client/state/selectors/get-timezones-by-continent.js
index c141086900..baabe039ce 100644
--- a/client/state/selectors/get-timezones-by-continent.js
+++ b/client/state/selectors/get-timezones-by-continent.js
@@ -7,4 +7,12 @@
 import { get } from 'lodash';

+/**
+ * Internal dependencies
+ */
+import { addReducers } from 'state/reducer-registry';
+import timezones from 'state/timezones/reducer';
+
+addReducers( { timezones } );
+
 /**
  * Return the timezones by continent data
diff --git a/client/state/selectors/get-timezones-labels.js b/client/state/selectors/get-timezones-labels.js
index 66ad78faa9..54bf52e410 100644
--- a/client/state/selectors/get-timezones-labels.js
+++ b/client/state/selectors/get-timezones-labels.js
@@ -7,4 +7,12 @@
 import { get } from 'lodash';

+/**
+ * Internal dependencies
+ */
+import { addReducers } from 'state/reducer-registry';
+import timezones from 'state/timezones/reducer';
+
+addReducers( { timezones } );
+
 /**
  * Return an object of timezones.
diff --git a/client/state/selectors/get-timezones.js b/client/state/selectors/get-timezones.js
index d96bd703d4..03e07ba9f0 100644
--- a/client/state/selectors/get-timezones.js
+++ b/client/state/selectors/get-timezones.js
@@ -11,4 +11,8 @@ import { get, map, toPairs } from 'lodash';
  */
 import { getTimezonesLabel } from 'state/selectors';
+import { addReducers } from 'state/reducer-registry';
+import timezones from 'state/timezones/reducer';
+
+addReducers( { timezones } );

 /**
diff --git a/client/state/timezones/actions.js b/client/state/timezones/actions.js
index c1e3bd29d7..177f9e427d 100644
--- a/client/state/timezones/actions.js
+++ b/client/state/timezones/actions.js
@@ -6,4 +6,8 @@

 import { TIMEZONES_RECEIVE, TIMEZONES_REQUEST } from 'state/action-types';
+import { addReducers } from 'state/reducer-registry';
+import timezones from 'state/timezones/reducer';
+
+addReducers( { timezones } );

 export const requestTimezones = () => ( {

Upon checking the effect of the patch above, I realized I am only half way there from making this useful. While dependency of selectors has been fixed in #20231, state/timezones/actions is still imported by wpcom data-layer and thus the reducer.

data-layer

It is certainly possible to move some handlers away from start-up. extensions-middleware.js serves as the example to do that. I will need to look up how the current file structure affects how we merge the handlers and load them asynchronously. This is what I am going to investigate next.

timdream commented 6 years ago

Update: I've set up a development branch to piece everything together.

https://github.com/Automattic/wp-calypso/compare/master...timdream:issue-19829

It currently consists the pull request I've submitted, and the previous code snippet that attempt to move timezones reducers and data-layer handlers out of the build chunk. Almost every other components are a lot complex than the timezones, so currently I am working on a codemod, hopefully to move the imports automatically and deterministically.

timdream commented 6 years ago

The development branch updated with WIP work on codemod to move data-layer imports. See also Automattic/calypso-codemods#2.

I've also noticed the fact a few data-layer handlers added back in #18098 has no corresponding action dispatcher/creators. There is essentially dead code currently. I wonder if I should push a quick pull request to not import them for now.

timdream commented 6 years ago

I've decided to change the approach of my script, so that we could have more confidence changing the code based on its result.

The script will scan through the entire client codebase and try to classify every file into tests, data-layer handlers, action creators/dispatchers, and reducers. With the classification, we will know how to move the imports around.

The next step here is to reduce the size of the last list, and produce the actual code mode that modify the code for the second and the third list.

The following action types are referenced by one data-layer handler but not any action dispatchers/ceators nor reducers. The data-layer handler is never called:

The following action types are referenced by one data-layer handler and many action dispatchers/creators. Imports of the data-layer handler should be able to safely move to action creators/dispatchers:

The following action types is only referenced by action dispatchers/creators and reducers. Imports of dispatchers/creators should be able to safely move to reducers:

The following action types has not been classified definitively to be safely modified by codemod:

timdream commented 6 years ago

On the other hand, this list look too simple and naïve. I should construct a DAG in which action files points to data-layer files to make it useful for the actual codemod.

timdream commented 6 years ago

Here is a better list, in the form of (data layer handler file): (action dispatchers/creators):

{
  "client/state/http/index.js": [
    "client/state/http/actions.js"
  ],
  "client/state/data-layer/third-party/directly/index.js": [
    "client/state/help/directly/actions.js"
  ],
  "client/state/data-layer/third-party/refer/index.js": [
    "client/state/refer/actions.js"
  ],
  "client/state/data-layer/wpcom/jetpack-onboarding/index.js": [
    "client/state/jetpack-onboarding/actions.js"
  ],
  "client/state/data-layer/wpcom/checklist/index.js": [
    "client/state/checklist/actions.js"
  ],
  "client/state/data-layer/wpcom/login-2fa/index.js": [
    "client/state/login/actions.js"
  ],
  "client/state/data-layer/wpcom/plans/index.js": [],
  "client/state/data-layer/wpcom/theme-filters/index.js": [
    "client/state/themes/actions.js"
  ],
  "client/state/data-layer/wpcom/users/index.js": [
    "client/state/users/actions.js"
  ],
  "client/state/data-layer/wpcom/timezones/index.js": [
    "client/state/timezones/actions.js"
  ],
  "client/state/data-layer/wpcom/privacy-policy/index.js": [
    "client/state/privacy-policy/actions.js"
  ],
  "client/state/data-layer/wpcom/read/teams/index.js": [
    "client/state/reader/teams/actions.js"
  ],
  "client/state/data-layer/wpcom/posts/revisions/index.js": [
    "client/state/posts/revisions/actions.js"
  ],
  "client/state/data-layer/wpcom/users/auth-options/index.js": [
    "client/state/login/actions.js"
  ],
  "client/state/data-layer/wpcom/videos/poster/index.js": [
    "client/state/ui/editor/video-editor/actions.js"
  ],
  "client/state/data-layer/wpcom/account-recovery/lookup/index.js": [
    "client/state/account-recovery/reset/actions.js"
  ],
  "client/state/data-layer/wpcom/account-recovery/request-reset/index.js": [
    "client/state/account-recovery/reset/actions.js"
  ],
  "client/state/data-layer/wpcom/read/streams/index.js": [
    "client/state/reader/streams/actions.js"
  ],
  "client/state/data-layer/wpcom/account-recovery/reset/index.js": [
    "client/state/account-recovery/reset/actions.js"
  ],
  "client/state/data-layer/wpcom/account-recovery/validate/index.js": [
    "client/state/account-recovery/reset/actions.js"
  ],
  "client/state/data-layer/wpcom/activity-log/deactivate/index.js": [
    "client/state/activity-log/actions.js"
  ],
  "client/state/data-layer/wpcom/activity-log/rewind/index.js": [
    "client/state/activity-log/actions.js"
  ],
  "client/state/data-layer/wpcom/sites/activity/index.js": [
    "client/state/activity-log/actions.js"
  ],
  "client/state/data-layer/wpcom/sites/blog-stickers/index.js": [
    "client/state/sites/blog-stickers/actions.js"
  ],
  "client/state/data-layer/wpcom/domains/countries-list/index.js": [],
  "client/state/data-layer/wpcom/me/devices/index.js": [
    "client/state/user-devices/actions.js"
  ],
  "client/state/data-layer/wpcom/sites/comment-counts/index.js": [],
  "client/state/data-layer/wpcom/sites/media/index.js": [
    "client/state/media/actions.js"
  ],
  "client/state/data-layer/wpcom/me/send-verification-email/index.js": [
    "client/state/current-user/email-verification/actions.js"
  ],
  "client/state/data-layer/wpcom/me/settings/index.js": [
    "client/state/user-settings/actions.js"
  ],
  "client/state/data-layer/wpcom/meta/sms-country-codes/index.js": [],
  "client/state/data-layer/wpcom/read/feed/index.js": [
    "client/state/reader/feed-searches/actions.js"
  ],
  "client/state/data-layer/wpcom/sites/simple-payments/index.js": [
    "client/state/simple-payments/product-list/actions.js"
  ],
  "client/state/data-layer/wpcom/read/recommendations/sites/index.js": [
    "client/state/reader/recommended-sites/actions.js"
  ],
  "client/state/data-layer/wpcom/sites/blog-stickers/add/index.js": [
    "client/state/sites/blog-stickers/actions.js"
  ],
  "client/state/data-layer/wpcom/concierge/schedules/appointments/index.js": [],
  "client/state/data-layer/wpcom/sites/blog-stickers/remove/index.js": [
    "client/state/sites/blog-stickers/actions.js"
  ],
  "client/state/data-layer/wpcom/sites/automated-transfer/eligibility/index.js": [
    "client/state/automated-transfer/actions.js"
  ],
  "client/state/data-layer/wpcom/sites/automated-transfer/initiate/index.js": [
    "client/state/automated-transfer/actions.js"
  ],
  "client/state/data-layer/wpcom/activity-log/rewind/restore-status/index.js": [
    "client/state/activity-log/actions.js"
  ],
  "client/state/data-layer/wpcom/me/settings/profile-links/index.js": [
    "client/state/profile-links/actions.js"
  ],
  "client/state/data-layer/wpcom/me/transactions/supported-countries/index.js": [],
  "client/state/data-layer/wpcom/me/notification/settings/index.js": [
    "client/state/notification-settings/actions.js"
  ],
  "client/state/data-layer/wpcom/sites/plugins/new/index.js": [
    "client/state/plugins/upload/actions.js"
  ],
  "client/state/data-layer/wpcom/me/block/sites/delete/index.js": [
    "client/state/reader/site-blocks/actions.js"
  ],
  "client/state/data-layer/wpcom/me/block/sites/new/index.js": [
    "client/state/reader/site-blocks/actions.js"
  ],
  "client/state/data-layer/wpcom/me/settings/profile-links/delete/index.js": [
    "client/state/profile-links/actions.js"
  ],
  "client/state/data-layer/wpcom/me/settings/profile-links/new/index.js": [
    "client/state/profile-links/actions.js"
  ],
  "client/state/data-layer/wpcom/read/following/mine/delete/index.js": [
    "client/state/reader/follows/actions.js"
  ],
  "client/state/data-layer/wpcom/read/sites/posts/follow/index.js": [
    "client/state/reader/conversations/actions.js"
  ],
  "client/state/data-layer/wpcom/read/sites/posts/mute/index.js": [
    "client/state/reader/conversations/actions.js"
  ],
  "client/state/data-layer/wpcom/read/site/comment-email-subscriptions/delete/index.js": [
    "client/state/reader/follows/actions.js"
  ],
  "client/state/data-layer/wpcom/read/site/post-email-subscriptions/new/index.js": [
    "client/state/reader/follows/actions.js"
  ],
  "client/state/data-layer/wpcom/read/site/post-email-subscriptions/delete/index.js": [
    "client/state/reader/follows/actions.js"
  ],
  "client/state/data-layer/wpcom/read/site/post-email-subscriptions/update/index.js": [
    "client/state/reader/follows/actions.js"
  ],
  "client/state/data-layer/wpcom/read/site/comment-email-subscriptions/new/index.js": [
    "client/state/reader/follows/actions.js"
  ],
  "client/state/data-layer/wpcom/sites/comments/replies/new/index.js": [],
  "client/state/data-layer/wpcom/sites/posts/replies/new/index.js": []
}

Empty array means the action is never dispatched, therefore the handler is never used. There are 58 files here already based on the data gather from my WIP script.

timdream commented 6 years ago

I am now working on a codemod that could automate what 0a9f9e6fdcc787f8a4551e113169b762cf025a7a does on the files above.

While I am working on this I have some second thoughts: the current data-layer middleware design is great in a way that hides itself from components that dispatch the action. The modification here asks developers to actively find out and import the data-layer handler a said action needs when implementing an action. This is an inferior developer experience.

The other route I have been thinking is to create a babel transform that could inject these imports. Yet I am not sure about how Babel transform can access all the information it needs to do that. Maybe we'll end up having to maintain an action type -> handler file list statically.

Let me know if I should work on that instead.

dmsnell commented 6 years ago

@timdream this is great work you are doing. it's also incredibly challenging and intrinsically fragile.

while I like the idea of loading these things asynchronously in theory, I suspect that we risk making the overall experience worse by doing this, and we risk spending a lot of effort in the meantime trying to make it happen.

when I look at the state directory I don't see it maybe as problematic as others do. I think it's quite efficient for what it does. that is, as long as we are hand-coding functions and systems to manage each type of data we have I'm not sure how much better we can get (not saying we can't get better, but the actual size of the state directory is already reasonable for how much it's doing). in other words, how much more overhead and more load-lag are we willing to introduce in order to push this paradigm further?

in #21015 I was able to cut out 7-10 kb just by pruning out the action types file which is totally redundant. I'd like to explore ways we can make these kinds of changes were instead of trying to optimize existing code structures we instead rip them out entirely.

two ways I suspect are viable are by normalizing how we interact with data and by moving data layer code into the server.

by normalizing I mean that there are plenty of cases where all we're doing is "shoving" data from the remote end into the Calypso app. these don't need their own reducers, selectors, and data layer code. it's possible to abstract this away with a kind of constrained remote data system. in #19060 I have explored such an apiData() system I think could replace many of our directories in the state tree. with that kind of system we could probably eliminate a significant amount of code altogether, which also means our builds are faster in addition to making smaller bundles. the code becomes more standardized and there's less of it to have to understand.

by moving to the server I mean taking the data layer to phase two of its adoption: open up another layer of providers. the wpcom providers make JSON API calls and handle all the negotiation. we could also make a system whereby we have some kind of /calypso endpoint on WordPress.com and it takes Redux actions as input and returns a list of Redux actions as a response. for example, maybe we intercept POST_REQUEST and send it to the /calypso API endpoint and it sends back { POST_ADD, COMMENT_ADD, COMMENT_ADD, COMMENT_ADD, COMMENT_COUNT_SET } in response, anticipating related data we want and eliminating the need in Calypso itself to handle all of the logic around translating the POST_REQUEST into API calls and then transforming the response and making related calls. This would mean that we can entirely eliminate sections of code. Also noteworthy is that adding PHP code on the server doesn't carry the same costs as adding JS to the client does. the server would become coupled to the client and we'd have to maintain the consistency of the data structures, but this is something we already do to some extend and maybe pragmatically is worth it.

lastly, and this would be a really nice boon but it's also a bit more controversial, is that by adding in some kind of sound type system we could eliminate most of our selectors. for all of the code where we are doing nothing but providing a specified interface into the state tree we could replace that by direct calls into state if we had a compiler to provide us the safety which the selectors themselves currently provide. the main difference is that with a type system (like with TypeScript or OCaml) then the constraints are maintained when we write the code and with Babel the constraints are pushed to our customers and maintained at runtime. Also, our selectors don't exactly give us the nice auto-complete that a sound type-system in a well-defined language would.

just some thoughts. this is good work, but I like stepping back and asking if it's the best fit for what we need.

tyxla commented 4 years ago

I think we can close this one - @sgomes has been working on state modularization for quite a bit now, and there is an entire project to track this project - https://github.com/Automattic/wp-calypso/projects/95.

I'll move this issue to the project and close it, so we have a reference for this great discussion.