mediacloud / web-tools

The shared repository for Media Cloud web apps (Explorer, Source Manager, Topic Mapper)
https://tools.mediacloud.org
Apache License 2.0
64 stars 30 forks source link

JS architecture for Explorer app #442

Closed rahulbot closed 7 years ago

rahulbot commented 7 years ago

Overall, I think the architecture should be a redux-form with nested FieldArrays holding the queries. Hitting the search button triggers a bunch of async results DataCards to show up. Each DataCard does multiple async loads to fetch the relevant data for each query, then stitches them together and renders them.

I think the state should look something like this:

{
  "explorer": {
    "queries": [
      { "name": "health",
        "color": "#443211",
        "startDate": dateObj,
        "endDate": dateObj,
        "collections": [4156],
        "sources": [1, 342],
        "query": "health",
        "index": 0,
      },
      ...
    ],
    "attention": [
      {"fetchStatus": null,
        "results": []
      },
      ....
    ],
    words: [
      {"fetchStatus": null,
        "results": []
      },
      ....
    ],
    geotags: [
      {"fetchStatus": null,
        "results": []
      },
      ....
    ],
}

Hitting search copies the queries into explorer.queries in the store, which triggers updates in all the aync containers. Each of them makes N requests to the server, one for each query, and then displays when they are all returned (we'll have to update asyncContainer to handle this multiple-fetch case). The reducers fill in the array based on the query index property. For instance, when the attention results reducer gets a response form the server, the action should still have the index in it under the meta property, and then the reducer will be the results in the explorer.attention array at the index indicated.

cindyloo commented 7 years ago

last friday, I think we discussed the reason to have the async containers send off each query one-by-one versus sending off a server call that loops through the queries and stiches the results together: the utility of have smaller, parallel, faster fetches. Is that what we gain? I ask b/c I am thinking we have to wait for the results to finish anyway to render the display, so is there really any speed gained?

Or is it because if a user tweaks any one query, it will only update that particular store item, and therefore only change the necessary rendered component/plot line...?

rahulbot commented 7 years ago

We definitely want all the calls to happen in parallel - ie. Query1 word count, Query2 word count, Query1 story sample, Query2 story sample should all happen in parallel. The question is whether to do them in parallel on the client in JS or on the server in Python. My gut was that we do it on the client in JS because it lets us be more flexible, and it probably easier.

I do think you are right about the optimization win in cases where they only change one query and click search again. If they only change query1 we shouldn't have to fetch query2 results again at all.

cindyloo commented 7 years ago

the async containers will be triggered from a change in our state.explorer.selected and kicking off a series of fetches that load data from the server into fields under

state.explorer.queries[index]
state.explorer.queries
state.explorer.queries[].definition
state.explorer.queries[].sentenceCount

in python we will extract a solrQueryFromRequest that pulls whatever fields from the request (whether form or JSON POST)

rahulbot commented 7 years ago

Latest idea is:

explorer {

    lastSearchTime: {timestamp},

    selected: { uid, keywords, startDate, endDate, ... }

    queries: [
        { uid, keywords, startDate, endDate, ... }
    ]

    words: {

        fetchStatus: [ boolean, boolean ],

        results: [ list, list, list ]

    }

    stories: {

        fetchStatus: [ boolean, boolean ],

        results: [ list, list, list ]

    }

    ...

}